Create simple and powerful spiders or crawlers with pyscalpel!

A spider weaving its web
A spider weaving its web
Photo by Divyadarshi Acharya on Unsplash

Nowadays, there are many usages where we need to scrape information from web pages like news feed, beautiful link rendering with image and text like social media do, research for data scientists, etc..

So it is important if there is a framework that help us to do this with little effort. The most known in the python ecosystem is probably scrapy. It is a veteran using the Twisted asynchronous backend (note that it has recently add partial support for asyncio). But I feel like its usage was heavy. We have to…


Kifurushi makes it simple to implement a network protocol.

a set of pillars maintaining a structure
a set of pillars maintaining a structure
Photo by Jon Tyson on Unsplash

This article aims to talk about my latest work in the python landscape. It is oriented for network protocol implementors.

Foreword

During my daily work, I see a lot of stuff related to the DNS protocol. I’m not working directly on it, but this caught my curiosity, and I started to study it. And the best way to learn is to.. practice! So I start interacted with the protocol using the excellent scapy library. It is a library implementing many network protocols and help you experiment as much as you like. I…


Write simple and complex CLI applications in python using click.

An image of a terminal running top command
An image of a terminal running top command
An image of my terminal running top command

As developers, we often used command line interface (CLI) applications for different purposes like writing files with vim or emacs, running containers with docker, researching a pattern in any file using grep, etc… When I began programming, I have always wondered how do we write one? Then after some researchs, I found click. It was created by the same author of Flask, Armin Ronacher, who was frustrated to not find a suitable library to write Flask CLI application.

Before diving into the usage of click, I want to present you…


A structured concurrency library that makes your live easy!

A woman trying to eat and read at the same time
A woman trying to eat and read at the same time
Photo by The Creative Exchange on Unsplash

Foreword

These last years there has been a new trend for concurrency in python called asyncio. It’s goal is to help us create more efficient concurrent programs than the ones we traditionally created with threads. It leverages the new keywords async/await introduced in python3.5 and achieve better concurrency through the usage of coroutines sometimes defined as lightweight threads.

Although asyncio brings some fresh air in the python concurrency world, it has confusing design decisions that make it difficult to use. …


Don’t be afraid of pythons, you can handle them!

Huw Cordey/Minden Pictures

Until recently I never had the issue of managing multiple python environments since I was always working with the default python2 or python3 installed on my Ubuntu laptop. But with my new job, things have changed, we have different projects using different versions of python, we use python2 and different versions of python3 and I also want to install the pypy interpreter that I’ve heard the most good about.

And that’s when the trouble started. Trying to compile pypy or a newer version of python3 was not successful. For the former…


A conductor in action
A conductor in action
Dan Ettinger. © Niedermueller

If you are new in python programming, sometimes you feel overwhelmed when it comes to manage package dependencies and publish your project on pypi or other form of package registry. There are too many tools involved in the ecosystem: virtualenv, pip, pipenv, flirt, twine, etc.. This is a bit frustrating because you don’t know what to use to begin and you very often have to juggle between several tools to get to your end.

Fortunately a new tool has arrived and solved all these issues, it is called poetry! This tutorial will be based on poetry 1.0.0 …


a robot playing piano
a robot playing piano
Photo by Franck V. on Unsplash

Last week I discovered nox a command line tool that automates testing in multiple python environments, similar to tox. It is considered by his creator like the descendant of tox.

Its main advantage over tox for me is that you can take advantage of all the power of the python language in your automation process, but it has others cool features. Also it is used by some great projects like google-cloud-python or urllib3.

Usage

Basics

For the example we are going to imagine a simple project with a module example.py that will have one function adding two numbers.

def addition(a, b)…

Kevin Tewouda

Déserteur camerounais résidant désormais en France. Passionné de programmation, sport, de cinéma et mangas. J’écris en français et en anglais dû à mes origines.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store