Pyenv: your Linux / Unix tool to manage different python versions

Don’t be afraid of pythons, you can handle them!

Huw Cordey/Minden Pictures

Until recently I never had the issue of managing multiple python environments since I was always working with the default python2 or python3 installed on my Ubuntu laptop. But with my new job, things have changed, we have different projects using different versions of python, we use python2 and different versions of python3 and I also want to install the pypy interpreter that I’ve heard the most good about.

And that’s when the trouble started. Trying to compile pypy or a newer version of python3 was not successful. For the former, the compilation has not be completed and for the latter, even if compilation succeeded, I have some weird errors when using some modules. I was really disappointed about these installation experiences and then I discovered pyenv. This tool literally saved my life and I will present you briefly in this tutorial.

Note that pyenv is a shell script initially created for Linux / Unix systems so it will not work on Windows. There is a Windows port called pyenv-win if you want to use it there but I will not talk about it. With my experience (I use Windows at home) you don’t need it if you install python from the official download page of python. Nevertheless I put the link here to be exhaustive.

Installation

Install pyenv is relatively simple.

Mac OS

Consider using homebrew to install pyenv. The commands will be the following:

$ brew update
$ brew install pyenv

Linux / Unix

First be sure to have git installed. If it is not the case, use your favorite package manager and install it. Example for Debian family:

$ sudo apt-get install git

After that you can run this command.

$ curl https://pyenv.run | bash

Configuration

If the previous command succeeded, you will see at the end of the output (the last six lines) something like the following:

# Load pyenv automatically by adding
# the following to ~/.bashrc:
export PATH="/home/ec2-user/.pyenv/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv virtualenv-init -)"

Note that the instructions will depend of the shell you are using. In my case I have a Redhat-like machine, so I will use the .bashrc script to finish my configuration. We will slightly differ from the previous instructions by defining an environment variable PYENV_ROOT to store the base pyenv directory since it is considered a good practice by the pyenv team.

Edit: Before pyenv 2.X, the init command used the option “-”instead of “ — path”.

$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(pyenv init --path)"' >> ~/.bashrc
$ echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

You should now restart your shell for changes to take effect.

$ exec "$SHELL"

Pyenv should be up and running now and if you type its name, you should see output like the following:

$ pyenv
pyenv 1.2.18
Usage: pyenv <command> [<args>]
Some useful pyenv commands are:
   activate    Activate virtual environment
   commands    List all available pyenv commands
...

If you ran into issues, you can look at detailed explanations here.

Usage

Yeah now that we have pyenv installed, we want to install different python versions. The command install — list lists all available python versions handled by pyenv. It is an impressive list, and I discovered some distributions I didn’t know exist like activepython 😳

$ pyenv install --list
2.1.3
2.2.3
..

Before we go on, let’s be honest, pyenv is not magic, it will try to download and build the python versions you specify but it will not work without installing the system dependencies your python interpreter needs. I will list the main dependencies for common linux distributions.

Mac OS

First be sure you have Xcode command line tools installed, if it is not the case run:

$ xcode-select --install

After that you can install the dependencies.

$ brew install openssl readline sqlite3 xz zlib

Debian family (debian, ubuntu, linux mint,..)

$ sudo apt-get update; sudo apt-get install --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Redhat family (Redhat, centos, Fedora before version 21)

$ sudo yum install gcc zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel

Fedora 22 and above

$ dnf install make gcc zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel

Yeah I see you, user of Arch Linux, openSUSE and others saying “once again, they forgot us in their documentation”. Don’t worry, you can have a look here to see the dependencies you need to install 😃

For pypy distributions, you may need to install other dependencies. You can have the list here.

Ok now we can try to install python 3.8

$ pyenv install 3.8.3

If all goes well, when you run the versions command you will see the installed version appearing.

$ pyenv versions
* system (set by /home/ec2-user/.pyenv/version)
  3.8.3

Note that the current activated versions are prefixed by an “*”. We can change it using the global command.

$ pyenv global 3.8.3
$ pyenv versions
  system
* 3.8.3 (set by /home/ec2-user/.pyenv/version)

We can run a bunch of other commands to prove that it is now python3.8 the default version.

$ pyenv version
3.8.3 (set by /home/ec2-user/.pyenv/version)
$ python --version
Python 3.8.3
$ python
Python 3.8.3 (default, May 21 2020, 18:02:42) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=8, micro=3, releaselevel='final', serial=0)
>>> exit()

Great! Everything works fine. Now let try to install the pypy. Don’t forget to install its dependencies like I said above.

$ pyenv install pypy3.6-7.3.1
$ pyenv versions
  system
* 3.8.3 (set by /home/ec2-user/.pyenv/version)
  pypy3.6-7.3.1

Awesome! Now I can compile pypy 😭 (If you remembered what I said at the beginning..). Again let’s change the current version to pypy and check it.

$ pyenv global pypy3.6-7.3.1
$ pyenv version
pypy3.6-7.3.1 (set by /home/ec2-user/.pyenv/version)
$ python --version
Python 3.6.9 (2ad108f17bdb, Apr 07 2020, 02:59:05)
[PyPy 7.3.1 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
$ python
Python 3.6.9 (2ad108f17bdb, Apr 07 2020, 02:59:05)
[PyPy 7.3.1 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.6.9 (2ad108f17bdb, Apr 07 2020, 02:59:05)\n[PyPy 7.3.1 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]'
>>> exit()

Everything is fine. Perfect!

Managing different python versions per project

In case you have different projects on your computer (which is often the case), it is not practical to change the global python version every time you change a project. For this case there is the local command which comes in handy. It helps you define the python versions you need on a specific project. For example if I have a project running on python3.6, I can do this.

# before local command
# I installed python3.6 and python3.7 for this example
$ cd project
$ pyenv versions
  system
  3.6.10
  3.7.7
  3.8.3
* pypy3.6-7.3.1 (set by /home/ec2-user/.pyenv/version)
# after local command
$ pyenv local 3.6.10
$ pyenv versions
  system
* 3.6.10 (set by /home/ec2-user/project/.python-version)
  3.7.7
  3.8.3
  pypy3.6-7.3.1

Now we see that the defaut interpreter for the project “project” is python3.6.

If you have a project that is supporting many python versions and you want to run tests locally with tools like tox or nox (Aw I have a nice introduction on nox here 😜) you will need to activate more than one interpreter at the same time. Just add the different python versions you want to support on the same line.

$ pyenv local 3.6.10 3.7.7 3.8.3
$ pyenv versions
  system
* 3.6.10 (set by /home/ec2-user/project/.python-version)
* 3.7.7 (set by /home/ec2-user/project/.python-version)
* 3.8.3 (set by /home/ec2-user/project/.python-version)
  pypy3.6-7.3.1

Pyenv just creates a file called .python-version with the list of local versions to activate. If you go back to your home directory and check the versions activated, you will see that they are different.

$ cd ~
$ pyenv versions
  system
  3.6.10
  3.7.7
  3.8.3
* pypy3.6-7.3.1 (set by /home/ec2-user/.pyenv/version)

Python package isolation

Just be aware that activating local python versions for your project doesn’t mean that the packages you will install will be isolated per project. No! You will still need tools like virtualenv for that purpose. The latest versions of pyenv embed a plugin to facilitate virtualenv creation called pyenv-virtualenv. You can read the README to know how to use it. Personally, I prefer to use poetry which is elegant, simple and complete. And again I have a nice introduction here for those who are curious 😁

Other commands

You can update pyenv with update command. If there are changes, you will see ouput like the following:

$ pyenv update
Updating /home/ec2-user/.pyenv...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 10 (delta 6), reused 1 (delta 1), pack-reused 0
Dépaquetage des objets: 100% (10/10), fait.
Depuis https://github.com/pyenv/pyenv
 * branch            master     -> FETCH_HEAD
   2c32a6b..cf81e5a  master     -> origin/master
Mise à jour 2c32a6b..cf81e5a
Fast-forward
 README.md                                               | 47 +++++++++++++++++++++++------------------------
 plugins/python-build/share/python-build/stackless-3.7.5 |  2 +-
 2 files changed, 24 insertions(+), 25 deletions(-)

You can uninstall a specific python version with the uninstall command.

$ pyenv uninstall 3.6.10

You can search the executable path for a (python) command that pyenv will invoke with the which command.

$ pyenv version
pypy3.6-7.3.1 (set by /home/ec2-user/.pyenv/version)
$ pyenv which pip3
/home/ec2-user/.pyenv/versions/pypy3.6-7.3.1/bin/pip3

You can learn more about the commands here and also look at the wiki.

Hope you enjoy this tutorial 😊

If you like my article and want to continue learning with me, don’t hesitate to follow me here and subscribe to my newsletter on substack 😉