Mike Slinn
Mike Slinn

A Python Virtual Environment For Every Project

Published 2021-04-09. Last modified 2023-06-14.
Time to read: 3 minutes.

This page is part of the posts collection, categorized under Bash, Python.

Python virtual environments are cheap to make and use – unless you are unfortunate enough to program in native Windows. I have adopted the habit of making a Python virtual environment (venv) for each significant Python project, plus a default venv for trivial Python work.

Why Virtualize Python?

It is better to use virtualized user- and project-specific Python instances, instead of working with a system-wide installation of Python. This allows you to install and upgrade Python packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent Python projects at the same time, without package version collisions.

Docker is over-sold. It adds unnecessary complexity to software projects. Instead of virtualizing the entire software environment, as docker attempts to do, virtualizing the Python programming environment as described in this blog post, Ruby rbenv, or node.js nvm are much easier and more productive approaches.

I think docker has been pushed hard in the media because it is a gateway technology to PaSS. This is a trend that PaSS vendors like AWS and Azure want to encourage, but customers are pushing back.

Python’s venv Virtualization Module

Venv is a tool to create isolated virtual Python environments. It has been included with Python since Python v3.3, which was released 11 years ago.

PEP 405 specifies Python’s venv virtualization module.

Deprecated Virtualization Modules

Python 3.6 was released 6 years ago. It deprecated the other virtualization modules, pyenv and virtualenv. Instead, use venv, as described in this blog post.

Extra Installation Steps for Ubuntu

Venv was included with Python on Ubuntu until Ubuntu 23.04 It virtualizes all the common Python executables.

Debian changed pip’s behavior as a result of PEP 668 – Marking Python base environments as “externally managed”. This affects Ubuntu because it is downstream. Starting with Ubuntu 23.04, you need to install venv by typing sudo apt install python3.xx-venv, where xx is the minor version number of Python that is installed. For Python 3.11, the following command is required:

Shell
$ yes | sudo apt install python3.11-venv

Some software projects use meta-packages to specify version-agnostic dependencies. Perhaps Python on Ubuntu will do so in the future.

In order to virtualize pip, venv needs to invoke ensurepip, which is not installed by default on Ubuntu. On Ubuntu systems, the ensurepip command is provided by a package called python3-pip.

Shell
$ yes | sudo apt install python3-pip

You could install both of the above packages together, of course:

Shell
$ yes | sudo apt install python3.11-venv python3-pip

Manually Creating a VEnv

The following demonstrates how to create a new virtual python environment in the ~/venv/default/ directory. Intermediate directories, such as venv in this example, will be created as required.

Shell
$ python3 -m venv ~/venv/default

Re-running this command at a later date will update the version of Python in an existing venv. Each time you run this command, all pip packages are removed.

At this point, the virtual environment just contained executable images for Python.

Shell
$ ls ~/venv/default/**
/home/mslinn/venv/default/lib64@  /home/mslinn/venv/default/pyvenv.cfg

/home/mslinn/venv/default/bin:
Activate.ps1  activate  activate.csh  activate.fish  pip*  pip3*  pip3.10*  python@  python3@  python3.10@

/home/mslinn/venv/default/include:

/home/mslinn/venv/default/lib:
python3.10/ 

VEnvs are Nearly Free

The cost of a venv is virtually free for all OSes except native Windows. This is because on all OSes except native Windows, the executable images are linked by default, so they do not require much storage space.

The ls command below shows that the python program in the default venv is linked to /usr/bin/python3.10.

Shell
$ ls -go ~/venv/default/bin/python
lrwxrwxrwx 1 18 Apr 9 06:01 /home/mslinn/venv/default/bin/python -> python3* 

Create a VEnv for Every Python Project

My projects are stored under the directory pointed to by $work.

My standard procedure when making a Python project called $work/blah is to also create a venv for it at within the project, at $work/blah/.venv/. I add an entry to .gitignore that merely consists of a line that says .venv/.

A bash alias could be defined called blah that makes the project directory current and activates the venv:

~/.bash_aliases
alias blah="cd $work/blah; source ./venv/bin/activate"
😁

Now you could type blah at a shell prompt, and you would be working on that project. Boom!

Deactivate a VEnv

Stop using venvs with deactivate. Notice that the prompt changes.

Shell
(aw) $ deactivate

$ 

Directory-Locked Python Virtualization

After setting up a Python virtual environment, a quick examination of the pip script shows that it is hard-coded to the directory that it was made for:

Shell
$ head -n 1 ~/venv/aw/bin/pip
#!/home/mslinn/venv/aw/bin/python 

For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created.

For Further Reading

Python Best Practices for a New Project in 2021. This article is already becoming dated. YMMV.

Summary

  • Demonstrated how to make an alias for working with Python virtual environments (venvs) that are coupled with Python projects.
  • Deactivating the current venv was demonstrated using the deactivate command, provided with every venv.
  • Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.