Published 2021-04-09.
Last modified 2023-06-14.
Time to read: 3 minutes.
Python virtual environments are cheap to make and use – unless you are unfortunate enough to program in native Windows. I have adopted the habit of making a Python virtual environment (venv) for each significant Python project, plus a default venv for trivial Python work.
Why Virtualize Python?
It is better to use virtualized user- and project-specific Python instances, instead of working with a system-wide installation of Python. This allows you to install and upgrade Python packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent Python projects at the same time, without package version collisions.
Docker is over-sold.
It adds unnecessary complexity to software projects.
Instead of virtualizing the entire software environment,
as docker attempts to do,
virtualizing the Python programming environment as described in this blog post,
Ruby rbenv,
or node.js nvm
are much easier and more productive approaches.
I think docker has been pushed hard in the media because it is a gateway technology to
PaSS.
This is a trend that PaSS vendors like AWS and Azure want to encourage, but
customers are pushing back.
Python’s venv Virtualization Module
Venv is a tool to create isolated virtual Python environments.
It has been included with Python since Python v3.3,
which was released 11 years ago.
PEP 405 specifies Python’s
venv
virtualization module.
Deprecated Virtualization Modules
Python 3.6 was released 6 years ago.
It deprecated the other virtualization modules,
pyenv and
virtualenv.
Instead, use venv, as described in this blog post.
Extra Installation Steps for Ubuntu
Venv was included with Python on Ubuntu until Ubuntu 23.04
It virtualizes all the common Python executables.
Debian changed pip’s behavior
as a result of
PEP 668 – Marking Python base environments as “externally managed”.
This affects Ubuntu because it is downstream.
Starting with Ubuntu 23.04, you need to install venv by typing sudo apt install python3.xx-venv,
where xx is the minor version number of Python that is installed.
For Python 3.11, the following command is required:
$ yes | sudo apt install python3.11-venv
Some software projects use meta-packages to specify version-agnostic dependencies. Perhaps Python on Ubuntu will do so in the future.
In order to virtualize pip, venv needs to invoke ensurepip,
which is not installed by default on Ubuntu.
On Ubuntu systems, the ensurepip command is provided by a package called python3-pip.
$ yes | sudo apt install python3-pip
You could install both of the above packages together, of course:
$ yes | sudo apt install python3.11-venv python3-pip
Manually Creating a VEnv
The following demonstrates how to create a new virtual python environment in the ~/venv/default/ directory.
Intermediate directories, such as venv in this example, will be created as required.
$ python3 -m venv ~/venv/default
Re-running this command at a later date will update the version of Python in an existing venv.
Each time you run this command, all pip packages are removed.
At this point, the virtual environment just contained executable images for Python.
$ ls ~/venv/default/** /home/mslinn/venv/default/lib64@ /home/mslinn/venv/default/pyvenv.cfg /home/mslinn/venv/default/bin: Activate.ps1 activate activate.csh activate.fish pip* pip3* pip3.10* python@ python3@ python3.10@ /home/mslinn/venv/default/include: /home/mslinn/venv/default/lib: python3.10/
VEnvs are Nearly Free
The cost of a venv is virtually free for all OSes except native Windows. This is because on all OSes except native Windows, the executable images are linked by default, so they do not require much storage space.
The ls command below shows that the python program in the default venv
is linked to /usr/bin/python3.10.
$ ls -go ~/venv/default/bin/python lrwxrwxrwx 1 18 Apr 9 06:01 /home/mslinn/venv/default/bin/python -> python3*
Create a VEnv for Every Python Project
My projects are stored under the directory pointed to by $work.
My standard procedure when making a Python project called $work/blah
is to also create a venv for it at within the project, at $work/blah/.venv/.
I add an entry to .gitignore that merely consists of a line that says .venv/.
A bash alias could be defined called blah
that makes the project directory current and activates the venv:
alias blah="cd $work/blah; source ./venv/bin/activate"
Now you could type blah at a shell prompt,
and you would be working on that project.
Boom!
Deactivate a VEnv
Stop using venvs with deactivate.
Notice that the prompt changes.
(aw) $ deactivate $
Directory-Locked Python Virtualization
After setting up a Python virtual environment,
a quick examination of the pip script
shows that it is hard-coded to the directory that it was made for:
$ head -n 1 ~/venv/aw/bin/pip #!/home/mslinn/venv/aw/bin/python
For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created.
For Further Reading
Python Best Practices for a New Project in 2021. This article is already becoming dated. YMMV.
Summary
- Demonstrated how to make an alias for working with Python virtual environments (venvs) that are coupled with Python projects.
- Deactivating the current venv was demonstrated using the
deactivatecommand, provided with every venv. - Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.