Mike Slinn
Mike Slinn

A Python Virtual Environment For Every Project

Published 2021-04-09. Last modified 2023-01-26.
Time to read: 4 minutes.

This page is part of the posts collection, categorized under Bash, Python.

Python virtual environments are cheap to make and use – unless you are unfortunate enough to program in native Windows. I have adopted the habit of making a Python virtual environment (venv) for each significant Python project, plus a default venv for trivial Python work.

Why Virtualize Python?

It is better to use virtualized user- and project-specific Python instances, instead of working with a system-wide installation of Python. This allows you to install and upgrade Python packages without using supervisor privileges. Also, virtualized instances allows you to work on many different independent Python projects at the same time, without package version collisions.

Docker is over-sold. It adds unnecessary complexity to software projects. Instead of virtualizing the entire software environment, as docker attempts to do, virtualizing the Python programming environment as described in this blog post, Ruby rbenv, or node.js nvm are much easier and more productive approaches.

I think docker has been pushed hard in the media because it is a gateway technology to PaSS. This is a trend that PaSS vendors like AWS and Azure want to encourage, but customers are pushing back.

Python’s venv Virtualization Module

Venv is a tool to create isolated virtual Python environments. It has been included with Python since Python v3.3, which was released 10 years ago.

PEP 405 specifies Python’s venv virtualization module.

Deprecated Virtualization Modules

Python 3.6 was released 6 years ago. It deprecated the other virtualization modules, pyenv and virtualenv. Instead, use venv, as described in this blog post.

Extra Installation Step for Ubuntu

Venv is included with Python, and it virtualizes all the common Python executables. In order to virtualize pip, venv needs to invoke ensurepip, which is not installed by default on Ubuntu.

On Ubuntu systems, the ensurepip command is provided by a package called python3-pip.

Shell
$ yes | sudo apt install python3-pip

Manually Creating a VEnv

The following demonstrates how to create a new virtual python environment in the ~/venv/default/ directory. Intermediate directories, such as venv in this example, will be created as required.

Shell
$ python3 -m venv ~/venv/default

At this point, the virtual environment just contained executable images for Python.

Shell
$ ls ~/venv/default/**
/home/mslinn/venv/default/lib64@  /home/mslinn/venv/default/pyvenv.cfg

/home/mslinn/venv/default/bin:
Activate.ps1  activate  activate.csh  activate.fish  pip*  pip3*  pip3.10*  python@  python3@  python3.10@

/home/mslinn/venv/default/include:

/home/mslinn/venv/default/lib:
python3.10/

VEnvs are Nearly Free

The cost of a venv is virtually free for all OSes except native Windows. This is because on all OSes except native Windows, the executable images are linked by default, so they do not require much storage space.

The ls command below shows that the python program in the default venv is linked to /usr/bin/python3.10.

Shell
$ ls -go ~/venv/default/bin/python
lrwxrwxrwx 1 18 Apr 9 06:01 /home/mslinn/venv/default/bin/python -> python3*

Create a VEnv for Every Python Project

I name each venv the same as the python project that it is dedicated to. My projects are stored under the directory pointed to by $work.

My standard procedure when making a Python project called $work/blah is to also create a venv for it at ~/venv/blah.

A bash alias could be defined called blah that activates the venv and makes the project directory current:

~/.bash_aliases
alias blah="source ~/venv/blah/bin/activate; cd $work/blah"
😁

Now you could type blah at a shell prompt, and you would be working on that project. Boom!

Script For Creating a VEnv

Here is a bash script that creates the venv and changes ~/.bashrc and ~/.bash_aliases for you. It assumes that you keep your projects under $work.

newVenv 
#!/bin/bash

function help {
  echo -e "$1$(basename "$0") - Create a Python virtual environment with a given name.

Usage:
$(basename "$0") venv_name

The new virtual environment will be created under ~/venv/.
If a project directory called \$work/venv_name exists before this script runs,
then a bash alias is created named after the venv."
  exit 1
}


if ! dpkg -l | grep -q python3-pip; then
  yes | sudo apt install python3-pip
fi

if [ -z "$1" ]; then help "Please specify a name for the virtual environment.\n\n"; fi
if [ "$1" == -h ]; then help; fi
VENV="$1"
shift

mkdir -p "$HOME/venv"
cd "$HOME/venv" || exit
python3 -m venv "$VENV"
DIR="$HOME/venv/$VENV"

echo "source $DIR/bin/activate" >> "$HOME/.bashrc"

echo
echo "Activation for '$VENV' in future shells was appended to $HOME/.bashrc"
printf "To activate the '$VENV' venv in this shell right now, type:\nsource ~/venv/$VENV/bin/activate\n"

if [ "$work" ] && [ -d "$work/$VENV" ]; then
  echo "alias $VENV='source $DIR/bin/activate; cd $work/$VENV'" >> "$HOME/.bash_aliases"
  echo "An alias called $VENV for future shells was appended to $HOME/.bash_aliases"
  printf "To define the alias in this shell right now, type:\nalias $VENV='source $DIR/bin/activate; cd $work/$VENV'\n"
else
  printf "To define an alias, type something like this:\nalias $VENV=\"source $DIR/bin/activate; cd $work/$VENV\"\n"
fi

This is the help message for the script:

shell — newVenv help message
$ newVenv -h
newVenv - Create a Python virtual environment with a given name.

Usage:
newVenv venv_name

The new virtual environment will be created under ~/venv/.
If a project directory called $work/venv_name exists before this script runs,
then a bash alias is created named after the venv.

Let's use the script to create a venv called aw:

shell — newVenv usage
$ newVenv aw
Activation for 'aw' in future shells was appended to /home/mslinn/.bashrc
To activate the 'aw' venv in this shell right now, type:
  source ~/venv/aw/bin/activate
To define an alias, type something like this:
alias aw="source /home/mslinn/venv/aw/bin/activate; cd /mnt/_/work/aw"

Script for Using a VEnv

Here is a script that can display the available Python virtual environments, and optionally activates one. It does not use bash aliases.

use 
#!/bin/bash

export SCRIPT_NAME="$0"

function help {
  echo "Usage:"
  for f in "$HOME"/venv/*; do
    if [ -d "$f" ]; then echo "  . $(basename "$SCRIPT_NAME") $(basename "$f")"; fi
  done
  return 2
}

unset PV
if [ "$1" == -h ]; then
  help
elif [ "$1" ]; then
  PV="$1"
else
  PV="default"
fi

if [ "$PV" ]; then
  DIR="$HOME/venv/$PV"
  if [ ! -d "$DIR" ]; then
    echo "Error: $DIR does not exist."
    return 1
  fi

  if [ ! -f "$DIR/bin/python" ]; then
    echo "Error: No Python virtual environment is installed in $DIR"
    return 1
  fi
  echo "Setting Python virtual environment to $DIR"
  source "$DIR/bin/activate"
fi

Here are examples of using the script to change virtual environments:

Shell
$ . use -h
Usage:
  . use aw
  . use default 

$ . use
Setting Python virtual environment to /home/mslinn/venv/default 

(default) $ . use aw
Setting Python virtual environment to /home/mslinn/venv/aw 

(aw) $

Notice that the last command above changed the shell prompt, in that (aw) was prepended to the normal prompt. To cause all future shells to use this virtual environment by default, the script adds a line to ~/.bashrc that looks like this:

Shell
$ echo "source ~/venv/aw/bin/activate" >> ~/.bashrc

Deactivate a VEnv

Stop using venvs with deactivate. Notice that the prompt changes.

Shell
(aw) $ deactivate

$

Activation By an Alias

Once again, we can use a bash alias, this time to invoke the use script. We can call the alias use, because bash aliases have precedence over bash scripts. This alias removes the need to type ., or source before the script name (which you know is use, if you have been following along). Also notice that the name of the default venv need not be specified.

Shell
$ alias use="source use"

$ use

(default) $ use aw

(aw) $ deactivate

$

You can add the alias to bash_aliases:

Shell
$ echo 'alias use="source use"' >> ~/.bash_aliases

$ source ~/.bash_aliases

$ alias | grep use
alias use='source use'
😁

Directory-Locked Python Virtualization

After setting up a Python virtual environment, a quick examination of the pip script shows that it is hard-coded to the directory that it was made for:

Shell
$ head -n 1 ~/venv/aw/bin/pip
#!/home/mslinn/venv/aw/bin/python

For virtualized environments, such as Docker, this means that a Python virtual environment created without Docker can only be used within a Docker image if the path to it is the same from within the Docker image as when it was created.

Updating Python

To update the version of Python in a venv, just run the same command that you used to create the venv in the first place.

shell — Updating Python in a venv
$ python3 -m venv ~/venv/default

For Further Reading

Python Best Practices for a New Project in 2021

Summary

  • Demonstrated how to make an alias for working with Python virtual environments (venvs) that are coupled with Python projects.
  • The newVenv bash script was demonstrated for making new Python virtual environments.
  • The use bash source script was demonstrated for activating a venv.
  • Deactivating the current venv was demonstrated using the deactivate command, provided with every venv.
  • The use alias for source use was demonstrated for more conveniently selecting a venv.
  • Locked directories mean that Python virtual environments should normally only be created in the same environment they are intended to be used.