Published 2021-04-05. Last modified 2021-04-14.
Time to read: 2 minutes.

This page is part of the django collection.

Django-oscar defines PIP dependencies with a setting called install_requires.

Django-oscar settings

install_requires = [
    'django>=2.2,<3.2',
    # PIL is required for image fields, Pillow is the "friendly" PIL fork
    'pillow>=6.0',
    # We use the ModelFormSetView from django-extra-views for the basket page
    'django-extra-views>=0.13,<0.14',
    # Search support
    'django-haystack>=3.0b1',
    # Treebeard is used for categories
    'django-treebeard>=4.3,<4.5',
    # Babel is used for currency formatting
    'Babel>=1.0,<3.0',
    # For manipulating search URLs
    'purl>=0.7',
    # For phone number field
    'phonenumbers',
    'django-phonenumber-field>=3.0.0,<4.0.0',
    # Used for oscar.test.newfactories
    'factory-boy>=2.4.1,<3.0',
    # Used for automatically building larger HTML tables
    'django-tables2>=2.3,<2.4',
    # Used for manipulating form field attributes in templates (eg: add
    # a css class)
    'django-widget-tweaks>=1.4.1',
]

According to the documentation, pip-tools/ uses install_requires to maintain requirements.txt.

Shell

$ pip install pip-tools

Layer 1: requirements.in

I could not get the install_requires setting to work with pip-tools. Instead, I was able to create a file called requirements.in to hold top-level dependencies, and pip-tools happily used it:

requirements.in

boto3==1.17.27
django
django-cors-headers==3.7.0
django-extensions
django-oscar>=3.0.2,<4.0.0
django_storages==1.11.1
django-grappelli==2.14.3
pip
pip-tools
psycopg2-binary==2.8.6
pycountry==20.7.3
python-decouple==3.4
sorl-thumbnail==12.6.3

Notice that pip-tools is an unpinned requirement :)

With requirements.in in place, a new requirements.txt can be generated using the pip-compile command provided by pip-tools. Here is the pip-compile help message:

Shell

(aw) $ pip-compile -h
Usage: pip-compile [OPTIONS] [SRC_FILES]...

  Compiles requirements.txt from requirements.in specs.

Options:
  --version                       Show the version and exit.
  -v, --verbose                   Show more output
  -q, --quiet                     Give less output
  -n, --dry-run                   Only show what would happen, don't change
                                  anything

  -p, --pre                       Allow resolving to prereleases (default is
                                  not)

  -r, --rebuild                   Clear any caches upfront, rebuild from
                                  scratch

  -f, --find-links TEXT           Look for archives in this directory or on
                                  this HTML page

  -i, --index-url TEXT            Change index URL (defaults to
                                  https://pypi.org/simple)

  --extra-index-url TEXT          Add additional index URL to search
  --cert TEXT                     Path to alternate CA bundle.
  --client-cert TEXT              Path to SSL client certificate, a single
                                  file containing the private key and the
                                  certificate in PEM format.

  --trusted-host TEXT             Mark this host as trusted, even though it
                                  does not have valid or any HTTPS.

  --header / --no-header          Add header to generated file
  --emit-trusted-host / --no-emit-trusted-host
                                  Add trusted host option to generated file
  --annotate / --no-annotate      Annotate results, indicating where
                                  dependencies come from

  -U, --upgrade                   Try to upgrade all dependencies to their
                                  latest versions

  -P, --upgrade-package TEXT      Specify particular packages to upgrade.
  -o, --output-file FILENAME      Output file name. Required if more than one
                                  input file is given. Will be derived from
                                  input file otherwise.

  --allow-unsafe / --no-allow-unsafe
                                  Pin packages considered unsafe: distribute,
                                  pip, setuptools.

                                  WARNING: Future versions of pip-tools will
                                  enable this behavior by default. Use --no-
                                  allow-unsafe to keep the old behavior. It is
                                  recommended to pass the --allow-unsafe now
                                  to adapt to the upcoming change.

  --generate-hashes               Generate pip 8 style hashes in the resulting
                                  requirements file.

  --reuse-hashes / --no-reuse-hashes
                                  Improve the speed of --generate-hashes by
                                  reusing the hashes from an existing output
                                  file.

  --max-rounds INTEGER            Maximum number of rounds before resolving
                                  the requirements aborts.

  --build-isolation / --no-build-isolation
                                  Enable isolation when building a modern
                                  source distribution. Build dependencies
                                  specified by PEP 518 must be already
                                  installed if build isolation is disabled.

  --emit-find-links / --no-emit-find-links
                                  Add the find-links option to generated file
  --cache-dir DIRECTORY           Store the cache data in DIRECTORY.
                                  [default: /home/mslinn/.cache/pip-tools]

  --pip-args TEXT                 Arguments to pass directly to the pip
                                  command.

  --emit-index-url / --no-emit-index-url
                                  Add index URL to generated file
  -h, --help                      Show this message and exit. 

Now I was able to update requirements.txt from requirements.in, and then upgrade all PIP packages like this:

Shell

(aw) $ pip-compile -U

(aw) $ pip install --upgrade -r requirements.txt

This could be written as one line.

Shell

(aw) $ pip-compile -U && \
pip install --upgrade -r requirements.txt

😁

Layer 2

I wanted to take advantange of the pip-tools layered requirements feature. Overtop the basic dependencies listed in requirements.in, I also wanted to manage development dependencies in dev.requirements.in and deployment dependencies in prod.requirements.in. The dev and prod layers are siblings.

Layer dev

There is no need to pin django-debug-toolbar because it is constrained by the django dependency in the lower layer. Jack Cushman, a pip-tools contributor, explains why the --generate-hashes option is important.

dev.requirements.in

-c requirements.txt
django-debug-toolbar
docutils
json5
pytest-django
PyYAML

Shell

(aw) $ pip-compile dev.requirements.in --generate-hashes --allow-unsafe

This produces dev.requirements.txt:

dev.requirements.txt

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile dev.requirements.in
#
asgiref==3.3.4
    # via
    #   -c requirements.txt
    #   django
django-debug-toolbar==3.2
    # via
    #   -c requirements.txt
    #   -r dev.requirements.in
django==3.1.8
    # via
    #   -c requirements.txt
    #   django-debug-toolbar
pytz==2021.1
    # via
    #   -c requirements.txt
    #   django
sqlparse==0.4.1
    # via
    #   -c requirements.txt
    #   django
    #   django-debug-toolbar

The warning to use --allow-unsafe seems unnecessary – I believe that --allow-unsafe should be the default behavior for pip-compile.

I spent some time digging into the reasons that pip-tools considers some packages “unsafe,” and as best I can tell it is because it was thought that pinning those packages could potentially break pip itself, and thus break the user's ability to recover from a mistake.

This seems to no longer be true, if it ever was. Instead, failing to use --allow-unsafe is unsafe, as it means different environments will end up with different versions of key packages despite installing from identical requirements.txt files.

 – Jack Cushman

Layer prod

I added gunicorn as a production dependency, and was surprised to find that it declares lists a specific version of the Pyton setuptools as a transitive dependency.

prod.requirements.in

-c requirements.txt
gunicorn
json5

Shell

(aw) $ pip-compile prod.requirements.in --generate-hashes --allow-unsafe

This produces prod.requirements.txt:

prod.requirements.txt

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --allow-unsafe --generate-hashes prod.requirements.in
#
gunicorn==20.1.0 \
    --hash=sha256:e0a968b5ba15f8a328fdfd7ab1fcb5af4470c28aaf7e55df02a99bc13138e6e8
    # via -r prod.requirements.in

# The following packages are considered to be unsafe in a requirements file:
setuptools==56.0.0 \
    --hash=sha256:08a1c0f99455307c48690f00d5c2ac2c1ccfab04df00454fef854ec145b81302 \
    --hash=sha256:7430499900e443375ba9449a9cc5d78506b801e929fef4a186496012f93683b5
    # via gunicorn

Share to X, Hacker News, LinkedIn.