Published 2021-04-05. Last modified 2021-04-14.
Time to read: about 2 minutes.
Django-oscar defines PIP dependencies with a setting called
install_requires = [ 'django>=2.2,<3.2', # PIL is required for image fields, Pillow is the "friendly" PIL fork 'pillow>=6.0', # We use the ModelFormSetView from django-extra-views for the basket page 'django-extra-views>=0.13,<0.14', # Search support 'django-haystack>=3.0b1', # Treebeard is used for categories 'django-treebeard>=4.3,<4.5', # Babel is used for currency formatting 'Babel>=1.0,<3.0', # For manipulating search URLs 'purl>=0.7', # For phone number field 'phonenumbers', 'django-phonenumber-field>=3.0.0,<4.0.0', # Used for oscar.test.newfactories 'factory-boy>=2.4.1,<3.0', # Used for automatically building larger HTML tables 'django-tables2>=2.3,<2.4', # Used for manipulating form field attributes in templates (eg: add # a css class) 'django-widget-tweaks>=1.4.1', ]
According to the documentation,
install_requires to maintain
$ pip install pip-tools
Layer 1: requirements.in
I could not get the
install_requires setting to work with
pip-tools. Instead, I was able to create a file called
requirements.in to hold top-level dependencies, and
pip-tools happily used it:
boto3==1.17.27 django django-cors-headers==3.7.0 django-extensions django-oscar>=3.0.2,<4.0.0 django_storages==1.11.1 django-grappelli==2.14.3 pip pip-tools psycopg2-binary==2.8.6 pycountry==20.7.3 python-decouple==3.4 sorl-thumbnail==12.6.3
pip-tools is an unpinned requirement :)
requirements.in in place, a new
requirements.txt can be generated using the
pip-compile command provided by
pip-tools. Here is the
pip-compile help message:
(aw) $ pip-compile -h Usage: pip-compile [OPTIONS] [SRC_FILES]... Compiles requirements.txt from requirements.in specs. Options: --version Show the version and exit. -v, --verbose Show more output -q, --quiet Give less output -n, --dry-run Only show what would happen, don't change anything -p, --pre Allow resolving to prereleases (default is not) -r, --rebuild Clear any caches upfront, rebuild from scratch -f, --find-links TEXT Look for archives in this directory or on this HTML page -i, --index-url TEXT Change index URL (defaults to https://pypi.org/simple) --extra-index-url TEXT Add additional index URL to search --cert TEXT Path to alternate CA bundle. --client-cert TEXT Path to SSL client certificate, a single file containing the private key and the certificate in PEM format. --trusted-host TEXT Mark this host as trusted, even though it does not have valid or any HTTPS. --header / --no-header Add header to generated file --emit-trusted-host / --no-emit-trusted-host Add trusted host option to generated file --annotate / --no-annotate Annotate results, indicating where dependencies come from -U, --upgrade Try to upgrade all dependencies to their latest versions -P, --upgrade-package TEXT Specify particular packages to upgrade. -o, --output-file FILENAME Output file name. Required if more than one input file is given. Will be derived from input file otherwise. --allow-unsafe / --no-allow-unsafe Pin packages considered unsafe: distribute, pip, setuptools. WARNING: Future versions of pip-tools will enable this behavior by default. Use --no- allow-unsafe to keep the old behavior. It is recommended to pass the --allow-unsafe now to adapt to the upcoming change. --generate-hashes Generate pip 8 style hashes in the resulting requirements file. --reuse-hashes / --no-reuse-hashes Improve the speed of --generate-hashes by reusing the hashes from an existing output file. --max-rounds INTEGER Maximum number of rounds before resolving the requirements aborts. --build-isolation / --no-build-isolation Enable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if build isolation is disabled. --emit-find-links / --no-emit-find-links Add the find-links option to generated file --cache-dir DIRECTORY Store the cache data in DIRECTORY. [default: /home/mslinn/.cache/pip-tools] --pip-args TEXT Arguments to pass directly to the pip command. --emit-index-url / --no-emit-index-url Add index URL to generated file -h, --help Show this message and exit.
Now I was able to update
requirements.in, and then upgrade all PIP packages like this:
(aw) $ pip-compile -U (aw) $ pip install --upgrade -r requirements.txt
This could be written as one line.
(aw) $ pip-compile -U && \ pip install --upgrade -r requirements.txt
I wanted to take advantange of the
pip-tools layered requirements feature. Overtop the basic dependencies listed in
requirements.in, I also wanted to manage development dependencies in
dev.requirements.in and deployment dependencies in
prod layers are siblings.
There is no need to pin
django-debug-toolbar because it is constrained by the
django dependency in the lower layer. Jack Cushman, a
pip-tools contributor, explains why the
--generate-hashes option is important.
-c requirements.txt django-debug-toolbar docutils json5 pytest-django PyYAML
(aw) $ pip-compile dev.requirements.in --generate-hashes --allow-unsafe
# # This file is autogenerated by pip-compile # To update, run: # # pip-compile dev.requirements.in # asgiref==3.3.4 # via # -c requirements.txt # django django-debug-toolbar==3.2 # via # -c requirements.txt # -r dev.requirements.in django==3.1.8 # via # -c requirements.txt # django-debug-toolbar pytz==2021.1 # via # -c requirements.txt # django sqlparse==0.4.1 # via # -c requirements.txt # django # django-debug-toolbar
Jack Cushman had this to say about
--allow-unsafeseems unnecessary — I believe that
--allow-unsafeshould be the default behavior for
pip-compile. I spent some time digging into the reasons that
pip-toolsconsiders some packages “unsafe,” and as best I can tell it is because it was thought that pinning those packages could potentially break
pipitself, and thus break the user's ability to recover from a mistake. This seems to no longer be true, if it ever was. Instead, failing to use
--allow-unsafeis unsafe, as it means different environments will end up with different versions of key packages despite installing from identical
gunicorn as a production dependency, and was surprised to find that it declares lists a specific version of the Pyton
setuptools as a transitive dependency.
-c requirements.txt gunicorn json5
(aw) $ pip-compile prod.requirements.in --generate-hashes --allow-unsafe
# # This file is autogenerated by pip-compile # To update, run: # # pip-compile --allow-unsafe --generate-hashes prod.requirements.in # gunicorn==20.1.0 \ --hash=sha256:e0a968b5ba15f8a328fdfd7ab1fcb5af4470c28aaf7e55df02a99bc13138e6e8 # via -r prod.requirements.in # The following packages are considered to be unsafe in a requirements file: setuptools==56.0.0 \ --hash=sha256:08a1c0f99455307c48690f00d5c2ac2c1ccfab04df00454fef854ec145b81302 \ --hash=sha256:7430499900e443375ba9449a9cc5d78506b801e929fef4a186496012f93683b5 # via gunicorn