Mike Slinn
Mike Slinn

Django Migrations

Published 2021-02-12. Last modified 2021-03-28.
Time to read: about 2 minutes.

This article is categorized under Django.

Each Django app can have its own migrations. Each migration is stored in a separate file, under the migrations directory for that app. I had installed django-oscar in the $oscar directory, and the django-oscar apps are found within the $oscar/lib/python3.8/site-packages/oscar/apps/ subdirectory.

Here is how I listed all the migration files, without the uninteresting __init__.py files:

Shell
(oscar) $cd $oscar/lib/python3.8/site-packages/oscar/apps/
(oscar) $ls **/migrations/*.py | grep -v __init__ address/migrations/0001_initial.py address/migrations/0002_auto_20150927_1547.py address/migrations/0003_auto_20150927_1551.py address/migrations/0004_auto_20170226_1122.py address/migrations/0005_regenerate_user_address_hashes.py address/migrations/0006_auto_20181115_1953.py analytics/migrations/0001_initial.py analytics/migrations/0002_auto_20140827_1705.py analytics/migrations/0003_auto_20200801_0817.py basket/migrations/0001_initial.py basket/migrations/0002_auto_20140827_1705.py basket/migrations/0003_basket_vouchers.py basket/migrations/0004_auto_20141007_2032.py basket/migrations/0005_auto_20150604_1450.py basket/migrations/0006_auto_20160111_1108.py basket/migrations/0007_slugfield_noop.py basket/migrations/0008_auto_20181115_1953.py basket/migrations/0009_line_date_updated.py catalogue/migrations/0001_initial.py catalogue/migrations/0002_auto_20150217_1221.py catalogue/migrations/0003_data_migration_slugs.py catalogue/migrations/0004_auto_20150217_1710.py catalogue/migrations/0005_auto_20150604_1450.py catalogue/migrations/0006_auto_20150807_1725.py catalogue/migrations/0007_auto_20151207_1440.py catalogue/migrations/0008_auto_20160304_1652.py catalogue/migrations/0009_slugfield_noop.py catalogue/migrations/0010_auto_20170420_0439.py catalogue/migrations/0011_auto_20170422_1355.py catalogue/migrations/0012_auto_20170609_1902.py catalogue/migrations/0013_auto_20170821_1548.py catalogue/migrations/0014_auto_20181115_1953.py catalogue/migrations/0015_product_is_public.py catalogue/migrations/0016_auto_20190327_0757.py catalogue/migrations/0017_auto_20190816_0938.py catalogue/migrations/0018_auto_20191220_0920.py catalogue/migrations/0019_option_required.py catalogue/migrations/0020_auto_20200801_0817.py catalogue/migrations/0021_auto_20201005_0844.py communication/migrations/0001_initial.py communication/migrations/0002_reset_table_names.py communication/migrations/0003_remove_notification_category_make_code_uppercase.py communication/migrations/0004_auto_20200801_0817.py customer/migrations/0001_initial.py customer/migrations/0002_auto_20150807_1725.py customer/migrations/0003_update_email_length.py customer/migrations/0004_email_save.py customer/migrations/0005_auto_20181115_1953.py customer/migrations/0006_auto_20190430_1736.py customer/migrations/0007_auto_20200801_0817.py offer/migrations/0001_initial.py offer/migrations/0002_auto_20151210_1053.py offer/migrations/0003_auto_20161120_1707.py offer/migrations/0004_auto_20170415_1518.py offer/migrations/0005_auto_20170423_1217.py offer/migrations/0006_auto_20170504_0616.py offer/migrations/0007_conditionaloffer_exclusive.py offer/migrations/0008_auto_20181115_1953.py offer/migrations/0009_auto_20200801_0817.py offer/migrations/0010_conditionaloffer_combinations.py order/migrations/0001_initial.py order/migrations/0002_auto_20141007_2032.py order/migrations/0003_auto_20150113_1629.py order/migrations/0004_auto_20160111_1108.py order/migrations/0005_update_email_length.py order/migrations/0006_orderstatuschange.py order/migrations/0007_auto_20181115_1953.py order/migrations/0008_auto_20190301_1035.py order/migrations/0009_surcharge.py order/migrations/0010_auto_20200724_0909.py order/migrations/0011_auto_20200801_0817.py partner/migrations/0001_initial.py partner/migrations/0002_auto_20141007_2032.py partner/migrations/0003_auto_20150604_1450.py partner/migrations/0004_auto_20160107_1755.py partner/migrations/0005_auto_20181115_1953.py partner/migrations/0006_auto_20200724_0909.py payment/migrations/0001_initial.py payment/migrations/0002_auto_20141007_2032.py payment/migrations/0003_auto_20160323_1520.py payment/migrations/0004_auto_20181115_1953.py payment/migrations/0005_auto_20200801_0817.py shipping/migrations/0001_initial.py shipping/migrations/0002_auto_20150604_1450.py shipping/migrations/0003_auto_20181115_1953.py voucher/migrations/0001_initial.py voucher/migrations/0002_auto_20170418_2132.py voucher/migrations/0003_auto_20171212_0411.py voucher/migrations/0004_auto_20180228_0940.py voucher/migrations/0005_auto_20180402_1425.py voucher/migrations/0006_auto_20180413_0911.py voucher/migrations/0007_auto_20181115_1953.py voucher/migrations/0008_auto_20200801_0817.py wishlists/migrations/0001_initial.py wishlists/migrations/0002_auto_20160111_1108.py wishlists/migrations/0003_auto_20181115_1953.py

showmigrations Subcommand

The showmigrations subcommand of ./manage.py is a handy way to display the same migrations, and it also shows an X next to the migrations that have already been applied.

Here is the showmigrations subcommand help:

Shell
(oscar) $./manage.py showmigrations -h
usage: manage.py showmigrations [-h] [--database DATABASE] [--list | --plan]
                                  [--version] [-v {0,1,2,3}]
                                  [--settings SETTINGS]
                                  [--pythonpath PYTHONPATH] [--traceback]
                                  [--no-color] [--force-color] [--skip-checks]
                                  [app_label [app_label ...]]

  Shows all available migrations for the current project

  positional arguments:
    app_label             App labels of applications to limit the output to.

  optional arguments:
    -h, --help            show this help message and exit
    --database DATABASE   Nominates a database to synchronize. Defaults to the
                          "default" database.
    --list, -l            Shows a list of all migrations and which are applied.
                          With a verbosity level of 2 or above, the applied
                          datetimes will be included.
    --plan, -p            Shows all migrations in the order they will be
                          applied. With a verbosity level of 2 or above all
                          direct migration dependencies and reverse dependencies
                          (run_before) will be included.
    --version             show program's version number and exit
    -v {0,1,2,3}, --verbosity {0,1,2,3}
                          Verbosity level; 0=minimal output, 1=normal output,
                          2=verbose output, 3=very verbose output
    --settings SETTINGS   The Python path to a settings module, e.g.
                          "myproject.settings.main". If this isn't provided, the
                          DJANGO_SETTINGS_MODULE environment variable will be
                          used.
    --pythonpath PYTHONPATH
                          A directory to add to the Python path, e.g.
                          "/home/djangoprojects/myproject".
    --traceback           Raise on CommandError exceptions
    --no-color            Don't colorize the command output.
    --force-color         Force colorization of the command output.
    --skip-checks         Skip system checks. 

Here is sample output for the showmigrations subcommand:

Shell
(oscar) $./manage.py showmigrations
address
 [X] 0001_initial
 [X] 0002_auto_20150927_1547
 [X] 0003_auto_20150927_1551
 [X] 0004_auto_20170226_1122
 [X] 0005_regenerate_user_address_hashes
 [X] 0006_auto_20181115_1953
admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
 [X] 0003_logentry_add_action_flag_choices
analytics
 [X] 0001_initial
 [X] 0002_auto_20140827_1705
 [X] 0003_auto_20200801_0817
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
 [X] 0010_alter_group_name_max_length
 [X] 0011_update_proxy_permissions
 [X] 0012_alter_user_first_name_max_length
basket
 [X] 0001_initial
 [X] 0002_auto_20140827_1705
 [X] 0003_basket_vouchers
 [X] 0004_auto_20141007_2032
 [X] 0005_auto_20150604_1450
 [X] 0006_auto_20160111_1108
 [X] 0007_slugfield_noop
 [X] 0008_auto_20181115_1953
 [X] 0009_line_date_updated
catalogue
 [X] 0001_initial
 [X] 0002_auto_20150217_1221
 [X] 0003_data_migration_slugs
 [X] 0004_auto_20150217_1710
 [X] 0005_auto_20150604_1450
 [X] 0006_auto_20150807_1725
 [X] 0007_auto_20151207_1440
 [X] 0008_auto_20160304_1652
 [X] 0009_slugfield_noop
 [X] 0010_auto_20170420_0439
 [X] 0011_auto_20170422_1355
 [X] 0012_auto_20170609_1902
 [X] 0013_auto_20170821_1548
 [X] 0014_auto_20181115_1953
 [X] 0015_product_is_public
 [X] 0016_auto_20190327_0757
 [X] 0017_auto_20190816_0938
 [X] 0018_auto_20191220_0920
 [X] 0019_option_required
 [X] 0020_auto_20200801_0817
 [X] 0021_auto_20201005_0844
communication
 [X] 0001_initial
 [X] 0002_reset_table_names
 [X] 0003_remove_notification_category_make_code_uppercase
 [X] 0004_auto_20200801_0817
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
customer
 [X] 0001_initial
 [X] 0002_auto_20150807_1725
 [X] 0003_update_email_length
 [X] 0004_email_save
 [X] 0005_auto_20181115_1953
 [X] 0006_auto_20190430_1736
 [X] 0007_auto_20200801_0817
flatpages
 [X] 0001_initial
offer
 [X] 0001_initial
 [X] 0002_auto_20151210_1053
 [X] 0003_auto_20161120_1707
 [X] 0004_auto_20170415_1518
 [X] 0005_auto_20170423_1217
 [X] 0006_auto_20170504_0616
 [X] 0007_conditionaloffer_exclusive
 [X] 0008_auto_20181115_1953
 [X] 0009_auto_20200801_0817
 [X] 0010_conditionaloffer_combinations
order
 [X] 0001_initial
 [X] 0002_auto_20141007_2032
 [X] 0003_auto_20150113_1629
 [X] 0004_auto_20160111_1108
 [X] 0005_update_email_length
 [X] 0006_orderstatuschange
 [X] 0007_auto_20181115_1953
 [X] 0008_auto_20190301_1035
 [X] 0009_surcharge (1 squashed migrations)
 [X] 0010_auto_20200724_0909
 [X] 0011_auto_20200801_0817
partner
 [X] 0001_initial
 [X] 0002_auto_20141007_2032
 [X] 0003_auto_20150604_1450
 [X] 0004_auto_20160107_1755
 [X] 0005_auto_20181115_1953
 [X] 0006_auto_20200724_0909
payment
 [X] 0001_initial
 [X] 0002_auto_20141007_2032
 [X] 0003_auto_20160323_1520
 [X] 0004_auto_20181115_1953
 [X] 0005_auto_20200801_0817
reviews
 [X] 0001_initial
 [X] 0002_update_email_length
 [X] 0003_auto_20160802_1358
 [X] 0004_auto_20170429_0941
sessions
 [X] 0001_initial
shipping
 [X] 0001_initial
 [X] 0002_auto_20150604_1450
 [X] 0003_auto_20181115_1953
sites
 [X] 0001_initial
 [X] 0002_alter_domain_unique
thumbnail
 [X] 0001_initial
voucher
 [X] 0001_initial
 [X] 0002_auto_20170418_2132
 [X] 0003_auto_20171212_0411
 [X] 0004_auto_20180228_0940
 [X] 0005_auto_20180402_1425
 [X] 0006_auto_20180413_0911
 [X] 0007_auto_20181115_1953
 [X] 0008_auto_20200801_0817
wishlists
 [X] 0001_initial
 [X] 0002_auto_20160111_1108
 [X] 0003_auto_20181115_1953 %}

Here is the first migration file, address/migrations/0001_initial.py:

address/migrations/0001_initial.py
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
 
from django.db import models, migrations
import oscar.models.fields
from django.conf import settings
 
 
class Migration(migrations.Migration):
 
    dependencies = [
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
    ]
 
    operations = [
        migrations.CreateModel(
            name='Country',
            fields=[
                ('iso_3166_1_a2', models.CharField(primary_key=True, max_length=2, verbose_name='ISO 3166-1 alpha-2', serialize=False)),
                ('iso_3166_1_a3', models.CharField(max_length=3, verbose_name='ISO 3166-1 alpha-3', blank=True)),
                ('iso_3166_1_numeric', models.CharField(max_length=3, verbose_name='ISO 3166-1 numeric', blank=True)),
                ('printable_name', models.CharField(max_length=128, verbose_name='Country name')),
                ('name', models.CharField(max_length=128, verbose_name='Official name')),
                ('display_order', models.PositiveSmallIntegerField(default=0, verbose_name='Display order', db_index=True, help_text='Higher the number, higher the country in the list.')),
                ('is_shipping_country', models.BooleanField(default=False, db_index=True, verbose_name='Is shipping country')),
            ],
            options={
                'ordering': ('-display_order', 'printable_name'),
                'verbose_name_plural': 'Countries',
                'verbose_name': 'Country',
                'abstract': False,
            },
            bases=(models.Model,),
        ),
        migrations.CreateModel(
            name='UserAddress',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('title', models.CharField(verbose_name='Title', max_length=64, blank=True, choices=[('Mr', 'Mr'), ('Miss', 'Miss'), ('Mrs', 'Mrs'), ('Ms', 'Ms'), ('Dr', 'Dr')])),
                ('first_name', models.CharField(max_length=255, verbose_name='First name', blank=True)),
                ('last_name', models.CharField(max_length=255, verbose_name='Last name', blank=True)),
                ('line1', models.CharField(max_length=255, verbose_name='First line of address')),
                ('line2', models.CharField(max_length=255, verbose_name='Second line of address', blank=True)),
                ('line3', models.CharField(max_length=255, verbose_name='Third line of address', blank=True)),
                ('line4', models.CharField(max_length=255, verbose_name='City', blank=True)),
                ('state', models.CharField(max_length=255, verbose_name='State/County', blank=True)),
                ('postcode', oscar.models.fields.UppercaseCharField(max_length=64, verbose_name='Post/Zip-code', blank=True)),
                ('search_text', models.TextField(editable=False, verbose_name='Search text - used only for searching addresses')),
                ('phone_number', oscar.models.fields.PhoneNumberField(verbose_name='Phone number', help_text='In case we need to call you about your order', blank=True)),
                ('notes', models.TextField(verbose_name='Instructions', help_text='Tell us anything we should know when delivering your order.', blank=True)),
                ('is_default_for_shipping', models.BooleanField(default=False, verbose_name='Default shipping address?')),
                ('is_default_for_billing', models.BooleanField(default=False, verbose_name='Default billing address?')),
                ('num_orders', models.PositiveIntegerField(default=0, verbose_name='Number of Orders')),
                ('hash', models.CharField(max_length=255, editable=False, db_index=True, verbose_name='Address Hash')),
                ('date_created', models.DateTimeField(auto_now_add=True, verbose_name='Date Created')),
                ('country', models.ForeignKey(verbose_name='Country', to='address.Country', on_delete=models.CASCADE)),
                ('user', models.ForeignKey(verbose_name='User', related_name='addresses', to=settings.AUTH_USER_MODEL, on_delete=models.CASCADE)),
            ],
            options={
                'ordering': ['-num_orders'],
                'verbose_name_plural': 'User addresses',
                'verbose_name': 'User address',
                'abstract': False,
            },
            bases=(models.Model,),
        ),
        migrations.AlterUniqueTogether(
            name='useraddress',
            unique_together=set([('user', 'hash')]),
        ),
    ]

sqlmigrate Subcommand

The SQL generated from the above migration file can be viewed with the sqlmigrate subcommand of manage.py.

Here is the sqlmigrate subcommand help:

Shell
(oscar) $./manage.py sqlmigrate -h
usage: manage.py sqlmigrate [-h] [--database DATABASE] [--backwards]
                              [--version] [-v {0,1,2,3}] [--settings SETTINGS]
                              [--pythonpath PYTHONPATH] [--traceback]
                              [--no-color] [--force-color] [--skip-checks]
                              app_label migration_name

  Prints the SQL statements for the named migration.

  positional arguments:
    app_label             App label of the application containing the migration.
    migration_name        Migration name to print the SQL for.

  optional arguments:
    -h, --help            show this help message and exit
    --database DATABASE   Nominates a database to create SQL for. Defaults to
                          the "default" database.
    --backwards           Creates SQL to unapply the migration, rather than to
                          apply it
    --version             show program's version number and exit
    -v {0,1,2,3}, --verbosity {0,1,2,3}
                          Verbosity level; 0=minimal output, 1=normal output,
                          2=verbose output, 3=very verbose output
    --settings SETTINGS   The Python path to a settings module, e.g.
                          "myproject.settings.main". If this isn't provided, the
                          DJANGO_SETTINGS_MODULE environment variable will be
                          used.
    --pythonpath PYTHONPATH
                          A directory to add to the Python path, e.g.
                          "/home/djangoprojects/myproject".
    --traceback           Raise on CommandError exceptions
    --no-color            Don't colorize the command output.
    --force-color         Force colorization of the command output.
    --skip-checks         Skip system checks. 

I tried using the --database option, so the generated SQL would refer to the proper database for my Frobshop web application, but after examining the generated SQL I discovered that it did not mention the database by name.

Shell
(oscar) $./manage.py sqlmigrate address 0001_initial
BEGIN;
--
-- Create model Country
--
CREATE TABLE "address_country" ("iso_3166_1_a2" varchar(2) NOT NULL PRIMARY KEY, "iso_3166_1_a3" varchar(3) NOT NULL, "iso_3166_1_numeric" varchar(3) NOT NULL, "printable_name" varchar(128) NOT NULL, "name" varchar(128) NOT NULL, "display_order" smallint NOT NULL CHECK ("display_order" >= 0), "is_shipping_country" boolean NOT NULL);
--
-- Create model UserAddress
--
CREATE TABLE "address_useraddress" ("id" serial NOT NULL PRIMARY KEY, "title" varchar(64) NOT NULL, "first_name" varchar(255) NOT NULL, "last_name" varchar(255) NOT NULL, "line1" varchar(255) NOT NULL, "line2" varchar(255) NOT NULL, "line3" varchar(255) NOT NULL, "line4" varchar(255) NOT NULL, "state" varchar(255) NOT NULL, "postcode" varchar(64) NOT NULL, "search_text" text NOT NULL, "phone_number" varchar(128) NOT NULL, "notes" text NOT NULL, "is_default_for_shipping" boolean NOT NULL, "is_default_for_billing" boolean NOT NULL, "num_orders" integer NOT NULL CHECK ("num_orders" >= 0), "hash" varchar(255) NOT NULL, "date_created" timestamp with time zone NOT NULL, "country_id" varchar(2) NOT NULL, "user_id" integer NOT NULL);
--
-- Alter unique_together for useraddress (1 constraint(s))
--
ALTER TABLE "address_useraddress" ADD CONSTRAINT "address_useraddress_user_id_hash_9d1738c7_uniq" UNIQUE ("user_id", "hash");
CREATE INDEX "address_country_iso_3166_1_a2_f395eed0_like" ON "address_country" ("iso_3166_1_a2" varchar_pattern_ops);
CREATE INDEX "address_country_display_order_dc88cde8" ON "address_country" ("display_order");
CREATE INDEX "address_country_is_shipping_country_f7b6c461" ON "address_country" ("is_shipping_country");
ALTER TABLE "address_useraddress" ADD CONSTRAINT "address_useraddress_country_id_fa26a249_fk_address_c" FOREIGN KEY ("country_id") REFERENCES "address_country" ("iso_3166_1_a2") DEFERRABLE INITIALLY DEFERRED;
ALTER TABLE "address_useraddress" ADD CONSTRAINT "address_useraddress_user_id_6edf0244_fk_auth_user_id" FOREIGN KEY ("user_id") REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "address_useraddress_hash_e0a6b290" ON "address_useraddress" ("hash");
CREATE INDEX "address_useraddress_hash_e0a6b290_like" ON "address_useraddress" ("hash" varchar_pattern_ops);
CREATE INDEX "address_useraddress_country_id_fa26a249" ON "address_useraddress" ("country_id");
CREATE INDEX "address_useraddress_country_id_fa26a249_like" ON "address_useraddress" ("country_id" varchar_pattern_ops);
CREATE INDEX "address_useraddress_user_id_6edf0244" ON "address_useraddress" ("user_id");
COMMIT; 

sqlcreate Subcommand

The sqlcreate subcommand generates the SQL to create the database.

Here is the sqlcreate subcommand help:

Shell
(oscar) $./manage.py sqlcreate -h
usage: manage.py sqlcreate [-h] [-R ROUTER] [--database DATABASE] [-D]
                             [--version] [-v {0,1,2,3}] [--settings SETTINGS]
                             [--pythonpath PYTHONPATH] [--traceback]
                             [--no-color] [--force-color]

  Generates the SQL to create your database for you, as specified in settings.py
  The envisioned use case is something like this: ./manage.py sqlcreate
  [--database=<databasename>] | mysql -u <db_administrator> -p ./manage.py
  sqlcreate [--database=<databasname>] | psql -U <db_administrator> -W

  optional arguments:
    -h, --help            show this help message and exit
    -R ROUTER, --router ROUTER
                          Use this router-database other then defined in
                          settings.py
    --database DATABASE   Nominates a database to run command for. Defaults to
                          the "default" database.
    -D, --drop            If given, includes commands to drop any existing user
                          and database.
    --version             show program's version number and exit
    -v {0,1,2,3}, --verbosity {0,1,2,3}
                          Verbosity level; 0=minimal output, 1=normal output,
                          2=verbose output, 3=very verbose output
    --settings SETTINGS   The Python path to a settings module, e.g.
                          "myproject.settings.main". If this isn't provided, the
                          DJANGO_SETTINGS_MODULE environment variable will be
                          used.
    --pythonpath PYTHONPATH
                          A directory to add to the Python path, e.g.
                          "/home/djangoprojects/myproject".
    --traceback           Raise on CommandError exceptions
    --no-color            Don't colorize the command output.
    --force-color         Force colorization of the command output. 

The output of the sqlcreate subcommand can be piped into the dbshell subcommand, and then all migrations can be applied, like this:

Shell
(oscar) $./manage.py sqlcreate | \
  ./manage.py dbshell && ./manage.py migrate

Resetting Migrations

I found myself resetting migrations several as I stumbled forward in my learning experience. Vitor Freitas wrote a helpful article that allowed me to write the following script for resetting migrations and setting up the database with my desired django-oscar shipping information:

reset
#!/bin/bash

# Reset migrations and set up the database with only US & Canada enabled for shipping

DB=ancient_warmth
set -e

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc" -delete

dropdb -U postgres -h localhost $DB
createdb -U postgres -h localhost $DB

./manage.py migrate --fake-initial  # See https://stackoverflow.com/a/37371482/553865
./manage.py makemigrations
./manage.py migrate

./manage.py oscar_populate_countries --no-shipping
bin/psql -c "update address_country set is_shipping_country = 't' where iso_3166_1_a2 in ('CA', 'US');"

DJANGO_SUPERUSER_EMAIL=admin@domain.com
DJANGO_SUPERUSER_PASSWORD=secret
./manage.py createsuperuser --noinput --username admin

This is my bin/psql script, used by the above:

bin/psql
#!/bin/bash

if [ "$1" == -t ]; then
  PREFIX="test_"
  shift
fi
PGPASSWORD=secret /usr/bin/psql \
  -U postgres \
  -h localhost \
  -d ${PREFIX}ancient_warmth \
  "$@"