How to Leapfrog a Massive Django/Python Upgrade Successfully

Migrating from Django 1.7 to 3.1 is no small task. In fact, one might call it a monster of a job. Besides having to deal with the upgrade of Django itself, you shouldn't forget that an upgrade like this implies an upgrade from Python 2 to 3 as well.

Although somebody is likely bold enough to try and upgrade directly from Django 1.7 to 3.2, it isn't what our team would recommend, and we're confident very few experienced Django developers will encourage a direct upgrade. Faced with such a massive multi-version upgrade, the best advice we can give you is to divide and conquer. Practically speaking, that means upgrade from 1.7 to 1.8, 1.8 to 1.9, 1.9 to 1.10, etc., until you reach your destination.

As far as upgrading Python is concerned, know that Django 1.11 was the last to support Python 2.7. However, Django 1.11 was already compatible with Python 3, so in the divide-and-conquer spirit, you should attempt the upgrade to Python 3 right after you migrated to Django 1.11.

Read the Release Notes

Before you attempt an upgrade of a particular Django version, it is prudent to review the release notes for the version that you are upgrading to.

Release notes for older versions are not where you might expect them to be, so you might only find them by searching for “Django 1.8 release notes” or looking through the docs. The release notes for Django 1.8 can be found here.

Backwards incompatible changes and deprecations are of particular importance when reviewing the release notes. If you go to the Django deprecation timeline, you can see all deprecations going back all the way to version 1.7.

Make sure that you have addressed all deprecation warnings before upgrading to the next version. Django will show deprecation warnings when you run tests with

python manage.py test 

or when you run the check command with

python manage.py check

Usually, the deprecation warnings are very explicit and most helpful. For example:

RemovedInDjango110Warning: Session verification will become mandatory in Django 1.10. Please add 'django.contrib.auth.middleware.SessionAuthenticationMiddleware' to your MIDDLEWARE_CLASSES setting when you are ready to opt-in after reading the upgrade considerations in the 1.8 release notes.

No Tests, No Go

Unit tests are vital when attempting upgrades of any nature. In the absence of unit tests, you only have time-consuming manual testing to exercise the code. In fact, in the absence of unit tests, do not upgrade! It would be wiser to delay the upgrade and add enough unit tests to cover at least 50% of the project.

Use pip-tools to Manage Requirements

Once you have reviewed the release notes of the version, fixed all deprecation warnings and updated test coverage, it is time to update the version of Django in your requirements.txt. Or perhaps not! If you are still stuck on version 1.7, chances are that you are not using something like pip-tools to generate the requirements.txt file for you. Rather than updating the version of each package individually, you can use the pip-compile command to compile a requirements.txt file for you from an existing setup.py or a requirements.in file.

When you install a particular version of Django, let's say Django 1.11, pip will install a whole bunch of dependencies along with it. When using pip-compile, simply specify the version requirement for Django itself in requirements.in (e.g, for Django 1.11 this would be Django<2.0). pip-compile will use this requirement to compile a requirements.txt file with specific versions for Django and all the dependent packages in turn.

When upgrading to Django 2.0, you only have to update the requirement to Django<2.1. You might have additional packages that you use in your project like Django Rest Framework or some packages to use AWS or Google Cloud APIs that also need to be specified in your requirements.in file, but that list of packages should be significantly smaller that the full list of versioned requirements that pip-compile will generate and save to requirements.txt. Checkout the Project's Github for detailed examples of how pip-tools can make your package version management significantly simpler.

After updating requirements.in to point to the next Django version and you've compiled a new requirements.txt file with pip-compile, go ahead and install the latest package versions with

pip install -r requirements.txt

Testing, Testing, Testing

Now you are ready to run your test suite again to see what is breaking in the new version. If you have acceptance tests in Selenium or Robot, run those additionally to exercise the code as much as possible. Once all deprecation warnings are fixed and all tests pass, you are ready for the next Django version. Rinse and repeat. 

When you reach Django 1.11, it's time to upgrade Python as well. At this point, it is recommended to completely remove the virtual environment you were using for Python 2.7 and recreate one for Python 3. Once you have a new Python 3 virtual env, use pip-compile to create a new requirements.txt file. Even though you have not upgraded any packages, pip-compile might highlight compatibility issues that relate to the Python 3 upgrade. 

Immediately after upgrading to Python 3, the project might be in a severely broken state. Knowing what the major changes were will help prepare you for this upgrade. Some notable changes are:

• The move to using unicode universally for strings
• Explicit use of byte strings
• Syntax of the print statement and the syntax of exceptions

There are several guides that will help you migrate from Python 2 to 3 and there are even useful tools like 2to3 that will convert code automatically for you.

Iterate calmly through the breakages and use the test suite as well as the command:

python manage.py check

to test if all issues are addressed. Additionally, to ensure Django starts up successfully, make sure to run:

python manage.py runserver

Finishing Up

When Django runs smoothly on Python 3, you are past the worst of it. What remains is a repeat of the process described above to upgrade Django itself.

Note you will still need to review any custom template changes manually since unit tests are not really meant to test the user interface. 

Attempting such a big upgrade might be daunting, but with a careful systematic approach in combination with the right tools, you can succeed. Good luck!