Humans are tool using creatures. When we want to drive in a nail, we use a hammer. When we want to dig a small hole, we use a shovel. When we want to connect something into a power outlet, we use a plug. Since there’s only one kind of hammer, shovel, or plug, it’s always easy to find the correct thing, right? Well, no, there are claw hammers, ball peen hammers, spades, shovels, 120V two-pronged and 3 pronged plugs, and those 220V plugs they use in Europe that I can never keep straight.
Happily, software systems and packages don’t suffer from these incompatible interface problems, right? Of course they do. Maintaining the ability for a system or package to operate correctly against different versions and environments can be quite challenging. And testing against all of the supported versions can drive one to distraction and convince maintainers to limit supported versions to what they can deal with.
In the Python world, tox (documentation) is a powerful testing tool that allows a project to test against many combinations of versioned environments. The django-coverage-plugin package (Github) uses tox to test against a matrix of Python versions (2.7, 3.4, 3.5, and 3.6) and Django versions (1.8, 1.9, 1.10, 1.11, 1.11tip, 2.0, 2.0tip), resulting in 25 valid combinations to test.
Preparing Your System Environments
tox needs to run from a virtual environment where it is installed and from which it’s run. As of Feb 2018, I would recommend a Python 2.7 environment so that you can use the detox package (see below) to parallelize your build’s workload. Installation of tox is usually into your base development environment and tox is usually included in your project’s requirements.txt file:
tox >= 1.8
detox
tox cannot install the different Python runtimes onto your system, so you’ll have to do so manually - usually by downloading and installing from your system’s package manager or directly from python.org. When versions of Python are installed in the usual way, there will be a command line executable with the version in its name (e.g. python2.7, python3.6, etc.). tox understands this naming convention and has built in configuration support for it.
Configuring List of Environments
Once you have your project’s initial environment and the necessary Python versions installed, it’s time to edit the tox config file called tox.ini. This configuration file is where you define the list of target environments (in the [tox] section using the envlist parameter), and then define the environments themselves, including required packages and commands to run using the [testenv] sections.
Here is the envlist configuration from django-coverage-plugin’s tox.ini:
[tox]
envlist =
py27-django{18,19,110,111,111tip},
py34-django{18,19,110,111,111tip,20},
py35-django{18,19,110,111,111tip,20,tip},
py36-django{18,19,110,111,111tip,20,tip},
check,doc
tox allows one to define configurations with list substitutions shell-style globs. tox will spin up a total of 27 different virtual environments given this envlist configuration. tox caches those virtual environments so that startup times are quick - except for the first time tox encounters a new environment configuration. The first 4 lines specify sets of environments that depend on the Python version (using tox’s built-in shorthand of pyVV) and versions of Django that support the given version of Python. The check, and doc environments do not rely on the pyVV version or other parts of the matrix, so they will run within the same virtual environment.
Configuring Main Environment
Configuration of an environment allows one to specifies dependencies (deps), the commands to run (commands), and other useful environment related items. Here’s the example of the default (testenv) environment configuration for the django-coverage-plugin:
[testenv]
deps =
unittest-mixins==1.1
django18: Django >=1.8, <1.9
django19: Django >=1.9, <1.10
django110: Django >=1.10, <1.11
django111: Django >=1.11, <2.0
Django111tip: https://github.com/django/django/archive/stable/1.11.x.tar.gz
django20: Django>=2.0,<2.1
djangotip: https://github.com/django/django/archive/master.tar.gz
commands =
python -c "import tests.banner"
python -m unittest {posargs:discover -b}
usedevelop = True
passenv = *
The deps config describes the packages to install into the target virtual environment, using the pip syntax. The above example installs unittest-mixins into all environments, and then specifies which version of Django to use for each target environment name. Because django-coverage-plugin is a testing tool, it’s important that it be tested against the latest development versions (aka tip) using the latest development branches. The commands config has tox run two commands: the first prints a banner showing the installed versions of Python and Django, and the second command actually runs the tests. tox allows passing of arguments down to test runs from the command line via the postargs variable. In this configuration, postargs is defined with a default value if not args are passed: ‘discover -b’. If you wanted to run a single test against all environments, you could execute: $ tox – tests.test_html.HtmlTest.test_simple
If you wanted to run that test against only one environment, you could execute:
$ tox -e py36-djangotip -- tests.test_html.HtmlTest.test_simple
Configuring Other Environments
The last two environments (check and doc) run useful operations. check runs flake8 against the source code, while doc builds a copy of the readme using rst2html.py from the excellent sphinx package.
[testenv:check]
deps =
flake8
commands =
flake8 --max-line-length=100 setup.py django_coverage_plugin tests
[testenv:doc]
deps =
sphinx
commands =
rst2html.py --strict README.rst /tmp/django_coverage_plugin_README.htm
Speeding Things Up with detox
A typical run of just $ tox on a django-coverage-plugin development environment that has already had tox build its virtual environments will take about 1 minute and 30 second on my 2013 Macbook Pro. However, if you use the detox package (which as of Feb 2018 only supports Python 2.7), the environments will run in parallel and the whole thing takes about 10 seconds. And 10 seconds is a much more manageable wait time for running a test suite during a development/debugging session.
Conclusion
tox is not a swiss army knife or multitool that can do a little bit of everything. Nor is it a fully tricked out table saw that you need for those testing enormous stand-alone applications. But if your package needs to work properly with a matrix of Python and/or other package dependencies, tox is the hex-key set that will fit your tests into the environments you need.
Photo credit: Photo by Barn Images on Unsplash