Testing

Unit tests

valjean uses pytest as a unit-test framework; some of the unit tests rely on the hypothesis package for property-based testing. The tests are defined in the tests folder and are automatically discovered by pytest.

You can run the unit-test suite with:

$ pytest

This will run the tests and produce a summary report. Additionally, pytest will calculate code coverage; a text report will be printed to stdout, and a nice HTML report will be written to tests/htmlcov.

The default pytest options are defined in the pytest.ini file, and coverage options are defined in .coveragerc, using pytest-doc. Extra options to pytest can be passed on the command line:

$ pytest tests/cosette/test_rlist.py  # run all tests in the named file
$ pytest tests/cosette/test_rlist.py::test_insert  # run this test only
$ pytest -k depgraph        # only run tests whose name matches "depgraph"
$ pytest -v                 # increase verbosity level by number of v
$ pytest --verbosity=N      # set verbosity test output
$ pytest -x                 # stop on the first test failure
$ pytest --ff               # run previously failed tests first

The --verbosity option sets all the valjean loggers to the verbosity level N. N=0 is equivalent to WARNING, N=1 to INFO (default) and N>1 to DEBUG. The verbosity level can also be increased by invoking the -v option, possibly multiple times. It is useful when debugging a failing test.

Property-based testing and the hypothesis package

In traditional unit testing, one verifies that the code gives the expected answer for a given, fixed set of input values. For instance, one may test a sort algorithm by verifying that it produces [1,2,3,4] when fed [4,2,3,1]. However, this does not check that the algorithm behaves sensibly on any of the following arguments:

  • empty lists;

  • long lists;

  • lists that are already sorted;

  • lists of objects other than integers…

You see the point. One could write additional, specific unit tests to address each of these limitations. However, it is impossible to make sure that we have not forgotten some important edge case; also, we will need to write essentially the same tests over and over, and nobody likes doing the same thing over and over. But wait a minute, isn’t “doing the same thing over and over” the kind of thing that computers are actually good at?

Enter property-based testing. The idea of this approach is that the developer should only check the expected properties of the code to be tested. For instance, in the case of the sort algorithm, the developer could check the following invariants about the list returned by the algorithm:

  • it is sorted;

  • it contains the same elements as the input list.

The testing library will generate a number of random inputs, call the sorting algorithm on each of them and check that the properties specified by the developer hold. If a counterexample is found, it will be shown to the user.

In Python, the hypothesis library offers a flexible and well-documented API for property-based testing. Additionally, hypothesis is well integrated with pytest. valjean unit tests leverage hypothesis whenever possible.

If you want some examples within valjean, a good place to start is the tests.cosette.test_rlist test module, which tests the invariants of the RList class.

Testing example docstrings with pytest

Sometimes the docstrings contain example code such as the following:

>>> print(1+2)
3

These examples are also automatically tested with pytest.

tox integration

There are specific tox test environments to run the unit tests. Check the page about using tox for continuous integration.