testing.md

Testing and Integration

• There are numerous articles in software literature about the motivation for unit tests and integration testing. Closer home The LSST policy described here is a useful source which addresses the kind of concerns we will have.
• Personally I think, the really big win for me (clearly writing unit tests is boring for people who could use the time to do more) is the increased confidence in generalizing/modifying/refactoring the codebase after a long break from developing the code without worrying that changes will break the code. Not surprisingly, talking to academics across many fields, this is the reason that seems to get the most buy-in from them.

Continuous Integration

• I think it is very useful to be able to use a continuous integration service. Travis is a popular one. The main advantage of this is to make sure the code works the way it is expected to with the install instructions.
• This is difficult from private repositories or with proprietary dependencies unless one pays for a travis account.
• Even if not doing it immediately, it may be worth keeping in mind how this might be done.
  • Useful to have a small data set for testing purposes in the repository. It cannot be large as the reposiory (on sites like github) has space limits and tends to get sluggish.
  • I have found it important to keep the data directory inside the python directory
  • It is useful to have a linter, and follow a style guide such as pep8. LSST DM has a style guide (which is close to pep8 for python and one for C++). Note that tools like autopep8 can help convert previously written code to pep8 compliant code.
  • If not possible, it is useful to host the data at a different location and have scripts in the code repository to curl in the data.
  • Since tests should be run often, it is important that the tests should take a small amount of time.
  • It is also useful to think about designing packages. DESC has a cookie-cutter for this purpose, and more general discussions regarding choices and purpose behind some of them are in shablona

Testing Framework in python

• unittest, nosetest, pytest are three well known frameworks.
• pytest seems to be the one that is becoming most popular. LSST seems to have moved to pytest. Can be used to test unittest classes and nose tests as well.

Testing

• Useful to organize tests (Like unittest classes, pytest fixtures): These are useful in constructing quantities once and running several non-destructive tests on these saving testing time and reusing code.
• Also useful for skipping tests if conditions are not met or having known failures or if there are known failures (xfail)
• If possible code producing random results should have a way of specifiying a random seed. In testing, it would be useful to fix that random seed and test results against old (human tested) results (technically this is an integration test and not a unit test).
• It is important to ensure that minimal dependencies are available, but you could escape tests that depend on other dependencies. Perhaps a good way to deal with these is to set up variables and use skipif as applicable
• In principle unit tests should test the logic of different conditionals but the results could be simple limiting cases or integration based tests.
• It is useful to use numpy.testing or pandas.util.testing assert methods
• I generally do tests on scalars, arrays, dataframes. It is also possible to do some tests on databases (but the ordering ambiguity makes a number of them impossible), but actually have never done tests on figures (except to check that they run and don't fail). However, it is possible to figure comparison using matplotlib.testing to test comparison of images. This is useful: for example, you could have all the figures in a paper saved, and then usei the imagecomparison features to ensure that further development of the code does not change enough to ring alarm bells (at least without you being aware of it!). An example of a book where this has been used is the astroml book.