Unittest2 test discovery and real dotted-path named test selection for Django

runner.py

"""
An alternative Django ``TEST_RUNNER`` which uses unittest2 test discovery from
a base path specified in settings, rather than requiring all tests to be in
``tests`` module of an app.

If you just run ``./manage.py test``, it'll discover and run all tests
underneath the ``TEST_DISCOVERY_ROOT`` setting (a path). If you run
``./manage.py test full.dotted.path.to.test_module``, it'll run the tests in
that module (you can also pass multiple modules).

And (new in this updated version), if you give it a single dotted path to a
package, and that package does not itself directly contain any tests, it'll do
test discovery in all sub-modules of that package.

This code doesn't modify the default unittest2 test discovery behavior, which 
only searches for tests in files named "test*.py".

"""
from django.conf import settings
from django.test import TestCase
from django.test.simple import DjangoTestSuiteRunner, reorder_suite
from django.utils.importlib import import_module
from django.utils.unittest.loader import defaultTestLoader



class DiscoveryRunner(DjangoTestSuiteRunner):
    """A test suite runner that uses unittest2 test discovery."""
    def build_suite(self, test_labels, extra_tests=None, **kwargs):
        suite = None
        discovery_root = settings.TEST_DISCOVERY_ROOT

        if test_labels:
            suite = defaultTestLoader.loadTestsFromNames(test_labels)
            # if single named module has no tests, do discovery within it
            if not suite.countTestCases() and len(test_labels) == 1:
                suite = None
                discovery_root = import_module(test_labels[0]).__path__[0]

        if suite is None:
            suite = defaultTestLoader.discover(
                discovery_root,
                top_level_dir=settings.BASE_PATH,
                )

        if extra_tests:
            for test in extra_tests:
                suite.addTest(test)

        return reorder_suite(suite, (TestCase,))

settings.py

"""
You need the ``BASE_PATH`` and ``TEST_DISCOVERY_ROOT`` settings in order for
this test runner to work.

``BASE_PATH`` should be the directory containing your top-level package(s); in
other words, the directory that should be on ``sys.path`` for your code to
import. This is the directory containing ``manage.py`` in the new Django 1.4
project layout.

``TEST_DISCOVERY_ROOT`` should be the root directory to discover tests
within. You could make this the same as ``BASE_PATH`` if you want tests to be
discovered anywhere in your project. If you want tests to only be discovered
within, say, a top-level ``tests`` directory, you'd set ``TEST_DISCOVERY_ROOT``
as shown below.

And you need to point the ``TEST_RUNNER`` setting to the ``DiscoveryRunner``
class above.

"""
import os.path

# This is correct for the Django 1.4-style project layout; for the old-style
# project layout with ``settings.py`` and ``manage.py`` in the same directory,
# you'd want to only call ``os.path.dirname`` once.
BASE_PATH = os.path.dirname(os.path.dirname(__file__))

# This would be if you put all your tests within a top-level "tests" package.
TEST_DISCOVERY_ROOT = os.path.join(BASE_PATH, "tests")

# This assumes you place the above ``DiscoveryRunner`` in ``tests/runner.py``.
TEST_RUNNER = "tests.runner.DiscoveryRunner"

This could easily be further improved to be able to do partial discovery within multiple given packages.

Thanks for the help, Carl. I've tried creating a new project under Django 1.4c1. I've uploaded it here: http://dl.dropbox.com/u/422013/testsite.tar.gz

(virtualenv)chase@chase-mint ~/testsite $ ./manage.py test
BASE_PATH:  /home/chase/testsite
TEST_DISCOVERY_ROOT:  /home/chase/testsite/tests
Creating test database for alias 'default'...

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Destroying test database for alias 'default'...
(virtualenv)chase@chase-mint ~/testsite $ ./manage.py test tests
BASE_PATH:  /home/chase/testsite
TEST_DISCOVERY_ROOT:  /home/chase/testsite/tests
Creating test database for alias 'default'...
TestCase1
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
Destroying test database for alias 'default'...
(virtualenv)chase@chase-mint ~/testsite $ ./manage.py test tests.more
BASE_PATH:  /home/chase/testsite
TEST_DISCOVERY_ROOT:  /home/chase/testsite/tests
Creating test database for alias 'default'...
TestCase2
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
Destroying test database for alias 'default'...

I'm expecting the first and second commands to find both tests, the one in test/__init__.py and the one in test/more.py. Do you have a whole project you can share where it's working?

@chase-seibert Ah, I didn't think to document how unittest2 test discovery itself works. By default, only files named test*.py are searched for tests. You can change this by passing the pattern argument to the discover method (it takes a shell glob, the default value is "test*.py"). It would be easy enough to modify the above runner to respect a TEST_DISCOVERY_PATTERN setting.

Nailed it. Thanks!

Firstly, thanks for this. When I'm more hip I might use nose or something but this works perfectly.

Secondly there is a tiny mismatch between runner.py and settings.py. DiscoveryRunner in settings and DiscoveryDjangoTestSuiteRunner in runner.py. Tripped me up for a few seconds.

Thanks @mallison, fixed.

Nose (with django-nose, so you don't lose the Django db setup/teardown stuff) is a great option, too. The main reason for this gist is that it's pre-testing something we're hoping to put in Django core in time for 1.5, and requiring nose isn't really an option for core.

In case anyone is interested, I've packaged this up for general consumption (apologies @carljm, I needed it quick): http://pypi.python.org/pypi/django-discover-runner

I'm stuck on how to use this with Jenkins. If I want my Jenkins server to use this plugin, then I replace the "./manage.py jenkins" command that Jenkins was running with "./manage.py test". But now I'm no longer using the django-jenkins plugin command, I don't get the XML output that I think Jenkins needs in order to display the test results. Am I being dense?