Unit-test framework requirements

testing-framework.md

• API to group test cases into test suites, e.g. describe and it

• setup/teardown hooks (before, beforeEach, after, afterEach)

• Support for promises: test case returns a promise, the test runner waits until the promise is resolved

• Output formatters: junit/xunit for CI, spec-like for humans (a list of failed tests is printed again at the end)

• Run all tests in a single file, run a single test case, run a single test suite.

• Capture stdout/stderr and associate it with tests. spec-like formatter should repeat the captured stdout/stderr for failed tests.

• Parameterised tests, for example

  it('returns true for prime numbers', [2,3,4], function(number) {
    expect(isPrime(number)).to.be.true();
  });

• Runs in node.js, browser, karma+phantomjs

• Nice to have: supported by IDEs like WebStorm

A little while ago I experimented with making minimal tests runnable as standalone scripts:

var assert = require('assert');
var foo = require('../lib/foo');

assert.equal(foo.bar(1), 2, 'foo.bar(1) should be 2');
assert.equal(foo.bar(2), 3, 'foo.bar(2) should be 3');

Worked ok, but if I had multiple failures I had to fix them one at a time to reveal the next failure. And the output was eerily silent when there are no failures. I wrote tapsert so that I could replace the first line with var assert = require('tapsert'); and have the tests magically output TAP, complete with multiple failures, without using a test runner or abandoning my vanilla style asserts.

Formatters: I've come to prefer TAP for both CI and my consumption.

Runners: I prefer smaller test files that are more focussed which allows you to narrow the tests to run by only specifying a single file. Ideally that file can be run as a standalone node script and doesn't require a special runner.