Unit-test framework requirements

testing-framework.md

• API to group test cases into test suites, e.g. describe and it

• setup/teardown hooks (before, beforeEach, after, afterEach)

• Support for promises: test case returns a promise, the test runner waits until the promise is resolved

• Output formatters: junit/xunit for CI, spec-like for humans (a list of failed tests is printed again at the end)

• Run all tests in a single file, run a single test case, run a single test suite.

• Capture stdout/stderr and associate it with tests. spec-like formatter should repeat the captured stdout/stderr for failed tests.

• Parameterised tests, for example

  it('returns true for prime numbers', [2,3,4], function(number) {
    expect(isPrime(number)).to.be.true();
  });

• Runs in node.js, browser, karma+phantomjs

• Nice to have: supported by IDEs like WebStorm

One thing twisted unit tests do is that promises can return promises... so a test can return a promise, which can resolve to another promise, etc... so the test doesn't complete until the promise resolves to a non-promise.

Honestly, I'm a fan of tap, as well. It could be that most of my tests are essentially integration tests, not unit tests, in that they test, for example, how strong-cluster-control integrates with node's cluster module, or how strong-pm integrates with strong-supervisor. In particular, the fact that node resources leak between tests, requiring more and more elaborate afterEach implementations to guarantee cleanup, and that the cleanup is pretty much guaranteed not to work in the case of failed tests (aka bugs), was a big pain when using mocha for the strong-cluster-control and strong-supervisor tests. They would have been much better written with tap, because tap waits for node to complete, which gives easy detection of resource leaks (node does it), and it gives guaranteed cleanup of resources on failure (because node also cleans up everything when assert forces node to exit).

There are two node tap implementations, at least, one works in the browser.

Many of the other things you mention could, I think, be best done as a layer on top of tap. For example, a wrapper around tap.test() that checks if the test function returns a promise, and waits for the promise chain to resolve before calling tap's done() callback.

I'm close to writing a tap wrapper, actually, because the biggest pain I have with node-tap (besides some small bugs and its abominable lack of documentation), is that when you use tap.test(), there is no easy way to only run matching tests.

And to your list of desired features, I would add:

• tests can be run WITHOUT a runner (for many reasons, among them that runners interact very poorly with debuggers such as node-inspector... or profilers, etc.).

A little while ago I experimented with making minimal tests runnable as standalone scripts:

var assert = require('assert');
var foo = require('../lib/foo');

assert.equal(foo.bar(1), 2, 'foo.bar(1) should be 2');
assert.equal(foo.bar(2), 3, 'foo.bar(2) should be 3');

Worked ok, but if I had multiple failures I had to fix them one at a time to reveal the next failure. And the output was eerily silent when there are no failures. I wrote tapsert so that I could replace the first line with var assert = require('tapsert'); and have the tests magically output TAP, complete with multiple failures, without using a test runner or abandoning my vanilla style asserts.

Formatters: I've come to prefer TAP for both CI and my consumption.

Runners: I prefer smaller test files that are more focussed which allows you to narrow the tests to run by only specifying a single file. Ideally that file can be run as a standalone node script and doesn't require a special runner.