A note describing some of the under-the-hood caveats of unit testing in Denali

denali-parallel-tests-and-unit-tests.md

tl;dr: Your Denali tests will run with normal Ava concurrency, except for unit tests. Unit tests will always be run serially, thanks to the quirks of parallel testing.

If you landed here, you might be wondering why your Denali app unit tests run serially - and there isn't even an option to allow them to run concurrently, which is the default for all other Denali tests, and for Ava in general.

The problem

Running tests concurrently means they cannot share state. For example:

let x;
let multiply = (y) => x * y;

test('when x = 2, multiply(2) returns 4', async (t) => {
  x = 2;
  await sleep(10); // simulate some async activity
  t.equals(multiply(2), 4);
});

test('when x = 4, multiply(2) returns 8', async (t) => {
  x = 4;
  await sleep(20); // simulate some async activity
  t.equals(multiply(2), 8);
});

In this test suit, the first test will likely always fail. Here's how it plays out:

1. Ava starts test one
2. x = 2
3. Test one waits for 10ms, so Ava starts test two
4. x = 4
5. Test two waits for 20ms, but while we wait, test one is back from sleep, so resume test one
6. Test one fails, because x = 4 when it runs multiply

In this case, it probably wouldn't be too hard to track this down. But in even a moderately complex web app's test suite, this quickly becomes nightmarishly difficult to trace.

Given the difficulty of these kinds of bugs to diagnose and isolate, Denali has to do whatever it can to prevent users from tripping up on them.

Parallel tests in Denali

Denali's dependency injection system relies on the container heavily. It allows for simple, concise dependency injection without verbose annotations on your classes, factory method wrappers, etc.

However, by it's very design, the container is shared state. It's where all the singletons for your app live.

Now imagine we want to test an email addon for Denali. Let's say we have a hypothetical Mailer class that manages the email templates, to, from, etc. That Mailer class delegates to another SMTPService to do the actual email server dance.

Let's imagine a classic unit test suite - when SMTPService responds with success, we want our Mailer to respond with true. When SMTPService returns an error from the SMTP server, we want Mailer to return false.

test('mailer sends an email', async (t) => {
  container.register('service:smtp', { send: () => 'ok!' });
  let result = await mailer.send('foo');
  t.equal(result, true);
});

test('mailer throws an error when smtp errors', async (t) => {
  container.register('service:smtp', { send: () => 'bad!' });
  let result = await mailer.send('foo');
  t.equal(result, false);
});

This test falls victim to the same bug in our multiply example above: because the container is shared state, the first test will likely always fail, since by the time the mailer in the first test talks to the SMTP mock, the second test will have started and mocked out failure.

Thus the core of the problem for Denali: since the container (a.k.a. shared state) is so pervasive, parallel tests are nearly impossible.

The solution

The result is that the appUnitTest helper forces your unit tests to be marked as serial, meaning Ava will run them one at a time. Between each test, the helper resets the state of the container, resulting in a clean state for each test.

Epilouge: app acceptance tests

You may have noticed that app acceptance tests do run in parallel though. What gives?

That's because when you run an app acceptance test, you never end up doing this:

import Foo from '../../app/bars/foo`;

Why does that matter?

Under the hood, each app acceptance test actually creates an entirely new copy of the app - container included. That means each app acceptance test has it's own container to work with - no shared state. And because the app is entirely self-contained behind t.context.app, we can ensure no state will leak across into other tests.

But unit tests make this impossible, because when you import Foo at the top of the file, once and outside any test, that leaves us stuck. You have imported a single copy of Foo to test - but each test actually needs it's own separate copy.

Basically, because you import Foo once at the top of the file, you have forced the use of shared state across all the tests in that file. But because that pattern doesn't appear in acceptance tests, the shared state isn't a problem, and we can safely parallelize them.