ScalaTest parallel testing

scalatest-parallelism.md

Docs

• Async testing
• Using the runner:

  If you include -P on the command line, Runner will pass a Distributor to the Suites you specify with -s. Runner will set up a thread pool to execute any Suites passed to the Distributor's put method in parallel. Trait Suite's implementation of runNestedSuites will place any nested Suites into this Distributor. Thus, if you have a Suite of tests that must be executed sequentially, you should override runNestedSuites as described in the documentation for Distributor.

  The -P option may optionally be appended with a number (e.g. "-P10" -- no intervening space) to specify the number of threads to be created in the thread pool. If no number (or 0) is specified, the number of threads will be decided based on the number of processors available.

Notes

• AsyncTestSuite mixin:

  • requires tests to return Future[Assertion]
  • executionContext by default is serial, i.e tests will run in one queue, single thread per test:

  When ScalaTest's serial execution context is called upon to execute a task, that task is recorded in a queue for later execution. For example, one task that will be placed in this queue is the task that transforms the Future[Assertion] returned by an asynchronous test body to the FutureOutcome returned from the test function. Other tasks that will be queued are any transformations of, or callbacks registered on, Futures that occur in your test body, including any assertions you map onto Futures. Once the test body returns, the thread that executed the test body will execute the tasks in that queue one after another, in the order they were enqueued. [...] This thread confinement strategy does mean, however, that when you are using the default execution context on the JVM, you must be sure to never block in the test body waiting for a task to be completed by the execution context. If you block, your test will never complete. This kind of problem will be obvious, because the test will consistently hang every time you run it.

  • can be overriden by

    implicit override def executionContext = ExecutionContext.Implicits.global

• parallel suites execution:

  • on by default in sbt: IntegrationTest / parallelExecution := true
  • number of threads by defauls is (number of available processors) ✖️ 2
  • can be controlled by IntegrationTest / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-P16") (16 can be any number)

• parallel tests execution (within the same suite) is controlled by the ParallelTestExecution mixin:

  By default (unless you mix in ParallelTestExecution), tests in an asynchronous style traits will be executed one after another, i.e., serially.

  If ParallelTestExecution is mixed in but parallel execution of suites is not enabled, asynchronous tests on the JVM will be started sequentially, by the single thread that invoked run, but without waiting for one test to complete before the next test is started. As a result, asynchronous tests will be allowed to complete in parallel, using threads from the executionContext.

  If you use an execution context backed by a thread-pool, such as global, however, even though tests will be started sequentially by one thread, they will be allowed to run concurrently using threads from the execution context's thread pool.

• 2 thread pools:

  • Distributor launcher suites and tests (controlled by -P)
  • executionContext in the Async... trait collects results

  On the JVM, if both ParallelTestExecution is mixed in and parallel execution is enabled in the build, tests in an async-style suite will be started in parallel, using threads from the Distributor, and allowed to complete in parallel, using threads from the executionContext.