minitest-parallel-db.md

tl;dr

We can run tests in parallel, even with a database:

Use Minitest's parallelize_me!.
Use database transactions.
Use my gem to help you (unless you're using Rails, because Rails's setup lets you do it reasonably easily without the gem).
The gem: https://github.com/ordinaryzelig/minitest-parallel-db.
[Slides from my lightning talk] (http://redningja.com/ok_rb_parallel_testing).

Why & Disclaimer

How do you test that your code is 100% thread-safe? I don't know either. But one way to help is to see if your tests can run in parallel.

But first, a disclaimer: If you're intersted in making your tests run faster, this is not exactly the perfect solution. Yes, it will make your tests run upto X times faster if you run them in X threads. But you're more likely to find better test speed in refactoring.

If, however, you're interested in making your code more threadsafe or making sure your tests run [independently and in isolation] (http://blog.zenspider.com/blog/2012/12/minitest-parallelization-and-you.html), keep reading.

The problem

Most production web apps with a database run multiple processes and handle multiple requests at the same time. So why can't we run our tests concurrently? Usually the problem is that tests often need to be run in isolatation. But I propose we can still run our tests in "enough" isolation while running our tests in parallel.

First let's explore the problem a bit. Take a look at the following example:

describe '2 similar tests' do

  it 'saves a record' do
    model = Model.create!
    Model.last.id.must_equal model.id
  end

  it 'edits a record' do
    model = Model.create!
    model.update_attributes(name: 'changed')
    Model.last.name.must_equal 'changed'
  end

end

Bear with me for a second and ignore the fact that these tests are badly written. The important thing I want to point out is the problem with running these tests in parallel. It is entirely possible for the test suite to play out chronologically like this:

order	saves a record	edits a record
1	`model = Model.create!`
2		`model = Model.create!`
3		`model.update_attributes(name: 'changed')`
4		`Model.last.name.must_equal 'changed'`
5	`Model.last.id.must_equal model.id`

See the problem in the execution of the code for #5? Model.last will return the model that was created on the right-hand side, but the test is expecting the model that was created in #1.

Database transactions

Some databases like PostgreSQL support these neat little things called transactions. If you've ever used ActiveRecord, you've already used transactions. And it's likely you've used it manually for something like this:

ActiveRecord::Base.transaction do
  begin
    # Do lots of things in the database that need to work FLAWLESSLY or not at all!
  rescue
    # If anything goes wrong, pretend nothing happened,
    # and put the database back to where it was before we started.
    raise ActiveRecord::Rollback
  end
end

Database transactions also have another neat feature: they are only visible (for the most part) to that connection. (Read about unique indexes and write locks for more info.)

Let's do another example to demonstrate:

order	Test 1	Test 2
1	`Model.create!`
2		`Model.create!`
3	`Model.count.must_equal 1`
4		`Model.count.must_equal 1`

By the time #3 is executed, 2 models have been created. HOWEVER, if each test runs their code within their own transaction, then the queries performed within those transactions are only visible to the connection that executed those queries. That means Test 1 can't see the Model that Test 2 created (and vice versa) until the transactions are committed. If we just ensure that the transaction is undone (called a 'rollback') after each test, then no other test will ever be able to see anything any other test does. So, given the 2 tests are run in parallel and each is run within a transaction that will be rolled back, these tests will pass no matter what order their code is run chronologically.

Minitest to the rescue

Minitest has a neat little feature, parallelize_me!, that will make all or some of your tests run in parallel. That takes care of the parallel part. If we can just ensure that each test is run inside a transaction, then we can get this idea to work. That's what my minitest-parallel-db gem tries to accomplish. It uses Minitest's parallelize_me! feature, database transactions, and ORMs' connection pools to run tests in parallel with a single database.

Rails

If you're using a recent version of Rails, and you're using Postgres, and you're using ActiveRecord, you're in luck. You don't even need my gem. At some point, Rails began using Minitest as the standard. That means you get parallel testing for free. If you haven't disabled transactional fixtures, you get your tests in transactions automatically. Here is what I did to get a brand new Rails app tested in parallel:

rails new project # Rails 4.1.1, Minitest 5.3.4.
Install 'pg' gem.
Edit config/database.yml to use postgres.
bin/rails g model user
rake db:migrate
Edit test/models/user_test.rb:

require 'test_helper'

Minitest.parallel_executor = Minitest::Parallel::Executor.new(5)

class UserTest < ActiveSupport::TestCase

  parallelize_me!

  5.times do |idx|
    test "creates a user #{idx}" do
      user = User.create!
      sleep 1
      assert_equal User.last.id, user.id
    end
  end

end

rake
Boom: Finished in 1.096245s, 4.5610 runs/s, 4.5610 assertions/s.

Caveat

You need to be careful with mocking because they are not guaranteed to be thread-safe. However, Minitest's mocking library is 95% thread-safe, and it mostly just depends on how you use it. Stubbing is isolated in a block, so as long as the object you're stubbing only exists within the test, you should be OK (i.e. don't stub methods on constants). A Mock object is just another local variable that only lives as long as the test. To demonstrate, I made a [gist with some examples of different libraries run in parallel] (https://gist.github.com/ordinaryzelig/f3a48b2456eac28bbbca).

Conclusion

Parallel testing seems to be within reach, but there is still work to be done. According to their readme, mocha doesn't appear to be immediately concerned about becoming thread-safe. [RSpec is in the process of becoming thread-safe] (rspec/rspec-mocks#380). But as I understand it, this work is being done to support multiple threads in production code, not parallel testing. For now, the only way I know of to accomplish this is with Minitest.

ordinaryzelig/minitest-parallel-db.md