Optimizing Rails fixtures

notes.md

I had a thought while riding to work today: how much faster would CI and tests be if we had a lot less fixtures. I had a suspicion that cutting down on our fixtures load would make CI a lot faster.

Let's test this hypothesis:

Given this test class:

class SomethingTest < ActiveSupport::TestCase
  test "something" do
  end
end

When I run Benchmark.ms { test('unit/something') }, I see this:

Tanda @ Rails 6.0.3.7 (test) :007 > Benchmark.ms { test('unit/something') }
Run options: --seed 62378

# Running:

-----------------------------
SomethingTest: test_something
-----------------------------
.

Finished in 0.965725s, 1.0355 runs/s, 0.0000 assertions/s.

1 runs, 0 assertions, 0 failures, 0 errors, 0 skips
3346.742656896822

ie. there's about 2.5s of unaccounted for time. And also, there's nearly 1s of test time not spent inside the test (we assume, since the test does nothing).

If I delete all the fixtures so that there's no fixtures for Rails to load, I get this:

Tanda @ Rails 6.0.3.7 (test) :008 > Benchmark.ms { test('unit/something') }
Run options: --seed 47190

# Running:

-----------------------------
SomethingTest: test_something
-----------------------------
.

Finished in 0.010474s, 95.4748 runs/s, 0.0000 assertions/s.

1 runs, 0 assertions, 0 failures, 0 errors, 0 skips
2046.133188996464

So it seems there's consistently around 900ms added to each test in loading fixtures. And then there's 2.5s in overhead in running the tests via a test console (so that may not be a problem in CI).

Let's make the test class bigger.

class SomethingTest < ActiveSupport::TestCase
  test "something" do
  end

  test "something2" do
  end

  test "something3" do
  end
end

With no fixtures:

Tanda @ Rails 6.0.3.7 (test) :010 > Benchmark.ms { test('unit/something') }
Run options: --seed 35867

# Running:

-----------------------------
SomethingTest: test_something
-----------------------------
.------------------------------
SomethingTest: test_something2
------------------------------
.------------------------------
SomethingTest: test_something3
------------------------------
.

Finished in 0.020280s, 147.9322 runs/s, 0.0000 assertions/s.

3 runs, 0 assertions, 0 failures, 0 errors, 0 skips
2448.007049970329

With fixtures:

Tanda @ Rails 6.0.3.7 (test) :016 > Benchmark.ms { test('unit/something') }
Run options: --seed 9946

# Running:

-----------------------------
SomethingTest: test_something
-----------------------------
.------------------------------
SomethingTest: test_something2
------------------------------
.------------------------------
SomethingTest: test_something3
------------------------------
.

Finished in 1.044996s, 2.8708 runs/s, 0.0000 assertions/s.

3 runs, 0 assertions, 0 failures, 0 errors, 0 skips
3062.3492379672825

So it takes basically the same amount of time if you have 1 test or 3 tests. That sounds weird. Fixtures are loaded in before setup, then the test runs in a transaction after they are loaded. But the docs say that method runs before every test.

Let's check:

class SomethingTest < ActiveSupport::TestCase
  def before_setup
    puts 'hi'
    super
  end

  test "something" do
  end

  test "something2" do
  end

  test "something3" do
  end
end

Output:

Tanda @ Rails 6.0.3.7 (test) :018 > Benchmark.ms { test('unit/something') }
Run options: --seed 41967

# Running:

hi
-----------------------------
SomethingTest: test_something
-----------------------------
.hi
------------------------------
SomethingTest: test_something2
------------------------------
.hi
------------------------------
SomethingTest: test_something3
------------------------------
.

Finished in 1.010500s, 2.9688 runs/s, 0.0000 assertions/s.

3 runs, 0 assertions, 0 failures, 0 errors, 0 skips
3092.0356039423496

🤷‍♂️ I guess there's something in Rails that makes the second fixture load faster than the first. I wonder if this works across files? I made two files in a dir that are identical.

Tanda @ Rails 6.0.3.7 (test) :020 > Benchmark.ms { test('unit/something/**') }
Run options: --seed 45892

# Running:

-----------------------------
SomethingTest: test_something
-----------------------------
.------------------------------
SomethingTest: test_something2
------------------------------
.------------------------------
SomethingTest: test_something3
------------------------------
.----------------------------------
SomethingElseTest: test_something3
----------------------------------
.---------------------------------
SomethingElseTest: test_something
---------------------------------
.----------------------------------
SomethingElseTest: test_something2
----------------------------------
.

Finished in 1.055187s, 5.6862 runs/s, 0.0000 assertions/s.

6 runs, 0 assertions, 0 failures, 0 errors, 0 skips
3486.501968000084

Yep. Looks like fixtures are loaded once (ever?) and then there's no real overhead for each additional test. So that means that decreasing our fixture load will probably not make CI faster (since we only incur the 1s overhead once).

So the original hypothesis was wrong. That said, we could shave up to 1s off each test run in a test console by cutting our fixture usage. Without a corresponding CI gain that's not very exciting though - the simplicity of using fixtures is pretty nice.

Turns out we use self.pre_loaded_fixtures = true so this is probably all wrong.