Possibly faster spec runs

exp.md

This is an experiment for possible faster boot time (and consequently faster feedback) when running tests. I ask your help to try this in your apps (any Rails 3.x app) and let me know how well it works.

How to

Add the following lines inside the configuration block in config/environments/test.rb:

config.cache_classes = true
def eager_load!; end

Notice that config.cache_classes needs to be true (which is the default), some tools may not be fine that. Needless to say, just do this for test environments.

Tests

The following two benches will work only for rspec running from the command line (hopefully you don't use rake to run your specs):

1. Pick up a spec file that runs all specs under 1s (for example, a file under spec/models or spec/helpers). Run the command below before and after you apply the eager_load! patch above (the bigger your application, the faster it should boot after you apply the patch):

   time rspec spec/models/SOME_MODEL_spec.rb

2. Try running your whole suite with and without the patch. In this case, you shouldn't see any perf difference and everything should work just fine (report if it doesn't!).

Let me know how 1) and 2) goes in the comments section below.

Without patch: 29.52 real 19.10 user 4.62 sys
With patch: 26.24 real 17.36 user 4.13 sys

• Single RSpec model spec

before: 11.34s, after: 9.00s✨

• Whole suite

(no difference)

Single spec
without patch
real 0m10.662s; user 0m8.287s; sys 0m0.934s;

with patch
real 0m7.848s; user 0m6.947s; sys 0m0.794s;

Whole suite
without patch
real 1m47.937s user 1m31.281s sys 0m3.974s

with patch
real 1m45.456s user 1m30.871s sys 0m3.989s
with a couple of failures

I ran 3 times

single spec

without change

real  0m4.127s
user  0m3.312s
sys 0m0.711s

real  0m4.140s
user  0m3.301s
sys 0m0.709s

real  0m4.097s
user  0m3.307s
sys 0m0.713s

with change

real  0m4.165s
user  0m3.316s
sys 0m0.713s

real  0m4.292s
user  0m3.328s
sys 0m0.725s

real  0m4.211s
user  0m3.323s
sys 0m0.727s

whole suite

without change

real  0m23.754s
user  0m13.202s
sys 0m1.562s

real  0m23.650s
user  0m12.893s
sys 0m1.710s

real  0m17.917s
user  0m12.983s
sys 0m1.505s

with change

real  0m20.474s
user  0m12.993s
sys 0m1.621s

eal 0m18.962s
user  0m12.867s
sys 0m1.493s

real  0m41.156s
user  0m13.029s
sys 0m1.794s

Singe test:
before: 15 seconds
after: 14 seconds

whole suite:
no difference

ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.4.0]

Spec file before

real    12.14s
user    10.15s
sys 1.63s

Spec file after

real    9.46s
user    7.98s
sys 1.35s

Spec file: 1 example, 0 failures. Same results with and without change.

Suite before

real    369.67s
user    164.59s
sys 13.26s

Suite after

real    366.51s
user    163.93s
sys 13.06s

Whole suite: 848 examples, 0 failures, 92 pending. Same results with and without change.

Most of the other comments are against pretty small sets, so I ran this against one of our larger projects (~1500 examples). This is Rails 3.2.6 on Ruby 1.9.3-p194.

I ran the tests several times. The results below are medians.

Model

Baseline

real    0m5.960s
user    0m5.360s
sys     0m0.524s

Patched

real    0m5.907s
user    0m5.386s
sys     0m0.515s

## Suite

Baseline

real    5m3.711s
user    4m58.806s
sys     0m3.704s

Patched

real    5m8.955s
user    5m2.633s
sys     0m3.643s

Entire suite

Baseline

Finished in 4 minutes 3.04 seconds
2699 examples, 0 failures, 3 pending

real  4m10.776s
user  4m2.966s
sys 0m6.156s

Patched

Finished in 4 minutes 3.63 seconds
2699 examples, 0 failures, 3 pending

real  4m10.949s
user  4m3.393s
sys 0m5.989s

Single spec file - 3 runs with each

Baseline

12 examples, 0 failures
Finished in 0.20853 seconds
Finished in 0.20797 seconds
Finished in 0.20495 seconds

real  0m4.882s 0m4.921s 0m4.914s
user  0m4.374s 0m4.365s 0m4.362s
sys   0m0.664s 0m0.705s 0m0.704s

Patched

12 examples, 0 failures
Finished in 0.26532 seconds
Finished in 0.27075 seconds
Finished in 0.26828 seconds

real  0m3.816s 0m3.749s 0m3.760s
user  0m3.428s 0m3.435s 0m3.440s
sys 0m0.528s 0m0.480s 0m0.484s