Possibly faster spec runs

exp.md

This is an experiment for possible faster boot time (and consequently faster feedback) when running tests. I ask your help to try this in your apps (any Rails 3.x app) and let me know how well it works.

How to

Add the following lines inside the configuration block in config/environments/test.rb:

config.cache_classes = true
def eager_load!; end

Notice that config.cache_classes needs to be true (which is the default), some tools may not be fine that. Needless to say, just do this for test environments.

Tests

The following two benches will work only for rspec running from the command line (hopefully you don't use rake to run your specs):

1. Pick up a spec file that runs all specs under 1s (for example, a file under spec/models or spec/helpers). Run the command below before and after you apply the eager_load! patch above (the bigger your application, the faster it should boot after you apply the patch):

   time rspec spec/models/SOME_MODEL_spec.rb

2. Try running your whole suite with and without the patch. In this case, you shouldn't see any perf difference and everything should work just fine (report if it doesn't!).

Let me know how 1) and 2) goes in the comments section below.

ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.4.0]

Spec file before

real    12.14s
user    10.15s
sys 1.63s

Spec file after

real    9.46s
user    7.98s
sys 1.35s

Spec file: 1 example, 0 failures. Same results with and without change.

Suite before

real    369.67s
user    164.59s
sys 13.26s

Suite after

real    366.51s
user    163.93s
sys 13.06s

Whole suite: 848 examples, 0 failures, 92 pending. Same results with and without change.

Most of the other comments are against pretty small sets, so I ran this against one of our larger projects (~1500 examples). This is Rails 3.2.6 on Ruby 1.9.3-p194.

I ran the tests several times. The results below are medians.

Model

Baseline

real    0m5.960s
user    0m5.360s
sys     0m0.524s

Patched

real    0m5.907s
user    0m5.386s
sys     0m0.515s

## Suite

Baseline

real    5m3.711s
user    4m58.806s
sys     0m3.704s

Patched

real    5m8.955s
user    5m2.633s
sys     0m3.643s

Entire suite

Baseline

Finished in 4 minutes 3.04 seconds
2699 examples, 0 failures, 3 pending

real  4m10.776s
user  4m2.966s
sys 0m6.156s

Patched

Finished in 4 minutes 3.63 seconds
2699 examples, 0 failures, 3 pending

real  4m10.949s
user  4m3.393s
sys 0m5.989s

Single spec file - 3 runs with each

Baseline

12 examples, 0 failures
Finished in 0.20853 seconds
Finished in 0.20797 seconds
Finished in 0.20495 seconds

real  0m4.882s 0m4.921s 0m4.914s
user  0m4.374s 0m4.365s 0m4.362s
sys   0m0.664s 0m0.705s 0m0.704s

Patched

12 examples, 0 failures
Finished in 0.26532 seconds
Finished in 0.27075 seconds
Finished in 0.26828 seconds

real  0m3.816s 0m3.749s 0m3.760s
user  0m3.428s 0m3.435s 0m3.440s
sys 0m0.528s 0m0.480s 0m0.484s