I tried to use nailgun which is supposed to speedup jruby usage by sharing a single jvm over multiple jruby instances. It all seems to be a bit broken but this is what works for me:
In .rvmrc change --1.9
to --Xcompat.version=1.9
to avoid a stupid bug
Run jruby --ng-server
in one terminal
Now env JRUBY_OPTS="$JRUBY_OPTS --ng" bundle exec rake spec
should give you somewhat better startup time (factor 2 for me).