ruby-1.9.3-p327 cumulative performance patch for rbenv

0-readme.md

ruby-1.9.3-p327 cumulative performance patch for rbenv

This installs a patched ruby 1.9.3-p327 with various performance improvements and a backported COW-friendly GC, all courtesy of funny-falcon.

Requirements

• rbenv
• ruby-build

You will also need a C Compiler. If you're on Linux, you probably already have one or know how to install one. On OS X, you should install XCode, and brew install autoconf using homebrew.

Ok, let's do this!

curl https://raw.github.com/gist/1688857/rbenv.sh | sh ; rbenv global 1.9.3-p327-perf

BONUS!

Putting the following in your shell config (eg. ~/.bash_profile) will make Rails even faster, but will increase its memory footprint. For most applications, this will probably work well for you:

export RUBY_GC_MALLOC_LIMIT=60000000
export RUBY_FREE_MIN=200000

If you have a particularly large rails app (say, >100k lines), this seems to improve it even more:

export RUBY_GC_MALLOC_LIMIT=1000000000
export RUBY_FREE_MIN=500000
export RUBY_HEAP_MIN_SLOTS=40000

More BONUS!

If you're feeling adventurous, you can set some additional compile flags before you run the command above. These may cause the build to fail. If so, you can try again without the march flag, and it'll probably work. This generally makes my ruby ~1.5% faster. See http://en.gentoo-wiki.com/wiki/Safe_Cflags/Intel for more guidance. I had to change it to -march=core2 on my Macbook Air 2011/11".

export CFLAGS="-march=native -O3 -pipe -fomit-frame-pointer"

This has to be run BEFORE you compile ruby.

2-1.9.3-p327-patched.sh

build_package_combined_patch() {
  local package_name="$1"

  {
    curl https://raw.github.com/gist/4136373/falcon-gc.diff | patch -p1
    autoconf
    ./configure --prefix="$PREFIX_PATH" $CONFIGURE_OPTS
    make -j 8
    make install
  } >&4 2>&1

}

install_package "yaml-0.1.4" "http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz"
install_package "ruby-1.9.3-p327" "http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p327.tar.gz" combined_patch

rbenv.sh

VERSION="1.9.3-p327"
curl https://raw.github.com/gist/1688857/2-$VERSION-patched.sh > /tmp/$VERSION-perf
rbenv install /tmp/$VERSION-perf

Thanks @samgranieri and @julian7, I forgot that line was in there. I've updated the gist.

I would change the 'install XCode' to reference the Command Line Tools instead. See my fork here: https://gist.github.com/3453141/76972c7185b579f5542d12cf56eae8d068c908ca

This patch is available in rvm stable, so if you don't want to live on the edge you can do rvm get stable instead.

@burke, new version is here https://gist.github.com/3721565

@burke I think you should update your instructions to use RVM stable rather than head, because the patch is already available in RVM stable and at the moment head seems quite unstable. It took me lots of digging and trial and error until I figured that I failed to compile Ruby with this patch because I was on the head version rather stable.

VERY cool! Here are my before stats, the base 1.9.3-p194:

mwhalen at Ajax in ~/Sites/askherzl on master!
± time rails r "puts 1"
Connecting to database specified by database.yml
1
rails r "puts 1"  4.23s user 1.19s system 54% cpu 10.009 total

And here is the same command after installing the patch, adding the suggested lines to my ~/.zshrc and re-running bundle:

mwhalen at Ajax in ~/Sites/askherzl on master!
± time rails r "puts 1"
Connecting to database specified by database.yml
1
rails r "puts 1"  2.95s user 0.90s system 58% cpu 6.549 total

That's pretty much an exact 35% increase in performance. Installing gem's feels a great deal quicker as well.

Any idea why execution would actually be slower with this patch? Using RVM on a retina MBP:

ruby-1.9.3-p194:

$ time rails runner 'puts 3'
real    0m7.454s
user    0m6.481s
sys 0m0.957s

ruby-1.9.3-p194-perf:

$ time rails runner 'puts 3'
real    0m12.410s
user    0m11.347s
sys 0m1.036s

ruby-1.9.3-p194-perf with exports

$ time rails runner 'puts 3'
real    0m9.609s
user    0m8.344s
sys 0m1.250s

@dvandersluis
the same with you.

Should we be using falcon's new patch now?

I tried benchmarking falcon's new patch on top of vanilla 1.9.3-p194 against 1.9.3-p194-perf: (using avgtime):

$ rvm current && avgtime -q -r5 -d -p rails runner "puts 1"
ruby-1.9.3-p194-perf

------------------------
Total time (ms): 46419.8
Repetitions    : 5
Sample mode    : 9320 (1 ocurrences)
Median time    : 9282.5
Avg time       : 9283.95
Std dev.       : 23.6096
Minimum        : 9257.65
Maximum        : 9321.53
95% conf.int.  : [9237.68, 9330.23]  e = 46.2739
99% conf.int.  : [9223.14, 9344.77]  e = 60.8142
EstimatedAvg95%: [9263.26, 9304.65]  e = 20.6943
EstimatedAvg99%: [9256.76, 9311.15]  e = 27.1969
Sorted times   : 
    [9257.65, 9261.26, 9282.5, 9296.83, 9321.53]

$ rvm current && avgtime -q -r5 -d -p rails runner "puts 1"
ruby-1.9.3-p194-falcon-new

------------------------
Total time (ms): 35149.3
Repetitions    : 5
Sample mode    : 6970 (2 ocurrences)
Median time    : 7032.1
Avg time       : 7029.87
Std dev.       : 52.5191
Minimum        : 6970.93
Maximum        : 7113.02
95% conf.int.  : [6926.93, 7132.8]  e = 102.936
99% conf.int.  : [6894.59, 7165.15]  e = 135.28
EstimatedAvg95%: [6983.83, 7075.9]  e = 46.0342
EstimatedAvg99%: [6969.37, 7090.37]  e = 60.4991
Sorted times   : 
    [6970.93, 6977.6, 7032.1, 7055.69, 7113.02]

There are some issues w/ RVM when installing these. If you rename the ruby (by calling it 1.9.3-perf) or whatever, it won't include the cflags patch. So you'll actually lose quite a bit of performance doing this. Be sure to include --patch cflags.patch Also it is best to include --disable-shared

kewl! will surely test it when I get home. Thanks for hard work.

kewl! will surely test it when I get home. Thanks for hard work.

BUILD FAILED

Inspect or clean up the working tree at /var/folders/50/nmb34mx94js1fd1z_s5fl8pr0000gn/T/ruby-build.20121130224625.6476
Results logged to /var/folders/50/nmb34mx94js1fd1z_s5fl8pr0000gn/T/ruby-build.20121130224625.6476.log

Last 10 log lines:
ossl_x509store.c:520: warning: ‘X509_STORE_CTX_set_flags’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:464)
ossl_x509store.c: In function ‘ossl_x509stctx_set_purpose’:
ossl_x509store.c:532: warning: ‘X509_STORE_CTX_set_purpose’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:460)
ossl_x509store.c: In function ‘ossl_x509stctx_set_trust’:
ossl_x509store.c:544: warning: ‘X509_STORE_CTX_set_trust’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:461)
ossl_x509store.c: In function ‘ossl_x509stctx_set_time’:
ossl_x509store.c:561: warning: ‘X509_STORE_CTX_set_time’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:466)
linking shared-object dl/callback.bundle
linking shared-object openssl.bundle
make: *** [build-ext] Error 2

Mother of God...

Running this patch against p327 is working. I'm running RVM. Here are my steps (pretty simple):

brew install autoconf automake
curl https://raw.github.com/gist/4136373/falcon-gc.diff > $rvm_path/patches/ruby/1.9.3/p327/falcon.patch
rvm install 1.9.3-p327 -n fast --patch falcon

Then recreate my gemsets and bundle of course.

Benchmarks?

p286:

time rake
...
real    0m48.761s
user    0m41.117s
sys     0m3.561s

time rails runner puts 'foobar'
....
real    0m4.939s
user    0m4.210s
sys     0m0.719s

p327 + patch:

time rake
...
real    0m20.775s
user    0m15.543s
sys     0m2.208s

time rails runner puts 'foobar'
...
real    0m2.610s
user    0m2.047s
sys     0m0.553s

Greatest patch ever!!

This is really great. I ran a few benchmarks against 2 separate apps. We're seeing a 28% - 49% speed increase depending on the test!

The rbenv installation gets build failures on Mountain Lion:

~Code/goodbre.ws master % curl https://raw.github.com/gist/1688857/rbenv.sh | sh ; rbenv global 1.9.3-p327-perf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   138  100   138    0     0    180      0 --:--:-- --:--:-- --:--:--   467
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   448  100   448    0     0   1194      0 --:--:-- --:--:-- --:--:--  1539
Downloading yaml-0.1.4.tar.gz...
-> http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
Installing yaml-0.1.4...
Installed yaml-0.1.4 to /Users/david/.rbenv/versions/1.9.3-p327-perf

Downloading ruby-1.9.3-p327.tar.gz...
-> http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p327.tar.gz
Installing ruby-1.9.3-p327...

BUILD FAILED

Inspect or clean up the working tree at /var/folders/tt/0zgjrcdd4zn0pvfmmp4s9n600000gq/T/ruby-build.20121206132529.63012
Results logged to /var/folders/tt/0zgjrcdd4zn0pvfmmp4s9n600000gq/T/ruby-build.20121206132529.63012.log

Last 10 log lines:
ossl_x509store.c: In function ‘ossl_x509stctx_set_flags’:
ossl_x509store.c:520: warning: ‘X509_STORE_CTX_set_flags’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:464)
ossl_x509store.c: In function ‘ossl_x509stctx_set_purpose’:
ossl_x509store.c:532: warning: ‘X509_STORE_CTX_set_purpose’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:460)
ossl_x509store.c: In function ‘ossl_x509stctx_set_trust’:
ossl_x509store.c:544: warning: ‘X509_STORE_CTX_set_trust’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:461)
ossl_x509store.c: In function ‘ossl_x509stctx_set_time’:
ossl_x509store.c:561: warning: ‘X509_STORE_CTX_set_time’ is deprecated (declared at /usr/include/openssl/x509_vfy.h:466)
linking shared-object openssl.bundle
make: *** [build-ext] Error 2

Looks like the problem was that my CONFIGURE_OPTS weren't getting picked up. If you're getting build failures on OSX, install it with the following so that it gets compiled correctly with support for the right OpenSSL and Readline libs:

brew install openssl readline
curl https://raw.github.com/gist/1688857/2-$VERSION-patched.sh > /tmp/1.9.3-p327-perf
CONFIGURE_OPTS="--with-openssl-dir=`brew --prefix openssl` --with-readline-dir=`brew --prefix readline`" rbenv install /tmp/1.9.3-p327-perf

Boy I sure wish that gist comments were editable. That should have been:

brew install openssl readline
curl https://raw.github.com/gist/1688857/2-1.9.3-p327-patched.sh > /tmp/1.9.3-p327-perf
CONFIGURE_OPTS="--with-openssl-dir=`brew --prefix openssl` --with-readline-dir=`brew --prefix readline`" rbenv install /tmp/1.9.3-p327-perf

Is this patch still relevant for 1.9.3-p362?

As expected applying it to p362 doesn't work.

I got issues when compiling, and I would see this:

checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: C compiler cannot create executables

I tracked down the issue to the "-march=native"-flag. When removed or changed to "-march=nocona" (on my Late 2008 Macbook with a Intel Core 2 Duo running Mountain Lion) it worked just fine.

I hope others won't hit this bug by reading this, because it has costed me some time tracking it down.

Mountain Lion+RVM users follow instructions by @cannikin.

And if you are getting Error running 'make ' you might have to append --with-gcc=gcc so the line looks like: ``rvm install 1.9.3-p327 -n fast --patch falcon --with-gcc=gcc`

ruby-build has S3-based mirrors for the various dependencies, which you can make use of in this gist, for times when sites like pyyaml.org go down.

To do so, change line 14 and 15 of 2-1.9.3-p327-patched.sh to be:

install_package "yaml-0.1.4" "http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz#36c852831d02cf90508c29852361d01b"
install_package "ruby-1.9.3-p327" "http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p327.tar.gz#96118e856b502b5d7b3a4398e6c6e98c" combined_patch

(I'm simply pulling these out of ruby-build's relevant manifest files here.)

That will attempt to download from the source and then fall-back to the S3 mirror if needed.

This hasn't been touched in a while, but with the new gist links, things will likely break. I've made a version of the rbenv definition file that should work, for the time being.
https://gist.github.com/umtrey/7592062

To get rid of the following error:

regparse.c:582:15: error: implicit conversion loses integer precision: 'st_index_t' (aka 'unsigned long') to 'int' [-Werror,-Wshorten-64-to-32]

In addition to https://gist.github.com/umtrey/7592062, I had to do the following first

brew update
brew tap homebrew/dupes
brew install apple-gcc42
export CC=gcc-4.2

Then rbenv install /tmp/1.9.3-p327-perf

Thanks @m-shirshendu! That did the trick.

Trying to install with ruby-install I got the following error (Using GCC 4.2):

....
compiling proc.c
compiling file.c
make: *** No rule to make target `pool_alloc.inc.h', needed by `gc.o'.  Stop.
!!! Compiling ruby 1.9.3-p327 failed!