Using Jemalloc 5 with Ruby.md

file.md

For years, people have been using jemalloc with ruby. There were various benchmarks and discussions. Legend had it that Jemalloc 5 didn't work as well as Jemalloc 3.

Then, one day, hope appeared on the horizon. @wjordan offered a config for Jemalloc 5.

Ubuntu/Debian

FROM ruby:3.1.2-bullseye
RUN apt-get update ; \
    apt-get install -y --no-install-recommends libjemalloc2 ;
ENV LD_PRELOAD=libjemalloc.so.2
ENV MALLOC_CONF='dirty_decay_ms:1000,narenas:2,background_thread:true'

Alpine

FROM ruby:3.2.3-alpine3.19
RUN apk add --update jemalloc
ENV LD_PRELOAD=libjemalloc.so.2
ENV MALLOC_CONF='dirty_decay_ms:1000,narenas:2,background_thread:true'

Heroku

1. add the apt buildpack (maintained by Heroku staff and owned by the Heroku namespace, but still considered "third-party"/"unofficial")

   heroku buildpacks:add --index 1 heroku-community/apt

2. in the same branch, create and deploy these two files:
   • Aptfile

     libjemalloc2
     

   • .profile

     export LD_PRELOAD=libjemalloc.so.2
     export MALLOC_CONF=dirty_decay_ms:1000,narenas:2,background_thread:true

• ENV vars are set in .profile instead of config:set so that jemalloc is only used at runtime and not compile time, and to avoid (harmless) noise when first starting a shell. see discussion here
• There is also the third-party heroku-buildpack-jemalloc, which will download, build, and install an arbitrary jemalloc version, which is useful if you need a version not distributed by Ubuntu for some reason, such as before the discovery of the improved MALLOC_CONF, necessitating the installation of version 3.6

a note on LD_PRELOAD

On some systems (maybe older?) it might be necessary to provide the full path to the file, such as:

LD_PRELOAD=/usr/lib/libjemalloc.so.2
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

MALLOC_CONF options

• dirty_decay_ms:1000,narenas:2,background_thread:true comes from dockerfile-rails. code discussion
• This is the original "tradeoff between memory and performance". it is likely identical to the above config on recent Jemalloc and Ruby versions, because the extra values are the defaults dirty_decay_ms:1000,muzzy_decay_ms:0,narenas:2,background_thread:true,thp:never
  • muzzy_decay is 0 by default (MALLOC_CONF=stats_print:true ruby -e "exit")
  • thp:never is default behavior on ruby starting in 2.6 https://bugs.ruby-lang.org/issues/14705
• This is "more heavily memory-optimized, like jemalloc 3.6" (the version used by Fullstaq Ruby) dirty_decay_ms:0,muzzy_decay_ms:0,narenas:2,background_thread:true,thp:never
• This is used by gitlab 1 2 narenas:2

Check if jemalloc is working:

MALLOC_CONF="stats_print:true" ruby -e "exit" # will produce no output if jemalloc not being used, a wall of text if it is
MALLOC_CONF="$MALLOC_CONF,stats_print:true" ruby -e "exit" # if a config is set, you can include it like this

Links!

• jemalloc docs:
  • https://github.com/facebook/jemalloc/blob/dev/TUNING.md
  • http://jemalloc.net/jemalloc.3.html
• https://github.com/gaffneyc/heroku-buildpack-jemalloc
• https://github.com/fullstaq-ruby/server-edition#vs-ld_preloading-jemalloc-yourself
• https://news.ycombinator.com/item?id=27873579
• https://bugs.ruby-lang.org/issues/14718
  • comment from jemalloc dev: https://bugs.ruby-lang.org/issues/14718#note-86
• https://github.com/code-dot-org/code-dot-org/blob/staging/cookbooks/cdo-jemalloc/attributes/default.rb
• https://github.com/zooniverse/panoptes/pull/4027/files
• https://gist.github.com/jjb/81866452fd979cae81297cb28713b4bf
• https://www.paulashraf.com/blog/jemalloc-ruby
• https://github.com/SamSaffron/allocator_bench
• https://andresakata.medium.com/benchmark-of-memory-allocators-on-a-multi-threaded-ruby-program-354ec4dc2e7e
• https://engineering.appfolio.com/appfolio-engineering/2018/2/1/benchmarking-rubys-heap-malloc-tcmalloc-jemalloc
• https://twitter.com/wgjordan/status/1440574986264006659
• fullstaq-ruby/server-edition#98
• https://bugs.ruby-lang.org/issues/15667
• https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html
• https://github.com/kaspergrubbe/grubruby-jemalloc
• discussion on adding jemalloc to official ruby docker images docker-library/ruby#182
• bonus: building jemalloc 3 from source: jemalloc/jemalloc#2394

Info from Shopify employee in a slack group (shared with permission):

We use version 5 without any tuning, mostly because for us latency (generally) has priority over memory usage

At Gusto, we're using jemalloc 3 via fullstaq ruby

Info from a GitHub employee in a slack group (shared with permission):

jemalloc3: no MALLOC_CONF afaict

Apparently the ruby binary can be patched, so that neither compiling ruby with jemalloc nor loading it with LD_PRELOAD is necessary:

apt install patchelf
patchelf --add-needed libjemalloc.so.2 /usr/local/bin/ruby
apt-get purge -y patchelf

taken from here, more info here

We've been using jemalloc since 2018 and we literally went from needing to restart our Ruby app every 3-4 days as the memory leaks to memory never leaking, ever and taking a third of without jemalloc. I do not understand why this is not the default for all Ruby builds!

MALLOC_CONF="stats_print:true" ruby -e "exit" is producing a wall of text so I am assuming jemalloc¹ is working.

-ljemalloc is not output when running ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS'], RbConfig::CONFIG['MAINLIBS']". I am assuming that both this is fine/jemalloc is still used and this occurs because Ruby wasn't installed --with-jemalloc. Is this correct?

Footnotes

1. Latest installed Debian version via apt-get install --no-install-recommends -y libjemalloc2 ↩

@agrberg yes, that sounds correct

Noting that Rails' docker entrypoint template now does the lookup dynamically:

LD_PRELOAD=$(find /usr/lib -name libjemalloc.so.2 -print -quit)

https://github.com/rails/rails/blob/aa66aece44f191e6330c04ce4942c3edf31d85d7/railties/lib/rails/generators/rails/app/templates/docker-entrypoint.tt#L5C1-L6C1

@bensheldon thanks for the update! I try to keep my Dockerfile and docker-entrypoint generally the same and up-to-date with Rails' ¹. Currently I have the following in my Dockerfile in the base step where ENV is being set

# Set production environment
ENV RAILS_ENV="production" \
    …
    LD_PRELOAD="libjemalloc.so.2" \
    MALLOC_CONF="dirty_decay_ms:1000,narenas:2,background_thread:true"

I believe removing these ☝️ in favor of adding the following 👇 to docker-entrypoint would result in the same configuration as both set LD_PRELOAD and the MALLOC_CONFIG options section says these are the current default values.

# Enable jemalloc for reduced memory usage and latency.
if [ -z "${LD_PRELOAD+x}" ]; then
    LD_PRELOAD=$(find /usr/lib -name libjemalloc.so.2 -print -quit)
    export LD_PRELOAD
fi

Footnotes

1. One minor addition others may have is the inclusion of .tool-versions when using ruby file: ".tool-versions" in the Gemfile. ↩

@agrberg welp, already changed in Rails with some commentary, though it's now doing a symlink strat: rails/rails#55252