Using Jemalloc 5 with Ruby.md

file.md

For years, people have been using jemalloc with ruby. There were various benchmarks and discussions. Legend had it that Jemalloc 5 didn't work as well as Jemalloc 3.

Then, one day, hope appeared on the horizon. @wjordan offered a config for Jemalloc 5.

Ubuntu/Debian

FROM ruby:3.1.2-bullseye
RUN apt-get update ; \
    apt-get install -y --no-install-recommends libjemalloc2 ;
ENV LD_PRELOAD=libjemalloc.so.2
ENV MALLOC_CONF='dirty_decay_ms:1000,narenas:2,background_thread:true'

Alpine

FROM ruby:3.2.3-alpine3.19
RUN apk add --update jemalloc
ENV LD_PRELOAD=libjemalloc.so.2
ENV MALLOC_CONF='dirty_decay_ms:1000,narenas:2,background_thread:true'

Heroku

1. add the apt buildpack (maintained by Heroku staff and owned by the Heroku namespace, but still considered "third-party"/"unofficial")

   heroku buildpacks:add --index 1 heroku-community/apt

2. in the same branch, create and deploy these two files:
   • Aptfile

     libjemalloc2
     

   • .profile

     export LD_PRELOAD=libjemalloc.so.2
     export MALLOC_CONF=dirty_decay_ms:1000,narenas:2,background_thread:true

• ENV vars are set in .profile instead of config:set so that jemalloc is only used at runtime and not compile time, and to avoid (harmless) noise when first starting a shell. see discussion here
• There is also the third-party heroku-buildpack-jemalloc, which will download, build, and install an arbitrary jemalloc version, which is useful if you need a version not distributed by Ubuntu for some reason, such as before the discovery of the improved MALLOC_CONF, necessitating the installation of version 3.6

a note on LD_PRELOAD

On some systems (maybe older?) it might be necessary to provide the full path to the file, such as:

LD_PRELOAD=/usr/lib/libjemalloc.so.2
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

MALLOC_CONF options

• dirty_decay_ms:1000,narenas:2,background_thread:true comes from dockerfile-rails. code discussion
• This is the original "tradeoff between memory and performance". it is likely identical to the above config on recent Jemalloc and Ruby versions, because the extra values are the defaults dirty_decay_ms:1000,muzzy_decay_ms:0,narenas:2,background_thread:true,thp:never
  • muzzy_decay is 0 by default (MALLOC_CONF=stats_print:true ruby -e "exit")
  • thp:never is default behavior on ruby starting in 2.6 https://bugs.ruby-lang.org/issues/14705
• This is "more heavily memory-optimized, like jemalloc 3.6" (the version used by Fullstaq Ruby) dirty_decay_ms:0,muzzy_decay_ms:0,narenas:2,background_thread:true,thp:never
• This is used by gitlab 1 2 narenas:2

Check if jemalloc is working:

MALLOC_CONF="stats_print:true" ruby -e "exit" # will produce no output if jemalloc not being used, a wall of text if it is
MALLOC_CONF="$MALLOC_CONF,stats_print:true" ruby -e "exit" # if a config is set, you can include it like this

Links!

• jemalloc docs:
  • https://github.com/facebook/jemalloc/blob/dev/TUNING.md
  • http://jemalloc.net/jemalloc.3.html
• https://github.com/gaffneyc/heroku-buildpack-jemalloc
• https://github.com/fullstaq-ruby/server-edition#vs-ld_preloading-jemalloc-yourself
• https://news.ycombinator.com/item?id=27873579
• https://bugs.ruby-lang.org/issues/14718
  • comment from jemalloc dev: https://bugs.ruby-lang.org/issues/14718#note-86
• https://github.com/code-dot-org/code-dot-org/blob/staging/cookbooks/cdo-jemalloc/attributes/default.rb
• https://github.com/zooniverse/panoptes/pull/4027/files
• https://gist.github.com/jjb/81866452fd979cae81297cb28713b4bf
• https://www.paulashraf.com/blog/jemalloc-ruby
• https://github.com/SamSaffron/allocator_bench
• https://andresakata.medium.com/benchmark-of-memory-allocators-on-a-multi-threaded-ruby-program-354ec4dc2e7e
• https://engineering.appfolio.com/appfolio-engineering/2018/2/1/benchmarking-rubys-heap-malloc-tcmalloc-jemalloc
• https://twitter.com/wgjordan/status/1440574986264006659
• fullstaq-ruby/server-edition#98
• https://bugs.ruby-lang.org/issues/15667
• https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html
• https://github.com/kaspergrubbe/grubruby-jemalloc
• discussion on adding jemalloc to official ruby docker images docker-library/ruby#182
• bonus: building jemalloc 3 from source: jemalloc/jemalloc#2394

Capturing some Rails Performance Slack commentary:

After switching from x86_64 to ARM64 on ECS Fargate (Graviton), memory usage started growing significantly. Turned out jemalloc takes in the build host's page size at compile time (GitHub Actions ARM64 runners have a 64KB-page kernel and Fargate runs on 4KB pages). Jemalloc couldn't release partially-used 64KB pages and RSS was increasing. Fixed it by rebuilding jemalloc with --with-lg-page=12 to force 4KB pages.

You can check kernel page size with getconf PAGE_SIZE, and check jemalloc page size with MALLOC_CONF=stats_print:true /bin/true 2>&1 | grep "Page size"

Compiling it in Docker:

ARG JEMALLOC_VERSION=5.3.1
ARG JEMALLOC_SHA256=3826bc80232f22ed5c4662f3034f799ca316e819103bdc7bb99018a421706f92
RUN set -eux; \
  curl -fsSL https://github.com/jemalloc/jemalloc/releases/download/$JEMALLOC_VERSION/jemalloc-$JEMALLOC_VERSION.tar.bz2 -o /tmp/jemalloc.tar.bz2; \
  echo "$JEMALLOC_SHA256  /tmp/jemalloc.tar.bz2" | sha256sum -c -; \
  cd /tmp && tar -xjf jemalloc.tar.bz2; \
  cd /tmp/jemalloc-$JEMALLOC_VERSION && ./configure --prefix=/usr/local --disable-static --with-lg-page=12 && make -j$(nproc) && make install_lib; \
  strip --strip-unneeded /usr/local/lib/libjemalloc.so.2; \
  rm -rf /tmp/jemalloc*
  
  
# and loading with
ENV LD_PRELOAD=/usr/local/lib/libjemalloc.so.2

jemalloc/jemalloc@6790c57