Make Docker Cache Gem Installs to speed up Bundle Install

README.md

Gem Install Dockerfile Hack

If you're hacking on your Gemfile and using Docker, you know the pain of having the bundle install command run after you've added or removed a gem. Using docker-compose you could mount a volume and stage your gems there, but this adds additional complexity and doesn't always really solve the problem.

Enter this imperfect solution:

What if we installed every gem into it's own Docker layer which would be happily cached for us?

gem-inject-docker does just that. It takes the list of gems used by your app via bundle list and transforms it into a list of RUN gem install <your gem> -v <gem version> statements and injects them into the Dockerfile at a point of your choosing.

It additionally caches this list, and only appends new gems to the end to prevent a full rebuild for all previously installged gems. If a gem is removed it is removed from the list of instructions and the cache. This does not seem to trigger Docker to rebuild all layers.

Pros

• Will absolutely increase the speed of your gem-update related Docker builds.

Cons

• Requires you to use a build script (see build.sh)
• Requires you to add ./tmp/docker-cache to your .gitignore file. If you do not do this, other people's builds might not benefit from the speed improvement
• Creates a bunch of Docker layers in the local cache

Using

1. Add a bin folder to your project
2. Add gem-inject-docker.rb and build.sh to that path
3. Make them executable: chmod +x bin/gem-inject-docker.rb bin/build.sh
4. Instead of running docker build . [...] run bin/build

The first build will take longer than you are used to as a new layer is created for every gem. This is normal (for this hack). Subsequent builds will benefit from this step, however.

build.sh

#!/usr/bin/env bash
./bin/gem-inject-docker > Dockerfile.injected && \
  docker build . -f Dockerfile.injected && \
  rm -f Dockerfile.injected

gem-inject-docker.rb

#!/usr/bin/env ruby
# HACK: This file is here to inject each gem
# dependency into the Dockerfile as a separate line.
#
# Doing so will increase your first build time, but
# will allow subsequent builds to complete rather
# quickly.
#
# Previously injected gems are cached to ./tmp/docker-cache
# in docker-gems.cache.
#
#
require 'fileutils'

VERSION_MATCH = /\(?((\d+)?\.(\d+)?\.(\d+)\.?(\d+)?)\)?/.freeze

# List the gems you wish not to inject here.
# This is useful when you have gems that are
# sourced from GitHub or another repo that
# will not work with `gem install`
IGNORED_GEMS = [
].freeze

# Where would you like the list of previously
# injected gems cached?
TMP_DIR = './tmp/docker-cache'.freeze
CACHE_FILE = File.join(TMP_DIR, 'docker-gems.cache').freeze

# This token should be placed at the point in
# your Dockerfile that you would like the gem
# install instructions injected
TOP_DELIMETER = '# --- INJECT GEMS HERE ---'.freeze

# Filters the gems listed in the IGNORED_GEMS
# above
def filter_ignored(gems)
  gems.reject { |g| IGNORED_GEMS.any? { |i| g.start_with?(i) } }
end

# Returns the list of gems in your Gemfile
def bundled_gems
  gem_list = `bundle list`.split("\n")
  gem_list[1..gem_list.length - 1].map do |gem|
    gem[4..-1]
  end
end

# Tranforms the list into Docker `gem install`
# instructions
def generate_docker_gem_install(gem_list)
  gem_list.map do |g|
    name = g.split(' ')[0]
    version = (VERSION_MATCH.match(g) || [])[1]
    "RUN gem install #{name} -v #{version}" if version && (!g.include? 'default')
  end.reject(&:nil?).join("\n")
end

# Tries to load the cached list of gems
def load_previous_gems
  File.read(CACHE_FILE).split("\n")
rescue StandardError
  []
end

# Saves the new list to the cachefile
def save_new_gem_list(list)
  FileUtils.mkdir_p TMP_DIR unless Dir.exist? TMP_DIR
  File.open(CACHE_FILE, 'w') do |file|
    file.write(list.join("\n"))
  end
end

# Loads the dockerfile
def load_dockerfile
  dockerfile = File.read('Dockerfile')
  return dockerfile if dockerfile.include? TOP_DELIMETER

  warn 'Dockerfile does not contain the injection point.'
  warn "Add #{TOP_DELIMETER} at the point you would like the gems inserted"
  exit(2)
end

# Injects the `gem install` instructions into
# the Dockerfile and returns the result
def inject_dockerfile
  dockerfile = load_dockerfile
  cached_gems = load_previous_gems
  required_gems = filter_ignored(bundled_gems)
  new_gems = required_gems - cached_gems
  removed_gems = cached_gems - required_gems
  new_list = (cached_gems + new_gems) - removed_gems

  save_new_gem_list(new_list)
  dockerfile.sub(TOP_DELIMETER, TOP_DELIMETER + "\n" +
    generate_docker_gem_install(new_list))
end

# output to STDOUT
$stdout.puts inject_dockerfile