James Hart hjhart

Effective Caching for Yarn, Bundler, and Rails Asset Pipeline in CircleCI

Phil Karlton said: "There are two hard problems in computer science: cache invalidation, naming things, and off-by-1 errors."

Well, maybe he didn't say exact that, but it's a decent joke anyway.

In this article I'll talk about the first of those problems, caching. And in doing so effectively, how we reduced our CircleCI build times by 33%. For those of you raising your eyebrows and saying "33% off of what?!", I'll direct your eyeballs to the graph below for some absolute numbers.

[ TODO: Show a bar graph of each job going down by a percentage ]

	# app/models/user.rb
	# username :string, null: false
	# email :string, null: false
	# password :string, null: false
	# admin :boolean, null: false, default: false

	class User < ApplicationRecord
	# Fields
	validates :admin, inclusion: { in: [true, false] }
	end

	update_homebrew_formula:
	description: Pulls in new homebrew formula and caches them
	steps:
	- run:
	name: Set up Cache Key
	command: date '+%Y %m %d' > ~/voom/dependency_checksum

	- restore_cache:
	keys:
	- v3-homebrew-{{ .Branch }}-{{ checksum "~/voom/dependency_checksum" }}

	precompile_assets:
	description: Precompile Rails Assets
	steps:
	- run:
	name: Set up Cache Key
	command: find ~/voom/app/javascript ~/voom/app/assets -type f -exec md5 -q {} \; > ~/voom/dependency_checksum

	- restore_cache:
	keys:
	- v3-assets-{{ arch }}-{{ .Branch }}-{{ checksum "~/voom/dependency_checksum" }}

	bundle_install:
	description: Bundle Install
	steps:
	- run:
	name: Expire cache key every month
	command: date '+%Y %m' > ~/voom/dependency_checksum

	- restore_cache:
	keys:
	- v1-bundle-{{ arch }}-{{ checksum "~/voom/dependency_checksum" }}-{{ checksum "~/voom/Gemfile.lock" }}

	bundle_install:
	description: Bundle Install
	steps:
	- restore_cache:
	keys:
	- bundle-{{ arch }}-{{ checksum "~/voom/Gemfile.lock" }}
	- bundle-{{ arch }}-

	- run:
	name: Install Bundler

	yarn_install:
	description: Yarn Install
	steps:
	- restore_cache:
	keys:
	- v1-yarn-{{ arch }}-{{ checksum "~/voom/yarn.lock" }}
	- v1-yarn-{{ arch }}-

	- run:
	name: "Yarn Install (Note: Bump cache prefix if this step takes over 30 seconds.)"

	git config --global alias.ahead "log origin/master..HEAD --oneline"
	for i in *; do echo $i && pushd $i && git ahead 2>/dev/null && popd; done

	# function happily borrowed from https://unix.stackexchange.com/a/259254
	bytesToHuman() {
	b=${1:-0}; d=''; s=0; S=(Bytes {K,M,G,T,E,P,Y,Z}iB)
	while ((b > 1024)); do
	d="$(printf ".%02d" $((b % 1024 * 100 / 1024)))"
	b=$((b / 1024))
	let s++
	done
	echo "$b$d ${S[$s]}"
	}