Caching http://registry.npmjs.org/ aggressively with squid
We run a lot of project in our automated test environment and they spend a lot of time doing npm install.
One way to speed up npm install and hammer the registry a little less is to use the http-version of the registry and to pass the requests through a proxy.
However, the npm requests include an authorization-header and the responses come with an ETag and Cache-Control: max-age=60, so with default squid settings there is a lot of roundtrips to registry.npmjs.org that result in 304's.
This is what the requests look like once npm has cached the first set of responses:
GET http://registry.npmjs.org/sntp HTTP/1.1
accept-encoding: gzip
authorization: Bearer d2fcf4e1-2660-42e2-8ea2-a0bae555b85d
version: 2.12.0
accept: application/json
referer: install
npm-session: f8ed55c4541125bc
user-agent: npm/2.12.0 node/v0.10.38 darwin x64
if-none-match: "BIQL559CK9WWT2W3XDIZP620E"
host: registry.npmjs.org
Connection: keep-alive
The response from the registry without proxies:
HTTP/1.1 304 Not Modified
server: CouchDB/1.5.0 (Erlang OTP/R16B03)
etag: "BIQL559CK9WWT2W3XDIZP620E"
Cache-Control: max-age=60
Accept-Ranges: bytes
Date: Fri, 26 Jun 2015 05:42:10 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-ams4123-AMS
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1435297330.212200,VS0,VE167
This squid refresh_pattern is aggressive enough to cache the requests effectively:
refresh_pattern registry.npmjs.org 900 20% 4320 ignore-auth ignore-private ignore-no-cache ignore-reload override-expire
With these settings you should see the following warnings when running service squid3 reload:
WARNING: use of 'ignore-auth' in 'refresh_pattern' violates HTTP
WARNING: use of 'ignore-reload' in 'refresh_pattern' violates HTTP
WARNING: use of 'override-expire' in 'refresh_pattern' violates HTTP
WARNING: use of 'override-lastmod' in 'refresh_pattern' violates HTTP
According to the source they are only printed when --enable-http-violations is enabled.
Even though --enable-http-violations is missing from the squid compile-time configuration, USE_HTTP_VIOLATIONS is enabled in the default Ubuntu package.
# squid3 -v
Squid Cache: Version 3.3.8
Ubuntu
configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security'
If you're not seeing those warning lines, to allow squid to shoot yourself in the foot, install squid from source with the
--enable-http-violations flag on:
apt-get build-dep squid3
wget http://www.squid-cache.org/Versions/v3/3.4/squid-3.4.13.tar.gz
./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --libexecdir=${prefix}/lib/squid3 --srcdir=. --disable-maintainer-mode --disable-dependency-tracking --disable-silent-rules --datadir=/usr/share/squid3 --sysconfdir=/etc/squid3 --mandir=/usr/share/man --enable-http-violations --enable-inline --enable-async-io=8 --enable-storeio=ufs,aufs,diskd,rock --enable-removal-policies=lru,heap --enable-delay-pools --enable-cache-digests --enable-underscores --enable-icap-client --enable-follow-x-forwarded-for --enable-auth-basic=DB,fake,getpwnam,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB --enable-auth-digest=file,LDAP --enable-auth-negotiate=kerberos,wrapper --enable-auth-ntlm=fake,smb_lm --enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group --enable-url-rewrite-helpers=fake --enable-eui --enable-esi --enable-icmp --enable-zph-qos --enable-ecap --disable-translation --with-swapdir=/var/spool/squid3 --with-logdir=/var/log/squid3 --with-pidfile=/var/run/squid3.pid --with-filedescriptors=65536 --with-large-files --with-default-user=proxy --enable-linux-netfilter
chmod 4755 /lib/squid3/pinger
cd /usr/sbin
mv squid3 squid3.old
ln -s squid squid3
service squid3 restart
When the HTTP violating rules are in place, squid no longer makes the round trip as eagerly and the responses from squid look like this:
HTTP/1.1 304 Not Modified
Date: Fri, 26 Jun 2015 05:07:38 GMT
Content-Type: application/json
Age: 1607
Warning: 110 squid/3.4.13 "Response is stale"
X-Cache: HIT from ci
X-Cache-Lookup: HIT from ci:3128
Via: 1.1 ci (squid/3.4.13)
Connection: keep-alive
No, I haven't had to work on squid since this event, but I'll be sure to take the latest version if I have to.