Created
January 10, 2013 06:11
-
-
Save xaviershay/4499828 to your computer and use it in GitHub Desktop.
Proof-of-concept squid proxy settings for proxying http://rubygems.org/ Tested on one project on my local machine both cold, and warm with internet turned off.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env ruby | |
# url rewriter for rubygems squid proxy | |
STDOUT.sync = true | |
while line = gets | |
url = line.split(' ')[0] | |
# Cargo-culted this conditional, not sure if it is necessary | |
response = if url | |
# Need to find a better way to cache this redirect, rather than hard-coding | |
# it. | |
if url.include?("http://rubygems.org/gems/") | |
"302:" + url.gsub( | |
"http://rubygems.org/", | |
"http://production.cf.rubygems.org/" | |
) | |
else | |
"" | |
end | |
else | |
"" | |
end | |
puts response | |
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
########################################################################## | |
# | |
# Proof-of-concept squid proxy settings for proxying http://rubygems.org/ | |
# Caches everything so is resilient to network outage or upstream outage. | |
# Works with bundler, gives a significant speed up when cache is warm (100%+) | |
# | |
########################################################################## | |
# Disable default patterns that prevent caching of query params | |
# refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 | |
# refresh_pattern . 0 20% 4320 | |
# Cache calls to resolve dependency list for at least 15 minutes, at max a day | |
# TODO: What is the actual behaviour of this? Desired is refresh after 15, but | |
# continue to work for a day if upstream is unavailable. | |
refresh_pattern -i /api/v1/dependencies 900 100% 86400 | |
# "Cache" redirects so that the proxy is completely isolated from the internet | |
# when it is warm. | |
url_rewrite_program rewrite.rb | |
# 1 Gb disk cache. Probably larger is better. | |
cache_dir ufs /usr/local/var/cache/squid 1024 16 256 | |
# BELOW THIS LINE IS DEFAULT SQUID CONFIG | |
# Recommended minimum configuration: | |
# | |
# Example rule allowing access from your local networks. | |
# Adapt to list your (internal) IP networks from where browsing | |
# should be allowed | |
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network | |
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network | |
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network | |
acl localnet src fc00::/7 # RFC 4193 local private network range | |
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines | |
acl SSL_ports port 443 | |
acl Safe_ports port 80 # http | |
acl Safe_ports port 21 # ftp | |
acl Safe_ports port 443 # https | |
acl Safe_ports port 70 # gopher | |
acl Safe_ports port 210 # wais | |
acl Safe_ports port 1025-65535 # unregistered ports | |
acl Safe_ports port 280 # http-mgmt | |
acl Safe_ports port 488 # gss-http | |
acl Safe_ports port 591 # filemaker | |
acl Safe_ports port 777 # multiling http | |
acl CONNECT method CONNECT | |
# | |
# Recommended minimum Access Permission configuration: | |
# | |
# Only allow cachemgr access from localhost | |
http_access allow localhost manager | |
http_access deny manager | |
# Deny requests to certain unsafe ports | |
http_access deny !Safe_ports | |
# Deny CONNECT to other than secure SSL ports | |
http_access deny CONNECT !SSL_ports | |
# We strongly recommend the following be uncommented to protect innocent | |
# web applications running on the proxy server who think the only | |
# one who can access services on "localhost" is a local user | |
#http_access deny to_localhost | |
# | |
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS | |
# | |
# Example rule allowing access from your local networks. | |
# Adapt localnet in the ACL section to list your (internal) IP networks | |
# from where browsing should be allowed | |
http_access allow localnet | |
http_access allow localhost | |
# And finally deny all other access to this proxy | |
http_access deny all | |
# Squid normally listens to port 3128 | |
http_port 3128 | |
# Leave coredumps in the first cache dir | |
coredump_dir /usr/local/var/cache/squid |
I would be interested to see how successful this is, have you run a report on using the logs produced ?
http://cord.de/tools/squid/calamaris/ is what i use to gather information about HITS/MISSES and how much is "saved".
I want a rubygems caching proxy to work. It seems like with the heavy use of RVM and various ruby interpreters, installing gems faster would be a significant gain.
I know I'm a year late but playing with Docker has got me really hoping for a way to cache gem downloads. Anyone ever come up with a good solution?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@xaviershay, what about the call that bundler makes for dependency resolution? I wonder if
bundle install --full-index
would use a cached gems index in that case? That might work.