More techniques - caching apt packages network-wide

This guide runs through setting up AptCacherNg on a proxy/server to provide apt caching to clients. The client proxy scripts allow for multiple proxies, and account for proxies being offline.

Note that, for obvious reasons, this only works on Debian-based machines (Ubuntu and derivatives as well).

Server (the cache/proxy)

Note: you will need port 3142 open to the clients using this cache/proxy

Install apt-cacher-ng

sudo apt install apt-cacher-ng avahi-daemon
sudo cp /etc/apt-cacher-ng/acng.conf /etc/apt-cacher-ng/acng.conf.bak

Configure sources

Edit /etc/apt-cacher-ng/acng.conf using your favorite editor (you need to be root/run as sudo)

Scrolling down a bit, you should see a section similar to the following:

Remap-debrep: file:deb_mirror*.gz /debian ; file:backends_debian # Debian Archives
Remap-uburep: file:ubuntu_mirrors /ubuntu ; file:backends_ubuntu # Ubuntu Archives
Remap-cygwin: file:cygwin_mirrors /cygwin # ; file:backends_cygwin # incomplete, please create this file or specify preferred mirrors here
Remap-sfnet:  file:sfnet_mirrors # ; file:backends_sfnet # incomplete, please create this file or specify preferred mirrors here
...

At the bottom of that section, add lines for your frequently-used repositories. We'll add Linux Mint (with its mirrors) and Docker as examples in this write-up.

For LM and Docker, do as follows:

Remap-lmrep:  file:lm_mirrors /lm ; file:backends_lm # Linux Mint Archives
Remap-dockerrep:  file:docker_mirrors /docker ; file:backends_docker # Docker Archives

Scrolling down a bit more, you'll see a section about a "precache". For Debian, Ubuntu, Debian Security, LM, and Docker, we'll add/modify this line as follows:

PrecacheFor: debrep/dists/*/*/*/Packages* uburep/dists/*/*/*/Packages* lmrep/dists/*/*/*/Packages* secdeb/dists/*/*/*/Packages* dockerrep/dists/*/*/*/Packages*

Finally, scrolling down even more, you should see a section about "pass-through". You'll need to set this to the following:

PassThroughPattern: ^(.*):443$

Create the file /etc/apt-cacher-ng/backends_lm with the following content:

http://packages.linuxmint.com/

Now, create the file /usr/lib/apt-cacher-ng/lm_mirrors with the following content:

Note: list copied from the "Repository mirrors" section of the Linux Mint website on December 6, 2020 so you may want to verify that this list is up-to-date

http://mirrors.evowise.com/linuxmint/packages/
https://mirrors.layeronline.com/linuxmint-packages/
http://muug.ca/mirror/linuxmint/packages/
http://mirror.scd31.com/mint/
http://mirror.csclub.uwaterloo.ca/linuxmint-packages/
http://mirrors.advancedhosters.com/linuxmint/packages/
http://mirror.clarkson.edu/linuxmint/packages/
http://mirrors.gigenet.com/linuxmint/repo/
http://mirrors.seas.harvard.edu/linuxmint-packages/
http://mirror.cs.jmu.edu/pub/linuxmint/packages/
http://mirrors.kernel.org/linuxmint-packages/
http://mirror.metrocast.net/linuxmint-packages/
http://mirror.os6.org/linuxmint.com/packages/
http://linuxmint.mirrors.pair.com/packages/
https://plug-mirror.rcac.purdue.edu/mint/
https://mirrors.sonic.net/mint/packages/
https://mirrors.syringanetworks.net/linuxmint-packages/
http://mirror.team-cymru.com/mint-packages/
http://mirror.pit.teraswitch.com/linuxmint/
http://reflection.oss.ou.edu/linuxmint/repos/
http://mintmirror.math.washington.edu/linuxmint-repo/
http://mirrors.usinternet.com/mint/packages/
http://mirrors.xmission.com/linuxmint/
http://mirror.inode.at/linuxmint/packages/
http://ftp.mgts.by/pub/linuxmint/packages/
http://mint.ipacct.com/packages/
http://mirrors.netix.net/LinuxMint/linuxmint-repo/
http://mirrors.uni-ruse.bg/linuxmint/packages/
http://mirrors.nic.cz/linuxmint-packages/
http://ucho.ignum.cz/linuxmint/
http://mirror.it4i.cz/mint/packages/
http://mirror.karneval.cz/pub/linux/linuxmint/packages/
http://mirrors.dotsrc.org/linuxmint-packages/
https://mirror.lelux.fi/linuxmint/packages/
https://mirror.cyberbits.eu/linuxmint/packages/
https://ftp.igh.cnrs.fr/pub/linuxmint/
http://mirror6.layerjet.com/linuxmint/packages/
http://mint.remi.lu/
http://mirror.bauhuette.fh-aachen.de/linuxmint/
http://ftp.fau.de/mint/packages/
http://linux-mint.froonix.org/
http://ftp-stud.hs-esslingen.de/pub/Mirrors/packages.linuxmint.com/
https://mirror.as20647.net/linuxmint-packages/
http://mirror.netcologne.de/linuxmint/packages/
https://mirror.pyratelan.org/mint-packages/
http://ftp.halifax.rwth-aachen.de/linuxmint/
http://mirror.wtnet.de/linuxmint/
https://ftp.wrz.de/pub/mintpackages/
http://ftp.otenet.gr/linux/linuxmint-packages/
http://mirrors.myaegean.gr/linux/linuxmint/packages/
http://ftp.cc.uoc.gr/mirrors/linux/linuxmint/packages/
http://mirror.greennet.gl/linuxmint/packages/
https://quantum-mirror.hu/mirrors/linuxmint/packages/
http://ftp.heanet.ie/pub/packages.linuxmint.com/
http://ftp.linux.edu.lv/mirrors/packages.linuxmint.com/
https://mirror.koddos.net/linuxmint/packages/
https://linuxmint-packages.mirror.liteserver.nl/
http://ftp.nluug.nl/os/Linux/distr/linuxmint/packages/
http://mintlinux.mirror.wearetriple.com/packages/
http://ftp.icm.edu.pl/pub/Linux/dist/linuxmint/packages/
https://mirror.fccn.pt/repos/pub/linuxmint_packages/
https://ftp.rnl.tecnico.ulisboa.pt/pub/linuxmint-packages/
https://mirrors.up.pt/linuxmint-packages/
http://mint.mirrors.telekom.ro/repos/
http://mirrors.powernet.com.ru/mint/packages/
http://mirror.truenetwork.ru/linuxmint-packages/
http://mirror.yandex.ru/linuxmint-packages/
https://fourdots.com/mirror/LinuxMint/packages/
http://mirror.pmf.kg.ac.rs/mint/packages.linuxmint.com/
http://ftp.energotel.sk/pub/linux/linuxmint-packages/
http://tux.rainside.sk/mint/packages/
https://ftp.cixug.es/mint/packages/
http://ftp.acc.umu.se/mirror/linuxmint.com/packages/
https://mirrors.c0urier.net/linux/linuxmint/packages/
https://mirror.linux.pizza/linuxmint/
http://mirror.zetup.net/linuxmint/packages/
https://mirror.init7.net/linuxmint/
http://linuxmint-packages.ip-connect.vn.ua/
https://mirrors.ukfast.co.uk/sites/linuxmint.com/packages/
http://www.mirrorservice.org/sites/packages.linuxmint.com/packages/
http://mirror.xeonbd.com/linuxmint.com-repo/
https://mirrors.bfsu.edu.cn/linuxmint/
http://mirrors.hust.edu.cn/linuxmint/
https://mirrors.tuna.tsinghua.edu.cn/linuxmint/
http://mirrors.ustc.edu.cn/linuxmint/
https://mirror-hk.koddos.net/linuxmint/packages/
http://mint.mirrors.estointernet.in/repos/
https://mintmirror.netfirst.in/
https://mirrors.piconets.webwerks.in/linuxmint-mirror/packages/
http://kartolo.sby.datautama.net.id/linuxmint/
https://mirror.deace.id/linuxmint/packages/
http://mirror.poliwangi.ac.id/linuxmint/
https://repo.iut.ac.ir/repo/Mint/
http://mirror.rasanegar.com/linuxmint/packages/
http://ftp.jaist.ac.jp/pub/Linux/linuxmint/packages/
https://mirror.hoster.kz/linuxmint/packages/
https://mirror.ps.kz/linuxmint/packages/
http://mirror.rise.ph/linuxmint/
http://mirror.0x.sg/linuxmint/
https://download.nus.edu.sg/mirror/linuxmint/
https://ftp.harukasan.org/linuxmint/
http://ftp.kaist.ac.kr/linuxmint/
http://free.nchc.org.tw/linuxmint/packages/
http://ftp.yzu.edu.tw/Linux/linuxmint/packages/
http://mirror1.ku.ac.th/linuxmint-packages/
https://mirror.kku.ac.th/linuxmint-packages/
http://mirror.dc.uz/linuxmint/
http://mirror.freedif.org/LinuxMint/packages/
https://mirror.aarnet.edu.au/pub/linuxmint-packages/
http://mirror.internode.on.net/pub/linuxmint-packages/
http://ucmirror.canterbury.ac.nz/linux/mint/packages/
http://mirror.xnet.co.nz/pub/linuxmint/packages/
https://mint.zero.com.ar/mintpackages/
http://mirror.ufscar.br/mint-archive/
https://mint.itsbrasil.net/packages/
http://repositorio.nti.ufal.br/mint/
http://mirror.ufam.edu.br/linuxmint-packages/
http://mint-packages.c3sl.ufpr.br/
http://mirror.cedia.org.ec/linuxmint-packages/
http://mirror.uta.edu.ec/linuxmint-packages/
http://mirror.epn.edu.ec/linuxmint-packages/
http://mirrors.epst-tlemcen.dz/mint/
http://mirror.retentionrange.co.bw/linuxmint/packages/
https://linuxmint.mirror.liquidtelecom.com/packages/
http://mirror.is.co.za/mirrors/linuxmint.com/packages/
http://mirror.ufs.ac.za/linuxmint-packages/
http://mirror.wiru.co.za/linuxmint-packages/

Also create the file /etc/apt-cacher-ng/backends_docker with the following content:

https://download.docker.com/linux/ubuntu

Finally, create the file /usr/lib/apt-cacher-ng/docker_mirrors with the following content:

https://download.docker.com/linux/ubuntu

Now, restart the service:

sudo service apt-cacher-ng restart

Edit the root crontab with sudo crontab -e (or, better yet, edit a crontab belonging to a user that will stay on the system) and add the following to the end of the file:

0 * * * * curl "http://127.0.0.1:3142/acng-report.html?forceRedownload=fRd&calcSize=cs&doDownload=dd&asNeeded=an&doMirror=Start+Mirroring#bottom"
30 0 * * * curl "http://127.0.0.1:3142/acng-report.html?forceRedownload=fRd&ignoreTradeOff=iTO&purgeNow=pN&doExpire=Start+Scan+and%2For+Expiration&calcSize=cs&asNeeded=an#bottom"

Client (the client)

Easy

This is the easiest way to hook a client up to the service. It uses the avahi-daemon installed on the server.

Note: While this is easy, it is highly susceptible to man-in-the-middle attacks. Use at your own risk. If you need more security, see the `Secure` section instead.

sudo apt install squid-deb-proxy-client

Secure

Note: you will need to access port 3142 on the cache/proxy server

Install prerequisites

sudo apt install netcat

Configure apt

Create the file /etc/apt/apt.conf.d/30detectproxy with the following content:

# Fail immediately if a file could not be retrieved. Comment if you have a bad
# Internet connection
Acquire::Retries 0;

# undocumented feature which was found in the source. It should be an absolute
# path to the program, no arguments are allowed. stdout contains the proxy
# server, stderr is shown (in stderr) but ignored by APT
Acquire::http::ProxyAutoDetect "/etc/apt/detect-http-proxy";

Create the file /etc/apt/detect-http-proxy with the following content:

Note `<your-proxy-1>` and `<your-proxy-2>` in the file, which you will replace with your proxy IPs/domains. Each new proxy goes on a new line. Use as many or as few as you need.

#!/bin/bash
# detect-http-proxy - Returns a HTTP proxy which is available for use

# Author: Lekensteyn <[email protected]>

# Supported since APT 0.7.25.3ubuntu1 (Lucid) and 0.7.26~exp1 (Debian Squeeze)
# Unsupported: Ubuntu Karmic and before, Debian Lenny and before

# Put this file in /etc/apt/detect-http-proxy and create and add the below
# configuration in /etc/apt/apt.conf.d/30detectproxy
#    Acquire::http::ProxyAutoDetect "/etc/apt/detect-http-proxy";

# APT calls this script for each host that should be connected to. Therefore
# you may see the proxy messages multiple times (LP 814130). If you find this
# annoying and wish to disable these messages, set show_proxy_messages to 0
show_proxy_messages=1

# on or more proxies can be specified. Note that each will introduce a routing
# delay and therefore its recommended to put the proxy which is most likely to
# be available on the top. If no proxy is available, a direct connection will
# be used
try_proxies=(
<your-proxy-1>:3142
<your-proxy-2>:3142
)

print_msg() {
    # \x0d clears the line so [Working] is hidden
    [ "$show_proxy_messages" = 1 ] && printf '\x0d%s\n' "$1" >&2
}

for proxy in "${try_proxies[@]}"; do
    # if the host machine / proxy is reachable...
    if nc -z ${proxy/:/ }; then
        proxy=http://$proxy
        print_msg "Proxy that will be used: $proxy"
        echo "$proxy"
        exit
    fi
done
print_msg "No proxy will be used"

# Workaround for Launchpad bug 654393 so it works with Debian Squeeze (<0.8.11)
echo DIRECT

Clear & update the client's cache

Finally, run the following:

sudo chmod +x /etc/apt/detect-http-proxy
sudo rm -rf /var/cache/apt/*
sudo rm -rf /var/lib/apt/lists/*
sudo mkdir -p /var/cache/apt/archives/partial
sudo touch /var/cache/apt/archives/lock
sudo chmod 640 /var/cache/apt/archives/lock
sudo chown _apt:root /var/cache/apt/archives/partial
sudo apt update

egg82/apt_cacher_ng.md