Mirror only a few Ubuntu releases using rsync

gb.archive.ubuntu.com

bionic
bionic-updates
bionic-backports
xenial
xenial-updates
xenial-backports

mirror-rsync.sh

#!/bin/bash
set -euo pipefail;

#A considerably simpler take on apt-mirror - parses the Packages files like apt-mirror, but then builds a file of .debs to download, which is then passed to rsync which does the rest.
#Saves the overhead of downloading each file over HTTP and considerably simpler to debug. Can now be configured using a file containing paths instead of running rsync many times in a loop.
#Just like apt-mirror, capable of running on only one Ubuntu release to save space.
#Author: Rob Johnson
#Date: 2017-09-20

syncDate=$(date +%F);
#File to build a list of files to rsync from the remote mirror - will contain one line for every file in the dists/ to sync
filename=packages-$syncDate.txt;
#Assumes a 'master source' file in /etc/mirror-rsync.d named for the masterSource value below. The file contains newline-separated
#entries of which dists/ to sync. See example in other file here.
masterSource='gb.archive.ubuntu.com';
#Adapt as necessary to your package mirror setup
localPackageStore="/srv/packagemirror/mirror/$masterSource/ubuntu";

if [ ! -f /etc/mirror-rsync.d/$masterSource ]; then
    echo "No master source file found at /etc/mirror-rsync.d/$masterSource, create one and add one line per dists/ entry to sync";
    exit 1;
fi

#Add a marker for a second APT mirror to look for - if the sync falls on its face, can drop this out of the pair and sync exclusively from the mirror until fixed
if [ -f /srv/packagemirror/lastSuccess ]; then
	rm -v /srv/packagemirror/lastSuccess;
fi

echo "$syncDate $(date +%T) Starting, exporting to /tmp/$filename";

#In case leftover from testing or failed previous run
if [[ -f /tmp/$filename ]]; then
	rm -v "/tmp/$filename";
fi

echo "$(date +%T) Syncing releases";
rsync --no-motd --delete-during --archive --recursive --human-readable --files-from="/etc/mirror-rsync.d/$masterSource" $masterSource::ubuntu "$localPackageStore/";

echo "$(date +%T) Generating package list";
#rather than hard-coding, use a config file to run the loop. The same config file as used above to sync the releases
while read release; do
	for repo in 'main' 'restricted' 'universe' 'multiverse'; do #Adapt if necessary
		for arch in 'amd64' 'i386'; do #Adapt if necessary
			if [[ ! -f $localPackageStore/$release/$repo/binary-$arch/Packages ]]; then #uncompressed file not found
				echo "$(date +%T) Extracking $release $repo $arch Packages file from archive";
				gunzip --keep "$localPackageStore/$release/$repo/binary-$arch/Packages.gz";
			fi
			echo "$(date +%T) Extracting packages from $release $repo $arch";
			if [[ -s $localPackageStore/$release/$repo/binary-$arch/Packages ]]; then
				grep 'Filename: ' "$localPackageStore/$release/$repo/binary-$arch/Packages" | sed 's/Filename: //' >> "/tmp/$filename";
			else
				echo "$(date +%T) Package list is empty, skipping";
			fi
		done
	done
done </etc/mirror-rsync.d/$masterSource

echo "$(date +%T) Deduplicating";

sort --unique "/tmp/$filename" > "/tmp/$filename.sorted";
rm -v "/tmp/$filename";
mv -v "/tmp/$filename.sorted" "/tmp/$filename";

echo "$(wc -l \"/tmp/$filename\" | awk '{print $1}') files to be sync'd";

echo "$(date +%T) Running rsync";

#rsync may error out due to excessive load on the source server, so try up to 3 times
set +e;
attempt=1;
exitCode=1;

while [[ $exitCode -gt 0 ]] && [[ $attempt -lt 4 ]];
do
	SECONDS=0;
	rsync --copy-links --files-from="/tmp/$filename" --no-motd --delete-during --archive --recursive --human-readable $masterSource::ubuntu "$localPackageStore/" 2>&1;
	exitCode=$?;
	if [[ $exitCode -gt 0 ]]; then
		waitTime=$((attempt*300)); #increasing wait time - 5, 10 and 15 minutes between attempts
		echo "rsync attempt $attempt failed with exit code $exitCode, waiting $waitTime seconds to retry";
		sleep $waitTime;
		let attempt+=1;
	fi
done

set -e;

#Exiting here will stop the lastSuccess file being created, and will stop APT02 running its own sync
if [[ $exitCode -gt 0 ]]; then
	echo "rsync failed all 3 attempts, erroring out";
	exit 2;
fi

echo "$(date +%T) Sync complete, runtime: $SECONDS s";

echo "$(date +%T) Deleting obsolete packages";

#Build a list of files that have been synced and delete any that are not in the list
find "$localPackageStore/pool/" -type f | { grep -Fvf "/tmp/$filename" || true; } | xargs --no-run-if-empty -I {} rm -v {}; # '|| true' used here to prevent grep causing pipefail if there are no packages to delete - grep normally returns 1 if no files are found

echo "$(date +%T) Complete";

rm -v "/tmp/$filename";

touch /srv/packagemirror/lastSuccess;

Hello I get the following error :

2020-04-15 14:01:54 Starting, exporting to /tmp/packages-2020-04-15.txt
14:01:54 Syncing releases
14:01:55 Generating package list
mirror-rsync.sh: line 32: read: read error: 0: Is a directory
14:01:55 Deduplicating
sort: cannot read: /tmp/packages-2020-04-15.txt: No such file or directory

Line that causes the problem :

while read release; do

Where does "release" come from ?

Thanks

@flavienbwk It comes from a file in /etc/mirror-rsync.d, a folder you'll need to create. You'll want to create a file in there named for the repo you intend to sync, e.g. ubuntu, then add the rsync URL as contents. You can create multiple files in the folder to sync multiple repos.

Thank you @Gargravarr2112, it still shows the same error. I created my file /etc/mirror-rsync.d/ubuntu with the following content :

http://fr.archive.ubuntu.com/ubuntu/ bionic main restricted
http://fr.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
http://fr.archive.ubuntu.com/ubuntu/ bionic universe
http://fr.archive.ubuntu.com/ubuntu/ bionic-updates universe
http://fr.archive.ubuntu.com/ubuntu/ bionic multiverse
http://fr.archive.ubuntu.com/ubuntu/ bionic-updates multiverse
http://fr.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
http://security.ubuntu.com/ubuntu bionic-security main restricted
http://security.ubuntu.com/ubuntu bionic-security universe
http://security.ubuntu.com/ubuntu bionic-security multiverse

It seems like the release keyword is not referenced anywhere. I don't see how the script can deduce to take this ubuntu file anywhere.

@flavienbwk Sorry, didn't see your reply until now. Also, I see I was wrong - I forgot how I wrote my own script, sorry!
The config file needs to have as the filename the mirror source. In your case, this means you create a file named 'fr.archive.ubuntu.com'. Within that file you would list the releases, which are 'bionic', 'bionic-updates' and 'bionic-backports'. You could do the same for 'security.ubuntu.com' but I feel it is better to pull security updates directly from Ubuntu to avoid a rapid update being behind. Pulling the main packages from a local mirror will dramatically reduce your required bandwidth in any case. I've added the file I used. Hope this helps!

So I tried your script there and ran into this problem.

rsync: link_stat "/bionic" (in ubuntu) failed: No such file or directory (2)

I'm not sure exactly how to resolve it, I don't understand where it's failing. all I did was remove the path part for the source file so it'd work from wherever and ofc the destination path. I get that same error if I run the rsync command directly after I replace the variables ofc.

@jms1989 you might want to use this script. It's a bit more robust for downloading mirrors.

@jms1989 probably best to do what @flavienbwk suggests, I don't recognise that error, nor do I use this script at the moment (don't have a use for it). I may revisit it in the future, but otherwise I'd be debugging it the same as you. You could try set -x to see what commands are actually being run when it fails.

To anyone finding this script - I have restarted development on it for my own needs and have created a repo: https://github.com/Gargravarr2112/mirror-rsync
Updated code (that actually works!) and instructions can be found there.