Clean up untagged manifests in private docker registry

cleanupManifestsWithoutTags.sh

#!/bin/bash

REGISTRY_DIR=YOUR_REGISTRY_DIR/data/docker/registry/v2/repositories
REGISTRY_URL=http://10.10.10.10:5000
#add --insecure to the curl command on line 17 if you use https with self-signed certificates

cd ${REGISTRY_DIR}
count=0

manifests_without_tags=$(comm -23 <(find . -type f -name "link" | grep "_manifests/revisions/sha256" | grep -v "\/signatures\/sha256\/" | awk -F/ '{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep "_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g'; echo; done | sort))

total_count=$(echo ${manifests_without_tags} | wc -w)

for manifest in ${manifests_without_tags}; do
	repo=$(find . | grep "_manifests/revisions/sha256/${manifest}/link" | awk -F "_manifest"  '{print $(NF-1)}' | sed 's#^./\(.*\)/#\1#')
	
	#should have error checking on the curl command, it might fail silently atm.
	curl -s -X DELETE ${REGISTRY_URL}/v2/${repo}/manifests/sha256:${manifest} > /dev/null
	
	((count++))
	echo "Deleted ${count} of ${total_count} manifests."
done

Thanks! Have been looking for months how to stop the registry from leaking disk space. Good job!

Thank you very many!

Thank you @mortensteenrasmussen for a very useful script. Especially line#10 is a strong piece of shell-engineering!
How should "curl -s -X ..." work without an access token? I had to get the token for every single repository and pass it to the curl command to get it work.

This was awesome. Just a note that:
As the curl command is silent, in my case at least, the docker registry storage delete option was set to false, so the script wasn't able to delete anything.
You might want to write those logs to somewhere like /var/log/cleanupScript.log to make sure you are cleaning what you mean.

Works perfectly. Thank you so much :)

@mortensteenrasmussen, thank you very much for this script.
Seven years later, there are still people like me, with the same problem, running terribly outdated docker registry version.
If there weren't for this script, I would have to devise the solution myself, which would take many hours, if not days.

The procedure in the script is correct but the script can be optimized in order to work much, much faster (in my setup more than 100 times faster).
Here is a fork with a modified script I wrote based on your solution:
https://gist.github.com/djosip/1c65c95d9a84acf696851451814529dd

@opusmagnum

Thank you @mortensteenrasmussen for a very useful script. Especially line#10 is a strong piece of shell-engineering!

I couldn't hold myself and had to react to the "Especially line#10 is a strong piece of shell-engineering".
The original script does work and will do the job but the line#10 is far from a good example of shell engineering.
I am sure the author had in mind to quickly solve the problem and not chasing the perfection and that's ok.

However, the original script, in my environment with 2881 manifest to delete, took around 12.53 hours.
After the rewrite, optimized script took only 4.48 minutes in the same environment with 2881 manifest to delete.
I did the correct benchmark using backup and restore just to make sure I got it right the first time.

Here is the optimized script with added error handling (the script was tested only in the GNU environment):
https://gist.github.com/djosip/1c65c95d9a84acf696851451814529dd