Skip to content

Instantly share code, notes, and snippets.

@eedugon
Forked from pbabics/gitlab-gc.sh
Created July 13, 2017 10:30
Show Gist options
  • Save eedugon/69da0397c142e56f4186ee691502e7ee to your computer and use it in GitHub Desktop.
Save eedugon/69da0397c142e56f4186ee691502e7ee to your computer and use it in GitHub Desktop.
Manual garbage collector for gitlab registry, it removes old revisions that are not referenced by any tag
#!/bin/bash
# This is a modification of gitlab-gc.sh script created by Peter Bábics (pbabics/gitlab-gc.sh)
# Improvements
# - Searching in all BASE_PATH, not fixing the search to a depth of 2
# - Directories without valid tags or revisions directories won't be processed (to avoid unexpected issues)
# - Logging in case there's nothing to delete
# - running registry-garbage-collect only when something has been deleted
# crontab example
# Run everynight the script, keeping 5 old revisions and running registry-garbage-collector at the end. All stdout logs sent to a file rotated every month
# 00 23 * * * { echo "STARTING EXEC $(date)"; /root/scripts/gitlab-registry-gc.sh -k 5 -r; } >> /root/scripts/logs/stdout_gitlab-registry-gc.$(date "+\%Y\%m").log 2>&1
BASE_PATH=/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories
DRY_RUN=0
KEEP_LAST_IMAGES=10
RUN_GARBAGE_COLLECTOR=0
GITLAB_CTL_COMMAND=`which gitlab-ctl`
if [ ! -x "${GITLAB_CTL_COMMAND}" ]; then
echo "Missing gitlab-ctl command"
exit 1
fi
while (( "$#" )); do
case "$1" in
"-b" | "--base-path")
BASE_PATH=$2
shift
;;
"-r" | "--run-gc")
RUN_GARBAGE_COLLECTOR=1
;;
"-d"|"--dry-run")
DRY_RUN=1
;;
"-k"|"--keep")
if ! ( echo $2 | grep -q '^[0-9]\+$') || [ $2 -eq 0 ]; then
echo "Invalid value for keep last images '$2'"
exit 1
fi
KEEP_LAST_IMAGES=$2
shift
;;
"-h"|"--help")
echo "Usage: ${0} [options]"
echo "Options:"
echo -e "\t-k NUM, --keep NUM"
echo -e "\t\tKeeps last NUM revisions, except current tags"
echo
echo -e "\t-d, --dry-run"
echo -e "\t\tEnables dry run, no changes will be made"
echo
echo -e "\t-b, --base-path"
echo -e "\t\tSets base path of Gitlab Registry repository storage"
echo
echo -e "\t-r, --run-gc"
echo -e "\t\tStarts garbage collector after revision removal"
exit 0
;;
*)
echo "Unknown argument: $1"
exit 1
;;
esac
shift
done
IFS=$'\n'
used_hashes=`mktemp`
marked_hashes=`mktemp`
found=""
#for repository in `find ${BASE_PATH} -mindepth 2 -maxdepth 2 -type d | sed "s#${BASE_PATH}/##"`; do
for repository in `find ${BASE_PATH} -name "_manifests" -type d | sed -e "s#${BASE_PATH}/##" -e "s#/_manifests##"`; do
echo "# Processing repository: $repository"
ls ${BASE_PATH}/${repository}/_manifests/tags/*/current/link >/dev/null 2>&1 || { echo "ERROR: Invalid repository. No tags/*/current/link files found"; echo; continue; }
test -d "${BASE_PATH}/${repository}/_manifests/revisions/sha256" || { echo "ERROR: Invalid repository. revisions/sha256 dir not found"; echo; continue; }
for tag_hash in ${BASE_PATH}/${repository}/_manifests/tags/*/current/link; do
cat "${tag_hash}" | cut -d':' -f2;
done > "${used_hashes}"
echo "# Removing revisions of $repository:"
ls -t ${BASE_PATH}/${repository}/_manifests/revisions/sha256 | fgrep -vf "${used_hashes}" | tail -n+${KEEP_LAST_IMAGES} | tee ${marked_hashes}
if test -s $marked_hashes ; then
if [ ${DRY_RUN} -ne 1 ]; then
cat ${marked_hashes} | sed "s#^#${BASE_PATH}/${repository}/_manifests/revisions/sha256/#" | xargs rm -rf && found="true"
fi
else
echo "Nothing to delete"
fi
echo
done
rm ${used_hashes}
rm ${marked_hashes}
if [ ${DRY_RUN} -eq 0 -a ${RUN_GARBAGE_COLLECTOR} -eq 1 ]; then
if test "$found"; then
"${GITLAB_CTL_COMMAND}" registry-garbage-collect
else
echo "Skipping execution of ${GITLAB_CTL_COMMAND} registry-garbage-collect because nothing was deleted"
fi
fi
@HaraldRieder
Copy link

Great! Thank you very much!!

@sradforth
Copy link

Am I using this correctly? I get these sorts of errors everywhere and no disk space was freed up...


# Processing repository: core/backend/api/feature-backplane
ERROR: Invalid repository. No tags/*/current/link files found

@itkroplis
Copy link

Hi! This scrip works for Gitlab-CE Omnibus?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment