Docker pruner. Deletes docker's overlay2 leftovers that survive 'docker system prune -af --volumes'

docker-pruner.sh

#!/usr/bin/env bash

set -eEuo pipefail

MARKER_FILE_NAME="${DOCKER_PRUNER_MARKER:-DOCKER-PRUNER-MARKER-FILE}"
DOCKER_PATH="${DOCKER_PATH:-/var/lib/docker/overlay2}"

function _used_dirs() {
    for docker_obj in $(docker ps -aq) $(docker image ls -aq); do
        lowerdir="$(docker inspect "$docker_obj" | jq '.[].GraphDriver.Data.LowerDir' -r)"
        for dir in ${lowerdir//:/ }; do
            dirname "$dir"
        done
        dirname "$(docker inspect "$docker_obj" | jq '.[].GraphDriver.Data.MergedDir' -r)"
    done
}

function used_dirs() {
    _used_dirs | sort
}

function all_dirs() {
    find "$DOCKER_PATH"/ -maxdepth 1 -type d | grep -Ev '^'"$DOCKER_PATH"'/?l?$' | sort
}

function unused_dirs() {
    grep -v -xF -f <(used_dirs) <(all_dirs)
}

function set_marker() {
    touch "$1"/merged/"$MARKER_FILE_NAME" 2> /dev/null || \
        touch "$1"/diff/"$MARKER_FILE_NAME"
}

function _check() {
    for container in $(docker ps -aq); do
        docker exec "$container" ls "/$MARKER_FILE_NAME" && echo "container: $container" || true
    done
    for image in $(docker image ls -aq); do
        docker run --rm -it --entrypoint ls "$image" "/$MARKER_FILE_NAME" && echo "image: $image" || true
    done
}

function check() {
    output="$(_check 2>&1 | grep -Ev /"$MARKER_FILE_NAME'?"': No such file or directory')"
    if [[ -n "$output" ]]; then
        echo "Problem detected !"
        echo "$output"
        exit 1
    fi
}

function usage() {
    echo -e "Usage:
    \t$0 list    -- List all directories that needs to be removed
    \t$0 marker  -- Put a marker on each directory, to check if the marker in found in a running container (detecting an issue with $0)
    \t$0 check   -- Check if a marker is found. Must have run $0 marker first to make sense
    \t$0 clear   -- Remove the directories"
}

if [[ $# == 0 ]]; then
    usage ; exit 0
fi

if [ "$EUID" -ne 0 ]
  then echo "Please run as root"
  exit 1
fi


if [[ "$1" == "list" ]]; then
    unused_dirs
elif [[ "$1" == "marker" ]]; then
    export -f set_marker
    export MARKER_FILE_NAME="$MARKER_FILE_NAME"
    unused_dirs | xargs -I {} bash -c "set_marker {}"
elif [[ "$1" == "check" ]]; then
    check
elif [[ "$1" == "clear" ]]; then
    unused_dirs | xargs -I {} find {} -delete
elif [[ "$1" == "fn" ]]; then
    $2 "$@"
fi

Hi @fayak! This seems to work well, well done.

A couple of things (you asked for feedback in this issue):

1. Would be nice if it checks that $DOCKER_PATH actually is a directory, and ideally also that it actually is the overlay2 directory (running against a directory that isn't that it will propose to remove everything in the directory)
2. Marker: Not all images actually have ls. Example: oliver006/redis_exporter@sha256:c5a968ba5f30bab5e12181cceb89c4ed37cb4fe4d85490b5b66c7cfe4750b5e8
3. If you could state a license to this that would be great!

I also wonder about the marker functionality: If I understand it correctly, it creates a file either in the containers filesystem, or in the images. For the latter I imagine it makes the image internally inconsistent (not matching the hash anymore). So that if you mark without removing, and then try to use that image at a later time, you could get issues?

Anyway, thanks for the script, it's very readable and well composed, IMO.

You are genius, this is only one way which helps me, thank you very much!

Great job!

Hi @fayak , after I use this tool to prune the overlay2, when I re-build a new image, I met below issue, do you have meet similar issue before?

#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 4.15kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/ubuntu:22.04
#3 ...

#4 [internal] load metadata for docker.io/pytorch/llvm:9.0.1
#4 DONE 0.4s

#3 [internal] load metadata for docker.io/library/ubuntu:22.04
#3 DONE 0.4s

#5 FROM docker.io/pytorch/llvm:9.0.1@sha256:946fe9e12ad235bc41a008b4a3033e5043008abc67ea600fc8acf45520ff9051
#5 CACHED

#6 [stage-0  1/49] FROM docker.io/library/ubuntu:22.04@sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
#6 CACHED

#7 [internal] load build context
#7 ERROR: stat /var/lib/docker/overlay2/y9i4ol147j3ih95mzagm24hps: no such file or directory
------
 > [internal] load build context:
------
ERROR: failed to solve: stat /var/lib/docker/overlay2/y9i4ol147j3ih95mzagm24hps: no such file or directory

Resolved by add one more line docker system prune -af. Thanks

Resolved by add one more line docker system prune -af. Thanks

@fayak
Should be added to the script, or is it possible to exclude build cache?

awesome script :)

Edit: Note that docker system prune will also remove networks, images etc.
As we only want to prune the build cache, use docker builder prune -f instead.

Just add it between line 80 and 81.

Hi @lug-gh do you know how to exclude build cache? I still struggle with the above issue...

Hi @lug-gh do you know how to exclude build cache? I still struggle with the above issue...

Unfortunately not, I had tried a few approaches but discarded them.
You can use the command docker buildx du to display the IDs of the build cache, unfortunately these do not always match the Overlay2 folder names. You probably have to dig a little deeper here.