git checkout master # setup Dockerfile
docker-compose build # uses cache (expected)
git checkout HEAD^1 # removes some line from Dockerfile
docker-compose build # uses cache (expected)
git checkout master # adds back same line to Dockerfile
docker-compose build --no-cache # does not use input from the cache (expected)
git checkout HEAD^1 # removes same line from Dockerfile
docker-compose build # has unexpected cache miss
possibly related? moby/moby#3199
Wild guess: Recency is not enough
"Use the most recent image from the build cache" is implemented here: https://github.com/docker/docker/blob/415dd8688618ac2b3872c6a5b5af3afcef2c3e10/daemon/daemon.go#L1363
"Use the most recent image from the build cache" will fail because the last known good build may have used a different image. It's not enough to sort by build cache item recency. Cache miss event should cause the builder to backtrack and try the next most recent available matching image, until it exhausts all available image options, at which point it should backtrack to the previous Dockerfile step and repeat the exhaustive search.
"Use the most recent image from the build cache" will fail after using
--no-cachebecause "last known most recently matching image" will naturally not have any relevant child images in the build cache. The "last known most recently matching image" will be whatever image was recently created during--no-cache, and therefore have none of the children from previous cached builds. Meanwhile the build cache is still present and available and has useful things inside.Confusion intensifies
.... turns out that I am not using "--no-cache" anyway, but still seeing lots of cache misses.
Tool for investigating the status of the docker build cache?
There should be more tools to display the status of the docker build cache. There should also be a tool to do a "dry run" of a docker build, which would display information about the docker build cache, without actually manipulating the docker build cache and without running an actual build.
Other related
"faster cache miss detection"
moby/moby#16821
"output reason for docker cache invalidation"
moby/moby#9294