Docker filesystem layers can be tricky to wrap your head around, and even more so how indoes work. Lets take an example and see what happens when you tail a file that exists in the image as part of your container entrypoint:
host$ docker run -d --rm --name test-inode debian tail -f /etc/issue
54bbfa8fa1f6751593dcf23103a1cdaec7fde8ffbcb3e31bab466b4f7a3581e7
Now make some changes to the file:
host$ docker exec -it test-inode /bin/bash
root@54bbfa8fa1f6:/# cat /etc/issue
Debian GNU/Linux 9 \n \l
root@54bbfa8fa1f6:/# ls -li /etc/issue
41813820 -rw-r--r-- 1 root root 26 Jul 13 2017 /etc/issue
root@54bbfa8fa1f6:/# echo "This container has been modified" >>/etc/issue
Lets view the logs of the tail command to see our new line:
host$ docker logs test-inode
Debian GNU/Linux 9 \n \l
Wait, where's the new line!? Lets go back in the container and look again:
host$ docker exec -it test-inode /bin/bash
root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 59 Feb 22 13:39 /etc/issue
root@54bbfa8fa1f6:/# cat /etc/issue
Debian GNU/Linux 9 \n \l
This container has been modified
root@54bbfa8fa1f6:/# exit
exit
We definitely changed the file:
host$ docker diff test-inode
C /etc/issue
C /root
A /root/.bash_history
The key can be seen in that ls command being run, the -i shows the current indoe number.
We can see the inode change when we first modifify the file.
root@54bbfa8fa1f6:/# ls -li /etc/issue
41813820 -rw-r--r-- 1 root root 26 Jul 13 2017 /etc/issue
root@54bbfa8fa1f6:/# echo "This container has been modified" >>/etc/issue
root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 59 Feb 22 13:39 /etc/issue
If we go back into the container and write another line, the inode doesn't change:
root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 59 Feb 22 13:39 /etc/issue
root@54bbfa8fa1f6:/# echo "Even more changes to the file" >>/etc/issue
root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 89 Feb 22 13:48 /etc/issue
What's happening is docker's copy-on-write is kicking in on the first change, copying
the /etc/issue file to the container specific layer, where it gets a new inode. The tail
command is still watching the inode of the /etc/issue that's part of the image filesystem,
which will never change.
The fix for this is to make some kind of change to the file to force the copy-on-write, even a timestamp change, so the tail is watching the inode of the file inside the container filesystem layer rather than that of the read only image layers:
host$ docker run -d --rm --name test-inode debian /bin/sh -c ":>>/etc/issue && tail -f /etc/issue"
ebc93a55aca487792b7a2bdabcdc0deb573302c1df97390884d3c97a0f643b28
That :>>/etc/issue outputs nothing to the file, but does change the modify timestamp. We can
see this in the docker diff that shows which files are now in the container specific layer:
host$ docker diff test-inode
C /etc/issue
Lets try our test again:
host$ docker exec -it test-inode /bin/bash
root@ebc93a55aca4:/# ls -li /etc/issue
43527824 -rw-r--r-- 1 root root 26 Jul 13 2017 /etc/issue
root@ebc93a55aca4:/# echo "This change should show in the logs" >>/etc/issue
root@ebc93a55aca4:/# ls -li /etc/issue
43527824 -rw-r--r-- 1 root root 62 Feb 22 13:55 /etc/issue
root@ebc93a55aca4:/# exit
exit
The inode didn't change since the file was already in the container specific layer, this is promising. Time for the moment of truth, lets check the logs:
host$ docker logs test-inode
Debian GNU/Linux 9 \n \l
This change should show in the logs
There it is, the tail command is now watching the file being modified in the container rather than the image's read-only layer.