docker volumes and overlayfs experiments

foo.md

With a docker volume

% sudo rm -r /tmp/overlay/upper; mkdir /tmp/overlay/upper
% docker volume create --driver local --opt type=overlay --opt o=lowerdir=/etc,upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work --opt device=overlay overlay-etc

# I've tried without --privileged, and with --security-opt label=disable, both still relabel all files.
% sudo docker run --volume overlay-etc:/etc2 --privileged ubuntu:latest date

# There should be *no* files in the upper, here, because "date" doesn't write anything.
# But dockerd calls lsetxattr(path, "security.selinux", ...) on every single file in the volume
# This triggers a write which causes overlayfs to copy_up to the upper... of every file :(
% ls /tmp/overlay/upper
abrt/                   crontab                     fstab                  jvm/                      modprobe.d/          pinforc           security/              tpm2-tss/
adjtime                 cron.weekly/                fstab.script*          jvm-common/               modules-load.d/      pkcs11/           selinux/               Trolltech.conf
aliases                 crypto-policies/            fuse.conf              kdump/                    motd                 pkgconfig/        sensors3.conf          trusted-key.key
alsa/                   crypttab                    fwupd/                 kdump.conf                motd.d/              pki/              sensors.d/             ts.conf

The full contents of the overlay's "lower" has been copied to the upper because of dockerd's behavior.

dockerd causes overlayfs to copy_up on every file as it calls lsetxattr() to label every path in the volume, as shown here via strace:

% sudo strace -s1500 -fe trace=mount,file -p 1922

2229  lsetxattr("/var/lib/docker/volumes/overlay-etc/_data/bashrc", "security.selinux", "system_u:object_r:container_file_t:s0", 37, 0 <unfinished .
..>
2217  lsetxattr("/var/lib/docker/volumes/overlay-etc/_data/bindresvport.blacklist", "security.selinux", "system_u:object_r:container_file_t:s0", 37,
 0 <unfinished ...>

overlayfs on host, then bind mount into docker

% sudo rm -r /tmp/overlay/upper; mkdir /tmp/overlay/upper
% sudo mount -t overlay -o lowerdir=/etc,upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work overlay /tmp/over-etc
% sudo docker run --volume /tmp/over-etc:/etc2 -it --privileged ubuntu:latest cp /etc2/bashrc /etc2/bashrc2
% ls /tmp/overlay/upper

# Do one write, copy bashrc to bashrc.backup
# This requires --privileged in order to write to the bind mount
% sudo docker run --volume /tmp/over-etc:/etc2 -it --privileged ubuntu:latest cp /etc2/bashrc /etc2/bashrc.backup
% ls /tmp/overlay/upper
bashrc.backup