Overlay bug with user namespaces + copy_up + dir removal (whiteout)

overlay-copy-up-and-delete-with-userns.md

overlayfs copy-up without userns

Setup: create proper directories, with a pre-existing dir (called dir) in lower, and then mount as overlay fs:

$ mkdir upper lower work merged upper lower/dir
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged

You can now create files in the merged/dir to force a copy up of lower/dir to upper/dir:

$ touch merged/dir/{1,2,3}
$ ls -l upper/dir
total 0
-rw-r--r-- 1 estesp estesp 0 Dec  9 15:18 1
-rw-r--r-- 1 estesp estesp 0 Dec  9 15:18 2
-rw-r--r-- 1 estesp estesp 0 Dec  9 15:18 3

You can see that dir has been "copied up" to upper and has the three files we touched.

Now I can remove the directory and it's contents from merged, which should convert the upper/dir entry to a character device which is overlay's notation for a "whiteout" at the upper layer.

$ rm -fR merged/*
$ ls -l upper/
total 0
c--------- 1 estesp estesp 0, 0 Dec  9 15:05 dir
$ ls merged/
$

Correctly, the directory is modified in upper to a char device 0,0 (the overlay fs whiteout), and ls merged shows no files (the whiteout is doing it's job)

overlayfs copy-up with userns

Setup: if using the same location, unmount merged and then blow away & recreate upper so that the whiteout and all content is gone. Now enter a user namespace with unshare:

$ unshare -m -p -f -U -r bash
# mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged

Now you can perform the same steps as above to create files in merged/dir, and notice that upper is properly populated.

The problem occurs when you try and remove this directory, which should replace it with a character device 0,0 (the overlay whiteout), but when inside a user namespace this particular step fails, and the removal gets a permission denied. Specifically the following unlinkat syscall gets EPERM after successfully removing the 1, 2, and 3 files:

unlinkat(4, "2", 0)                     = 0
unlinkat(4, "1", 0)                     = 0
unlinkat(4, "3", 0)                     = 0
close(4)                                = 0
unlinkat(AT_FDCWD, "merged/dir", AT_REMOVEDIR) = -1 EPERM (Operation not permitted)

Of course, the real magic is happening in the kernel (fs/overlay/*) driver at this point, and something in that codepath regarding the replace of upper/dir as a real dir with a char device is failing or getting a permissions issue.

Interestingly, if you don't place files in merged/dir you can remove it, meaning if upper/dir does not exist, creating the char device file works properly in that same location. This is the output verifying that (when nothing has existing in upper yet + user namespaces):

# ls -al upper/
total 8
drwxr-xr-x 2 root root 4096 Dec  9 15:14 .
drwxr-xr-x 6 root root 4096 Dec  9 15:14 ..
c--------- 1 root root 0, 0 Dec  9 15:14 dir

Note that ownership looks like root because I'm in a user namespace with remapped root to my user via unshare.

@estesp I cannot even touch merged/dir/1 inside the userns with kernel 4.4.6, could you double check this is still working so I can reproduce and git bisect the kernel?

alright, reproduced on rawhide with 4.5.0 kernel :/

on rawhide with 4.5.0 kernel I also get a weird result:

[root@01-rawhide ~]# ls -la merged/dir/
total 0
drwxr-xr-x. 1 root root 33 Mar 20 12:35 .
drwxr-xr-x. 1 root root 17 Mar 20 12:32 ..
-rw-r--r--. 1 root root  0 Mar 20 12:35 1
-rw-r--r--. 1 root root  0 Mar 20 12:35 2
-rw-r--r--. 1 root root  0 Mar 20 12:35 3
[root@01-rawhide ~]# ls -la upper/dir/
total 0
drwxr-xr-x. 2 root root 33 Mar 20 12:35 .
drwxr-xr-x. 3 root root 17 Mar 20 12:32 ..
-rw-r--r--. 1 root root  0 Mar 20 12:35 1
-rw-r--r--. 1 root root  0 Mar 20 12:35 2
-rw-r--r--. 1 root root  0 Mar 20 12:35 3
[root@01-rawhide ~]# rm -fR merged/*
rm: cannot remove 'merged/dir': Operation not permitted
[root@01-rawhide ~]# ls -la upper/dir/
total 0
drwxr-xr-x. 2 root root  6 Mar 20 12:35 .
drwxr-xr-x. 3 root root 17 Mar 20 12:32 ..
# merged/dir is also empty

The rm -rf merged/* fails with EPERM but files are deleted regardless and no whiteouts are present 😕