Reproduce tar incompatibility between Go 1.8 and Go <= 1.7

README.md

Go Tar incompatibility

This gist demonstrates a bug in Go's archive/tar handling of tar files. Under certain conditions, tar files generated with Go 1.7 and before cannot be correctly read with Go 1.8+. While first encountered in Docker, which makes heavy use of tar files, this gist attempts to make a minimal reproduction without the Docker code base.

To reproduce this, you'll need respective versions of Go installed in /usr/local/go1.7.6 and /usr/local/go1.8.3. Other methods of version switching will work, but the environment variables assume this setup. The files print the version and GOROOT to make it easier to verify that the right version is being picked up for the reproduction.

This was reported as moby/moby#34092. The comment at moby/moby#34092 (comment) explains the issue as related to docker/moby. This seems to be related to the fix in https://go-review.googlesource.com/c/31444/ and https://go-review.googlesource.com/c/32234/.

Conditions

The only two conditions for this are a path with a length greater than 100, with at least two components (i.e. not a but a/b) and a uid with an octal value of 010000000 or greater.

See out.go for specific details. After going through the example below, you can play with the values to confirm the above conditions.

Generate the problem tar file

To begin, we'll use out.go to generate a bad.tar.

We generate such a tar as follows:

$ GOROOT=/usr/local/go1.7.6/ /usr/local/go1.7.6/bin/go run out.go > bad.tar
2017/07/13 18:05:20 writing tar go1.7.6
2017/07/13 18:05:20 GOROOT /usr/local/go1.7.6/
2017/07/13 18:05:20 writing entry path of length 101 with aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/foo

At this point, let's also use the exact same code, but with Go 1.8 to generate a tar:

$ GOROOT=/usr/local/go1.8.3 /usr/local/go1.8.3/bin/go run out.go > good.tar
2017/07/13 18:17:31 writing tar go1.8.3
2017/07/13 18:17:31 GOROOT /usr/local/go1.8.3
2017/07/13 18:17:31 writing entry path of length 101 with aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/foo

Behavior with GNU tar

The first problem with this file is that we lose that obnoxious path name if we try to read it with regular GNU tar:

$ tar tvf bad.tar
---------- 2097152/0         0 1969-12-31 16:00 foo

Only the basename foo is preserved in GNU tar. When generated with Go 1.8, we get the expected behavior:

$ tar tvf good.tar
---------- 2097152/0         0 1969-12-31 16:00 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/foo

Reading bad tar with Go 1.7

Within this gist, we also have a file in.go. Let's try to read this file with Go 1.7 and see what happens:

$ GOROOT=/usr/local/go1.7.6 /usr/local/go1.7.6/bin/go run in.go < bad.tar
2017/07/13 18:08:30 reading tar go1.7.6
2017/07/13 18:08:30 GOROOT /usr/local/go1.7.6
&tar.Header{Name:"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/foo", Mode:0, Uid:2097152, Gid:0, Size:0, ModTime:time.Time{sec:62135596800, nsec:0, loc:(*time.Location)(0x539980)}, Typeflag:0x0, Linkname:"", Uname:"", Gname:"", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}, ChangeTime:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}, Xattrs:map[string]string(nil)}

While the output is a little less nice that tar, we can see that we read the original file path just fine, even though tar loses that information.

Reading bad tar with Go 1.8

The problem for docker and other Go users arises when we try to read this file with Go 1.8+:

$ GOROOT=/usr/local/go1.8.3 /usr/local/go1.8.3/bin/go run in.go < bad.tar
2017/07/13 18:10:16 reading tar go1.8.3
2017/07/13 18:10:16 GOROOT /usr/local/go1.8.3
&tar.Header{Name:"foo", Mode:0, Uid:2097152, Gid:0, Size:0, ModTime:time.Time{sec:62135596800, nsec:0, loc:(*time.Location)(0x53ed20)}, Typeflag:0x0, Linkname:"", Uname:"", Gname:"", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}, ChangeTime:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}, Xattrs:map[string]string(nil)}

We get roughly the same result as GNU tar, but ultimatelycannot read files that were generated with previous versions of Go.

Reading good tar with Go 1.8

As a control, we can use the same command from above to read the good.tar:

$ GOROOT=/usr/local/go1.8.3 /usr/local/go1.8.3/bin/go run in.go < good.tar
2017/07/13 18:22:49 reading tar go1.8.3
2017/07/13 18:22:49 GOROOT /usr/local/go1.8.3
&tar.Header{Name:"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/foo", Mode:0, Uid:2097152, Gid:0, Size:0, ModTime:time.Time{sec:62135596800, nsec:0, loc:(*time.Location)(0x53ed20)}, Typeflag:0x0, Linkname:"", Uname:"", Gname:"", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}, ChangeTime:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}, Xattrs:map[string]string(nil)}

Note that, as expected, we see the right path, since this was generated with Go 1.8.

in.go

package main

import (
	"archive/tar"
	"fmt"
	"log"
	"os"
	"runtime"
)

func main() {
	log.Println("reading tar", runtime.Version())
	log.Println("GOROOT", runtime.GOROOT())
	tr := tar.NewReader(os.Stdin)
	hdr, err := tr.Next()
	if err != nil {
		log.Fatalln(err)
	}

	fmt.Printf("%#v\n", hdr)
}

out.go

package main

import (
	"archive/tar"
	"log"
	"os"
	"runtime"
	"strings"
)

// If this file is run with Go 1.7 and earlier, it will generate an invalid tar
// file that will be read incorrectly with GNU tar and Go 1.8+.
func main() {
	log.Println("writing tar", runtime.Version())
	log.Println("GOROOT", runtime.GOROOT())

	basename := "foo"
	path := strings.Repeat("a", 100-len(basename)) + "/" + basename
	tw := tar.NewWriter(os.Stdout)
	hdr := tar.Header{
		Name: strings.Repeat("a", 100-len(basename)) + "/" + basename, // overflow path to force GNU prefix
		Uid:  010000000,                                               // octal string greater than 7 bytes.
		// Note that if we use 010000000 - 1, the condition is not encountered
	}

	log.Println("writing entry path of length", len(path), "with", path)
	if err := tw.WriteHeader(&hdr); err != nil {
		log.Fatalln(err)
	}

	// no data required.

	if err := tw.Flush(); err != nil {
		log.Fatalln(err)
	}
}