Skip to content

Instantly share code, notes, and snippets.

@bradfitz

bradfitz/build-diff.md

Last active June 11, 2026 18:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bradfitz/34b5a28559388b2c71f941f430f106fe to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bradfitz/34b5a28559388b2c71f941f430f106fe to your computer and use it in GitHub Desktop.
Download ZIP
Raw
build-diff.md

tailscale v1.98.4: tgz vs docker image build comparison

Comparison of the tailscaled and tailscale (CLI) binaries shipped in two artifacts that both claim to be Tailscale v1.98.4:

  • tgz: https://pkgs.tailscale.com/stable/tailscale_1.98.4_amd64.tgz
  • docker: tailscale/tailscale:v1.98.4 (digest sha256:25cde9ad…)

Both binaries report tailscale.com v1.98.4 and were compiled with Go 1.26.3, but every other relevant signal differs: build flags, CGO setting, embedded VCS metadata, and a non-trivial slice of the dependency graph. Neither pair is byte-identical.

Binary hashes

binary sha256
tgz tailscaled aaef867dc2ed94aae433abd68902f06984332f4f4eb7c9f1ea149b3ffbdae3e2
tgz tailscale 957d73713f10fc18239378d126608a1a7535d6f5ffd69dcd12a3d903c674408b
docker tailscaled daa1a85808fa1b3f2a9fbb78d116a16ad6fb43ed34d60523729e65abf5d63026
docker tailscale 1fb7007d123578c329a1617a6784569fe6c79304306caa40e2844c7712ceb623

All four are statically linked, x86-64, with debug info present.

Build configuration differences

Extracted via go version -m:

setting tgz docker
-tags (recorded) tailscale_go,osusergo,netgo ts_kube,ts_package_container
CGO_ENABLED 1 0
DefaultGODEBUG tlsmlkem=0 (unset)
main-module h1: hash present absent
vcs=git stripped revision=01c6b9661c9e001674c650fa655e56ee1fda4d90, time=2026-05-28T19:16:35Z, modified=false

The tgz build looks like the standard Tailscale release: CGO on, Tailscale Go fork with osusergo/netgo so DNS and user lookups use the pure-Go implementations even though CGO is enabled. The docker build is a CGO-off container build using ts_kube and ts_package_container tags, and it leaves VCS provenance embedded.

GODEBUG: ML-KEM is disabled in the tgz, enabled in the docker build

Both tgz binaries carry build DefaultGODEBUG=tlsmlkem=0, baked in at link time via //go:debug tlsmlkem=0 somewhere in the build (or an equivalent linker flag). The docker binaries have no DefaultGODEBUG setting at all, so they inherit Go 1.26's stock default, which is tlsmlkem=1 — ML-KEM-768 hybrid post-quantum key exchange is offered in TLS 1.3 client hellos and accepted server-side.

Practical effect:

  • tgz tailscaled / tailscale: no post-quantum hybrid in TLS. Connections use only classical X25519 (or P-256) ECDH unless the user explicitly overrides with GODEBUG=tlsmlkem=1 at runtime.
  • docker tailscaled / tailscale: post-quantum hybrid (X25519MLKEM768) is on by default and will be negotiated against any TLS 1.3 peer that supports it.

This is the single most user-visible runtime-behavior difference between the two artifacts — the rest of the divergence is build-graph drift, but this one changes wire behavior for outbound TLS (control-plane connections to controlplane.tailscale.com, log uploads, ACME, etc.). Likely intentional on the tgz side to avoid CH size / middlebox issues with the larger ML-KEM client hello; the docker build appears to have simply not picked up whatever source flips the default.

The published docker image is built by build_docker.sh invoking github.com/tailscale/mkctr. That script prepends ./tool to PATH so the go binary in use is ./tool/go, i.e. github.com/tailscale/go — the same Tailscale Go fork that produces the tgz. The repo's standalone Dockerfile uses upstream golang:1.26-alpine and explicitly states it is not what builds published images.

The tailscale_go build tag is on both, despite what -tags shows

Initial reading of the -tags field made it look like the docker build was missing the tailscale_go build tag. It is not.

As of github.com/tailscale/go commit 179b46cade ("cmd/go: set 'tailscale_go' build tag", 2026-03-31, on the tailscale.go1.26 branch which is current HEAD), the Tailscale Go toolchain auto-appends tailscale_go to BuildContext.ToolTags in src/cmd/go/internal/cfg/tailscale.go:

func init() {
    BuildContext.ToolTags = append(BuildContext.ToolTags, "tailscale_go")
}

ToolTags participate in //go:build file selection but are not recorded in runtime/debug.BuildInfo — that build -tags=... setting echoes only the user-supplied -tags flag. So:

  • tgz: tailscale_go was passed explicitly, so it appears in -tags and the //go:build tailscale_go files are compiled in.
  • docker: tailscale_go is added implicitly via ToolTags; it does not appear in -tags but the same files are still compiled in.

Verified empirically: both tailscaled binaries contain the symbol TS_PERMIT_TOOLCHAIN_MISMATCH and the panic format string from tailscale.com/assert_ts_toolchain_match.go, which is gated by //go:build tailscale_go. If the docker build had skipped the tag, those strings would not be in the binary. Therefore both binaries run the toolchain-rev assertion in init() on startup, and both are confirmed to be compiled by the Tailscale Go fork.

Dependency differences

The go.mod snapshot baked into the docker build lags the tgz build on many direct and transitive dependencies — the docker version is consistently older on every diff. Some lag by patch versions, several lag by multiple minor versions, and one dependency is present in the tgz tailscaled but absent from the docker tailscaled.

Shared between both binaries (tailscale and tailscaled)

module tgz docker
github.com/aws/aws-sdk-go-v2 v1.41.5 v1.41.0
github.com/aws/aws-sdk-go-v2/config v1.32.5 v1.29.5
github.com/aws/aws-sdk-go-v2/credentials v1.19.5 v1.17.58
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 v1.16.27
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 v1.8.2
github.com/aws/aws-sdk-go-v2/service/ssm v1.45.0 v1.44.7
github.com/aws/aws-sdk-go-v2/service/sso v1.30.7 v1.24.14
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 v1.28.13
github.com/aws/smithy-go v1.24.2 v1.24.0
github.com/go-json-experiment/json 2025-10-27 2025-08-13
github.com/jsimonetti/rtnetlink v1.4.1 v1.4.0
golang.org/x/exp 2026-02-18 2025-06-20
golang.org/x/time v0.15.0 v0.12.0

tailscaled-only

module tgz docker
github.com/creachadair/msync v0.8.1 v0.7.1
github.com/insomniacslk/dhcp 2024-01-29 2023-12-06
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 (absent)

tailscale (CLI)-only

module tgz docker
github.com/mattn/go-colorable v0.1.14 v0.1.13
go.yaml.in/yaml/v2 v2.4.3 v2.4.2

The same Go version (1.26.3), Tailscale Go fork, and main module version (tailscale.com v1.98.4) are used by both, so the dependency drift comes from the go.mod / go.sum snapshot present in each build's source tree, not from toolchain differences.

Takeaway

Both artifacts are legitimately v1.98.4 and both are compiled with the Tailscale Go fork. They are not, however, the same binary or even the same build:

  • Same: Go toolchain (Tailscale's fork of Go 1.26.3), main module version, tailscale_go build constraint active.
  • Different: CGO on (tgz) vs off (docker); osusergo/netgo only in tgz; container-specific tags (ts_kube, ts_package_container) only in docker; VCS provenance embedded only in docker; main-module h1: hash recorded only in tgz; DefaultGODEBUG=tlsmlkem=0 baked into the tgz (so ML-KEM hybrid TLS is off by default), absent from the docker build (so ML-KEM is on by default); ~15 dependency versions, with the docker go.mod snapshot trailing the tgz one on every divergent entry.

Practically, a downstream consumer comparing SBOMs across the two artifacts should expect them to differ even when the version tag is identical, and any CVE scan that pivots on dependency versions will produce different findings for the two.

Reproduction

Artifacts and intermediate SBOMs are in ~/sbom/:

  • tailscale_1.98.4_amd64.tgz, expanded to tailscale_1.98.4_amd64/
  • docker-fs.tar (full image filesystem export), with tailscaled and tailscale extracted under docker-fs/usr/local/bin/
  • sbom-tgz-tailscale{,d}.txt, sbom-docker-tailscale{,d}.txt — output of go version -m on each binary

Commands used:

curl -sSL -o tailscale_1.98.4_amd64.tgz \
  https://pkgs.tailscale.com/stable/tailscale_1.98.4_amd64.tgz
tar -xzf tailscale_1.98.4_amd64.tgz

docker pull tailscale/tailscale:v1.98.4
cid=$(docker create tailscale/tailscale:v1.98.4)
docker export "$cid" -o docker-fs.tar
docker rm "$cid"
mkdir -p docker-fs
tar -xf docker-fs.tar -C docker-fs usr/local/bin/tailscaled usr/local/bin/tailscale

for b in tailscale_1.98.4_amd64/tailscale{,d} docker-fs/usr/local/bin/tailscale{,d}; do
  go version -m "$b"
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment