Why PR #13608 golang:test-all fails, why it fails now, and what the failure teaches us about recursive Dagger workspaces

pr-13608-golang-vcs-briefing.md

When a workspace marker becomes a broken Git repository

Root-cause briefing for golang:test-all on dagger/dagger#13608

This briefing answers two related questions:

1. Why is the check failing?
2. Why did it start failing now?

The short answer is:

The generic Go test tool creates a deliberately incomplete .git/HEAD as a workspace-boundary marker for nested Dagger clients. The new Go client e2e harness later feeds that projected workspace back into Dagger's Go module builder. Go mistakes the marker for a real Git repository, tries to collect VCS metadata, and fails because the repository is only a marker.

This is not fundamentally a notify problem, a test assertion failure, or a new Go-toolchain regression. It is a composition bug: two individually reasonable behaviors meet for the first time in this PR and disagree about what .git means.

The cast of characters

Four systems participate:

• The generic Go toolchain module discovers Go modules and creates a small, filtered /ws tree in which to run each module's tests.
• Dagger workspace detection walks upward until it finds .git; that entry defines the workspace boundary.
• The new sdk/go/e2e harness obtains CurrentWorkspace() and dynamically serves toolchains/engine-dev from it.
• The Go command automatically stamps VCS metadata into main binaries when -buildvcs=auto, the default.

The problem lives at the boundary between these systems, not inside any one of them.

The learning journey

1. Start with the check, not a theory

gh pr checks identified one failure:

golang:test-all  fail
Errored in 58.9s.
Run `dagger trace 81ee336159c92c177f1e0a593958480b`

The trace showed two failed tests:

github.com/dagger/dagger/sdk/go/e2e › TestAgainstEngines › dev
github.com/dagger/dagger/sdk/go/e2e › TestBootstrap

Both failed during the same setup operation:

serve engine-dev module:
failed to load dependencies as modules:
failed to call module "notify" to get functions:
call constructor: exit code: 1

At first glance, this looks like a notify module failure.

That is the first misleading clue.

2. notify is the messenger, not the cause

Both tests call the shared newHarness. Before either test builds a CLI, starts an engine, or runs an assertion, the harness does this:

dag.CurrentWorkspace().
    ModuleSource("/toolchains/engine-dev").
    AsModule().
    Serve(ctx)

Serving engine-dev loads its dependency graph. That graph includes notify, alpine, wolfi, go, codegen, and dagger-cli.

The detailed trace showed both the notify and alpine Go runtime builds failing. Dependency loading is concurrent; notify happened to be the error reported at the top. Nothing in its Slack or Discord logic ran.

Zooming into the shared failed span revealed the useful error:

error obtaining VCS status: exit status 128
    Use -buildvcs=false to disable VCS stamping.

The failing command was Dagger's standard Go module-runtime build:

go build -ldflags "-s -w" -o /runtime .

That command comes from core/sdk/go_sdk.go.

We have now moved one layer down:

not "notify constructor is broken"
but "Go cannot inspect the repository while building a module runtime"

3. Follow the filesystem, because Go is complaining about Git

The trace shows how the generic Go test tool constructs the test container:

1. Select only the files required by the Go module and its //go:test:include directives.

2. Mount that projection at /ws.

3. Create /ws/.git/HEAD with:

   ref: refs/heads/main
   

4. Change the working directory to /ws/sdk/go/e2e.

5. Run otelgotest.

The relevant source is in the pinned github.com/dagger/go toolchain:

# Nested Dagger clients find the workspace boundary by walking up to .git.
.withNewFile(".git/HEAD", "ref: refs/heads/main\n")

The comment is important. The tool is not trying to construct a Git repository. It is constructing a Dagger workspace-boundary marker.

Dagger only needs the marker to exist. Its workspace detector explicitly walks upward looking for .git; it does not validate the repository contents at that stage. See core/workspace/detect.go.

So, for Dagger's purpose, this is sufficient:

/ws/.git/HEAD

For Git's purpose, it is not.

A real repository also needs coherent metadata such as configuration, refs, objects, and an index or valid empty-repository structure. .git/HEAD alone is like putting a sign saying “library” on an empty room: it is enough to find the room, but not enough to check out a book.

4. The marker escapes into a second build

If the projected workspace were used only to run ordinary tests, the story would stop there. The new harness does something unusual and important:

1. The Go test starts inside the projected /ws.
2. It connects to Dagger.
3. It asks for CurrentWorkspace().
4. It loads engine-dev from that workspace.
5. Dagger's Go SDK builds Go runtime binaries for engine-dev and its dependencies from the same projected tree.

The boundary marker has now crossed an abstraction boundary. It began life as “a hint for Dagger workspace discovery,” but the nested Go build sees it as ordinary source-tree content.

flowchart TD
    A["golang:test-all selects sdk/go/e2e"] --> B["Generic Go tool projects selected files into /ws"]
    B --> C["Tool adds only /ws/.git/HEAD<br/>as a Dagger workspace marker"]
    C --> D["otelgotest runs the new e2e harness"]
    D --> E["newHarness asks CurrentWorkspace()<br/>to serve engine-dev"]
    E --> F["Dagger Go SDK builds engine-dev dependencies"]
    F --> G["go build uses -buildvcs=auto"]
    G --> H["Go sees /ws/.git and assumes a Git repository"]
    H --> I["Go runs git status --porcelain"]
    I --> J["Git exits 128:<br/>the marker is not a repository"]

Loading

5. Why Go turns this into a hard failure

Go's default -buildvcs=auto behavior stamps repository information into a main binary when the package, module, and working directory appear to belong to the same repository.

The implementation:

1. detects .git;
2. selects the Git VCS driver;
3. runs git status --porcelain;
4. reads the current revision and commit time; and
5. adds fields such as vcs.revision and vcs.modified to the binary's build information.

See the Go 1.26.3 sources for setBuildInfo and gitStatus.

auto does not mean “ignore every Git error.” Once Go finds what looks like a repository and the git executable is available, failure to obtain its status is fatal.

The minimal reproduction is tiny:

example/
├── go.mod
├── main.go
└── .git/
    └── HEAD    # contains: ref: refs/heads/main

Results:

$ go build .
error obtaining VCS status: exit status 128
    Use -buildvcs=false to disable VCS stamping.

$ git status --porcelain
fatal: not a git repository (or any of the parent directories): .git

$ go build -buildvcs=false .
# succeeds

This exactly reproduces the CI error without Dagger, engine-dev, or notify. That isolates the failing invariant:

A directory containing only .git/HEAD is simultaneously valid as Dagger's current workspace marker and invalid as Go's perceived Git repository.

Why now?

The marker itself is not new.

The previous github.com/dagger/go pin already contained the same withNewFile(".git/HEAD", ...). The new pin still does. Both use Go 1.26 by default. Therefore neither the toolchain pin update nor a new Go VCS-stamping policy explains the timing.

What changed is the route taken by the tests.

Before this PR	With this PR
golang:test-all selected only e2e/**.	It selects e2e/**,sdk/go/**, so it discovers and runs sdk/go/e2e.
Go SDK tests used the dedicated go-sdk-dev.test path.	Go client tests become standard modules under the generic Go tool.
The outer Dagger graph installed the development client/engine environment before running go test.	The test itself connects back to Dagger and dynamically serves engine-dev from CurrentWorkspace().
The synthetic .git/HEAD marker was not reused as source for another Go module build.	The projected workspace, marker included, becomes input to nested Go runtime builds.

The selection change is visible in dagger.json. The new dynamic module load is in sdk/go/e2e/harness_test.go.

So the precise answer to “why now?” is:

This PR is the first time a test running inside the generic Go tool's marker-only workspace turns around and asks Dagger to compile Go modules from that same workspace.

No single component suddenly regressed. A previously latent incompatibility became reachable.

The recent history cleanup is also unrelated: it changed commit structure and DCO trailers, while preserving the source tree byte-for-byte. It merely caused CI to run again.

Why an existing test passes

There is a useful control case. The Helm e2e tests also perform nested Dagger module work under the generic Go tool, but they explicitly include the real repository metadata:

//go:test:include ../../.git

See e2e/helm/helm_test.go.

For that module, the synthetic HEAD is laid over a complete .git directory, so git status has the data it needs. That explains why the generic mechanism is not failing everywhere.

It also gives us a likely tactical fix, but not necessarily the best architectural fix.

Fix directions and their tradeoffs

1. Include the real .git directory in sdk/go/e2e

Add the same //go:test:include ../../../.git pattern used by Helm.

• Benefit: Small PR-local change with an existing repository precedent.
• Cost: Copies repository metadata into the test input, increases coupling to checkout shape, and may behave differently for worktrees or synthetic workspaces.
• Interpretation: Turn the marker into the real thing.

2. Remove .git before converting the workspace directory into a module source

Once CurrentWorkspace() has already been found, the marker has completed its job. The harness could derive a directory without .git, then call AsModuleSource with an explicit toolchains/engine-dev source root.

• Benefit: Keeps module-runtime builds hermetic and removes the false VCS signal at the boundary where it becomes harmful.
• Cost: Slightly more explicit module-source plumbing in the test harness.
• Interpretation: Do not let an internal workspace marker escape into a consumer that assigns it different semantics.

3. Build Dagger Go module runtimes with -buildvcs=false

Dagger could change the global runtime build in core/sdk/go_sdk.go.

• Benefit: Module runtime builds become independent of checkout metadata.
• Cost: Global behavior change; maintainers should first decide whether VCS build information in module runtime binaries is intentional or useful.
• Interpretation: Make hermeticity a property of the Go SDK runtime builder.

4. Stop representing a workspace boundary as a fake Git repository

The generic Go tool could eventually pass an explicit Dagger workspace boundary instead of manufacturing .git/HEAD.

• Benefit: Removes the semantic collision at its origin.
• Cost: Requires a broader workspace-selection mechanism for nested clients, not just a PR-local adjustment.
• Interpretation: A sentinel should not masquerade as a real object owned by another tool.

For this PR, options 1 or 2 are narrow. Option 2 better preserves the conceptual boundary: .git/HEAD is a Dagger discovery implementation detail, not an input to module compilation. Options 3 and 4 are broader follow-up design questions.

Root cause, stated at three levels

Immediate failure

go build runs git status --porcelain, which exits 128 because .git contains only HEAD.

Trigger

The new sdk/go/e2e harness dynamically serves engine-dev from the generic Go test container's projected CurrentWorkspace.

Underlying design problem

One layer creates a partial representation of a Git repository as a private sentinel, then exposes it as ordinary filesystem state to another layer that correctly interprets .git according to Git's public contract.

The durable lesson

Filtered workspaces are not passive bags of files. They are executable inputs to downstream tools, and those tools infer behavior from special paths.

Whenever we synthesize a well-known path such as .git, go.mod, package.json, or .env, we inherit the semantic contract of that path for every later consumer—not only the consumer for which we created it.

Here, the system worked until the workspace became recursive:

Dagger builds a workspace
→ code inside that workspace asks Dagger to build from the workspace
→ the implementation detail returns as user-visible source

That recursion is why the bug was latent, why the error message named the wrong module, and why it appeared now.