compose-go v3 yaml.Node refactoring — initial plan + current postMergeLegacy/mapstructure removal + docker/compose#13799 validation strategy

gist_compose_go_v3_refactoring.md

compose-go v3 — refactoring plan: yaml.Node based loader

This document consolidates the full refactoring story for compose-spec/compose-go's switch from a map[string]any loader to a yaml.Node based pipeline (v3 module path). It supersedes the incremental notes (plan.md at commit 60e91d3 and the subsequent "postMergeLegacy / mapstructure removal" addendum): the design choices that emerged in those two passes are folded in from the start.

It has three sections:

• Motivation — why the legacy map[string]any loader is unfit and what the new pipeline guarantees.
• Target architecture & implementation phases — a single ordered plan: types, pipeline, public API, file layout, and the work to do on each phase.
• Validation strategy — the docker/compose#13799 draft-PR loop that gates every iteration on the real downstream consumer's CI.

---

1. Motivation

1.1 Limitations of the legacy loader

The legacy loader parses each Compose file into a map[string]any, merges the resulting maps, runs interpolation on the merged dictionary, and finally decodes it into typed Go structs through mapstructure. The pipeline discards per-node provenance early:

1. Lost context after merge. Once two files are merged into a single map[string]any, there is no way to tell which leaf came from which file. The information needed to interpret that leaf correctly — the working directory of its source file, and the environment variables that were in scope when it was parsed — is no longer reachable.

2. include: cannot honor its own context. An include: directive may redefine the working directory and provide additional environment variables via include.env_file. With the legacy loader the included file's content is folded into the global map and the include-specific working directory and environment are dropped. Any later operation that should consume them (path resolution, environment resolution of a service declared in the included file) operates on the wrong context.

3. WithServicesEnvironmentResolved is incomplete. It iterates the typed service.EnvFiles but its lookup only sees project.Environment and service.Environment. If a service comes from a file included with include.env_file: …, the variables the include provided are not reachable. env_file: ${VAR} declared in an included service cannot be resolved when VAR was provided by the include itself.

4. Poor diagnostics. Errors reported during decoding or validation reference only the merged document, not the original file. A user inspecting a loaded Project cannot ask "which file does this value come from, and on which line?".

5. mapstructure dependency. Map-based decoding requires a third-party reflection library and a parallel set of DecodeMapstructure methods on every custom type, duplicating the logic that already exists for YAML decoding. The four global hooks (decoderHook, cast, nameServices, secretConfigDecoderHook) sit between the merged tree and the typed project and behave subtly differently from a real UnmarshalYAML.

6. Marshal / unmarshal round-trips. Schema validation goes through map[string]any, which means the merged tree has to be marshaled and re-parsed for validation purposes — wasteful and a source of subtle differences between the validated representation and the decoded one.

1.2 What the new pipeline guarantees

• A single in-memory representation of the merged project as a *yaml.Node tree. Interpolation, path resolution, canonicalisation, default-value injection, schema and semantic validation, and typed decoding all operate on that tree.
• A NodeContext attached to every node (via a map[*yaml.Node]*NodeContext carried by ComposeModel) that records the source file, working directory, scoped environment, and parent context. Merge is leaf-preserving so the context survives every pass.
• Typed decoding via Decode(project) plus UnmarshalYAML(*yaml.Node) on every custom type. No mapstructure, no DecodeMapstructure, no Transform helper, no global decoder hooks.
• Lazy env_file / label_file reading: LoadWithContext returns a project whose EnvFile.Context carries enough information to resolve the file later, but does not open it. Consumers that only manipulate a subset of services never pay for env files they do not use.
• A diagnostic surface (project.OriginOf(path) → {Source, Line, Column}) that lets compose config, IDEs, and language servers explain where each value comes from.

---

2. Target architecture

2.1 Core types

// types/node_context.go
type NodeContext struct {
    Source     string   // yaml file path the node was parsed from
    WorkingDir string   // base directory to resolve relative paths
    Env        Mapping  // environment variables in scope for interpolation
    Parent     *NodeContext
}

// Origin pairs a node context with a position inside its source file.
type Origin struct {
    Source string
    Line   int
    Column int
}

// loader/model.go
type Layer struct {
    Root    *yaml.Node
    Context *types.NodeContext
}

type ComposeModel struct {
    layers        []*Layer
    contexts      map[*yaml.Node]*types.NodeContext
    merged        *yaml.Node          // set once mergeLayers has run
    dict          map[string]any      // lazy, computed by Dict()
    opts          *Options
    configDetails types.ConfigDetails
    loadedFiles   []string
}

func (m *ComposeModel) Merged() *yaml.Node     // the merged tree
func (m *ComposeModel) Dict() map[string]any   // lazy via schema.NodeToInterface(m.merged)

Invariants:

• Once a node is registered in contexts, the entry survives every later operation (merge, interpolation, path resolution, canonicalisation, defaults injection, schema validation).
• MergeNodes never clones leaf scalar nodes. Only mappings and sequences may be cloned, and only when actually modified.
• Included trees are never mutated by pre-processing passes. No path rewriting, no bare-variable expansion. Whatever the loader needs at the end, it derives by walking the merged tree and consulting contexts[node].
• dict is only ever computed on demand and is never used as the source of truth inside the loader.

2.2 Pipeline of LoadWithContext

LoadWithContext(ctx, configDetails, opts...)
  │
  └── load(ctx, configDetails, opts) -> *ComposeModel
        ├── parseLayers            // []*Layer with per-file NodeContext
        │                          // multi-document YAMLs split into N layers
        ├── expandIncludes         // adds extra Layers, parent context preserved
        │                          // emits "include" Listener events
        ├── resolveSecretsConfigsEnvLayer
        │                          // canonical content: nodes, per layer
        ├── inferExtends           // resolves extends in-tree
        ├── mergeLayers            // single *yaml.Node, contexts intact
        ├── stripResetOverride     // !reset / !override semantics
        ├── interpolate            // per-scalar, NodeContext.Env via contexts[node]
        ├── resolvePaths           // per-path scalar, NodeContext.WorkingDir
        │                          // env_file.path EXCLUDED — resolved lazily
        ├── canonicalizeNode       // ports, volumes, secrets/configs, dns, gpus,
        │                          // devices, ssh, extra_hosts, depends_on,
        │                          // env_file, extends, ulimits, legacy external
        ├── setDefaultValuesNode
        ├── omitEmptyNode
        ├── stripVersion
        ├── enforceUnicityNode
        ├── validateNode           // semantic checks (yaml.Node)
        └── schema.ValidateNode    // JSON Schema on the yaml.Node tree

  └── ModelToProject(model) -> *types.Project
        ├── model.Merged().Decode(project)     // UnmarshalYAML on every type
        ├── nameServices/Networks/Volumes/...  // post-decode key-naming pass
        ├── processProjectExtensions           // yaml round-trip for x-extensions
        ├── attachEnvFileContexts
        ├── attachLabelFileContexts
        ├── normalizeProject                   // operates on *Project
        └── checkConsistency

WithServicesEnvironmentResolved and WithServicesLabelsResolved are not called by LoadWithContext. The consumer calls them explicitly when (and only when) it actually needs the resolved environments.

2.3 Public API

Symbol	Shape
loader.load (private)	(ctx, configDetails, opts) -> (*ComposeModel, error)
loader.LoadWithContext	unchanged signature, body = load + ModelToProject
loader.LoadModel	(ctx, configDetails, opts) -> (*ComposeModel, error) — replaces LoadModelWithContext
loader.ModelToProject	(model *ComposeModel) -> (*types.Project, error)
loader.LoadAnnotatedYaml	rebuilt on top of load + the diagnostic API
cli.ProjectOptions.LoadModel	returns *ComposeModel
loader.Transform	not introduced — no mapstructure detour
mapstructure hooks (decoderHook, cast, nameServices, secretConfigDecoderHook)	not introduced
go.mod	github.com/go-viper/mapstructure/v2 never added to v3

2.4 File layout

loader/
  model.go              # Layer, ComposeModel, accessors
  loader.go             # LoadWithContext, LoadModel, load
  resolve.go            # parseLayers, expandIncludes, mergeLayers
  include.go            # contextual graft + "include" listener event
  extends.go            # pre-merge extends resolution
  interpolate.go        # per-scalar interpolation
  paths.go              # single-pass path resolution
  canonical_node.go     # canonicalisation on yaml.Node
  defaults_node.go      # default-value injection on yaml.Node
  omit_empty_node.go    # omit-empty pass on yaml.Node
  normalize_project.go  # post-decode normalize on *Project
  annotate.go           # diagnostic / provenance output

validation/
  validate_node.go      # semantic validation on yaml.Node

schema/
  validate_node.go      # JSON Schema on yaml.Node
  node_to_interface.go  # bridge used for Dict() and error formatting only

types/
  node_context.go       # NodeContext, Origin
  yaml.go               # NodeError, NodeErrorf, WrapNodeError,
                        # WithSource, resolveYAMLNode, hasKey, findYAMLKey
  *.go                  # every custom type implements UnmarshalYAML(*yaml.Node)

override/
  merge_node.go         # yaml.Node based merger
  unicity_node.go       # EnforceUnicity on yaml.Node
  reset_override.go     # !reset / !override on yaml.Node

---

3. Implementation phases

Phases are sequenced so that each one ends on a green test suite and a green docker/compose#13799 CI run (see §4). One commit per phase or per sub-step.

Phase 1 — Skeleton: NodeContext, Layer, ComposeModel

• types/node_context.go: NodeContext, Origin.
• loader/model.go: Layer, ComposeModel, Merged(), Dict() (lazy via schema.NodeToInterface).
• types/yaml.go: shared helpers — NodeError, NodeErrorf, WrapNodeError, WithSource, resolveYAMLNode, hasKey, findYAMLKey.

Phase 2 — Parsing and merging

• parseLayers: one Layer per parsed file. Multi-document YAML files expand into one Layer per document from the start, so OCI round-trips (publish/reload) round-trip cleanly.
• override.MergeNodes: leaf-preserving merge into a single *yaml.Node. Mappings and sequences may be cloned; scalars never are.
• mergeLayers populates ComposeModel.merged and never drops entries from contexts.

Phase 3 — include: as a contextual graft

• Each include: entry is parsed as its own Layer with its own NodeContext (Source, WorkingDir, Env from include.env_file, Parent set).
• The graft is appended to layers — the included tree is not mutated, no path rewriting, no bare-variable expansion.
• opts.ProcessEvent("include", …) is emitted at graft time so downstream listeners (publish, OCI checks) see the include.

Phase 4 — extends: without path rewriting

• Resolved pre-merge, in-tree, using NodeContext rather than path string surgery. Targets reachable through includes use the include's context.

Phase 5 — !reset / !override semantics

• stripResetOverride runs on the merged tree, after merge, before interpolation. Tags are consumed from the YAML.

Phase 6 — Per-node interpolation

• One pass that walks the merged tree. Each scalar consults contexts[node].Env. Compose-style escape sequences and defaults are handled per scalar. No marshal/unmarshal.

Phase 7 — Path resolution

• Single pass driven by NodeContext.WorkingDir. Covers build, volumes (bind sources), env_file.path is excluded — resolved lazily, see Phase 13.

Phase 8 — Canonicalisation on yaml.Node

• loader/canonical_node.go, walker keyed by tree.Path, mirroring every entry in transform/canonical.go: ports, volumes, secrets/configs, dns / StringOrList, gpus, devices, ssh, extra_hosts, depends_on, env_file, extends, ulimits, legacy external: {name}.

Phase 9 — Default-value injection on yaml.Node

• loader/defaults_node.go, ported from transform/defaults.go.

Phase 10 — OmitEmpty on yaml.Node

• loader/omit_empty_node.go. Empty scalars and empty mappings/sequences are removed pre-decode; empty slices that the schema demands keep their representation (see commit c9079e9).

Phase 11 — Semantic validation on yaml.Node

• validation/validate_node.go, ported from validation/validation.go. Includes EnforceUnicityNode covering every indexer of override/uncity.go.

Phase 12 — Schema validation on yaml.Node

• schema.ValidateNode(merged) runs JSON Schema directly on the merged tree. schema.NodeToInterface exists only as a bridge for the lazy Dict() accessor and error formatting; the validation itself does not go through it.

Phase 13 — UnmarshalYAML(*yaml.Node) on every custom type

Clusters, in order, harvested and adapted from prior art on compose-spec/compose-go#854 (commit e074115 ships the bulk of the implementations and the types/yaml.go helpers — these are imported, the loader-pipeline choices of that PR are not):

• Cluster 1 — primitives: bytes.go, cpus.go, duration.go, command.go, options.go.
• Cluster 2 — collections: mapping.go, labels.go, stringOrList.go, hostList.go.
• Cluster 3 — service fields: device.go, ssh.go, healthcheck.go, models.go.
• Cluster 4 — types/types.go: ports, volumes, secrets, configs, extends, env_file, ulimits.

Each type's single UnmarshalYAML covers what mapstructure's decoderHook + cast + per-type DecodeMapstructure used to do:

• string ↔ bool/int/float coercion: per-type tag handling.
• string-or-list / string-or-mapping shapes: per-type.
• EnvFile.Context: written here from contexts[node].
• secret.content is already in canonical form (Phase 8 / step resolveSecretsConfigsEnvLayer), so the legacy secretConfigDecoderHook shape never appears.

Phase 14 — ModelToProject(*ComposeModel) -> *types.Project

func ModelToProject(model *ComposeModel) (*types.Project, error) {
    project := newProject(model)
    if err := model.Merged().Decode(project); err != nil { return nil, err }
    nameServices(project); nameNetworks(project); ...     // ex-mapstructure hook
    processProjectExtensions(project, model.Merged())     // yaml round-trip
    attachEnvFileContexts(project, model)
    attachLabelFileContexts(project, model)
    normalizeProject(project)                              // ex-loader/normalize
    checkConsistency(project)
    return project, nil
}

No Transform. No mapstructure hooks. processProjectExtensions is the yaml round-trip from PR #854 (one Marshal/Unmarshal per x- extension, isolated to this single helper).

Phase 15 — Lazy env_file / label_file resolution

• EnvFile.Context carries the NodeContext resolved at decode time.
• WithServicesEnvironmentResolved(ctx, project, opts) uses EnvFile.Context.WorkingDir for base directory and EnvFile.Context.Env (which already includes include.env_file-provided variables) for lookups.
• Same shape for WithServicesLabelsResolved.

Phase 16 — Diagnostic API

origin, ok := project.OriginOf("services.api.image")
// origin == {Source: "/path/to/override.yaml", Line: 14, Column: 12}

• Backed by ComposeModel.contexts + the merged tree's Line/Column.
• Powers cmd --debug (annotated yaml output, already wired on 2609dbc), compose config provenance comments, and IDE jump-to-definition.

Phase 17 — Cleanup

• Delete loader/post_merge.go, the legacy load/loadYamlModel/loadYamlFile/processRawYaml/ convertToStringKeysRecursive/fixEmptyNotNull/ loadIncludeConfig paths in loader/loader.go, loader/include.go, loader/extends.go, loader/environment.go, loader/omitEmpty.go, loader/normalize.go.
• go mod tidy — confirm mapstructure never enters go.mod on v3.
• Update CHANGELOG and README examples to the new public API (LoadModel, ModelToProject(*ComposeModel), no Transform).

---

4. Validation strategy: docker/compose#13799

A library-only test suite is not enough: compose-go's contract is defined by what docker/compose actually does with a loaded project (compose up, compose config, compose publish, OCI round-trips). A draft PR kept open on docker/compose runs the full docker/compose CI against our in-flight compose-go branch.

4.1 Wiring

docker/compose#13799 (branch test-compose-go-v3, marked DNM) adds a replace directive in go.mod:

// Test compose-spec/compose-go#874 (v3 yaml.Node-based loader)
replace github.com/compose-spec/compose-go/v3 => github.com/ndeloof/compose-go/v3 v3.0.0-<timestamp>-<hash>

Every compose-go commit on ndeloof/v3-yaml-node-context is followed by a one-line bump of that pseudo-version on ndeloof/test-compose-go-v3, signed-off and pushed. GitHub Actions re-runs the docker/compose CI matrix:

• validate (lint), validate (docs) — fast.
• binary build and bin-image-test build per OS/arch — compile parity.
• e2e (plugin, oldstable | stable) / e2e (standalone, oldstable | stable) — 10–12 min each, the real signal: every interesting compose-go behaviour ends up exercised through pkg/e2e (publish, OCI round-trip, secrets from include, env interpolation, profiles, …).

4.2 Why a draft PR rather than a local checkout

• The CI runs against docker:dind and a fresh registry container, reproducing things impossible to reproduce locally without Docker-on-Docker.
• Each push gets a stable artefact link in the PR for triage.
• Reviewers (and future contributors) can see exactly which compose-go pseudo-version a given docker/compose run was tested against.
• PR commenting integrates with the change history — useful when a regression appears after a specific compose-go commit.

4.3 Workflow per compose-go change

1. Focused commit on ndeloof/v3-yaml-node-context.
2. go test ./..., golangci-lint run, make deepcopy (no diff) locally.
3. Push to ndeloof/v3-yaml-node-context. Wait for compose-spec/compose-go#874 CI green.
4. In docker/compose@test-compose-go-v3, bump the replace pseudo-version to point at the new compose-go commit, go mod tidy, commit (signed-off), push.
5. Wait for docker/compose#13799 CI. If any e2e job fails, inspect the log (gh api .../jobs/<id>/logs), reproduce locally if possible, fix on compose-go, repeat.
6. Once docker/compose#13799 is green, the compose-go commit is confirmed safe.

4.4 Regressions surfaced by the loop (used as design guards)

The loop has already caught three regressions invisible to the compose-go unit-test suite. Each design choice above is set so the regression cannot reappear:

1. TestSecretFromInclude — secret.environment: VAR not resolved when VAR came from include.env_file. Guard: Phase 3 keeps the included NodeContext, Phase 8's resolveSecretsConfigsEnvLayer resolves per layer (40c3140).
2. TestPublishChecks/refuse_to_publish_with_local_include — "include" Listener event missing. Guard: Phase 3 emits opts.ProcessEvent("include", …) from loadIncludeEntry (45cca9c).
3. TestPublish (OCI extends round-trip) — loadYamlFileNode silently kept only the last document of a multi-document YAML. Guard: Phase 2's parseLayers returns one Layer per document (be24303).

Each fix also landed as a compose-go-side regression test (TestInclude_EnvFile_ResolvesEnvironmentBackedSecret, TestInclude_EmitsIncludeListenerEvent, TestLoadMultiDocumentYaml, …) so the failure stays caught even if the docker/compose draft PR were eventually closed.

4.5 Gating per phase

• After every Phase ending — bump and run docker/compose#13799.
• Phase 13 (UnmarshalYAML per cluster): the e2e suite is the only reliable way to catch UnmarshalYAML bugs that mapstructure's weaker coercion used to mask. Each cluster ends on a green docker/compose#13799 run.
• Phase 17 cannot be declared complete until a full docker/compose#13799 run is green on the cleanup commit.

The replace directive in docker/compose@test-compose-go-v3 is the single dial; everything else follows automatically.