den-schema: Standalone Schema Library Design Spec

2026-05-10-den-schema-standalone-design.md

den-schema: Standalone Schema Library

Standalone flake library extracting den's schema system into a generic typed record registry with extension points, strict-by-default validation, identity hashing, cross-instance references, introspection, and declarative methods.

Supersedes: 2026-05-09-schema-lib-extraction-design.md (earlier exploration). Companion to: 2026-05-09-aspects-lib-extraction-design.md (the aspects library). The two libraries are independent but composable — den's schema extensions wire them together.

Goals

1. Standalone flake library at ../den-schema — usable without den. Any Nix project needing a typed record registry with extension points can use it. Exposes both lib for programmatic use and flakeModules.default for flake-parts integration.
2. Kind registry — declare named record types (kinds), merge kind modules from multiple sources, build submodule types from them.
3. Extension API — any module can extend any kind: schema.host.options.vpnAlias = mkOption { ... };
4. Strict by default — undeclared freeform keys error with fix guidance. Per-kind opt-out via _module.freeformType. Global opt-out via mkSchema { strict = false; }.
5. Identity hashing — auto-computed id_hash from primitive options with three-layer precedence for key selection.
6. Instance registries — mkInstanceRegistry builds typed attrsOf options per kind. Replaces den's hostsOption/homesOption.
7. Cross-instance references — mkRefType validates references at eval time and resolves to the target instance.
8. Kind introspection — _meta.kindNames, _meta.kindMeta for tooling, documentation generation, and diag.
9. Documentation generation — renderDocs produces markdown reference from schema metadata.
10. Declarative methods — schemaFn declares functions on entities with automatic config argument resolution.

Non-Goals

• Aspect resolution, include tree expansion — that's the aspects library.
• Policy dispatch, enrichment, topology construction — that's the pipeline.
• Pipeline-specific fields on entities (resolved, mainModule, hasAspect) — den wires these via schema extensions through extraModules.
• Cross-entity relationships as explicit declarations — nesting is structural (registries inside registries), not declared metadata.

Public API

mkSchema

Single entry point. Returns a module option whose type is a freeform attrset of schema entry types. Each key is a kind.

schema = mkSchema {
  strict = true;          # default — undeclared keys error
  baseModule = { };       # module imported into every kind
};

Arguments:

• strict (bool, default true) — when true, injects mkStrictModule into every kind. Kinds opt out individually via _module.freeformType.
• baseModule (module, default {}) — module imported into every kind's merge. Shared options across all kinds without manual imports.

baseModule vs den's schema.conf: baseModule is a static argument set at mkSchema call time — it cannot be extended by downstream modules after the fact. Den's schema.conf is a schema kind that is user-extensible: any module can write den.schema.conf.options.foo = mkOption { ... } and it flows to all entity kinds via imports = [ den.schema.conf ].

For den's integration, baseModule alone doesn't replace conf. Den continues to use the import-based pattern: conf remains a regular schema kind, and entity kinds import it via extraModules:

mkInstanceType den.schema "host" {
  extraModules = [({ ... }: { imports = [ den.schema.conf ]; })];
};

baseModule is for library-level shared options that don't need downstream extensibility (e.g., _identity.keys defaults). conf remains an extensible shared base, owned by den, wired through the standard import pattern.

The returned option is a types.submodule with:

• freeformType = types.lazyAttrsOf schemaEntryType — each key is a kind
• options._meta — introspection (read-only, internal)

Kind declaration

Kinds are declared through the module system. Each kind's value is a deferred module defining options and config:

# Declare kinds:
schema.host = {
  options.name = mkOption { type = str; };
  options.addr = mkOption { type = str; };
};
schema.user = {
  options.userName = mkOption { type = str; };
};

# Extend from anywhere:
schema.host.options.vpnAlias = mkOption { type = str; default = config.name; };

# Shared base via mkSchema config — no manual imports needed:
# mkSchema { baseModule = { options.description = mkOption { type = str; default = ""; }; }; }

mkInstanceType

Builds a types.submodule for instances of a kind:

mkInstanceType = schema: kind: { extraModules ? [] }:
  lib.types.submodule ({ name, config, ... }: {
    imports = [ schema.${kind} ] ++ extraModules;
    config._module.args.${kind} = config;
    options.name = lib.mkOption {
      type = lib.types.str;
      default = name;
    };
  });

The schema kind module controls strictness and freeform — the instance type doesn't set freeformType. The kind's strict/freeform setting flows through to instances automatically.

extraModules is where consumers add kind-specific options beyond the schema. For den, this includes instantiate, intoAttr, mainModule, resolved, users, and critically, cross-entity _module.args bindings:

# Den's user type — host binding comes from extraModules:
mkInstanceType den.schema "user" {
  extraModules = [({ config, ... }: {
    # Cross-entity binding: user instances receive their parent host
    config._module.args.host = hostConfig;
  })];
};

# Den's home type — multiple cross-entity bindings:
mkInstanceType den.schema "home" {
  extraModules = [({ name, config, ... }: {
    # home instances may reference both host and user
    config._module.args.host = hostByName;
    config._module.args.user = userByName;
  })];
};

mkInstanceType only injects _module.args.${kind} = config (self-reference). Cross-entity bindings are the consumer's responsibility via extraModules. This is intentional — the schema library doesn't know about entity nesting relationships, and different consumers wire cross-entity context differently (den's home type does builtins.split "@" name to resolve host/user from the instance name).

mkInstanceRegistry

Wraps mkInstanceType into a complete option:

mkInstanceRegistry = schema: kind: { extraModules ? [], description ? "${kind} instances" }:
  lib.mkOption {
    inherit description;
    default = {};
    type = lib.types.attrsOf (mkInstanceType schema kind { inherit extraModules; });
  };

Den's current two-level nesting (system → hosts) is den-specific. The registry itself is a flat attrsOf. Den wraps it:

# Den's usage:
options.den.hosts = mkOption {
  type = attrsOf (submodule ({ name, ... }: {
    freeformType = attrsOf (mkInstanceType den.schema "host" {
      extraModules = [ /* den-specific: instantiate, intoAttr, users, mainModule */ ];
    });
  }));
};

mkRefType

Cross-instance references. Validates that a referenced instance exists and resolves to the instance:

mkRefType = instances:
  lib.types.str // {
    check = v: instances ? ${v};
    merge = loc: defs:
      let key = lib.mergeOneOption loc defs;
      in if instances ? ${key} then instances.${key}
         else throw "${lib.showOption loc}: '${key}' not found";
  };

Input value is a string (the instance key). Resolved value is the instance attrset. Validation happens in the type's merge.

# Declaration:
schema.service.options.host = mkOption {
  type = mkRefType config.hosts;
  description = "Host this service runs on";
};

# Usage — resolves to the instance:
config.services.nginx.host.addr  # direct access, no lookup step

Safe for cross-references because Nix is lazy — holding a reference doesn't force evaluation. Cycles only break on == comparison, which is what id_hash solves.

schemaFn

Declarative methods on entities. Declares a function whose arguments are automatically resolved from the instance's config:

schemaFn = description: type: fn: {
  inherit description type fn;
};

Usage via the methods sidecar on a kind:

schema.host.methods.secretPath = schemaFn
  "Resolve secret path for this host"
  (types.functionTo types.str)
  ({ name, ... }: secret: "/secrets/${name}/${secret}.age");

The entry type extracts methods from defs (same pattern as includes/excludes in den), generates both the option declaration and config wiring:

• options.secretPath = mkOption { description = ...; type = ...; readOnly = true; }
• config.secretPath = fn { name = config.name; ... } — args resolved via builtins.functionArgs

Multiple modules extending the same kind can each add methods. They compose via the deferred module merge.

renderDocs

Pure function generating markdown reference from schema metadata:

renderDocs = schema: /* markdown string with table per kind, row per option */;

Intentionally simple — markdown table per kind. Consumers needing richer docs use _meta.kindMeta directly.

Schema Entry Type

Core type that each kind's value resolves to. A customized deferredModule merge that:

1. Extracts sidecar fields (methods) before merge
2. Injects baseModule (from mkSchema config)
3. Injects mkStrictModule kind when strict = true
4. Injects identityModule kind (id_hash computation)
5. Injects methodsModule (option + config wiring from extracted methods)
6. Merges all kind definitions through standard deferredModule.merge

schemaEntryType = { strict, baseModule }:
  let base = lib.types.deferredModule;
  in base // {
    merge = loc: defs:
      let
        kind = lib.last loc;

        # Extract methods sidecar (same pattern as den's includes/excludes)
        allMethods = lib.foldl' (acc: d:
          if builtins.isAttrs d.value && d.value ? methods
          then acc // d.value.methods
          else acc
        ) {} defs;

        strippedDefs = map (d:
          if builtins.isAttrs d.value
          then d // { value = builtins.removeAttrs d.value [ "methods" ]; }
          else d
        ) defs;

        # Inject base, strict, identity modules
        injected =
          lib.optional (baseModule != {}) {
            file = "den-schema/base";
            value = baseModule;
          }
          ++ lib.optional strict {
            file = "den-schema/strict-default";
            value = mkStrictModule kind;
          }
          ++ [{
            file = "den-schema/identity";
            value = identityModule kind;
          }]
          ++ lib.optional (allMethods != {}) {
            file = "den-schema/methods";
            value = methodsModule allMethods;
          };

        merged = base.merge loc (strippedDefs ++ injected);
      in {
        __functor = _: { ... }: { imports = [ merged ]; };
      };
  };

Key difference from den's current schemaEntryType: No pipeline concerns. No includes/excludes extraction, no isEntity computation, no resolvedCtx, no collisionPolicy. Those stay in den as schema extensions.

The functor wrapping (__functor) is preserved — it's how schema kinds become importable modules in entity types (imports = [ schema.host ]).

Den's pipeline sidecar extraction: Den's current schemaEntryType extracts includes/excludes from defs before merge and exposes them as attributes on the merged result ({ __functor; includes; excludes; isEntity; }). Post-extraction, den has two options:

1. Wrap the entry type — den defines its own denSchemaEntryType that delegates to den-schema's schemaEntryType but adds sidecar extraction around it. Den's version extracts includes/excludes/isEntity from defs, strips them, passes the stripped defs to den-schema's merge, and attaches the extracted values to the result alongside the functor.

2. Use a separate module option — instead of sidecar fields on schema entries, den moves includes/excludes to a parallel option (den.pipelineWiring.host.includes = [...]) that the pipeline reads independently from the schema.

Option 1 is closer to the current architecture and lower migration risk. The schema library doesn't need to provide a hook — den wraps the type externally.

Identity System

Three-layer precedence for determining which options contribute to id_hash:

Layer 1: Explicit _identity.keys (highest priority)

Module option on every kind, set through the module system:

options._identity.keys = lib.mkOption {
  type = lib.types.listOf lib.types.str;
  default = [];
  description = "Explicit identity keys. Empty = use reflection.";
  apply = lib.unique;
};

• [] (default) — use reflection (Layers 2+3)
• Non-empty — use exactly those keys
• Multiple modules adding keys merge naturally via mkMerge

Composable: a schema composed from three flake inputs, each extending host with their own identity-relevant fields, all merge without coordination.

Layer 2: identity = false exclusion (opt-out)

Individual options declare themselves non-identity:

schema.host.options.description = mkOption {
  type = str;
  identity = false;  # excluded from id_hash reflection
};

The identity module checks opt.identity or true.

Layer 3: Automatic reflection (default)

All non-internal primitive options (str, int, bool) are included. Same as den's current behavior.

Identity module

identityModule = kind: { config, options, ... }: {
  options._identity.keys = lib.mkOption {
    type = lib.types.listOf lib.types.str;
    default = [];
    apply = lib.unique;
  };

  options.id_hash = lib.mkOption {
    readOnly = true;
    internal = true;
    type = lib.types.str;
    default =
      let
        explicitKeys = config._identity.keys;
        isPrimitive = name: opt:
          !(lib.hasPrefix "_" name)
          && (opt ? type)
          && builtins.elem (opt.type.name or "") ["str" "int" "bool"]
          && !(opt.internal or false)
          && (opt.identity or true);
        reflectedKeys = lib.sort (a: b: a < b)
          (builtins.attrNames (lib.filterAttrs isPrimitive options));
        identityKeys =
          if explicitKeys != [] then lib.sort (a: b: a < b) explicitKeys
          else reflectedKeys;
        encode = k: "${k}=${toString config.${k}}";
      in
      builtins.hashString "sha256"
        "${kind}|${lib.concatMapStringsSep "|" encode identityKeys}";
  };
};

How it's consumed

id_hash enables safe entity comparison without deep structural ==:

# Compare entities:
builtins.filter (h: h.id_hash != host.id_hash) allHosts

# Policy targeting:
if builtins.elem ctx.host.id_hash targetHashes then fire else []

Kind prefix prevents cross-kind collisions — a host named "foo" and a user named "foo" hash differently.

Strict Mode

mkStrictModule produces a freeform type that throws on undeclared keys with actionable error messages:

mkStrictModule = kind: { lib, ... }: {
  _module.freeformType = lib.mkOptionType {
    name = "strict";
    typeMerge = _outer: {
      merge = path: decls:
        let
          decl = lib.pipe decls [
            lib.head
            (lib.getAttr "value")
            lib.attrsToList
            lib.head
          ];
        in
        throw ''
          STRICT MODE: "${decl.name}" is not declared on ${kind}.
          Fix: schema.${kind}.options.${decl.name} = lib.mkOption { ... };
        '';
    };
  };
};

Injected at low priority when strict = true on mkSchema. Per-kind opt-out:

schema.host._module.freeformType = lib.types.attrsOf lib.types.anything;

Global opt-out:

mkSchema { strict = false; }

Methods Module

Generated by the entry type from extracted methods sidecar:

methodsModule = allMethods: { config, ... }: {
  options = lib.mapAttrs (name: m: lib.mkOption {
    inherit (m) description type;
    readOnly = true;
  }) allMethods;

  config = lib.mapAttrs (name: m:
    let
      args = builtins.functionArgs m.fn;
      resolved = lib.mapAttrs (n: _: config.${n}) args;
    in m.fn resolved
  ) allMethods;
};

The builtins.functionArgs reflection resolves config values matching the function's argument names. Users write the function; the library wires the config.

Kind Introspection

_meta is a declared option on the schema submodule (following _module convention — metadata about the structure, not part of it):

options._meta = lib.mkOption {
  readOnly = true;
  internal = true;
  type = lib.types.submodule {
    options.kindNames = lib.mkOption {
      type = lib.types.listOf lib.types.str;
      description = "All kind names in the schema";
    };
    options.kindMeta = lib.mkOption {
      type = lib.types.functionTo lib.types.raw;
      description = "Per-kind introspection: options, types, identity keys";
    };
  };
};

kindNames — builtins.attrNames filtered to exclude _-prefixed keys.

kindMeta — evaluates a throwaway instance via lib.evalModules to reflect on options:

kindMeta = kind:
  let dummy = lib.evalModules { modules = [ schema.${kind} ]; };
  in {
    optionNames = builtins.attrNames dummy.options;
    options = dummy.options;
    hasIdentity = dummy.options ? id_hash;
    identityKeys = dummy.config._identity.keys;
  };

Lazy — evalModules only fires when accessed. Sees schema-level options only, not den-specific extraModules additions.

Documentation Generation

renderDocs = schema:
  let
    kinds = schema._meta.kindNames;
    renderKind = kind:
      let meta = schema._meta.kindMeta kind;
      in ''
        ## ${kind}

        | Option | Type | Default | Description |
        |--------|------|---------|-------------|
        ${lib.concatMapStringsSep "\n" (renderOption meta.options) meta.optionNames}
      '';
    renderOption = options: name:
      let
        opt = options.${name};
        defaultStr =
          if opt ? defaultText then
            if builtins.isAttrs opt.defaultText then opt.defaultText.text or "—"
            else toString opt.defaultText
          else "—";
      in "| ${name} | ${opt.type.name or "?"} | ${defaultStr} | ${opt.description or ""} |";
  in
  lib.concatMapStringsSep "\n\n" renderKind kinds;

Simple markdown tables. Consumers needing richer formatting use _meta.kindMeta directly.

Kind Relationships

Relationships between kinds are structural, not declared. They emerge from how registries nest:

options.hosts = mkInstanceRegistry schema "host" {
  extraModules = [({ config, ... }: {
    options.users = mkInstanceRegistry schema "user" {
      extraModules = [
        ({ ... }: { config._module.args.host = config; })
      ];
    };
  })];
};

Host contains users because the host instance type contains a user registry. No _schema.parent / _schema.children declarations needed.

If introspection needs to answer "what kinds nest under host?" it can reflect on the registry structure.

This matches how den works today — hostType contains options.users as a nested attrsOf userType.

File Layout

den-schema/
  flake.nix                   — nixpkgs only, exposes lib + flakeModules
  nix/lib/
    default.nix               — public API surface (~40 lines)
    entry-type.nix            — schemaEntryType (~60 lines)
    identity.nix              — id_hash, _identity.keys (~45 lines)
    strict.nix                — mkStrictModule (~20 lines)
    ref-type.nix              — mkRefType (~15 lines)
    methods.nix               — schemaFn, methodsModule (~25 lines)
    introspect.nix            — _meta kindNames/kindMeta (~30 lines)
    docs.nix                  — renderDocs (~25 lines)
  nix/flakeModule.nix         — flake-parts integration (~10 lines)
  templates/
    ci/
      flake.nix               — nix-unit + den-schema
      justfile
      tests/                  — per-phase test files
    demo/
      flake.nix               — standalone usage examples

Estimated total library code: ~270 lines across 8 files.

Flake Structure

# flake.nix
{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

  outputs = { nixpkgs, ... }:
    let
      lib = nixpkgs.lib;
      schemaLib = import ./nix/lib { inherit lib; };
    in {
      inherit (schemaLib) lib;
      flakeModules.default = ./nix/flakeModule.nix;
    };
}

# nix/flakeModule.nix
{ lib, ... }:
let schemaLib = import ./lib { inherit lib; };
in {
  options.schema = schemaLib.mkSchema {};
}

Consumers choose their integration style:

• flake-parts: imports = [ den-schema.flakeModules.default ]; → config.schema.host = { ... };
• programmatic: den-schema.lib.mkSchema { strict = true; baseModule = ...; }

Phased Delivery

Phase 1: Core + Default Propagation (B+H)

Library code:

• entry-type.nix — schemaEntryType with base module injection, sidecar extraction
• identity.nix — id_hash with three-layer precedence
• strict.nix — mkStrictModule
• default.nix — mkSchema, mkInstanceType, mkInstanceRegistry
• flakeModule.nix — flake-parts integration

Tests:

• kind-declaration — declare a kind, verify options merge from multiple modules
• multi-module-extension — extend same kind from 3 separate modules
• strict-default — undeclared key errors with fix guidance
• freeform-opt-in — override strict with freeformType, verify open keys work
• identity-hash — same entity = same hash, different = different
• identity-cross-kind — host "foo" and user "foo" differ
• identity-custom-options — custom schema options auto-included
• identity-explicit-keys — _identity.keys overrides reflection
• identity-opt-out — identity = false excludes an option
• shared-base — baseModule imported by all kinds
• instance-type — mkInstanceType produces correctly typed instances
• instance-registry — mkInstanceRegistry creates typed attrsOf
• instance-strict — instances respect kind's strict/freeform setting
• default-propagation — config.x = mkDefault val flows through to instances
• global-strict-toggle — mkSchema { strict = false; } disables strict everywhere

Demo template: Minimal fleet with host + user kinds, typed registries, strict validation, default propagation.

Phase 2: Relationships + References (A+D)

Library code:

• ref-type.nix — mkRefType

Tests:

• nesting-structure — nested mkInstanceRegistry establishes parent-child
• ref-type-valid — valid reference resolves to instance
• ref-type-invalid — invalid reference errors with message
• ref-type-composition — refs work across composed schemas

Demo template addition: Service kind referencing hosts via mkRefType.

Phase 3: Introspection + Docs (C+F)

Library code:

• introspect.nix — _meta option
• docs.nix — renderDocs

Tests:

• introspect-kind-names — _meta.kindNames lists all kinds
• introspect-kind-meta — _meta.kindMeta returns options, types, identity keys
• introspect-methods — methods flagged distinctly from values
• docs-render — renderDocs produces expected markdown

Demo template addition: nix run .#schema-docs package.

Phase 4: Methods (E)

Library code:

• methods.nix — schemaFn, methodsModule generation

Tests:

• method-declaration — schemaFn creates working method on instance
• method-args-resolution — functionArgs resolved from config
• method-composition — multiple modules add methods to same kind
• method-introspection — methods appear in kindMeta

Demo template addition: Methods on host kind (secretPath, description).

After Phase 4: Den Integration

• Den adds den-schema as flake input
• modules/options.nix replaces 160-line schemaEntryType with mkSchema
• Entity types use mkInstanceType + mkInstanceRegistry
• Pipeline concerns (resolved, mainModule, includes/excludes, collisionPolicy) become schema extension modules via extraModules
• nix/strict.nix removed — strict-by-default handles it
• nix/lib/schema-util.nix — schemaEntityKinds derived from _meta.kindNames or stays as-is (pipeline concern)
• Den's test suite validates no regressions

What Moves From Den

Current location	Moves to	Notes
modules/options.nix schemaEntryType (~160 lines)	entry-type.nix (~60 lines)	Drops pipeline concerns
modules/options.nix schemaOption (~8 lines)	default.nix (mkSchema)
nix/lib/strict.nix (~30 lines)	strict.nix (~20 lines)	Error message references schema lib
modules/options.nix id_hash in resolvedCtx (~35 lines)	identity.nix (~45 lines)	Adds _identity.keys + identity=false

What stays in den as schema extensions

# Den adds pipeline concerns via extraModules on mkInstanceType:
extraModules = [({ config, options, ... }: {
  options.resolved = mkOption { ... };      # pipeline computes this
  options.mainModule = mkOption { ... };    # pipeline computes this
  options.hasAspect = mkOption { ... };     # aspect library computes this
  options.aspect = mkOption { ... };        # aspect registry lookup
  options.collisionPolicy = mkOption { ... };
  options.instantiate = mkOption { ... };
  options.intoAttr = mkOption { ... };
})];

What the schema library eliminates from den

• schemaEntryType's custom merge with includes/excludes/isEntity extraction — pipeline adds these via its own extension
• The resolvedCtx module interleaving identity, pipeline, and typing concerns
• The functor wrapping complexity mixing schema and pipeline
• nix/strict.nix — strict-by-default replaces per-kind opt-in

Review Criteria

• Stress test against den's internal API: Verify that mkSchema + mkInstanceType + mkInstanceRegistry can express everything den's current schemaEntryType + hostsOption + homesOption do, including den's two-level system-keyed nesting, entity-derived bindings in _module.args, and the den.schema.conf shared base pattern.
• Pipeline concern separation: Verify no pipeline concepts leak into the schema library — resolved, mainModule, includes, excludes, isEntity, collisionPolicy must all be expressible via extraModules without schema library changes.
• Module system compatibility: Verify schemaEntryType merge plays correctly with the NixOS module system's deferredModule semantics, fixed-point evaluation, and _module.* conventions.
• Composability: Verify schemas from multiple flake inputs merge correctly — kind extensions, identity keys, methods all compose without coordination.

Open Questions

1. identity = false mechanism. The identity attribute on mkOption is custom — it's not a standard NixOS module system option attribute. Need to verify it survives the module system's option processing pipeline (specifically mergeOptionDecls in lib/modules.nix). If custom attributes are stripped, fall back to a wrapper: mkIdentityOpt false (mkOption { ... }) that sets an internal marker. Action: Phase 1 should include an early spike test against mergeOptionDecls before building the full identity system on this assumption. If stripped, switch to the wrapper approach before coding the rest of identity.

2. Methods functionArgs edge cases. If a method's function uses ... (e.g., { name, ... }: ...), builtins.functionArgs returns only the named args, which is correct. But if a method arg name doesn't match any config key, the config.${n} lookup will fail at eval time. The methods module should validate arg names against available options and produce a clear error (e.g., "method 'secretPath' references config key 'secretRoot' which is not declared on kind 'host'"). Action: Phase 4 test suite must include a test case for this failure mode.

3. _meta.kindMeta throwaway evaluation cost. lib.evalModules on a kind module is lazy but not free. If multiple consumers access kindMeta for the same kind, is the evaluation cached by Nix's thunk memoization, or does each access re-evaluate? If re-evaluated, consider memoizing in the _meta computation.

4. mkRefType and option apply ordering. The ref type resolves in merge, which runs before apply. If the instances attrset is itself lazily constructed (common in module system), circular dependencies between ref resolution and instance construction could arise. Need test coverage for this case.