Companion to: HOAG Pipeline Architecture
den-schema (github:denful/den-schema) provides the entity data model — kinds, instances, identity hashing, validation, cross-references. This document specifies what den-schema provides to the HOAG pipeline, what gaps exist, and how the integration bridge works.
mkSchemaOption defines kinds. mkInstanceRegistry creates typed instance registries. Every instance gets id_hash via mkIdentityModule — a stable SHA-256 of "${kind}|key1=val1|..." over sorted primitive fields. The HOAG pipeline uses id_hash for entity comparison (not as vertex IDs — those use "${kind}:${name}" for readability).
mkInstanceRegistry takes extraModules which can contain nested mkInstanceRegistry calls. This establishes parent-child topology structurally:
options.hosts = schemaLib.mkInstanceRegistry schema "host" {
extraModules = [({ config, ... }:
let hostConfig = config;
in {
options.users = schemaLib.mkInstanceRegistry schema "user" {
extraModules = [
({ ... }: { config._module.args.host = hostConfig; })
];
};
}
)];
};hosts.igloo.users.tux nests users inside hosts. The _module.args.host = hostConfig injection gives children access to their parent's config.
mkRefType instances creates a coercion type: string key in → resolved instance out. Used for cross-entity relationships:
options.upstream = lib.mkOption {
type = schemaLib.mkRefType config.services;
};
# config.services.gateway.upstream = "api";
# config.services.gateway.upstream.port → 8080Custom sidecars with inferred merge strategies (++ for lists, // for attrsets). Built-in sidecars: methods and validators. Schema-level computed fields derive from extracted sidecars.
validate → derive → apply on mkInstanceRegistry:
mkValidator name pred message— tagged predicates (name as identity, pred as function, message for errors)validateInstancesreturns Either:{ right = instances; }or{ left = [ { name; validator; message; } ]; }derive/deriveEitherfor post-validation enrichment
den-schema's id_hash is the most mature implementation of Palmer et al.'s (2024) intensional function pattern: a mechanism for making functions and data inspectable, comparable, and hashable by their definition-site identity and captured values. This pattern appears independently across the entire stack:
| Component | Identify (program point) | Inspect (closure) | Equality |
|---|---|---|---|
| den-schema entities | kind prefix |
primitive field values | id_hash (SHA-256) |
| flake-aspects | aspect-chain ++ [name] |
meta, __args |
identity.key |
| scope-engine nodes | node ID string | decls |
string equality |
| Den policies | policy.name |
functionArgs of fn |
name + context hash |
| Den validators | validator.name |
pred, message |
name equality |
| Den constraints | type + identity prefix |
from, replacement |
type + target |
Six independent implementations of the same pattern. den-schema should own the general form, since it already solved the hard parts (key selection, primitive filtering, hash computation). Downstream libraries consume den-schema's identity exports.
The core of Palmer's pattern: a name (program point) plus serializable fields (closure contents) → a stable hash for comparison.
# Proposed addition to den-schema's identity.nix
mkIdentity = { name, fields ? {} }:
"${name}:${builtins.hashString "sha256" (builtins.toJSON fields)}";This is a simplification of what mkIdentityModule already does — strip the NixOS module system wiring, keep the "${prefix}:${hash(fields)}" core.
Palmer's full pattern: a function that is callable AND inspectable AND comparable.
mkIntensional = { name, closure ? {}, fn }:
let identity = mkIdentity { inherit name; fields = closure; };
in {
inherit name fn closure identity;
__functor = self: self.fn;
# Palmer's three eliminators:
# identify = self.name (program point — where it was defined)
# inspect = self.closure (captured values — what it closes over)
# apply = self.fn or self (the function itself)
};flake-aspects — aspect identity:
# Aspect identity uses mkIdentity with the definition-site path
aspectIdentity = aspect: den-schema.identity.mkIdentity {
name = lib.concatStringsSep "/" (aspect.meta.aspect-chain ++ [aspect.name]);
};
# Parametric wrappers are intensional functions
parametricWrapper = den-schema.identity.mkIntensional {
name = nameFromLoc;
closure = builtins.functionArgs fn;
inherit fn;
};scope-engine — attribute dispatch already uses string names as program points (Palmer's identify). No code change needed — document the correspondence.
Den policies — convergence dedup uses Palmer's Search monad pattern:
# Policy dedup in convergence: same name + same context = skip
policyKey = p: ctx: den-schema.identity.mkIdentity {
name = p.name;
fields = ctx;
};Den validators — already intensional (mkValidator name pred message). Can optionally wrap with mkIntensional for the hash-based identity field.
The identity exports are self-contained — no dependency on schema, instance, validation, or the module system:
# Import just the identity module, not the full den-schema:
identity = import den-schema/nix/lib/identity.nix { inherit lib; };
identity.mkIdentity { name = "my-thing"; fields = { x = 1; }; }
# → "my-thing:abc123..."This gives flake-aspects and scope-engine a lightweight dependency on den-schema's identity module (~30 lines), not on the full library.
Palmer's Theorem 5.12 (closure consistency) proves: two functions with the same program point and equivalent closures will produce equivalent results. This validates our dedup strategy — mkIdentity produces the same hash for the same inputs, so dedup (a: a.identity) is formally sound. One proof covers all consumers because they all use the same mechanism.
| Palmer formal concept | den-schema implementation | Proven property |
|---|---|---|
Program point ℓ |
name parameter |
Unique per definition site (Lemma 4.3) |
Closure expression e' |
fields / closure parameter |
Inspectable without application |
Lazy substitution θ |
scope-engine's inherited-context accumulation |
Bisimilar to eager application (§4.2) |
| Conservative equality | mkIdentity hash comparison |
Same point + same closure → same behavior (Theorem 5.12) |
| Intensional Search monad | Policy convergence with dedup | Continuation dedup is sound for idempotent operations (§3) |
den-schema already uses the { right } / { left } protocol internally. The HOAG pipeline and den-schema's validation both benefit from a small set of Either composition utilities. These replace the Bend dependency in the demo with ~20 lines of purpose-built code:
# Proposed addition to den-schema's validate.nix or a new either.nix
# Constructors
right = value: { right = value; };
left = value: { left = value; };
# Short-circuit pipe: first left stops the chain
pipe = fns: value:
builtins.foldl' (acc: f:
if acc ? left then acc else f acc.right
) { right = value; } fns;
# Accumulate all errors (don't short-circuit)
collectErrors = fns: value:
let
results = map (f: f value) fns;
errors = builtins.filter (r: r ? left) results;
in if errors == [] then { right = value; }
else { left = map (r: r.left) errors; };
# Map over right
mapR = f: e: if e ? right then { right = f e.right; } else e;
# Chain (flatMap on Either)
chain = f: e: if e ? right then f e.right else e;Rationale: den-schema's validation is one-directional — we validate/transform data, never "put it back." Bend's bidirectional lens protocol ({ get, set }) is unused. The Either utilities are the useful subset: short-circuit composition, error accumulation, functor mapping. Tagged validators (mkValidator name pred message) already implement the intensional function pattern (Palmer et al., 2024) — name as definition-site identity, pred as function, message as inspectable metadata.
den-schema has all the data the HOAG pipeline needs but no API to extract it as algebraic graph fragments. Three specific gaps:
1. No registry topology introspection. _meta.kindNames returns kind names (["host" "user"]) but not how registries nest. Nesting is implicit in extraModules composition — opaque to introspection. There is no _meta.topology that says "user registries nest inside host registries."
2. No reference-edge metadata. mkRefType resolves eagerly at merge time — the reference relationship is consumed. There is no { from = "service:nginx"; to = "host:igloo"; } edge record surviving merge. The target instance is available but the reference relationship is lost.
3. No automatic parent pointers. _module.args.${kind} = config injects the current kind's config. Parent injection (_module.args.host = hostConfig) is manual via extraModules. Instances don't carry a _parent attribute.
The bridge lives in Den, not in den-schema. Den's wiring layer already knows the nesting structure (config.den.hosts, config.den.homes) and constructs the algebraic graph explicitly:
# Note: `homes` (standalone home-manager configs) omitted from v1.
# They can be added as root-level nodes with no parent when needed.
buildScopeGraphs = hosts:
let
engine = scope-engine;
hostNames = lib.attrNames hosts;
hostIds = map (name: "host:${name}") hostNames;
userEntries = lib.concatMap (hostName:
let host = hosts.${hostName};
in map (userName: {
vertex = "user:${userName}@${hostName}";
parent = "host:${hostName}";
}) (lib.attrNames (host.users or {}))
) hostNames;
# Parent graph: child→parent edges (entity nesting)
parentGraph = engine.overlay
(engine.vertices hostIds)
(lib.foldl' engine.overlay engine.empty
(map (u: engine.edge u.vertex u.parent) userEntries));
# Import graph: parent→child edges (for bottom-up pipe data flow)
importGraph = lib.foldl' engine.overlay engine.empty
(map (u: engine.edge u.parent u.vertex) userEntries);
# Decls: entity config per vertex ID
decls = lib.listToAttrs (
(map (name: { name = "host:${name}"; value = hosts.${name}; }) hostNames)
++ (map (u: {
name = u.vertex;
value = hosts.${lib.removePrefix "host:" u.parent}.users.${
lib.head (lib.splitString "@" (lib.removePrefix "user:" u.vertex))
} or {};
}) userEntries)
);
# Types: entity kind per vertex ID
types = lib.listToAttrs (
(map (name: { name = "host:${name}"; value = "host"; }) hostNames)
++ (map (u: { name = u.vertex; value = "user"; }) userEntries)
);
in { parent = parentGraph; import = importGraph; inherit decls types; };Design decision: hand-written bridge, not den-schema API. The bridge is ~30 lines and specific to Den's entity structure. Adding introspection to den-schema would be a larger change for a single consumer. If other projects need scope graph extraction from den-schema registries, we can add _meta.topology later.
Import edge policy: buildScopeGraphs automatically creates import edges from parent to children (host:igloo → user:tux@igloo) for bottom-up pipe data flow. This implements Open Question #3 option (a) from the HOAG spec. Additional import edges from explicit pipe routing declarations or policy effects are added by the synthesize function.
No breaking changes. The existing API is sufficient. Four enhancements, two near-term:
| Enhancement | What | When | Size |
|---|---|---|---|
| Identity primitives | mkIdentity, mkIntensional in identity.nix — Palmer's intensional function pattern |
Before HOAG integration — foundational for all downstream | ~18 lines |
| Either utilities | right, left, pipe, collectErrors, mapR, chain in either.nix |
Before HOAG integration (replaces Bend in demo) | ~20 lines |
_meta.topology |
Optional introspection: { kind, parentKind, childKinds } per registry |
Deferred — if other consumers need scope graph extraction | |
mkRefType edge metadata |
Optional sidecar recording { field, targetKind, targetKey } per reference |
Deferred — if import edges need to be derived from refs automatically |
The identity primitives and Either utilities are near-term additions (~38 lines total). Identity is foundational — flake-aspects and scope-engine both consume it. Either replaces the Bend dependency in the demo.