Skip to content

Instantly share code, notes, and snippets.

@sini

sini/2026-05-27-clan-6655-service-coordination.md

Last active May 27, 2026 22:30
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save sini/e425d8accaceaeb1c9b5e2b61b2bfee4 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save sini/e425d8accaceaeb1c9b5e2b61b2bfee4 to your computer and use it in GitHub Desktop.
Download ZIP
Service coordination for Clan — addressing clan-core#6655 with scope graphs, composable blocks, and fleet visualization
Raw
2026-05-27-clan-6655-service-coordination.md

Service Coordination, Dependency Graphs, and Composable Blocks — Addressing clan-core#6655

Date: 2026-05-27 Context: clan-core#6655 — "The future and scope of Clan relative to other Nix-based self-hosting solutions"

Related: Den × Clan Collaboration Proposal

Related: Gen Ecosystem Integration Opportunities for Clan-core

Related: [den/gen-internal] Den Pragmatic Fleet Features — Learning from Clan, Bridging Strategy

The Problem (As Stated by Clan's Community)

Issue #6655 identifies a clear gap: Clan services can't discover, depend on, or coordinate with each other. The concrete symptoms reported:

  1. Monitoring service hardcodes nginx on port 80. No way to say "I need a reverse proxy" without hardcoding which one or what port. (TakodaS)

  2. SSL depends on PKI, Grafana depends on DNS, but nothing enforces or wires these. curl local-machine.clan/grafana fails because DNS isn't configured — and the monitoring service has no way to declare that it needs DNS. (TakodaS)

  3. Backup isn't automatic. Borgbackup exists as a service, but other services can't declare "I have state, back me up." There's no mechanism for one service to advertise state to another. (TakodaS)

  4. Port collisions. Qubasa acknowledges needing collision-free port allocation but says it's "not high priority right now." (Qubasa)

  5. Reverse proxy pluggability abandoned. The team concluded that abstracting across nginx/caddy/traefik is too hard, so nginx is hardcoded. (Qubasa)

  6. No service blocks. TakodaS repeatedly asks for something like selfhostblocks' composable blocks — auth, reverse proxy, backup as plugins that layer onto services. hsjobeki confirms exports are being built for this. (TakodaS, hsjobeki)

  7. sjdevries explicitly mentions den as having a desirable VM testing workflow and iteration speed.

The thread also includes ibizaman (selfhostblocks author) discussing convergence, and hsjobeki confirming that the exports system under development targets exactly these problems.

How This Maps to Gen/Den Capabilities

Service Dependencies → Scope Graph Import Edges

The core missing piece is declared dependencies between services that are resolved structurally.

Clan services currently discover each other through roles.*.machines — manual machine lists inside perInstance. There's no way for monitoring to say "I need DNS" and have the system wire it up.

In a scope graph, this is an import edge:

service:monitoring ──I──→ service:dns       (monitoring imports dns)
service:monitoring ──I──→ service:pki       (monitoring imports pki for SSL)
service:grafana    ──I──→ service:monitoring (grafana imports monitoring data)

gen-scope's resolution calculus follows these edges with specificity ordering (D < I < P) and well-formedness (P*.I*). If monitoring declares it needs DNS but DNS isn't deployed, resolution fails with a clear error: "service:monitoring imports service:dns, but no scope matching service:dns exists in the graph."

This is what hsjobeki describes as "services that can discover and coordinate with each other across boundaries via contracts" — the scope graph is the structural backbone for those contracts.

Service Blocks → Den Aspects with Class Keys

TakodaS asks for composable blocks: reverse proxy, auth, backup as plugins that layer onto services. This is exactly what den aspects are.

# A reverse-proxy "block" as a den aspect
den.aspects.reverse-proxy = {
  meta.provides = [ "http-proxy" "https-proxy" ];

  # Settings: which upstream, what domain, SSL?
  settings = { lib, ... }: {
    options.upstream = lib.mkOption { type = lib.types.str; };
    options.domain = lib.mkOption { type = lib.types.str; };
    options.ssl = lib.mkOption { type = lib.types.bool; default = true; };
  };

  # The aspect emits nginx config — but the CLASS KEY is what matters.
  # If someone registers a "caddy" class, they write a caddy aspect instead.
  # The service doesn't care which reverse proxy — it just includes the block.
  nixos = { settings, ... }: {
    services.nginx.virtualHosts.${settings.domain} = {
      forceSSL = settings.ssl;
      locations."/".proxyPass = settings.upstream;
    };
  };
};

# Auth block
den.aspects.authelia-sso = {
  meta.requires = [ "http-proxy" ];  # needs a reverse proxy
  meta.provides = [ "sso-auth" ];
  nixos = { ... }: { /* authelia config */ };
};

# Backup block
den.aspects.auto-backup = {
  meta.requires = [ ];
  meta.provides = [ "backup" ];
  # Discovers all services that declare state via pipe
  pipe.state-discovery = {
    gather = sel.attrs { provides = "stateful"; };
    channel = "backup-targets";
  };
  nixos = { pipes, ... }: {
    services.borgbackup.jobs = lib.genAttrs pipes.backup-targets (svc: {
      paths = svc.statePaths;
    });
  };
};

The key insight from Qubasa's reverse proxy discussion: they concluded that abstracting nginx/caddy/traefik in one module is impossible. They're right — but that's a false dilemma. The solution isn't one module that generates all proxy configs. It's the class system: register nginx as a class, register caddy as a class, and let the aspect emit into whichever class the user has registered. The service says "I need a reverse proxy." The class routing determines which reverse proxy it gets.

# User A: uses nginx (default)
den.classes.reverse-proxy = { backend = "nginx"; };

# User B: uses caddy
den.classes.reverse-proxy = { backend = "caddy"; };

# The grafana service doesn't care — it just includes the reverse-proxy block
den.aspects.grafana.includes = [ den.aspects.reverse-proxy ];

Port Collision → gen-derive Constraint Rules

Qubasa mentions needing "python code for collision-free ports." In gen-derive, this is a constraint rule with a negative application condition:

# Conceptual — rule that fires when two services claim the same port
gen-derive.mkRule {
  condition = ctx: ctx ? port && ctx ? service;
  nac = ctx:  # don't fire if no collision
    let users = servicesUsingPort ctx.port;
    in builtins.length users <= 1;
  produce = ctx:
    let users = servicesUsingPort ctx.port;
    in actions.error "Port ${toString ctx.port} claimed by ${concatStringsSep ", " users}";
}

Rules compose, run at Nix evaluation time (not Python), and catch violations that emerge after tag/membership resolution.

Backup State Discovery → Pipes

TakodaS wants borgbackup to automatically discover and back up all stateful services. In den, this is a pipe:

# Any service can advertise state
den.aspects.matrix-synapse.pipe.state = {
  channel = "backup-targets";
  data = { paths = [ "/var/lib/matrix-synapse" ]; };
};

den.aspects.nextcloud.pipe.state = {
  channel = "backup-targets";
  data = { paths = [ "/var/lib/nextcloud" ]; };
};

# Backup service gathers from the pipe
den.aspects.borgbackup.nixos = { pipes, ... }: {
  services.borgbackup.jobs.auto = {
    paths = lib.concatMap (s: s.paths) (pipes.backup-targets or []);
  };
};

Pipe data flows through the scope graph — only services on the same host (or explicitly imported) contribute. Adding a new stateful service means adding a pipe.state declaration; borgbackup picks it up automatically.

DNS/SSL Dependency Chain → Resolution Ordering

TakodaS's grafana problem: monitoring needs DNS, DNS needs data-mesher, none of this is declared or enforced. In the scope graph:

service:grafana
  ├──I──→ service:monitoring
  │         ├──I──→ service:dns
  │         │         └──I──→ service:data-mesher
  │         └──I──→ service:pki
  └──I──→ service:reverse-proxy

gen-scope's resolution follows import chains. If data-mesher isn't deployed, resolving grafana's dependencies fails at service:dns → service:data-mesher with: "service:dns imports service:data-mesher, which is not in scope."

gen-graph's topoSort orders the dependency chain for deployment sequencing. den-diagram renders it as a Mermaid graph so you can see the full chain before deploying.

Fleet Visualization → den-diagram

sjdevries explicitly mentions den's VM workflow. den-diagram adds fleet topology visualization on top:

nix run .#diagrams        # renders service topology as Mermaid/C4/DOT
nix run .#diagrams.diff   # what changed since last deployment

For the monitoring example, the diagram would show:

graph LR
  subgraph host:server1
    grafana -->|needs| monitoring
    monitoring -->|needs| dns
    monitoring -->|needs| pki
    dns -->|needs| data-mesher
    grafana -->|needs| reverse-proxy
  end
Loading

Missing dependencies visible at a glance. TakodaS's "curl fails because DNS isn't wired" problem would show up as a missing edge in the diagram before deploying.

Kubernetes → Multi-Target Class System

TakodaS asks about kubernetes for production uptime/redundancy. Qubasa says "managing kubernetes is quite a task." Den's class system makes this a class registration, not a separate architecture:

# Same service, additional output target
den.aspects.matrix-synapse = {
  nixos = { ... }: { /* NixOS config for bare metal */ };
  kubernetes = { ... }: {
    # Helm chart or k8s manifests for the same service
    deployment.matrix-synapse.spec.replicas = 2;
  };
};

The aspect graph resolves once; class routing materializes into both NixOS configs and k8s manifests. This is already demonstrated in den's terranix-demo template (terraform config from the same aspect graph).

Convergence with selfhostblocks

ibizaman is already in the #6655 thread discussing how selfhostblocks' contracts relate to Clan's exports. The key convergence points:

selfhostblocks concept Clan equivalent (planned) Den/gen equivalent
Contracts (RFC 189) Exports system gen-scope import edges + gen-schema refinements
Blocks (reverse proxy, auth, backup) Not yet designed Aspects with class keys
Secrets provider abstraction vars backends gen-vars with gen-derive backend selection
NixOS VM tests Eval tests + VM tests nix-unit + den's VM template

selfhostblocks' contracts RFC and Clan's exports system are solving the same problem gen-scope solves. A shared resolution layer (gen-scope) could underpin all three projects' contract/export systems, avoiding three independent implementations of name resolution, dependency ordering, and conflict detection.

What We're Offering

Immediate (no commitment from Clan)

  1. den-diagram adapter (~100-200 lines) — fleet visualization for Clan inventories. Service topology, dependency chains, export flow. Posted as a standalone flake Clan can evaluate without adopting anything.

  2. Dependency graph POC — model the monitoring→dns→data-mesher→pki chain from #6655 as a gen-scope graph. Demonstrate that resolution catches the "missing DNS" error structurally. Benchmark against the current exports system.

  3. Port collision validator — gen-derive rules that detect port conflicts across services at eval time. Replaces the planned Python solution with pure Nix.

If there's interest

  1. Service block aspects — reverse-proxy, auth, backup as den aspects that compose via the class system. Demonstrates pluggable reverse proxy without the abstraction problem Qubasa described.

  2. Bidirectional bridge — Clan services usable as den aspects, den aspects usable as Clan services. Both communities access each other's work.

  3. Pipe-based state discovery — borgbackup automatically gathers state paths from all services that declare them. The "automated backups" feature #6655 asks for.

Longer-term

  1. Shared resolution layer — gen-scope as the common dependency resolution engine for Clan exports, selfhostblocks contracts, and den pipes. One well-tested implementation instead of three.

Settings Cascade: Typed, Hierarchical, Auditable

One of the problems surfaced in #6655 is configuration layering — services need defaults, environments need overrides, and individual hosts need per-machine tuning. Den's settings pattern handles this with a three-layer cascade, already running in production on 8 hosts across 3 environments.

How It Works

1. Aspects declare typed settings:

# modules/den/aspects/disk/impermanence.nix
den.aspects.disk.impermanence = {
  settings = {
    wipeRootOnBoot = lib.mkOption {
      type = lib.types.bool;
      default = true;
    };
    wipeHomeOnBoot = lib.mkOption {
      type = lib.types.bool;
      default = false;
    };
  };

  nixos = { host, ... }:
    let
      wipeRoot = host.settings.disk.impermanence.wipeRootOnBoot;
      wipeHome = host.settings.disk.impermanence.wipeHomeOnBoot;
    in
    { /* NixOS config using resolved settings */ };
};

2. Hosts override settings by path:

# modules/den/hosts/axon-01.nix
den.hosts.x86_64-linux.axon-01 = {
  environment = "prod";

  settings = {
    services.k3s.clusterName = "axon";
    services.bgp.localAsn = 65001;
    disk.impermanence = {
      wipeRootOnBoot = true;
      wipeHomeOnBoot = true;   # override the aspect default (false)
    };
  };
};

3. Scope-engine resolves the cascade: aspect defaults → environment defaults → host overrides. The resolution is a DAG traversal with import edges for delegation (e.g., a dev environment can import prod settings as a base).

Aspects read their resolved settings via host.settings.<category>.<aspect>.<key> — the path mirrors the aspect tree. The host schema auto-discovers all aspect settings declarations and generates a nested submodule type, so typos and type errors are caught at eval time.

What This Gives Clan's Service Block Problem

The #6655 thread discusses how services like monitoring hardcode assumptions (port 80, nginx, specific SSL). With den's settings pattern:

  • The monitoring aspect declares settings.port, settings.ssl, settings.reverseProxy as typed options with sensible defaults
  • The prod environment overrides ssl = true for all hosts
  • Individual hosts can override port = 8443 where needed
  • The aspect code reads the resolved value — it never hardcodes

This is the same layering Clan's role interface → role settings → machine settings provides, but extended with environment-level defaults and scope-engine cascade resolution with provenance tracking (settingSources tells you which layer provided each value).

Next Evolution: Settings as Scope Graph Collection

The nix-config pattern above is shipped and working. The next iteration pushes settings deeper into the gen library stack, making the scope graph itself the resolution engine:

Aspects declare settings schema with per-field merge strategies:

config.aspects.nginx.settings = {
  port = { type = types.int; default = 80; };           # "replace" — host wins
  workers = { type = types.positive; default = 4; };     # "replace"
  allowedOrigins = {
    type = types.listOf types.str;
    default = [];
    merge = "append";    # accumulates across layers, not replaces
  };
  locations = {
    type = types.attrsOf (types.submodule { ... });
    default = {};
    merge = "recursive"; # nested attrset merge per-key
  };
};

The scope graph composes via D > I > P traversal (Neron 2015):

D:  { port = 8080; workers = 16; }        ← host:web1 (most specific)
I:  { allowedOrigins = [ "cdn.example" ]; } ← imported environment
P:  { port = 443; allowedOrigins = [ "*.example.com" ]; } ← env:prod
base: { port = 80; workers = 4; }          ← schema defaults

Result: {
  port = 8080;                              # replace: host wins
  workers = 16;                             # replace: host wins
  allowedOrigins = [ "cdn.example" "*.example.com" ];  # append: both layers
}

No mkDefault, no NixOS module priorities — the scope graph's structural ordering replaces priority annotations entirely. Validation is lazy (Chitil 2012): fields validate on access, not assignment, with blame identifying both the aspect (schema source) and scope level (value source).

Per-field merge strategies directly address #6655's problems:

  • "replace" for ports, toggles, server names — host override wins cleanly
  • "append" for IP allowlists, trusted origins, package lists — layers accumulate instead of clobbering
  • "recursive" for nginx location blocks, firewall rules — nested config merges per-key across layers

This is the missing piece between Clan's three-layer merge (which always does full-attrset replace) and what the monitoring/grafana/auth services actually need (mixed strategies per field).

Full Implementation (Current Shipped Pattern)

The complete settings pattern in a production nix-config:

Design spec for scope-graph-native settings: Gen Type Unification — covers pluggable entry types, aspects on gen-schema, the "neron" traverse mode, and per-field merge strategies. ~360 lines of library changes across gen-schema, gen-aspects, gen-algebra, and gen-scope.

Live Demos: What This Looks Like in Practice

The best way to understand what den-diagram produces is to see it. These are real, materialized outputs from production flakes — not mockups.

Interactive Fleet Viewer

The den-diagram IR viewer renders fleet topology interactively from JSON IR. Load either of these real fleet captures:

Paste either URL into the viewer to explore the scope hierarchy, click nodes to inspect attributes, and trace data flow edges between hosts.

Static Rendered Diagrams

den-diagram also renders to Mermaid markdown that GitHub displays natively. These are committed, versioned fleet documentation generated from the same aspect graph that produces NixOS configurations:

Fleet-level views:

  • fleet-demo diagrams — the demo template's rendered output, showing the full pipeline from aspects to Mermaid
  • @sini's fleet overview — a production fleet rendered as Mermaid, showing all hosts, their aspects, and cross-host data flow

Host-level views:

  • cortex (workstation) — a workstation host showing all aspects, user-level home-manager config, and devshell contributions
  • uplink (edge server) — an edge server showing NixOS services and networking aspects

Note: host-level views currently show the aspect/class graph per host. Pipe flow visualization (cross-host data edges, producer/consumer annotations) is available in the fleet-level view and the interactive IR viewer — host-level pipe rendering is on the roadmap.

What Clan Would Get

The same rendering stack applied to Clan's inventory would produce:

  • Service topology — which machines participate in which roles, with cross-role data flow edges (borgbackup server↔client, zerotier controller→peer)
  • Dependency chains — the monitoring→dns→data-mesher→pki chain from #6655, visible at a glance, with missing edges highlighted
  • Export flow — Sankey diagrams showing which services consume which exports from which other services
  • Per-machine projection — "what is m1 responsible for?" across all services
  • Fleet diff — what changed between two inventory snapshots

The adapter from Clan inventory to den-diagram IR is ~100-200 lines. The rendering stack (Mermaid, C4, DOT, PlantUML, Sankey, treemap, JSON) works unchanged — it operates on the IR, not on den internals.

The Opportunity

Issue #6655 has 12 comments over 4 months from users, maintainers, and the selfhostblocks author. The community consensus is clear:

  • Service coordination is the #1 priority
  • Composable service blocks are needed
  • Current export/dependency story is insufficient
  • Multiple projects (Clan, selfhostblocks, den) are solving overlapping problems

The gen libraries provide a formal resolution layer that could underpin all three projects' coordination systems. den's aspect model demonstrates composable service blocks that solve the reverse proxy / auth / backup problems #6655 describes. den-diagram provides the visualization Clan doesn't have.

The worst case is that our POCs don't convince anyone and we've built useful tools for den anyway. The best case is shared infrastructure that benefits all three communities.

Companion documents:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment