LAYERED KNOWLEDGE SYSTEM — GIST v2

Karpathy-style compressed spec. Read once, execute against. Deterministic by default; testable by construction.

TL;DR

Tiered knowledge graph where JSON is truth, wiki is a pure function of JSON, rollups splice mechanically, retrieval traverses by rule, and an Opus-led audit keeps coherence as the graph grows. Every non-deterministic surface (LLM calls) is bounded, logged, and replayable. Tier slots exist from day one; deployments activate slots, never add them. Solo and enterprise run the same architecture at different activation levels.

v2 CHANGES (vs v1)

Render determinism separated from content determinism; generation_method tracked per field.
projection_stale removed; staleness derived from projection_hash vs children's hashes. No bit to drift.
Edge traversal is rule-based via a versioned edge-semantics registry. No LLM at retrieval time.
Entity resolution is an explicit deterministic cascade: normalize → vector → LLM-verify, with named thresholds.
Rollup is mechanical section splicing; LLM only regenerates sections whose keyed inputs changed.
Shadow-mode graduation is a measurable gate with explicit metrics, not a judgement call.
Approval queue is a first-class node kind with an explicit state machine.
Model routing is capability-class based; concrete model names live in a versioned routing table.
Added: failure modes, cost envelope, storage migration protocol, test contracts per tier.

CORE PRINCIPLES

JSON is source of truth. Wiki is a pure function of JSON.
Field-level provenance. Every body section carries generation_method ∈ {template, llm, rollup_splice, human, adapter} and generator_id. Tests assert which methods are permitted in which slots.
Five tier slots in schema from day one. Deployments flip activation flags; they never add slots.
Knowledge flows up as abstractions. Authority flows down as context. Peers get lateral visibility through typed edges with a one-hop traversal limit.
Tier promotion is a controlled mutation that preserves node identity. Never a rebuild.
Shadow mode is the default. Ingest → extract → propose → review → commit. Graduation is per-wiki, gated by measurable thresholds.
Route models by task character, not size. Capability classes, not model names, are wired into code.
Determinism first. When a choice exists between a deterministic algorithm and an LLM call, the algorithm wins unless a test shows the algorithm is wrong.

THE FIVE TIERS

Tier	Name	Scope	Character	Solo	Small Team	Org	Division	Enterprise
L0	Persona	Individual	Identity, working style, relationships	1	N	N	N	N
L1	Project/Knowledge	Bounded domain	Full structured wiki, where authoring happens	few-many	many	many	many	many
L2	Team/Department	Group	70% rollup, 30% native team context	0-1 portfolio	1 portfolio	many teams	many teams	many teams
L3	Division/Executive	Functional group	Mostly strategic-native, thin rollup	inactive	inactive	1 synthesis	many divisions	many divisions
L4	Enterprise	Whole org	External context, cross-division posture	inactive	inactive	inactive	1 synthesis	many strategic axes

Tier slots always exist in the schema. Activation is a flag.

NODE ENVELOPE (universal, v2)

{
  "id": "node_<ulid>",
  "tier": "L0|L1|L2|L3|L4",
  "tier_active": true,
  "kind": "persona|project|projection|strategic|entity|approval",
  "title": "...",
  "owner": {"type": "individual|role|system", "ref": "..."},
  "acl": {
    "read": ["role:...", "individual:...", "tier:L2+"],
    "write": ["individual:..."],
    "project_to": ["role:...", "tier:L3"]
  },
  "vines": {
    "parent": "node_id | null",
    "children": ["node_id", ...],
    "lateral": [{"type": "depends-on|supersedes|references|shares-entity-with", "target": "node_id", "note": "..."}],
    "entities": ["entity_id", ...]
  },
  "provenance": {
    "created_at": "...", "updated_at": "...",
    "source": "human|agent|rollup|import|promotion",
    "source_refs": [...],
    "derived_from": ["node_id", ...],
    "promoted_from_tier": "L2"
  },
  "body": {
    "<section_key>": {
      "content": ...,
      "generation_method": "template|llm|rollup_splice|human|adapter",
      "generator_id": "local-gemma-4|flash-3.0|sonnet|human:<id>",
      "input_hash": "sha256...",
      "content_hash": "sha256...",
      "generated_at": "..."
    }
  },
  "body_hash": "sha256...",          // hash over ordered section content_hashes
  "projection_hash": "sha256...",    // hash of child body_hashes at last rollup
  "schema_version": "2.0"
}

Staleness is derived, not stored: a node is stale iff hash(sorted(children.body_hash)) ≠ projection_hash. No bit to drift.

TIER-SPECIFIC BODY SHAPES

Sections are keyed, ordered, and independently hashed. This is what makes mechanical splicing possible.

L0 persona: role, expertise, active_contexts, communication_preferences, relationships, current_focus.
L1 project: facts, decisions, open_questions, artifacts, timeline. Each item inside a section is addressable as <section_key>.<item_id>.
L2 projection: summary, key_decisions, risks, dependencies, status_signals (all rollup_splice), plus native_content (human).
L3/L4 strategic: posture, cross_cutting_themes, external_context, decision_log, native_content. Native dominates at higher tiers.
Entity: canonical_name, aliases, type, attributes, first_seen, last_referenced. Identity only; narrative lives in referencing nodes.
Approval (new kind): proposed_action, origin_finding, state (pending|approved|rejected|expired), reviewer, decided_at, rationale.

EDGE TYPES (three kinds, don't conflate)

Vertical (parent/children in vines): rollup/drilldown between tiers. One parent max, many children.
Lateral (lateral array): typed peer edges, within or across tiers. One hop max during retrieval.
Entity (entities array): references to canonical entity nodes. Rename/merge propagates by reference, never by rewrite.

EDGE SEMANTICS REGISTRY (new)

Retrieval traversal is governed by a versioned rule file, not an LLM decision.

// edge_semantics.v1.json
{
  "query_intents": {
    "causality":     {"follow": ["depends-on", "supersedes"],           "direction": "outgoing"},
    "provenance":    {"follow": ["supersedes", "references"],           "direction": "incoming"},
    "context":       {"follow": ["references", "shares-entity-with"],   "direction": "both"},
    "entity_scope":  {"follow": ["shares-entity-with"],                 "direction": "both"}
  },
  "defaults": {"max_hops": 1, "fallback_intent": "context"}
}

The retrieval conductor classifies the query into one intent (rule-based matcher first, cheap classifier fallback, final fallback to context), then the registry deterministically decides traversal. Test contract: for each intent, a fixture graph + golden expected-node-set.

WIKI PROJECTION

Pure template function over JSON. Walks sections in their declared order; renders frontmatter + summary + keyed sections + links panel (grouped: parent, children, lateral-by-type, entities).

Render determinism contract: given the same JSON at the same schema_version, bytes-identical markdown out. Golden-file tested per tier kind.

Humans never write wiki. They write through structured editors that produce JSON, OR they write prose into note fields inside structured items — those render as markdown but never drive retrieval logic.

WRITE PATH (shadow mode — numbered stages with schema'd I/O)

Each stage has a schema'd input and output. Each is a contract test.

Adapter. Raw source → NormalizedEvent{source, ingestion_id, ts, raw, extracted_text}. Per-source adapter, pure function, deterministic.
Extraction (capability: extract_small). NormalizedEvent → ExtractionProposal{candidate_facts, candidate_entities, candidate_links, target_node_hints}. No writes. LLM call with seed logged for replay.
Entity resolution. Deterministic cascade, stops at first decision:
1. Normalize. NFKC unicode → case-fold → whitespace collapse → punctuation strip. Exact match on normalized name → bind.
2. Vector. Cosine similarity against entity embeddings. >= ENT_HIGH (default 0.92) → bind. < ENT_LOW (default 0.75) → propose new entity.
3. LLM verify (capability: extract_small). Ambiguous zone [ENT_LOW, ENT_HIGH) only. Output: bind | propose_new | escalate_to_review.
Node routing (capability: route_cheap). Decides L1 placement or proposes a new node. Conservative: ambiguous → review queue.
Shadow write. Commits to shadow namespace with provenance + diff against current authoritative.
Graduation gate (see §Shadow-Mode Graduation). Per-wiki, measurable. Only graduated shadow entries become authoritative.
Rollup enqueue. On authoritative commit, parent chain's projection_hash stops matching children's product → rollup worker picks up.
Wiki re-render. Deterministic, on every authoritative JSON commit.

ROLLUP WORKER (mechanical splicing)

Capability: rollup_mid. Cadence classes by rolling-window change count (defaults: hot = ≥5 children changed in 24h, warm = ≥1 in 7d, cold = none in 7d). Classes drive queue priority; they are derived, not stored.

Algorithm:

For each projection section s, compute inputs(s) from child body sections via the projection's section-mapping spec (versioned per projection kind).
Compute new_input_hash(s) = hash(inputs(s)).
If new_input_hash(s) == prev_input_hash(s) → splice verbatim (copy content + content_hash from prior projection).
Else → call rollup_mid with ONLY that section's inputs + prior content as continuity context. Record new input_hash, content_hash, generator_id, generated_at.
Recompute body_hash and projection_hash.

Tested guarantees:

Sections with unchanged inputs are byte-identical to the previous rollup.
LLM is never asked to preserve — preservation is mechanical.
L1→L2 rollups dominate volume. L2→L3 and L3→L4 run sparser.

READ PATH

Identity → ACL context (readable node set).
Vector search scoped to readable nodes; rank; return top-k IDs + summaries.
Query intent classified; edge semantics registry decides lateral traversal. One hop max. Upward traversal only on explicit context intent.
Conductor (capability: route_cheap) selects materialization set.
Materialization returns JSON bodies. Agents work with JSON, not wiki.

SHADOW-MODE GRADUATION (measurable gate)

Per-wiki. A shadow proposal graduates when all of:

Accepted-proposal rate ≥ GRAD_ACCEPT_RATE (default 0.85) over a sliding window of GRAD_WINDOW proposals (default 50).
Zero critical audit findings against graduated content in the prior audit cycle.
Human override not vetoed during a GRAD_COOLDOWN window (default 7 days).

Thresholds live in per-wiki config. Graduation states: shadow → probation → authoritative. Downgrade to shadow on audit regression. Test contract: simulated accept/reject streams produce deterministic state transitions.

APPROVAL QUEUE (first-class)

Opus recommends; humans approve. Approvals are nodes of kind=approval.

State machine: pending → (approved | rejected | expired). Expiry defaults: 7d routine, 24h critical.

Invariants (tested):

No graph write from an audit finding without a matching approval node in state approved.
Rejection does not delete the originating finding — it records a counter-decision with rationale.
Batch approvals allowed; each grouped approval still produces one approval node per target.

MODEL ROUTING (capability classes)

Code depends on capability classes. A versioned routing table maps classes to concrete models. Swapping models is a config change, not a code change.

Capability class	Tasks
`extract_small`	Extraction, entity-resolution LLM gate
`route_cheap`	Node routing, retrieval conductor, ACL anomaly check
`synth_mid`	Cross-tier synthesis
`rollup_mid`	Rollup regeneration (section-scoped)
`audit_drift`	Projection drift audit
`audit_structural`	Vine health, schema pressure
`audit_wide_context`	Entity consistency across wikis
`audit_manager`	Audit coordination and continuity

Current routing table (routing.v1.json, April 2026 lineup):

{
  "extract_small":      "local-gemma-4",
  "route_cheap":        "gemini-flash-lite-3.1",
  "synth_mid":          "gemini-flash-3.0",
  "rollup_mid":         "gemini-flash-3.0",
  "audit_drift":        "claude-sonnet-4-6",
  "audit_structural":   "claude-opus-4-6",
  "audit_wide_context": "gemini-pro-3.1",
  "audit_manager":      "claude-opus-4-6"
}

Every capability class declares a fallback model. Health-check failure routes to fallback; if no fallback, stage pauses (never bypasses).

MANAGER/WORKER AUDIT

Cadence: weekly at corporate scale, weekly-to-fortnightly at solo.

Manager (capability: audit_manager): small persistent context (~30-50k tokens). Holds audit plan, shard assignments, prior findings summaries. Never holds node content. Plans shards, assigns workers, collates findings, detects cross-shard patterns, produces report, maintains continuity log.

Workers: scoped shard + specific task + structured output. Never prose; always findings JSON:

{
  "findings": [
    {
      "type": "projection_drift|entity_conflict|vine_anomaly|schema_pressure|acl_drift",
      "node_ref": "node_id",
      "severity": "critical|high|medium|low",
      "evidence": {...},
      "proposed_action": {...}
    }
  ],
  "meta": {"coverage": "...", "confidence": "...", "escalations": [...]}
}

Overlap rules:

Boundary nodes: each worker gets summaries (not bodies) of its shard's neighbors.
Cross-shard entity refs: each worker gets compact summaries of how its referenced entities appear elsewhere.
Strategic redundancy: ~20% double-covered, rotated weekly → full coverage over 4-6 weeks.

Deterministic shard assignment. Given the same (node_ids, rotation_week), shards partition identically. Test contract: pure-function test on the partitioner.

Replayability. Manager's plan, worker inputs, and RNG seeds are stored. Re-running an audit over the same graph snapshot produces the same findings set. LLM wording may vary, but finding existence for severity ≥ high is stability-tested against goldens.

Continuity log: each audit writes a compact summary the next audit's manager reads. Accumulates knowledge about where to look.

GROWTH STAGES (activation path)

Stage 0 — Solo, L0+L1. Personal persona, project wikis. Inactive L2/L3/L4 slots.

Stage 1 — Solo + portfolio, L0+L1+L2. Single L2 portfolio node activates. Transition triggers: cross-project queries common, >5-7 L1s, external sharing begins.

Stage 2 — Small team, multi-author L1s. L0 multiplies. ACLs acquire role semantics. Still L0+L1+L2.

Stage 3 — Multi-team org. L2 multiplies (teams). Original portfolio PROMOTES to L3 (preserving identity). L3 activates with one node.

Stage 4 — Division scale. L3 multiplies. Original cross-org node PROMOTES to L4. L4 activates with one node. Embedded DBs migrate via the migration protocol.

Stage 5 — Enterprise. L4 gains substructure. Full governance active.

PROMOTION OPERATION

A specific controlled mutation:

Node's tier field updates to new tier.
Node's provenance.promoted_from_tier records history.
Node re-parents (new parent at higher tier, or null if top).
Existing children's parent references remain pointing at this node.
Rollup logic updates (node now rolls up from tier-below instead of directly from L1s).
Audit trail entry created.

Only this operation can change a node's tier. It is not a general-purpose capability.

STORAGE MIGRATION PROTOCOL (new)

Stage 4 LanceDB→Qdrant and SQLite→Kuzu is a migration, not additive work. Applies to any index/store migration.

Dual write. New store begins receiving writes alongside old store. Writer guarantees happen-before on both stores for the same transaction. Reads still served by old store.
Backfill. Stream existing data to new store. Idempotent and re-runnable.
Consistency audit. Diff old vs new on a sample (≥1%). All diffs resolved before proceeding.
Shadow read with compare. Reads hit both stores; responses compared; old store wins; diffs logged.
Cutover. New store becomes authoritative. Old store remains read-only for ROLLBACK_WINDOW (default 14 days).
Decommission. After rollback window elapses with zero escalations.

Rollback at any step: revert to prior authoritative store; flip dual-write flag; audit entry.

FAILURE MODES (new)

Failure	Detection	Degraded mode	Recovery
Rollup worker behind	`projection_hash` mismatch age > SLA	Read-path emits `stale: true` flag with age	Worker catches up
Extractor down	Adapter backlog > threshold	Adapters keep enqueuing; no shadow writes	Drain on restart
Entity resolution ambiguity flood	Review queue depth > threshold	New refs auto-quarantined; no rollup over quarantined refs	Human triage; bulk-bind UI
Audit manager context overrun	Plan file size > cap	Split into sub-plans; multi-pass run	Plan compaction pass
LLM provider outage	Capability health check fails	Fallback model in routing table; else stage pauses	Provider returns or routing update
Schema migration in flight	`schema_version` mismatch	Read upcasts; write rejects downlevel	Migration completes

Silent degradation is forbidden. Every degraded mode surfaces a flag to the read path.

COST ENVELOPE (new)

Worst-case daily LLM invocations per L1 wiki with N nodes and ingest rate I events/day:

Extraction: I calls (one per event) at extract_small.
Entity resolution: I * p_ambiguous calls at extract_small. With good normalization, p_ambiguous ≈ 0.10–0.20.
Node routing: I calls at route_cheap.
Rollup (L1→L2): ≤ changed_sections_per_day calls at rollup_mid. Bounded by sections_per_projection × stale_L2s.
Audit: (shards_per_week × workers_per_shard) / 7 calls. 20 shards × 2 workers ≈ 6/day.
Audit manager: ~1 plan + 1 collate per cycle.

Token cost is dominated by extraction × event rate, not by graph size. Graph size shows up only in rollup breadth on days when many L1s change, which is itself bounded by change rate. This is the load-bearing argument for scale.

Publish a dashboard exposing each line as a metric. Budget alarms per capability class.

TEST CONTRACTS

For every tier and every stage, required tests:

Envelope schema: JSON-Schema validation; accept/reject vectors.
Renderer: golden markdown files per tier kind; byte-equality.
Section splicing: unchanged-input sections produce byte-identical output over N rollup runs.
Entity cascade: normalize/vector/LLM decision table with golden inputs.
Edge semantics: each query intent → expected traversal set on a fixture graph.
Graduation gate: simulated accept/reject streams produce deterministic state transitions.
Approval state machine: exhaustive transition coverage.
Audit shard partitioner: pure-function test.
Severity-stability: ≥high-severity finding existence stable across LLM seeds.
Migration protocol: dual-write consistency on a seeded dataset.
Failure-mode flags: each degraded mode raises the expected read-path flag.

Coverage target: every non-deterministic surface has a deterministic envelope around it that is tested.

HARD RULES

Wiki is never source of truth. Human edits route through JSON-producing forms.
Lateral edges never auto-traverse more than one hop in retrieval.
All five tier slots exist in the schema from day one, even at solo scale, even when inactive.
Opus recommends; humans approve via explicit approval nodes. Opus has no direct write authority.
Entity narrative lives in referencing nodes, not in entities themselves.
Tier promotion preserves node identity. Do not rebuild promoted nodes.
Shadow-mode graduation is per-wiki with measurable thresholds. Not a global flag flip.
No silent degradation. Every degraded mode surfaces a flag.
No LLM call at retrieval traversal time. The registry decides.
No LLM preservation of content. Preservation is mechanical splicing.
No code depends on a concrete model name. Only on a capability class.

IMPLEMENTATION PRIORITY (MVP)

JSON schema for envelope + all five tier body shapes + the approval kind.
Section-keyed wiki renderer (template function). Golden tests.
Edge semantics registry file + retrieval conductor that consumes it.
Two L1 wikis with real content (recommended: AI conversation transcripts + build artifacts — load-bearing at solo scale).
One L2 projection above them. Section splicer + projection mapping spec.
Entity layer with ~10 canonical entities + deterministic resolution cascade.
Rollup worker for the L2 projection (section-scoped, splicing first).
Shadow-mode extractor for one source type + graduation gate.
Approval queue + state machine.
Single audit pass: manager + 2 workers + continuity log.
Failure-mode dashboard with at least 3 degraded-mode flags wired end-to-end.

Everything else (worker fleet expansion, additional tiers, governance, infrastructure migration) is additive to this foundation.

WHAT IS INVARIANT ACROSS SCALES

JSON schema
Wiki renderer
Entity layer mechanics
Typed edges + one-hop rule
Edge semantics registry format
Shadow-mode ingestion
Manager/worker audit pattern (even with a tiny fleet)
Tier slot existence (activation differs)
Capability-class routing (table content differs)

WHAT SCALES CONTINUOUSLY

Worker fleet size
Rollup cadence class thresholds
Index backing (embedded ↔ production DBs via migration protocol)
Ingestion throughput
Entity resolution confidence thresholds

WHAT CHANGES CHARACTER

Tier activation count (3 at solo, 5 at enterprise)
Authority model (privacy zoning ↔ role-based with elevation logging)
Governance overhead (inline ↔ architecture review + audit-of-audit)
Strategic redundancy in audit (absent at solo, rotated at enterprise)
Specialist model access (scheduled budgets ↔ on-demand)

BEHAVIORAL EXPECTATIONS FOR CLAUDE INSTANCES

When ingesting this gist and extending the system:

Preserve the invariants above.
Prefer deterministic algorithms over LLM calls. LLM calls require a deterministic envelope: goldens, seeds, or stability tests.
When adding features, ask: does this respect the wiki-is-projection rule, the one-hop rule, the shadow-mode discipline, and the no-silent-degradation rule?
Optimize token cost via retrieval scoping and rollup abstraction before reaching for bigger context windows.
When scaling, activate tier slots; do not add them.
When a solo deployment grows, execute promotion operations; do not rebuild.
Treat the entity layer as load-bearing at every scale.
Route by capability class; never hard-code a model name.
Surface every degraded mode. Silent failure is a bug.

tyoung515-svg/layered-knowledge-system-gist-v2.md

Select an option

No results found