You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Broad retrieval over everything not modeled as a node
"What else is relevant?"
Orchestration
Agent/LLM with both tools registered
Natural-language answers combining all sources
Data Flow
[Source system] → Structured Graph ([cadence] ETL): [entities] from fact tables; [reference data] from dimension/reference tables.
Documents → Extraction + Review → Curated Graph: [event/decision/policy types] extracted from [document types], reviewed by [who] before insertion.
Documents → Semantic Search Collection (bulk pipeline): all [remaining document types], indexed for retrieval — no entity extraction.
Choosing the Semantic Search Backend
[Does the organization already have enterprise search indexing these documents? If so, name it and use it — no new infrastructure, no embedding pipeline to build. If not, evaluate a dedicated vector store, but only after confirming the existing option is genuinely insufficient. See references/hybrid-architecture.md.]
Create a "[Project] [Domain]" collection containing:
[Document type 1]
[Document type 2]
[Relevant channels/spaces]
If backend limitations emerge later (custom chunking, metadata filtering, etc.), migrate to a dedicated vector store. Start simple.
This document defines the schema (ontology) for the [Project Name] world model — structured nodes from [source system], curated unstructured nodes, their relationships (edges), and example traversals.
4. Ontology
4.1 Structured Nodes (from [source system])
One subsection per node type. Repeat.
[Entity Name]
[One-line description of what this entity represents.]
Field
Type
Description
[id_field]
PK
Primary Key
[field]
[type]
[description]
[fk_field]
FK
Foreign Key → [OtherEntity]
[Dimension/Reference Node]
[DimensionName]:[value1, value2, value3, ...]
(Flag any "dimension" that is really just a text/code column in the source rather than a real reference table — it still becomes its own concept here, populated from distinct values; see references/source-mapping-patterns.md.)
(list curated-to-structured and curated-to-curated edges)
4.4 Node Counts (Estimated)
Node Type
Count
Notes
[Structured entity 1]
~[N]
[source/basis for estimate]
[Structured entity 2]
~[N]
[Curated node type]
~[N]/year
[basis, e.g. "1-2 per source per year"]
Document
~[N]/year
A sizing input for 5_data_mapping.md, not a precision requirement.
5. Example Traversals
5.1 [Entity] View
This traversal shows how one structured entity's attributes, relationships, and linked context come together.
graph TD
E[Entity instance<br>key attributes]
E -- "relation" --> R[Related entity]
E -- "relation" --> C[Curated node<br>e.g. linked Event]
Loading
5.2 Explanation Chain Traversal
This traversal shows how to diagnose a variance by walking from a structured aggregate back through curated nodes to the document that explains it.
graph TD
S1[Step 1: Start at a structured aggregate<br>e.g. an account/metric total]
S2[Step 2: Find contributing structured factors<br>e.g. rate/volume/mix components]
S3[Step 3: Find linked curated node<br>e.g. the Event/Decision that changed a factor]
S4[Step 4: Follow to source Document]
S5[Step 5: Semantic search for broader context<br>related memos, threads, tickets]
S1 -- "reverse traversal" --> S2
S2 -- "impacts (inverse)" --> S3
S3 -- "documented_in" --> S4
S5 -. "synthesized explanation" .-> S1
Loading
This is a reachability/path query. If it needs ranking, variable depth, or multi-path enumeration beyond a fixed-depth chain of lookups, it becomes rai-graph-analysis territory once the ontology is implemented.
[Project Name]: Implementation Decisions, Documents, & Next Steps
This document tracks resolved and open implementation decisions, the document-gathering plan for the curated pipeline, ontology areas parked for later phases, and immediate next steps.
6. Implementation Decisions
#
Question / Topic
Status
Decision / Approach
Rationale
1
[Grain of the primary snapshot/activity entity]
Resolved
[e.g. Weekly, with a monthly rollup]
[why this grain]
2
[Aggregation level of the core fact table]
Resolved
[e.g. Full grain, pending prototype validation]
[why]
3
[A many-to-many edge's grain, e.g. a network/relationship edge]
Open
Pending [validation needed]
[why it's not resolved yet]
4
How curated nodes get created
Resolved
Extraction ([LLM/analyst]) + human review before insertion
See references/explanatory-node-pattern.md
5
Semantic search backend
Resolved
[Reuse existing enterprise search / new vector store]
[why]
6
Historical backfill window
Resolved
[N years/periods]
[why this boundary, e.g. a known data-quality cutoff]
Add a row for every non-obvious modeling or architecture choice. See references/decision-log-and-scoping.md.
7. Curated Document Sources — Documents to Gather
Phase 1 Document Collection
The smallest set that proves the pattern end to end for at least one use case from 2_use_cases.md.
1. [Document Type Name]
Scope: [what's included]
Time Range: [range]
Frequency: [cadence]
Estimated Count: [~N documents]
Target Data to Extract: [which curated node types/fields come from this document type]
2. [Next document type]
(repeat)
Phase 2 Document Collection (Future)
To be added after Phase 1 is operational:
[Document type]
[Document type]
8. Ontology Gaps — Parked for Future
Nodes/edges identified as potential additions but out of scope for the current phase:
[Node/edge name]: [what it would enable] — parked because [reason: data doesn't exist yet / volume needs validation / stakeholder not yet engaged].
[Next parked item]
9. Next Steps
Review this spec and gather sign-off.
Confirm source tables for each node type (see 5_data_mapping.md).
Design the ETL pipeline (source system → RAI).
Build a bounded prototype with a subset of data.
Validate traversal performance on the prototype graph.
Iterate on the schema based on what the prototype reveals.
This document maps structured data from [source system] to the [Project Name] ontology, including join keys, field definitions, and any categorical coding-scheme mapping.
8. Source Table Mapping
8.1 Primary Source Tables
Table Path
Purpose
Grain
Key Columns
[schema.table]
[purpose]
[grain]
[key columns]
[schema.table]
[purpose]
[grain]
[key columns]
Pull the real column list and types from INFORMATION_SCHEMA.COLUMNS (or the source system's equivalent) before filling this in — never infer types from names or examples.
8.2 Join Keys Between Tables
[Table A] → [Table B]:
[exact join expression, including any COALESCE/cast/date-truncation]
[Table A] → [Table C]:
[exact join expression]
8.3 Ontology Node → Source Table Mapping
One block per structured node from 3_ontology_and_traversals.md.
[Entity Name]
Ontology Node/Field
Primary Source Table
Source Columns / Mapping Logic
[Entity]
[source table]
[column], [column]
• [Entity.field]
[source table]
[column] — [derivation logic if computed]
Dimension Nodes
Ontology Node/Field
Primary Source Table
Source Columns / Mapping Logic
[DimensionConcept]
[source table]
DISTINCT [column] — text column promoted to a concept, no reference table exists
9. Coding-Scheme Mapping
Only needed if the domain has a categorical scheme (a chart of accounts, a classification/product code, a regulatory category) owned by a governance function or external counterpart, separate from pricing/transactional metadata. See the hub-concept pattern in references/source-mapping-patterns.md.
9.1 Scheme Structure
[How the code is structured — segments, hierarchy levels, code ranges and what they mean]
9.2 [Entity] → Coding-Scheme Mapping Table
[Entity]
Measure
Code
Name
Notes
[Entity value]
[measure type]
[code]
[name]
9.3 Architectural Decision: Hub Pattern
Decision: [Pricing/policy metadata] and [the coding scheme] are separate nodes, both linked through the hub of [Hub Concept, e.g. Product/Service].
Why: keeps [pricing metadata] independently maintainable from [the coding scheme]; the hub is the natural place to reason about everything related to [the entity]; no direct edge to keep in sync when either side changes.
4_implementation_and_decisions.md is the spec's memory — it's what keeps a design review from re-litigating the same question twice, and what tells a future maintainer why the model looks the way it does, not just what it looks like.
Decision log format
One row per non-obvious choice, four columns:
#
Question / Topic
Status
Decision / Approach
Rationale
1
Grain of the primary activity/snapshot entity
Resolved
Weekly, with a monthly rollup for dedup
Weekly is the atomic grain; monthly avoids double-counting when a stakeholder asks "how many were active this month"
2
Whether to keep transaction-level detail or pre-aggregate
Resolved
Full grain, pending a 1-month prototype validation
The reasoner's compute model tolerates full grain; validate before committing at scale
3
Grain of a many-to-many network edge (e.g. peer-to-peer transfers)
Open
Pending volume validation
Full grain is preferred for network analysis, but needs a size check before committing
4
How curated nodes get created
Resolved
Extraction (LLM or analyst) + human review before insertion
Keeps a finance/compliance-grade layer from silently drifting — see explanatory-node-pattern.md
5
Semantic search backend
Resolved
Reuse existing enterprise search
Already deployed and indexing the relevant documents; no new infrastructure
6
Historical backfill window
Resolved
[N years], chosen to avoid a known data-quality boundary
State the boundary explicitly (a system migration, a schema change, a known bad period)
Every row should be something a reviewer would otherwise ask about — if a decision is truly obvious, it doesn't need a row. Open rows are not a weakness in the spec; they're the explicit list of what blocks moving from spec to build, which is exactly what this document is for.
Phasing the curated-document collection plan
Don't try to source every document type on day one. Phase by how much of the pattern each phase proves:
Phase 1 — the smallest set of document types that lets the causal-chain traversal actually run end to end for at least one real use case from 2_use_cases.md. Pick document types that are already gathered somewhere (regularly published reports, standing notification channels) over ones that require new process to collect.
Phase 2+ — expansion once Phase 1 is operational: additional document types, additional stakeholder groups, longer history.
For each Phase 1 document type, record: scope (which documents), time range, frequency, an estimated count, and what specifically gets extracted from it (which curated node types and fields). This is the direct input to designing the extraction pipeline in hybrid-architecture.md's data-flow step 2.
Parking scope, not dropping it
When a candidate node, edge, or use case is identified but out of scope for the current phase, write it down under a "parked" heading with one line on why it's parked (data doesn't exist yet, volume needs validation, stakeholder not yet engaged) rather than letting it disappear from the conversation. This does two things: it keeps the current spec honestly scoped (the ontology in 3_ontology_and_traversals.md should only include what's actually being built), and it gives the next phase a ready-made backlog instead of starting from a blank page.
Next steps
Close with a short, concrete, ordered list — not a restatement of the whole plan. Typical shape: get sign-off on this spec, confirm source tables for each node (should already be done in 5_data_mapping.md by this point), design the ETL pipeline, build a bounded prototype, validate traversal performance, iterate on the schema based on what the prototype reveals. Keep it to what happens next, not the full roadmap — that's what production-scaling.md is for once the prototype is working.
Curated unstructured nodes (an Event, Decision, Policy, Regulation, or similar named occurrence extracted from documents) are modeled with the exact same PyRel constructs as structured nodes — there is no separate "unstructured concept" class. What makes them distinct is how they're populated and how they carry provenance.
Definition
Use model.Concept with identify_by exactly as you would for a structured node:
Event=model.Concept("Event", identify_by={"event_id": Integer})
Event.event_type=model.Property(f"{Event} has type {String:event_type}")
Event.title=model.Property(f"{Event} has title {String:title}")
Event.effective_date=model.Property(f"{Event} has effective date {Date:effective_date}")
Decision=model.Concept("Decision", identify_by={"decision_id": Integer})
Decision.decided_date=model.Property(f"{Decision} was decided on {Date:decided_date}")
The difference from a structured concept: no model.Table() binding. A structured concept loads from a source table on a schedule; a curated concept loads from a reviewed extraction batch, loaded once or on its own (usually much slower) cadence. Mark this in a comment at the point of definition — e.g. # curated: no source table, loaded from reviewed extraction batches — so the distinction is legible to the next person reading the ontology, since nothing in the type system enforces it.
Linking to structured nodes
Link curated nodes into the structured graph the same way you'd link any two concepts — a functional Property FK when the curated node points to exactly one structured entity, a Relationship when it can point to several or carries its own payload:
Curated nodes also cross-link each other — an Event can be documented_in a Document, from an Organization, and so on. Model these the same way.
Provenance: making "human-reviewed" real
A spec that says curated nodes are "LLM-extracted and human-reviewed" is a design intent, not a guarantee — nothing stops an implementation from becoming a script that writes hand-typed rows straight into production tables with no review step at all. If the spec claims review, the ontology needs to carry evidence of it:
Every curated concept should have a functional Property FK to a Document concept (minimum bar for provenance — "where did this fact come from")
If answers need to be auditable (someone should be able to check the claim against the source), add fields that let them: a quote or span, an extraction confidence, which extraction run/model produced it
If there's a genuine review step, model it as data, not process: a status property (draft / approved / rejected), reviewed_by, reviewed_at. Downstream queries should only traverse approved curated nodes — that's what makes "human-reviewed" an enforced property of the graph rather than a claim in a comment.
Without these fields, a curated layer that looks governed in the spec can quietly become an ungoverned one in practice — anyone can write a row, nothing distinguishes a proposed fact from an approved one, and there's no way to trace a synthesized answer back to what a human actually signed off on.
Designing the explanation-chain traversal
The signature traversal for a world model walks backward from a structured aggregate to the curated node (and ultimately the document) that explains it:
For example: a ledger balance, traced back through the pricing/fee structure that computed it, to the event that changed that pricing, to the notification document that announced it. This is a multi-hop reachability query, not a single join — write it as a chain of lookups in PyRel for a fixed-depth chain (2-4 hops, known structure). If the use case needs variable-depth traversal, ranking by relevance/recency, or enumerating multiple candidate paths, that's a graph algorithm, not a fixed query — hand off to rai-graph-analysis once the ontology exists.
Model both directions of the chain during Step 3 (3_ontology_and_traversals.md): the forward edge (curated node → what it affects) is what you define in PyRel; the traversal usually needs to run in reverse (start from the structured node, find what explains it), so make sure both directions are actually queryable, not just documented as a diagram.
The world model has four working parts. Getting the boundary between them right — what becomes a graph node, what stays a document, what gets a search index — is the highest-leverage decision in the whole spec.
Deciding what belongs in which layer
If the fact is...
Put it in...
Because
A record from a system of record, refreshed on a schedule
Structured graph
It's already structured, keyed, and joinable — modeling it as anything else throws that away
A discrete, named occurrence that changes a structured fact (a rate change, a policy update, a pricing decision) and is worth linking to multiple structured nodes
Curated unstructured graph
It's a first-class entity with its own identity and edges, not just a passage of text
Context that explains or supports an answer but isn't itself a recurring, linkable entity (a one-off memo, a Slack thread, a support ticket)
Semantic search collection
Modeling every document as a node doesn't scale and most of them are read once, not traversed
The test for "does this deserve to be a curated node, not just search text": will more than one use case need to traverse to or from it (not just retrieve it)? If yes, model it. If a use case only ever needs to quote it, leave it in search.
Choosing a semantic search backend
Before evaluating a dedicated vector store, check whether the organization already has enterprise search indexing the relevant documents (a workplace search product, an internal knowledge index, an existing RAG pipeline). If so:
No new infrastructure to deploy or operate
No embedding pipeline to build and maintain — it's handled natively
Can usually be scoped to a domain-relevant collection without a new system
Only build/adopt a dedicated vector store if a concrete limitation shows up later — custom chunking strategy, metadata filtering the existing tool can't do, latency requirements it can't meet. Start with what's already deployed; a vector database is a fallback, not a default.
The orchestration pattern
The structured graph and the search collection are not fused by a hand-written retrieval function. In practice, both surface as separate callable tools registered on one serving agent:
A graph-query tool (or a small set of stored procedures) that runs traversals against the RAI model
A semantic-search tool that queries the document index
The agent (LLM) decides at answer time which tool(s) to call and how to combine their outputs into a synthesized answer — the fusion logic lives in the agent's reasoning, not in a bespoke Python function that merges result sets. Don't design a custom "combine graph result + search result" module; design the tool interfaces well enough that the agent can compose them itself.
Data flow
Three independent pipelines feed the model, each on its own cadence:
Source system → structured graph, on a scheduled ETL cadence (weekly, daily — whatever matches the source's own refresh rate). This pipeline reads from a curated mart, never raw source tables directly (see production-scaling.md).
Documents → extraction → curated graph, an extraction-plus-human-review pipeline. Candidate nodes are proposed (by an LLM or an analyst) and reviewed before they enter the graph — see explanatory-node-pattern.md for how to make that review step real rather than assumed.
Documents → search collection, a bulk indexing pipeline with no entity extraction — just chunking and embedding (or whatever the search backend natively does) for retrieval.
Keep these three pipelines decoupled. A document can flow into both the curated pipeline (if it contains extractable, linkable facts) and the search collection (for full-text/semantic retrieval) at once — they're not mutually exclusive.
This document is a conceptual overview of the [Project Name] world model expressed in PyRel — for design review, not the implementation itself. Once approved, build the real model with rai-ontology's Greenfield Build Workflow and rai-pyrel for syntax/queries/rules.
Before writing this file, verify current PyRel constructor syntax against rai-pyrel/rai-ontology. Never copy constructor names from an older spec, whitepaper, or previous project — the API surface shown below (model.Concept, model.Property, model.Relationship) is what to confirm as current, not assume.
The code below uses generic placeholder names (Entity, RelatedEntity, Category...) — replace them with the project's real concept/property names; keep the constructs (Concept / Property / Relationship, the f-string reading pattern) as-is.
Functional attributes and FKs (each maps the concept to exactly one value):
Entity.name=model.Property(f"{Entity} has {String:name}")
Entity.related=model.Property(f"{Entity} linked to {RelatedEntity:related}") # functional FKEntity.is_flagged=model.Property(f"{Entity} is flagged") # unary boolean flag
3. Relationships (model.Relationship)
Many-to-many links and payload-bearing edges:
Entity.sent_to=model.Relationship(
f"{Entity:sender} sent to {Entity:recipient} in {Week:week} totaling {Float:amount}"
)
Event.impacts=model.Relationship(f"{Event} impacts {RelatedEntity}")
Decision.changes=model.Relationship(f"{Decision} changes {RelatedEntity}")
Note what identify_by does and does not guarantee: it gives each concept instance a stable identity, not a business-rule constraint. If the spec requires validation beyond identity (e.g. "every active Entity must map to a valid RelatedEntity"), state that requirement explicitly here and confirm with rai-pyrel how it's enforced — don't assume the ontology definition alone enforces it.
A world model spec is usually validated first on a bounded prototype — a sample of customers, a month of history, a hand-seeded curated layer — before it runs at full scale. This reference captures what changes (and what doesn't) between the two, so the prototype-phase decisions in 4_implementation_and_decisions.md don't quietly become permanent.
What does NOT change going to production
The ontology and model logic are size-agnostic. If 3_ontology_and_traversals.md and ontology.md are well-designed, they don't need a redesign at scale — production is an ETL and compute-sizing exercise layered on top of the same concepts, properties, and relationships. The guiding principle throughout: keep the RAI-readable layer a deliberately small, curated mart, even as the real sources grow into the hundreds of millions or billions of rows.
What does change
Dimension
Prototype
Production
Data
Bounded, hand-built snapshots in a personal/dev schema
Governed pipelines off the real sources, in a shared production schema
Freshness
One-off snapshot, single time period
Scheduled incremental refresh, multi-period history
Coverage
A sample
The full population
Access
Single user, personal database
Any granted role, RBAC + masking
Curated "why" layer
Hand-seeded curated nodes
Real extraction pipeline + human review
Ops
Manual runs, ad-hoc recovery
Monitoring, alerting, scheduled jobs, runbooks
Why raw-source binding doesn't survive contact with scale
This is the single biggest force shaping the production architecture, and it's worth stating as a hard constraint rather than a preference: RAI-on-raw is a non-starter at real scale.
Binding a model.Table() directly to a source requires CHANGE_TRACKING=TRUE, which requires ownership the modeling team usually doesn't hold on production source tables
A source exposed only as a view can't carry change tracking at all, full stop
The first query against a newly-bound table triggers a full initial sync — fine for a table with thousands of rows, a non-starter for one with hundreds of millions or billions
Point/join lookups on huge unclustered tables (a token/reference join across a many-billion-row table) are prohibitively expensive until those tables are clustered or pre-aggregated
The fix is the same at every scale: materialize an owned, change-tracked staging/mart layer between the raw sources and the model, and never point the model at raw sources directly. This mart is where aggregation, clustering, and stratified sampling happen — the model only ever reads from it.
Sampling that preserves structure
When a prototype samples down from a huge source, a flat LIMIT is not neutral — it can silently collapse a categorical dimension to a single value, or (for network/graph data) break the very structure a use case needs to analyze. Two corrections:
Stratified sampling for categorical coverage: QUALIFY ROW_NUMBER() OVER (PARTITION BY <category> ORDER BY ...) <= N instead of a bare LIMIT, so every category survives into the sample
Ego-network sampling for graph/network data (a peer-to-peer or relationship graph): sample around seed nodes and include their neighborhoods, rather than uniform-random rows, so the sampled graph still has real structure to analyze
Turning a hand-seeded curated layer into a governed one
A prototype's curated layer is often literally hand-typed rows written once into the target tables — useful for proving the traversal works, but not what the spec's "LLM-extraction + human review" language actually describes. Moving to production means building the real pipeline: ingest the Phase 1 document types from 4_implementation_and_decisions.md, extract candidate curated nodes, route them through an actual review/approval step (see the status/reviewed_by fields in explanatory-node-pattern.md), and schedule ingestion so new documents keep the graph current as they arrive.
Compute and cost, briefly
Different workloads in a world model (structured-graph queries, curated-graph queries, any graph-algorithm or predictive workload layered on top) can have very different compute profiles — a query-serving workload wants a warm, right-sized engine for low latency; a one-off model-training or full-population scoring job wants a larger engine sized for throughput and can tolerate auto-suspend between runs. Size and monitor each accordingly rather than running everything on one default-sized engine; see rai-health for engine sizing and monitoring guidance once the model is live.
Designs a "world model" — a hybrid architecture pairing a structured RAI graph with a curated unstructured graph (events, decisions, policies extracted from documents) and semantic search over everything else, so questions get answered with "why," not just "what." Produces the upfront technical-specification package — problem framing, use-case catalog, ontology and causal traversals, an implementation-decision log, source-to-ontology data mapping, and a PyRel translation overview — before any ontology code is written. Use when scoping a knowledge graph for root-cause or variance analysis, when structured transactional data needs explaining with unstructured context (policies, decisions, notifications, memos, tickets), or when asked to design a "world model," "causal graph," or similar, even if not phrased that way. Not for authoring the PyRel ontology itself (see rai-ontology, rai-pyrel) or routing questions to a reasoner once the model exists (see rai-discovery).
World Model
Summary
What: Scopes and specifies a world model — a hybrid knowledge architecture pairing a structured graph (facts from systems of record), a curated unstructured graph (events, decisions, policies extracted from documents and linked into the structured layer), and semantic search (broad retrieval over everything not worth modeling as a node) — purpose-built to answer why something happened, not just report what happened. The deliverable is a six-document technical-specification package a team reviews and signs off on before any ontology code is written.
When to use:
Scoping an analytics initiative that must explain drivers/root causes behind a metric — variance analysis, "why did X change," audit trails
Designing a knowledge graph that must combine structured transactional data with unstructured context (policies, decisions, notifications, memos, tickets)
Asked to build a "world model," "causal graph," "knowledge graph for explainability," or similar — even without those exact words
Producing the upfront design-review package (problem statement, use cases, ontology, decision log, data mapping) before committing to an implementation
When NOT to use:
Writing or evolving the actual PyRel ontology code — see rai-ontology
PyRel syntax, data loading, queries, rules — see rai-pyrel
Routing a specific question to a reasoner once the model exists — see rai-discovery
Selecting or running a graph algorithm (centrality, reachability, paths) — see rai-graph-analysis
Deploying the finished model to production — see rai-deployment
Overview: Frame the gap → catalog use cases across stakeholders → design the structured + curated-unstructured ontology and its causal traversals → log implementation decisions and scope the curated-document pipeline → map sources to the ontology → translate to a PyRel overview. Each step produces one file of a six-file spec package (see Six-Document Deliverable). Steps 3 and 6 hand off to rai-ontology/rai-pyrel for the real implementation once the spec is approved.
Quick Reference
Architecture layers
Layer
Populated from
Answers
Notes
Structured graph
A governed, curated/aggregated mart — never raw multi-billion-row sources directly
"What happened," with precise numbers
Concept/Property/Relationship bound via model.Table(); see the CHANGE_TRACKING pitfall below
Curated unstructured graph
Documents, via extraction + review, linked into the structured graph
"What caused it" — the events/decisions/policies behind a number
Conceptual PyRel translation: concepts, properties, relationships, example queries, constraints
Step 6
Copy the skeletons in examples/output_templates/ into the new spec directory and fill them in per project.
The World Model Design Workflow
Interaction mode: before starting, ask the user which mode they prefer — guided (confirm each document before moving to the next; best when the user has domain context to share along the way) or one-shot (produce the full six-file package in one pass for review). This mirrors rai-ontology's greenfield workflow.
Output lands in a new directory (e.g. <project>_docs/), one file per step below.
Step 1 — Frame the gap → 1_overview.md
State what current tooling reports vs. what it can't explain: which dashboards, tables, or predictive models exist today, and which "why" questions they can't answer. List what's missing — causal relationships between structured facts, external events, internal decisions, unstructured context — in whatever shape the domain actually takes (don't force this exact list). Then lay out the hybrid architecture as a "why this component" table (purpose, what it answers) and a data-flow list: source system → structured graph (cadence), documents → curated graph (extraction + review), documents → search collection (bulk, no entity extraction). Name the actual candidate systems for each layer — the warehouse, the document sources, an existing enterprise search tool if one exists — rather than leaving them generic. See Hybrid Architecture for the layer-boundary test, how to choose (or avoid building) a search backend, and the tool-registration orchestration pattern.
Step 2 — Catalog use cases → 2_use_cases.md
Group use cases by stakeholder or department, not by data source. For each: a question phrased the way that stakeholder would actually ask it, a traversal sketch noting which layer(s) it touches and the ontology hop, and a synthesized answer shape. Close with a summary matrix — rows are question types ("what happened," "why did it happen," "what's the impact of X," "what should we know," "show the audit trail"), columns are the three layers, cells mark which layer is primary. This matrix is the fastest sanity check that the three-layer split earns its keep: if one layer never comes up as primary, reconsider whether the use cases actually need it.
Step 3 — Design the ontology and traversals → 3_ontology_and_traversals.md
Model the structured nodes first — apply rai-ontology's concept/property/relationship decision table. Then layer curated unstructured nodes on top using the Explanatory Node Pattern. Document edges in all three directions that can occur (structured-to-structured, curated-to-structured, curated-to-curated), and give a rough node-count estimate per type — a sizing input for Step 5, not a precision requirement. Close with 1-2 example traversals as diagrams: one entity view (all the context linked to one structured node) and one explanation chain (from a structured aggregate back through curated nodes to the document that explains it). An explanation-chain traversal is a reachability/path query — if it needs ranking, distance, or multi-hop enumeration beyond a simple joined lookup, that becomes rai-graph-analysis territory once the ontology is built.
Step 4 — Log decisions and scope the curated pipeline → 4_implementation_and_decisions.md
Keep a running decision log with four columns — topic, status (Resolved/Open), decision/approach, rationale. Every non-obvious modeling or architecture choice earns a row here, not just a mention in prose. List curated-document sources to gather, phased by priority (phase 1 = the smallest set that proves the pattern; later phases = expansion), with volume/frequency estimates. Park out-of-scope ontology extensions explicitly rather than silently dropping them. End with concrete next steps. See Decision Log & Scoping for the phasing heuristic and a worked excerpt.
Step 5 — Map sources to the ontology → 5_data_mapping.md
For every structured node, name the primary source table(s), the join keys between them, and the source column → ontology field mapping — pull real column names and types from the source schema, never infer them from names (same rule as rai-ontology Step 2). If the domain has a categorical coding scheme owned by a governance function or external counterpart (a chart of accounts, a product catalog, a classification system), document it as its own mapping table and route it through a shared hub concept rather than wiring it directly to pricing/policy metadata — see the hub pattern in Source Mapping Patterns. Flag anywhere a "dimension" is really just a text column in the source rather than a real reference table — it still becomes its own concept, populated from distinct values.
Step 6 — Translate to PyRel → ontology.md
Restate the Step 3 ontology as PyRel: one Concept per node type (with identify_by), one Property per functional attribute or FK, one Relationship per many-to-many or payload-bearing link — using current PyRel v1 syntax (model.Concept(...), model.Property(f"..."), model.Relationship(f"...")). Verify the exact syntax against rai-pyrel before writing this file; never copy constructor names from an older spec or whitepaper (see Common Pitfalls — specs go stale, the API doesn't wait for them). This file is a conceptual overview for reviewers, not the implementation — once approved, build the real model with rai-ontology's Greenfield Build Workflow and rai-pyrel for query/rule syntax. Before scaling past a bounded prototype, review Production Scaling — the ontology doesn't need to change, but the source binding and the curated pipeline do.
Common Pitfalls
Mistake
Cause
Fix
Binding the structured graph straight to raw source tables
model.Table() requires CHANGE_TRACKING=TRUE plus ownership; multi-billion-row views can't carry change tracking at all, and the first query triggers a full sync
Materialize an owned, change-tracked curated/aggregated mart between raw sources and the model; keep the RAI-readable layer deliberately small even as sources scale into the billions
Calling the curated layer "LLM-extracted + human-reviewed" in the spec, then shipping hand-typed literals with no review fields
Documented design intent is not the same as an enforced pipeline
If the spec claims a review step, add explicit status/reviewed_by/reviewed_at fields (or a staging table) to every curated concept — otherwise say "hand-seeded" and scope the real pipeline as an open decision item
Curated nodes with no link back to the source document, or a Document concept too thin to verify the fact
Provenance modeled as a title-level pointer with no quote, span, or confidence
Every curated concept (Event, Decision, Policy...) should carry a functional Property FK to Document at minimum; add span/offset/confidence fields if answers need to be auditable
Random LIMIT sampling when building a bounded prototype from a huge source
A naive LIMIT can collapse a whole categorical dimension to a single value, silently destroying structure
Sample stratified per category (QUALIFY ROW_NUMBER() OVER (PARTITION BY <category> ...)) so every segment survives into the prototype
Modeling a recurring reference/category column as a plain string wherever it's used
Loses the ability to traverse or join on that dimension as a first-class node
Promote recurring categorical text columns to their own Concept, populated from distinct values, joined via lookup
Wiring pricing/policy metadata directly to an accounting or coding scheme with a direct edge
Tightly couples two schemas that change independently and are usually owned by different teams
Route both through a shared hub concept with separate relationships to each, so either can evolve on its own
Forcing every "what if X changes" use case into an optimization/solver framing
Parametric recomputation over the structured graph looks similar to optimization but has no decision variables or constraints
Reserve rai-prescriptive-problem for genuine constrained-decision questions; route "what if the rate/policy changes by X" scenarios through rai-pyrel recomputation instead
Copying ontology code from an old spec or whitepaper verbatim into ontology.md
PyRel's API surface evolves; a document's constructor names may already be legacy syntax
Always verify current constructor names via rai-pyrel/rai-ontology before writing the PyRel translation — never trust a spec's code block as current syntax
Structured vs. curated-unstructured vs. semantic-search: what belongs in which layer, choosing a search backend, the tool-registration orchestration pattern
Step 1, or when deciding whether a new vector store is actually needed
5_data_mapping.md is where the conceptual ontology in 3_ontology_and_traversals.md gets tied down to real, physical columns. Three recurring patterns show up in this step across domains.
Promote recurring category columns to concepts
A source table frequently has a "dimension" that isn't actually backed by a reference table — it's just a constrained text or code column (a status, a category, a type code) reused across many rows. Model it as its own Concept anyway, populated from the distinct values in the source:
This is worth doing even though there's no physical reference table, because it's what makes the dimension traversable and joinable — "group transactions by category and show me the category's other attributes" needs the category to be a node, not a repeated string. Document in the mapping table that the "source" for this concept is a DISTINCT projection of a column, not a standalone table, so nobody goes looking for a reference table that doesn't exist.
Document join keys explicitly, not just per-table columns
List each join between source tables as its own row with the literal join condition, not just "these tables relate somehow":
Real join keys are rarely clean 1:1 FKs — expect COALESCEs across renamed/legacy key columns, type casts, and date-truncation joins (e.g. joining a transaction-grain table to a monthly snapshot table on DATE_TRUNC('month', ...)). Write the exact expression, because it's what the ETL step actually implements — a vague description here becomes an ambiguous implementation later.
The hub-concept pattern for coding schemes
When a domain has two categorical structures that are both attached to the same underlying entity but are governed and evolve independently — e.g. a pricing/fee schedule (owned by a commercial or product team) and an accounting or classification code (owned by a finance/governance function) — don't wire them to each other directly. Route both through a shared hub concept:
Benefits: pricing metadata and the coding scheme stay independently maintainable (either can change without touching the other), the hub is the natural place to reason about "everything related to this product," and there's no direct edge to keep in sync when either side's structure changes. This pattern generalizes beyond finance — anywhere two independently-owned classification/reference schemes both attach to the same core entity, prefer a shared hub over a direct edge between them.
Real-schema grounding
Pull the actual column list and types from the source before writing this file — INFORMATION_SCHEMA.COLUMNS or the equivalent for the source system — and keep that result available through the rest of the workflow. Types matter for the PyRel translation in Step 6: a numeric column with decimal precision maps to Float/Decimal differently than an integer, and a DATE column is not the same as a DATETIME/TIMESTAMP even if a design doc's prose calls it a "timestamp." Never infer a type from a column's name or from a handful of example values — the same rule rai-ontology applies to any greenfield build applies here.
