Fully local temporal knowledge graph: Graphiti + Ollama on a single RTX 5090 — working config and all the traps

graphiti-ollama-local.md

Local knowledge graph on a single RTX 5090: Graphiti + Ollama without the cloud

Technical writeup. Audience: anyone tracking issue #868 who wants to reproduce a fully local, temporal knowledge-graph memory layer for an LLM chat stack — no OpenAI key, no hosted Graphiti, one GPU.

---

1. What was built and why

The goal was a persistent memory layer for a self-hosted LibreChat + Ollama setup: not "stuff the last N turns into the prompt," but a real temporal knowledge graph where facts become entities and relationships, survive across sessions, and can be retrieved by semantic search rather than by chunk overlap. Concretely: the assistant should be able to store "Yurii lives in Kyiv" once, and surface it in an unrelated conversation a week later. Graphiti (getzep/graphiti) does the entity/edge extraction and bi-temporal bookkeeping on top of Neo4j; we already ran Neo4j and Ollama, so the only new moving part was a small service wrapping graphiti-core.

The hard constraint was no cloud. Everything — the extraction LLM, the embedder, the graph store — runs on one workstation with a single RTX 5090 (32 GB VRAM) and a system-level Ollama. That constraint is what makes this non-trivial: Graphiti is written and documented against the OpenAI API, and its "local" path is under-documented and partly broken. Most of the work below is reconciling Graphiti's assumptions with what Ollama actually supports, and dealing with the fact that a 27B model doing structured extraction on one GPU is slow — slow enough to break the transport layer if you call it naively.

---

2. Stack with versions

Everything runs in one Docker Compose project (librechat_default network), except Ollama which runs as a host system service so it owns the GPU directly.

Component	Version / tag	Notes
Graphiti	graphiti-core==0.29.2	Released 2026-06-08; pinned. The library, not the official server image.
Neo4j	neo4j:5.26-community	With APOC plugin (NEO4J_PLUGINS=["apoc"]). Graphiti requires APOC.
Ollama	host service on :11434	Runs outside Docker (/usr/local/bin/ollama serve), owns the GPU.
Extraction LLM	qwen3.6:27b	Custom local tag, 17 GB, ID a50eda8ed977. See the model-name note below.
Embedder	bge-m3	1024-dim. Served by the same Ollama.
Wrapper service	FastAPI + fastmcp	Our own main.py, exposes REST + MCP/SSE.
Chat front end	LibreChat (librechat-dev:latest)	Connects to the wrapper over MCP SSE.
Vector RAG (adjacent)	pgvector/pgvector:0.8.0-pg15 + rag_api	Pre-existing; separate from the graph memory.

Python deps for the wrapper (services/graphiti/requirements.txt):

graphiti-core==0.29.2
fastapi>=0.115.0
uvicorn>=0.44.0
pydantic-settings>=2.4.0
httpx>=0.28.1

Dockerfile is intentionally boring:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001"]

A note on the model tag. qwen3.6:27b is not a real upstream Qwen release — there is no qwen3.6. It's a custom/renamed local tag (ollama list confirms it: 17 GB, real, resolves fine). It is hardcoded in three places that must stay in sync: docker-compose.yml (LLM_MODEL), services/graphiti/main.py (default), and librechat.yaml (Ollama endpoint model list). If you reproduce this, substitute your own extraction model — qwen3:32b or similar — but be aware a 27–32B class model is roughly the floor for Graphiti's extraction prompts to produce valid structured output reliably.

---

3. Rejected approaches and why

mem0. The original plan (B1 of the build) was mem0 in Docker for "memory types" — long/episodic/semantic/working, with decay and reinforcement — wired to both pgvector and Neo4j. It was dropped before installation. Reasons: (a) the value proposition overlapped almost entirely with what we already had — pgvector + bge-m3 already gave us vector memory, and Neo4j already gave us the graph; mem0 would have been a third store to keep consistent with the other two. (b) What we actually wanted was relationships between facts over time (entity A mentioned with entity B, fact superseded by a newer fact), which is Graphiti's bi-temporal graph model, not mem0's typed-memory model. mem0's decay/reinforcement framing solves a different problem than "build a queryable graph of who/what/when." Adding mem0 on top of Neo4j+pgvector was more integration surface for less of the thing we needed, so it was cut and we went straight to Graphiti on the existing Neo4j.

The official Graphiti MCP server image. Graphiti ships an official mcp_server/ and a container image. It does not work against Ollama for our use, for one decisive reason: it does not expose structured_output_mode as a configuration knob (env or otherwise). Graphiti's default client path uses OpenAI structured outputs via json_schema (constrained decoding), which Ollama's OpenAI-compatible endpoint does not support — extraction fails on every episode. The official image gives you no way to switch the client to the generic path. There is also a related limitation that the official server's LLM client is configured post-instantiation (it mutates client.llm_client.config.base_url / .api_key / .model after construction) and doesn't surface the generic-client + JSON-object combination at all. Rather than fork the image, we wrote a ~200-line FastAPI wrapper that constructs the client ourselves. That also let us add MCP via fastmcp in the same process and control the ingestion concurrency model (section 4), which the official image doesn't give you either.

---

4. Bugs and workarounds

4.1 The client config — the one that actually makes Ollama work

This is the core of the whole writeup. Use OpenAIGenericClient, not OpenAIClient, and force structured_output_mode="json_object":

from graphiti_core import Graphiti
from graphiti_core.llm_client import LLMConfig
from graphiti_core.llm_client.openai_generic_client import OpenAIGenericClient
from graphiti_core.embedder.openai import OpenAIEmbedder, OpenAIEmbedderConfig

llm = OpenAIGenericClient(                        # NOT OpenAIClient
    config=LLMConfig(
        api_key="ollama",                         # dummy; Ollama ignores it but the client requires a value
        model="qwen3.6:27b",                      # both model and small_model are required:
        small_model="qwen3.6:27b",                # if unset, Graphiti falls back to gpt-4.1-nano,
        base_url="http://host.docker.internal:11434/v1",   # which does not exist in Ollama → hard failure
    ),
    structured_output_mode="json_object",         # critical: json_object, NOT json_schema
)

embedder = OpenAIEmbedder(
    config=OpenAIEmbedderConfig(
        embedding_model="bge-m3",
        api_key="ollama",
        base_url="http://host.docker.internal:11434/v1",
        embedding_dim=1024,                        # bge-m3 = 1024; must match or Neo4j vector index is wrong
    )
)

graphiti = Graphiti(
    uri="bolt://neo4j:7687",
    user="neo4j",
    password="<your-password>",
    llm_client=llm,
    embedder=embedder,
)
await graphiti.build_indices_and_constraints()    # idempotent; must run at startup (see 4.3)

Why each line matters:

• OpenAIGenericClient + json_object — OpenAIClient emits response_format with a JSON schema (json_schema mode), which Ollama's /v1 endpoint rejects/ignores; extraction then returns unparseable output and every add_episode fails. OpenAIGenericClient with json_object asks only for valid JSON and parses it itself. This is the single switch that makes local extraction work at all.
• Both model and small_model — Graphiti routes some internal/cheaper calls through small_model. LLMConfig's default small_model is gpt-4.1-nano. If you set only model, those calls silently target gpt-4.1-nano, which Ollama doesn't have → failure on a subset of operations that's annoying to diagnose. Set both to the same local tag.
• api_key="ollama" — dummy, but the client constructor still wants a non-empty key. Ollama ignores it.
• embedding_dim=1024 — must match bge-m3's true dimension. Get this wrong and Neo4j's vector index dimension won't match the vectors you write, so similarity search misbehaves.

4.2 Docker networking: container → host Ollama

Ollama runs on the host, the wrapper runs in a container. From inside the container, localhost is the container, not the host, so http://localhost:11434 fails. Use host.docker.internal and publish the gateway route explicitly:

  graphiti:
    container_name: graphiti
    build: ./services/graphiti
    restart: always
    ports:
      - "8001:8001"
    extra_hosts:
      - "host.docker.internal:host-gateway"     # <-- without this, host.docker.internal won't resolve on Linux
    depends_on:
      - neo4j
    environment:
      - NEO4J_URI=bolt://neo4j:7687             # container-to-container: service name, not localhost
      - NEO4J_USER=neo4j
      - NEO4J_PASSWORD=<your-password>
      - OLLAMA_BASE_URL=http://host.docker.internal:11434/v1
      - LLM_MODEL=qwen3.6:27b
      - EMBEDDER_MODEL=bge-m3

Two distinct address regimes, easy to mix up:

• Container → host service (Ollama): host.docker.internal:11434, and you must add extra_hosts: host.docker.internal:host-gateway on Linux or the name doesn't resolve. (The same fix was needed earlier on the rag_api container for the same reason.)
• Container → container (Neo4j): use the compose service name, bolt://neo4j:7687. Both containers are on librechat_default.

Neo4j itself:

  neo4j:
    container_name: neo4j
    image: neo4j:5.26-community
    restart: always
    ports: ["7474:7474", "7687:7687"]
    environment:
      - NEO4J_AUTH=neo4j/<your-password>
      - NEO4J_PLUGINS=["apoc"]                   # Graphiti needs APOC
    volumes:
      - neo4j_data:/data
      - neo4j_logs:/logs

4.3 "node <uuid> not found" on first write

Symptom: add_memory fails with Error calling tool 'add_memory': node <uuid> not found, search_memory returns nothing, the graph stays empty even though MCP is connected and the tools are invoked.

Root cause: we were passing an explicit uuid into add_episode. Graphiti interpreted that as "attach to an existing node with this uuid," went looking for it, didn't find it (empty DB), and failed instead of creating a new node. Fix: do not pass uuid to add_episode — let Graphiti mint it.

Adjacent requirement surfaced during diagnosis: build_indices_and_constraints() must actually run at startup, idempotently, in the FastAPI lifespan hook, against the (possibly empty) database. On a fresh DB the fulltext/range indexes and constraints have to exist before the first write. It's safe to call every boot — it no-ops if they already exist (index or constraint already exists is logged at INFO and is fine). After fixing the uuid passing, the first real write produced 5 Entity + 1 Episodic node and search returned exact facts.

4.4 MCP -32001 timeout — extraction is slower than the transport

Symptom: add_memory over MCP fails with error -32001 (timeout). The graph write does eventually happen, but the MCP client gives up first.

Cause: extraction on qwen3.6:27b takes ~70–80 s per episode (section 5). The MCP timeout in librechat.yaml was 30000 (30 s). Synchronous add_episode blocks the whole call until extraction finishes, so the transport times out every time.

Two-part fix:

1. Raise the MCP timeout in librechat.yaml for the graphiti-memory server: 30000 → 180000 (180 s).

   mcpServers:
     graphiti-memory:
       type: sse
       url: http://graphiti:8001/mcp/sse
       timeout: 180000        # was 30000

2. Make ingestion fire-and-forget. Both the MCP tool add_memory and the REST /messages endpoint now schedule add_episode as a background task and return queued immediately (HTTP 202). The caller never blocks on extraction.

import asyncio

_background_tasks: set[asyncio.Task] = set()   # strong refs — see below

async def _ingest_episode(**kwargs) -> None:
    try:
        await _graphiti.add_episode(**kwargs)   # NOTE: no uuid= here (4.3)
        logger.info("Episode ingested (group=%s, name=%s)", kwargs.get("group_id"), kwargs.get("name"))
    except Exception:
        logger.exception("Background add_episode failed")   # failures go to logs, never swallowed

def _spawn_ingest(**kwargs) -> None:
    task = asyncio.create_task(_ingest_episode(**kwargs))
    _background_tasks.add(task)                 # hold a strong ref
    task.add_done_callback(_background_tasks.discard)

Two non-obvious details that bit us:

• The _background_tasks set is load-bearing. asyncio.create_task only keeps a weak reference to the task. Without holding a strong ref, the event loop can GC an in-flight task mid-run, and background writes vanish silently — no error, just missing nodes. The set keeps them alive until they finish.
• Wrap the background body in try/except + logger.exception. Fire-and-forget means a crash in the task has nowhere to surface; without the wrapper, a failed extraction disappears with no trace. With it, failures land in the graphiti container logs and successes log Episode ingested.

After this, the API returns {"queued": 1} in ~0.01 s and the graph fills in asynchronously.

---

5. Metrics and what we tried for speed

Numbers from the working setup (single RTX 5090, qwen3.6:27b Q-quantized 17 GB model, bge-m3 embeddings, one episode at a time):

Measurement	Value
Extraction latency per episode	~70–80 s
API response latency (after async fix)	~0.01 s (returns queued, work continues in background)
Graph growth from one short fact	+4 Entity, +1 Episodic (count 6 → 11)
Old MCP timeout (failed)	30 s — always lost the race
New MCP timeout	180 s

Verification method: snapshot MATCH (n) RETURN count(n) before, submit one fact, poll every 10 s. Count held at 6 through t=70 s and jumped to 11 at t=80 s — i.e. the whole 70–80 s is the model doing extraction, after which the graph commit is effectively instant.

Note that the ~70–80 s figure is for a short single fact; per-episode latency scales with content density, not a fixed cost — a light paragraph ran ~140 s, and a dense multi-entity episode (a book passage yielding 7–8 Entity + 1 Episodic) took ~350 s, so on bulk/batch ingestion the realistic planning number is the hundreds-of-seconds end of that range, not 70–80 s.

On "speeding it up": be honest — we did not make extraction faster, we made it not matter. The ~70–80 s is dominated by a 27B model running Graphiti's multi-step extraction prompts on one GPU; that's the real cost of cloud-free structured extraction at this model size. The lever we actually pulled was latency hiding: async fire-and-forget ingestion so the user-facing call returns immediately and the graph catches up in the background, plus a generous transport timeout so nothing in the chain gives up. That's the right trade for a memory layer (writes are not interactive), but it does not reduce GPU time.

Levers we identified but did not implement, in rough order of expected payoff — treat these as untested hypotheses if you're reproducing:

• Split model / small_model — point small_model at a genuinely smaller local model (e.g. a 3B/7B) for Graphiti's cheaper internal calls, keeping the 27B only for the main extraction. We set both to the same tag for correctness first; this is the obvious next optimization.
• Keep the model warm — ensure Ollama isn't paying cold-load on each episode (model resident, keep_alive tuned). Part of the per-episode time may be load, not compute.
• Batch / concurrency — the async design already lets multiple episodes ingest concurrently; whether Ollama on one GPU benefits or just serializes is unverified.
• Smaller / more quantized extraction model — directly trades extraction quality for speed; risky, since below ~27B the structured-output reliability that drove the model choice degrades.

---

6. What's next

• Reduce real extraction cost, starting with the small_model split above — the single biggest knob, and currently untouched.
• Wire the reranker. OpenAIRerankerClient (graphiti_core.cross_encoder.openai_reranker_client) is imported but search currently runs without a tuned reranking pass; retrieval quality on a larger graph will need it.
• Multi-group memory. Everything currently lands in group_id="default". Per-user / per-project groups are supported by the API (group_id is already threaded through add_memory, search_memory, get_recent_memories, clear_memory) but not yet used by the front end.
• Confirm the MCP path end-to-end from the agent UI. All ingest testing went through the REST /messages endpoint (same background code as the MCP tool) because the agent UI couldn't be driven programmatically; the MCP tool path should be confirmed once from a real chat.
• The dispatcher layer (B2). The broader build calls for a coordinator model that decides what is worth committing to memory, rather than storing every turn — otherwise the 70–80 s/episode cost and graph noise both grow unbounded.

---

Appendix: minimal repro checklist

1. Neo4j 5.26 with APOC, reachable at bolt://neo4j:7687 on the compose network.
2. Ollama on the host with a 27B-class extraction model and bge-m3; container reaches it via host.docker.internal:11434 + extra_hosts: host.docker.internal:host-gateway.
3. graphiti-core==0.29.2, client = OpenAIGenericClient, structured_output_mode="json_object", both model and small_model set, embedding_dim=1024.
4. build_indices_and_constraints() in the FastAPI lifespan; never pass uuid to add_episode.
5. Ingest async (background task + strong-ref set + try/except logging); MCP timeout: 180000.
6. Validate with a before/after MATCH (n) RETURN count(n); expect the count to rise ~70–80 s after submitting a fact.