A pattern for building personal knowledge bases using LLMs.
This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.
Most people's experience with LLMs and documents looks like RAG: you upload a collection of files, the LLM retrieves relevant chunks at query time, and generates an answer. This works, but the LLM is rediscovering knowledge from scratch on every question. There's no accumulation. Ask a subtle question that requires synthesizing five documents, and the LLM has to find and piece together the relevant fragments every time. Nothing is built up. NotebookLM, ChatGPT file uploads, and most RAG systems work this way.
The idea here is different. Instead of just retrieving from raw documents at query time, the LLM incrementally builds and maintains a persistent wiki — a structured, interlinked collection of markdown files that sits between you and the raw sources. When you add a new source, the LLM doesn't just index it for later retrieval. It reads it, extracts the key information, and integrates it into the existing wiki — updating entity pages, revising topic summaries, noting where new data contradicts old claims, strengthening or challenging the evolving synthesis. The knowledge is compiled once and then kept current, not re-derived on every query.
This is the key difference: the wiki is a persistent, compounding artifact. The cross-references are already there. The contradictions have already been flagged. The synthesis already reflects everything you've read. The wiki keeps getting richer with every source you add and every question you ask.
You never (or rarely) write the wiki yourself — the LLM writes and maintains all of it. You're in charge of sourcing, exploration, and asking the right questions. The LLM does all the grunt work — the summarizing, cross-referencing, filing, and bookkeeping that makes a knowledge base actually useful over time. In practice, I have the LLM agent open on one side and Obsidian open on the other. The LLM makes edits based on our conversation, and I browse the results in real time — following links, checking the graph view, reading the updated pages. Obsidian is the IDE; the LLM is the programmer; the wiki is the codebase.
This can apply to a lot of different contexts. A few examples:
- Personal: tracking your own goals, health, psychology, self-improvement — filing journal entries, articles, podcast notes, and building up a structured picture of yourself over time.
- Research: going deep on a topic over weeks or months — reading papers, articles, reports, and incrementally building a comprehensive wiki with an evolving thesis.
- Reading a book: filing each chapter as you go, building out pages for characters, themes, plot threads, and how they connect. By the end you have a rich companion wiki. Think of fan wikis like Tolkien Gateway — thousands of interlinked pages covering characters, places, events, languages, built by a community of volunteers over years. You could build something like that personally as you read, with the LLM doing all the cross-referencing and maintenance.
- Business/team: an internal wiki maintained by LLMs, fed by Slack threads, meeting transcripts, project documents, customer calls. Possibly with humans in the loop reviewing updates. The wiki stays current because the LLM does the maintenance that no one on the team wants to do.
- Competitive analysis, due diligence, trip planning, course notes, hobby deep-dives — anything where you're accumulating knowledge over time and want it organized rather than scattered.
There are three layers:
Raw sources — your curated collection of source documents. Articles, papers, images, data files. These are immutable — the LLM reads from them but never modifies them. This is your source of truth.
The wiki — a directory of LLM-generated markdown files. Summaries, entity pages, concept pages, comparisons, an overview, a synthesis. The LLM owns this layer entirely. It creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. You read it; the LLM writes it.
The schema — a document (e.g. CLAUDE.md for Claude Code or AGENTS.md for Codex) that tells the LLM how the wiki is structured, what the conventions are, and what workflows to follow when ingesting sources, answering questions, or maintaining the wiki. This is the key configuration file — it's what makes the LLM a disciplined wiki maintainer rather than a generic chatbot. You and the LLM co-evolve this over time as you figure out what works for your domain.
Ingest. You drop a new source into the raw collection and tell the LLM to process it. An example flow: the LLM reads the source, discusses key takeaways with you, writes a summary page in the wiki, updates the index, updates relevant entity and concept pages across the wiki, and appends an entry to the log. A single source might touch 10-15 wiki pages. Personally I prefer to ingest sources one at a time and stay involved — I read the summaries, check the updates, and guide the LLM on what to emphasize. But you could also batch-ingest many sources at once with less supervision. It's up to you to develop the workflow that fits your style and document it in the schema for future sessions.
Query. You ask questions against the wiki. The LLM searches for relevant pages, reads them, and synthesizes an answer with citations. Answers can take different forms depending on the question — a markdown page, a comparison table, a slide deck (Marp), a chart (matplotlib), a canvas. The important insight: good answers can be filed back into the wiki as new pages. A comparison you asked for, an analysis, a connection you discovered — these are valuable and shouldn't disappear into chat history. This way your explorations compound in the knowledge base just like ingested sources do.
Lint. Periodically, ask the LLM to health-check the wiki. Look for: contradictions between pages, stale claims that newer sources have superseded, orphan pages with no inbound links, important concepts mentioned but lacking their own page, missing cross-references, data gaps that could be filled with a web search. The LLM is good at suggesting new questions to investigate and new sources to look for. This keeps the wiki healthy as it grows.
Two special files help the LLM (and you) navigate the wiki as it grows. They serve different purposes:
index.md is content-oriented. It's a catalog of everything in the wiki — each page listed with a link, a one-line summary, and optionally metadata like date or source count. Organized by category (entities, concepts, sources, etc.). The LLM updates it on every ingest. When answering a query, the LLM reads the index first to find relevant pages, then drills into them. This works surprisingly well at moderate scale (~100 sources, ~hundreds of pages) and avoids the need for embedding-based RAG infrastructure.
log.md is chronological. It's an append-only record of what happened and when — ingests, queries, lint passes. A useful tip: if each entry starts with a consistent prefix (e.g. ## [2026-04-02] ingest | Article Title), the log becomes parseable with simple unix tools — grep "^## \[" log.md | tail -5 gives you the last 5 entries. The log gives you a timeline of the wiki's evolution and helps the LLM understand what's been done recently.
At some point you may want to build small tools that help the LLM operate on the wiki more efficiently. A search engine over the wiki pages is the most obvious one — at small scale the index file is enough, but as the wiki grows you want proper search. qmd is a good option: it's a local search engine for markdown files with hybrid BM25/vector search and LLM re-ranking, all on-device. It has both a CLI (so the LLM can shell out to it) and an MCP server (so the LLM can use it as a native tool). You could also build something simpler yourself — the LLM can help you vibe-code a naive search script as the need arises.
- Obsidian Web Clipper is a browser extension that converts web articles to markdown. Very useful for quickly getting sources into your raw collection.
- Download images locally. In Obsidian Settings → Files and links, set "Attachment folder path" to a fixed directory (e.g.
raw/assets/). Then in Settings → Hotkeys, search for "Download" to find "Download attachments for current file" and bind it to a hotkey (e.g. Ctrl+Shift+D). After clipping an article, hit the hotkey and all images get downloaded to local disk. This is optional but useful — it lets the LLM view and reference images directly instead of relying on URLs that may break. Note that LLMs can't natively read markdown with inline images in one pass — the workaround is to have the LLM read the text first, then view some or all of the referenced images separately to gain additional context. It's a bit clunky but works well enough. - Obsidian's graph view is the best way to see the shape of your wiki — what's connected to what, which pages are hubs, which are orphans.
- Marp is a markdown-based slide deck format. Obsidian has a plugin for it. Useful for generating presentations directly from wiki content.
- Dataview is an Obsidian plugin that runs queries over page frontmatter. If your LLM adds YAML frontmatter to wiki pages (tags, dates, source counts), Dataview can generate dynamic tables and lists.
- The wiki is just a git repo of markdown files. You get version history, branching, and collaboration for free.
The tedious part of maintaining a knowledge base is not the reading or the thinking — it's the bookkeeping. Updating cross-references, keeping summaries current, noting when new data contradicts old claims, maintaining consistency across dozens of pages. Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don't get bored, don't forget to update a cross-reference, and can touch 15 files in one pass. The wiki stays maintained because the cost of maintenance is near zero.
The human's job is to curate sources, direct the analysis, ask good questions, and think about what it all means. The LLM's job is everything else.
The idea is related in spirit to Vannevar Bush's Memex (1945) — a personal, curated knowledge store with associative trails between documents. Bush's vision was closer to this than to what the web became: private, actively curated, with the connections between documents as valuable as the documents themselves. The part he couldn't solve was who does the maintenance. The LLM handles that.
This document is intentionally abstract. It describes the idea, not a specific implementation. The exact directory structure, the schema conventions, the page formats, the tooling — all of that will depend on your domain, your preferences, and your LLM of choice. Everything mentioned above is optional and modular — pick what's useful, ignore what isn't. For example: your sources might be text-only, so you don't need image handling at all. Your wiki might be small enough that the index file is all you need, no search engine required. You might not care about slide decks and just want markdown pages. You might want a completely different set of output formats. The right way to use this is to share it with your LLM agent and work together to instantiate a version that fits your needs. The document's only job is to communicate the pattern. Your LLM can figure out the rest.
https://github.com/ojuschugh1/sqz
Pre-injection context compression and session deduplication for AI coding agents
Real session stats: 3,003 compressions · 178,442 tokens saved · 24.7% avg reduction · up to 92% on output the model already has
Docs · Install · How It Works · Supported Tools · MCP Proxy · Benchmark · Changelog · Discord
AI coding agents spend most of their input tokens on tool output: build logs, test runs,
git status, the same file read again after every edit. All of it goes into the context window raw, and it is re-sent on every turn after that.sqz compresses tool output before it enters the context. Per-command formatters keep what an agent acts on (the failing test, the assertion, the
file:line) and drop what it does not. Anything the model already has in context comes back as a short reference instead of the content. Source code, stack traces and secrets pass through untouched.Real output from the release binary (
assets/demo.tape). Regenerate withvhs assets/demo.tape.Single Rust binary, deterministic, no LLM calls, works offline. Every compressed result can be recovered byte-exact with
sqz expand.Note
Name disambiguation: this repo, ojuschugh1/sqz, is an independent project and is not affiliated with any other similarly named or working tool or other compression projects that shorten "squeeze". If you installed
sqz/sqz-cli/sqz-mcpfrom crates.io, npm, PyPI, or Homebrew, it comes from this repository.Token Savings
One developer's week, measured from actual
sqz gainoutput:Per-command compression
Single-command compression (measured via
cargo test -p sqz-engine benchmarks):Session-level with dedup
Anything the model already has in its context comes back as a reference. Measured with the release binary against a throwaway database, using the fixtures in
sqz/tests/quality_bench.rsanddemo/:git statusrun againcargo testoutput, nothing changedA word on what actually repeats. A user who counted 92 of their own Claude Code sessions (script) found identical whole-file re-reads in 1 of 542 reads, and line-range re-reads in 8.4% of them. So sqz does not lead with "the same file read five times": the repeats that happen in practice are unchanged command output, line ranges of a file already read, and files re-read after a small edit, and each has its own reference type (details). Sessions with noisy command output and repeated commands see the biggest wins.
Install
Prebuilt binaries (no compiler required — works on every platform):
Build from source via Cargo:
sqz-cliprovides thesqzbinary;sqz-mcpprovides the MCP server.sqz-engineis a library dependency — it compiles automatically and does not need to be installed separately.Build from source (
cargo install sqz-cli) works too, but needs a C toolchain:build-essential(apt) or equivalentxcode-select --install)cargo installfails withlinker link.exe not found. If you don't already have them, use the PowerShell or npm install above instead.Then initialize:
--globalwrites to~/.claude/settings.json(the user scope per theAnthropic scope table),
so the sqz hook fires in every Claude Code session on this machine. This is
the common case on first install. Your existing
permissions,env,statusLine, and unrelated hooks in~/.claude/settings.jsonarepreserved — sqz merges its entries rather than overwriting.
Plain
sqz init(project scope) is useful when you want sqz active onlyinside one repo.
After installing,
sqz doctorchecks the whole chain — binary, database,shell hook, which clients are detected vs actually routed through sqz, and
whether anything was compressed recently — and prints the fix for each gap
it finds.
See what sqz would have saved, before installing any hook
If you already use Claude Code or Kiro, your transcripts are on disk. Replay
them through the sqz engine and get a counterfactual estimate on your own
sessions — computed locally, nothing uploaded:
$ sqz discover --replay sqz discover — counterfactual replay (last 7 days) ──────────────────────────────────────────────────────── Sessions replayed: 24 Tool outputs: 43 Tokens (original): 116,598 Tokens (compressed): 92,560 Estimated avoidable: 24,038 tokens (20.6%) Largest opportunities: source tokens in avoidable web_fetch 58,102 12,288 read 37,085 10,486It scans
~/.claude/projectsand~/.kiro/sessions/cli, or point itanywhere with
--transcripts PATH. Each session replays against a freshthrowaway cache, so dedup references never pretend to span sessions, and the
report says "estimate" because that's what it is: these outputs were not
compressed at the time.
Only using one agent? Pass
--only(or--skip) to limit whichconfigs are written:
Accepted names:
claude,cursor,windsurf,cline,gemini,kiro,opencode,codex. Aliases (claude-code,gemini-cli,roo,kiro-cli) also work.--onlyand--skipcan't be combined.Manual installation (preserve comments in your config)
sqz initround-trips your config file through a JSON parser to mergethe sqz entry, which drops any comments in your
opencode.jsonc(andthe analogous JSON-with-comments files other tools accept). If you've
commented your config carefully and want to keep them, install by hand
instead.
OpenCode — two steps:
Drop the plugin file in place.
sqzprints the generated TS tostdout so you don't have to hand-write the path-escaping logic:
Add the MCP entry to your existing
opencode.jsoncyourself.Append this block inside the top-level
mcpobject (create themcpobject if it doesn't exist):Comments in the rest of your file stay put. OpenCode auto-discovers
the plugin file; no
pluginarray entry needed (adding one causesdouble-loading, see issue #10).
Other tools — Claude Code, Cursor, Windsurf, Cline, Gemini CLI,
and Codex use plain JSON configs without comment support, so the
automated path is non-destructive there. Use
sqz init --only <tool>for those.
That's it. Shell hooks installed, AI tool hooks configured.
How It Works
sqz installs a PreToolUse hook that intercepts bash commands before your AI tool runs them. The output gets compressed transparently — the AI tool never knows.
What gets compressed:
What doesn't get compressed:
Supported Tools
sqz initsqz initsqz initsqz initsqz initsqz initsqz initsqz initsqz initsqz initsqz init --ci+ commitsqz-mcp proxywraps it, compresses its tool resultsCompress Any MCP Server
sqz-mcp proxysits between your agent and any stdio MCP server and compresseswhat flows back: tool results go through the full sqz pipeline (dedup refs on
repeats, safe-mode for stack traces and secrets, error results untouched), and
verbose tool descriptions get compacted so
tools/liststops eating yourcontext window. Everything stays reversible — the proxy injects an
sqz_expandtool so the agent can recover any original byte-exact.
Wrap a server by prefixing its command in your MCP config:
{ "mcpServers": { "github": { "command": "sqz-mcp", "args": ["proxy", "--", "npx", "-y", "@modelcontextprotocol/server-github"] } } }Flags:
--lazy-toolsshortens every tool description to one sentence andinjects an
sqz_tool_helptool that serves the full original docs on demand(some MCP servers spend 10-40k tokens on
tools/listalone);--no-desckeeps descriptions verbatim;
--no-cachedisables dedup refs. Works withevery MCP client (Claude Code, Cursor, Windsurf, Zed, Codex, Kiro, ...)
because the client just sees a normal MCP server. Full guide, including the
server's own
sqz_read_file/sqz_grep/sqz_list_dirtools and per-clientconfig: MCP context compression.
CLI
Dedup Escape Hatch
When sqz sees the same content twice, it returns a compact
§ref:HASH§tokeninstead of the full text. Most models handle this fine, but some (e.g., GLM 5.1)
can't parse the ref format and loop. Four ways to work around this:
Recall: search everything sqz has seen
Everything that flows through sqz is indexed locally (SQLite FTS5, on your
machine, nothing leaves it). When your agent compacts its context and loses
that error message from an hour ago, search for it instead of re-running
the command:
Agents get the same thing as the
sqz_recallMCP tool.Track Your Own Savings
Run
sqz gainin your shell any time to see your own daily breakdown (see theToken Savings section above for what the output looks like), and
sqz statsfor the full cumulative report:
$ sqz stats 📊 sqz compression stats ────────────────────────────────────────────────── 178,442 tokens saved ↓ 24.7% average reduction Compressions 3,003 Tokens in 721,840 Tokens out 543,398 Tokens saved 178,442 Avg reduction 24.7% 🗄️ Cache ────────────────────────────────────────────────── Entries 43 Size 39.1 KBAdd
--breakdownto see exactly which commands consume the most tokens:Honest accounting: cost and regret
Token reduction is not automatically billed-cost reduction — with provider
prompt caching, most context re-transmits at a ~90% discount, and compression
that drops something the agent needed costs extra turns instead
(arXiv:2607.12161). sqz measures both
sides instead of hand-waving:
sqz stats --costestimates dollars saved under a prompt-cached billingmodel (cache write ×1.25 once, cache read ×0.10 per subsequent turn), with
every assumption printed and overridable (
--price-in,--reread-turns,--cache-write-mult,--cache-read-mult). This is the conservativeestimate: without caching the same savings would bill at full input price
on every turn.
sqz statsreports regret signals: quick re-runs (the agent re-producedbyte-identical output within 2 minutes — the repeat bought nothing) and
ref expands (
§ref§tokens recovered to original bytes). Both are proxiesfor "compression dropped something the model needed." If one command
keeps showing up, its formatter needs work — file an issue.
sqz's design already avoids the failure modes that study measured:
compression is deterministic and query-agnostic (never invalidates provider
prompt caches), only new tool output is touched (history is never rewritten),
piped or redirected commands are never intercepted, and the 16-token net-win
gate skips compressions that wouldn't pay for their own markers.
Per-project filtering:
Stats are stored locally in SQLite under
~/.sqz/sessions.db— nothing leaves your machine.How Compression Works
Per-command formatters — 45+ commands across 12 ecosystems get purpose-built compression:
-jsonstream), go build, go vet, golangci-lintUnknown commands fall through to the generic compression pipeline — no output is ever left uncompressed.
Table compactor — aligned-column output from tools without a dedicated formatter (
ps aux,netstat, database CLIs) collapses its padding runs to two-space separators. The detector is strict — indented lines, code, YAML, and JSON never match.Structural summaries — code files compressed to imports + function signatures + call graph (~70% reduction). The model sees the architecture, not implementation noise.
Dedup cache — SHA-256 content hash, persistent across sessions. Byte-identical output the model already has = 13-token reference. A line range of a file the model already has in full (
sed -n '40,80p',sqz_read_filewithoffset/limit) =§ref:HASH:L40-80§. A re-read after a small edit = only the changed lines. References are only served while the original is still in context (30-minute window, tunable viaSQZ_REF_TTL_SECS, reset on compaction).JSON pipeline — strip nulls → project out debug fields → flatten → collapse arrays → TOON encoding (lossless compact format)
Safe mode — stack traces, secrets, migrations detected by entropy analysis and routed through with 0% compression
Measured quality, including what each stage drops and what it never touches: quality benchmark. For the full technical details, see docs/.
Configuration
Per-project database
Everything sqz persists (stats, dedup cache, sessions) lives in one SQLite
file,
~/.sqz/sessions.dbby default. SetSQZ_DB_PATHto keep itper-project instead:
Every surface honors it — the shell hook,
sqz stats/gain/expand, andthe MCP server (set it under
envin your MCP config). Prefer absolutepaths: relative values resolve against whatever directory the process runs
from. The parent directory is created if missing (mode 0700, like
~/.sqz).Privacy
Development
License
Elastic License 2.0 (ELv2) — use, fork, modify freely. Two restrictions: no competing hosted service, no removing license notices.
Links
mcp-name: io.github.ojuschugh1/sqzStar History
Contributors
Thanks to everyone who has contributed code, fixes, and ideas to sqz:
And to everyone who filed the detailed bug reports behind our fixes. Precise repros make this project better with every release. Want to join them? PRs are reviewed fast: see the open issues to get started.
Contributor grid made with contrib.rocks.