Skip to content

Instantly share code, notes, and snippets.

@joonan30
Last active August 31, 2026 08:31
Show Gist options
  • Select an option

  • Save joonan30/cbce305684d079dbe9a3fbaefe4e3959 to your computer and use it in GitHub Desktop.

Select an option

Save joonan30/cbce305684d079dbe9a3fbaefe4e3959 to your computer and use it in GitHub Desktop.
LLM Wiki: building a compounding knowledge base of academic papers with AI agents (revised 2026-08)
# LLM Wiki — [YOUR FIELD]
A personal knowledge base of [YOUR FIELD] papers, following [Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/1dd0294ef9567971c1e4348a90d69285):
```
Original PDF → sources/*.md → wiki/{category}/*.md → wiki/overviews/ + wiki/concepts/
```
Language policy: all wiki content is in English. Conversation can be in any language.
This file is the single source of truth for how agents work in this folder. Claude Code and Codex both read it — keep `CLAUDE.md` as a symlink to this file (`ln -s AGENTS.md CLAUDE.md`) and keep the wording agent-neutral.
---
## Startup checklist
At the start of every session in this folder, before acting on any ingest, search, or synthesis request:
1. Re-read this file.
2. Re-read [any registries or mapping files that change often — list them here].
3. Do not rely on memory from a previous session for anything that moves.
---
## The rules
### The four core rules
1. No web search. Never use `WebSearch` or `WebFetch` to fill a gap. Every answer must be grounded in papers actually present here. Use them only when the user explicitly asks, for that one task.
2. Answer from the wiki first. `sources/` and `wiki/` are the only sources of truth.
3. If the wiki is insufficient, re-read the original PDF in `papers/` and extract more detail. Then update the wiki so the next question does not need the PDF.
4. If no paper exists on the topic, say so: "I don't have a paper on this — please give me the PDF." Do not improvise.
These apply to every response, overview pages included: cite only papers that exist in this wiki.
### Five more rules
5. A paper is not ingested until it is connected to the synthesis layer. See "The synthesis requirement" below.
6. No tiers, no placeholder pages. Every admitted paper meets the same standard: read from the original, with concrete methods, results, limitations, and synthesis links. Never create a page from an abstract, a metadata record, a reference-list mention, or a filename. If the original cannot support a complete page, leave the paper out and record the gap in the day's log.
7. Exhaustive means exhaustive. When the user asks for all of something, process the whole set before reporting. State the denominator and the numerator. Count extraction failures, empty text, and timeouts as unchecked, not as passing. Never sample and report as if you surveyed. Never negotiate the scope down because it is large — start, checkpoint, and finish.
8. Confidential material stays out of `wiki/`. Unpublished manuscripts, drafts under review, and embargoed work must not appear in `wiki/` at all — not as a page, not as a title, not as a tag, not as a "relevance to our manuscript" section on a published paper's page. Those notes go in `agenda/`. Published work with a DOI is ordinary wiki content.
9. PDFs only. The canonical full text of a paper is an exact PDF in `papers/`. Never save publisher HTML or a browser text snapshot as a substitute. If an exact PDF cannot be obtained, keep the paper on a missing-PDF list rather than ingesting a degraded copy.
Also: correction, erratum, and retraction notices are not papers. Titles beginning `Author Correction:`, `Publisher Correction:`, `Correction:`, `Erratum`, or `Retraction` never get a page. Match the correction word only when followed by `:` or `to`, so real papers about correcting something survive.
[Optional: list any publishers or venues you have decided not to cite, and enforce it with a DOI-prefix sweep plus a body-text scan — a preprint DOI will pass a DOI check even when the paper was published in an excluded venue.]
---
## Repository structure
```
your-llm-wiki/
├── AGENTS.md # This file. CLAUDE.md is a symlink to it
├── index.md # Generated catalog: categories, page counts, synthesis coverage
├── indexes/ # Generated per-category page listings
├── logs/ # Daily work logs — the narrative record
│ ├── {YYYY-MM-DD}-{agent}-{host}.md
│ ├── README.md # generated folder guide and topic index
│ └── reports/ # audit reports written for a person to read
├── scripts/ # Extraction, validation, index and log builders
├── papers/ # Original PDFs, canonical storage (cp, never symlink)
├── papers-supplementary/ # Publisher supplements + manifests
├── sources/ # PDF summaries, English
├── wiki/ # Knowledge layer, English. Obsidian vault root
│ ├── {category}/
│ ├── concepts/
│ ├── overviews/
│ └── questions/
├── agenda/ # Project execution notes, plans, handoffs, unpublished work
└── materials/ # Non-paper reference material
```
### The three synthesis layers
- `overviews/` — encyclopedic topic pages. Declarative noun-phrase titles. Never a question title.
- `concepts/` — stable definitions, methods, mechanisms. Written once, pointed at by many paper pages.
- `questions/` — paper-anchored research questions. The title is a question; the body has `## Question`, `## Sharper follow-up`, `## What the knowledge base holds`, `## Tentative answer`.
A question-form synthesis page belongs in `questions/`, never in `overviews/`.
### `agenda/`
`agenda/{project-or-context}/` holds work-facing notes: execution plans, handoff documents, analysis backlogs, decision records, meeting follow-ups, training material, and notes on unpublished manuscripts. No schema requirements.
A handoff document includes: why the method or paper is needed and what it enables, the analyses to run, the expected results, a decision section stating when to continue or pause or stop if results are weak, and the concrete deliverables. No agent-workflow notes, no personal absolute paths, no relative-date titles.
Put durable decisions about how this wiki operates in `agenda/llm-wiki-ops/` as short ADR-style notes.
### `materials/`
Non-paper reference material, with no filename convention, no category, and no PDF frontmatter requirement: journal scope and author-guideline snapshots, writing-style profiles, lecture notes, curated meeting notes, professional-activity records.
Do not link `materials/` from `wiki/`, `sources/`, or `index.md`. It is outside the paper knowledge graph by design. Keep raw audio and raw transcripts outside this repository entirely; only curated notes come in.
---
## File naming convention
All three tiers (PDF, source, wiki) share one stem:
```
{first-author-lastname}-{year}-{first-5-title-tokens}.{ext}
```
- Tokens, not words. Lowercase the title, then take runs of alphanumerics. Every non-alphanumeric character is a separator: `self-report` is two tokens. Digits count.
- Year is 4 digits. Consortium papers use the consortium name.
- Let the ingest script generate the stem. Do not hand-build it.
- Do not bulk-rename legacy stems to match a newer rule — every wikilink and index entry would have to follow. Fix only the cases where one paper's three tiers disagree with each other.
Example: `pollard-2006-an-rna-gene-expressed`
---
## Categories
> Edit this section. Define 5–10 categories that match your domain. Start small; split when one passes ~500 files.
| Category | Includes |
|---|---|
| `[your-category-1]` | [what goes here, and what deliberately does not] |
| `[your-category-2]` | [...] |
| `concepts` | Methods, mechanisms, reusable definitions |
| `overviews` | Encyclopedic synthesis pages |
| `questions` | Paper-anchored research-question pages |
| `other` | Cross-cutting, miscellaneous |
Classify by method, not topic: a methylation paper studying a phenotype goes to `methylation`, not to the phenotype's category. Write the exclusions into each row — "what would I deliberately leave out when reading about this?" is the question that keeps categories from blurring.
---
## Adding a paper
### Step 0 — locate the PDF
If the user did not give a path, look in `~/Downloads/`. Identify each file from its first page (title, authors, DOI). If a mapping is ambiguous, ask instead of guessing. Never move, delete, or overwrite files in the user's directories.
### Step 1 — copy to `papers/` and extract text
Copy, never symlink. `pdf_path` is always an absolute path inside `papers/`; `pdf_filename` matches its basename. Never put an external path in `pdf_path`.
Default extractor is [opendataloader-pdf](https://github.com/opendataloader-project/opendataloader-pdf): it reconstructs two-column paragraphs as continuous prose, separates figure captions, tidies references, and strips page furniture, where simpler extractors interleave figure-panel scaffolding into the body. It needs Java. Wrap it in a helper that discovers a working Python and Java on the current machine, applies a timeout, and falls back to `pypdf` then `pdftotext`.
```bash
pdfinfo "/path/to/paper.pdf" | grep Pages # check length first
bash scripts/extract_pdf_text.sh "/path/to/paper.pdf" 1-25 400000
```
The arguments are a page range and a character cap. A small character cap truncates the paper mid-Methods and still reports success. Be generous with the cap and widen the range for long PDFs.
For gene or relation extraction rather than summarization, PyMuPDF is the better tool: it keeps columns pure, needs no Java, and is much faster.
Clean the raw output with a bounded contract — list exactly what may be removed (page furniture, running headers, line-break hyphenation, empty pseudo-tables) and remove nothing else.
Record the provenance. Mandatory on every new or re-extracted record:
```yaml
source_format: pdf
text_extractor: opendataloader-pdf-{version}
text_extracted_date: YYYY-MM-DD
```
Never ingest without naming the tool. Update these fields on re-extraction, and note the re-extraction in the day's log rather than in the page body. Do not invent values for legacy records that lack them.
### Step 2 — write `sources/{stem}.md`
```yaml
---
title: "Paper Title"
authors: Author List
year: YYYY
doi: DOI
category: [your-category]
pdf_path: /absolute/path/to/papers/{stem}.pdf
pdf_filename: {stem}.pdf
source_collection: publisher-or-repository-collection
source_format: pdf
text_extractor: opendataloader-pdf-{version}
text_extracted_date: YYYY-MM-DD
---
## One-line Summary
## 1. Document Information
## 2. Key Contributions
## 3. Methodology and Architecture
## 4. Key Results and Benchmarks
## 5. Limitations and Future Work
## 6. Related Work
## 7. Glossary
```
That section list is the whole schema. Do not add a section holding the raw extraction dump — the canonical full text is the PDF, and copying it in drags publisher boilerplate into the search index.
`source_collection` names a stable source class such as a publisher or repository collection. Processing metadata is forbidden in paper frontmatter: no `tier`, no `curation_level`, no batch or run names, no agent names, no dated batch identifiers, no status tags. Batch processing is an execution strategy and must be invisible in the knowledge layer; run provenance belongs in the log.
### Step 3 — write `wiki/{category}/{stem}.md`
```yaml
---
title: "Exact English Title"
authors: Author list
year: YYYY
doi: DOI
source: {stem}.md
category: [your-category]
pdf_path: /absolute/path/to/papers/{stem}.pdf
pdf_filename: {stem}.pdf
source_collection: publisher-or-repository-collection
source_format: pdf
text_extractor: opendataloader-pdf-{version}
text_extracted_date: YYYY-MM-DD
tags: []
---
## Summary
## Key Contributions
## Methodology and Architecture
## Results
## Related Papers
- [[overviews/some-page]] — relationship
```
### Step 4 — connect to the synthesis layer
Mandatory. See the next section.
### Step 5 — log it, then rebuild the indexes
Append one entry to the day's log file with a topic slug in the heading, then regenerate the catalog and the log index:
```bash
python3 scripts/build_hierarchical_index.py --all --apply
python3 scripts/build_logs_index.py --apply
```
Do not put ingest narrative into `index.md`.
---
## The synthesis requirement
A source and wiki pair with no link into `overviews/` or `concepts/` is a synthesis-orphan. It behaves like a RAG cache entry, not a wiki, and defeats the purpose of the system. Every ingest, routine ones included, must end with:
1. At least one bidirectional link between the new page and the single most relevant existing overview or concept page — `Related Papers` on the paper, `Related Pages` on the synthesis page. One-way links are invisible from the side that would use them.
2. One sentence in that synthesis page's body placing the paper in context. Not just a bullet in a list. If the paper only restates what the page already says, a link is enough, but say so in the log.
3. A supersede check: does the paper strengthen, narrow, contradict, or replace an existing claim? If it contradicts or replaces one, append a `supersede` entry to the day's log.
4. If no overview or concept covers the topic, record the gap in the day's log instead of skipping silently. Create the anchor when the topic is clearly recurring.
Audit the drift with a read-only script that lists orphans. That list is the backlog.
### High-impact mode
When a paper can change an active argument, review, benchmark interpretation, or project decision:
- Process one paper, or a very small set, at a time.
- Read the relevant existing synthesis pages before writing.
- Update the overview and concept page bodies, not just the links.
- Create the concept anchor if one is missing.
- State the claim delta explicitly: strengthen, narrow, contradict, replace, or unchanged.
- If the paper's conclusion lives in a figure (heatmap direction, effect-size plot, architecture diagram), render that figure to PNG and read it as an image rather than trusting text extraction.
- If the right emphasis is unclear, ask for a short interpretation checkpoint rather than burying the decision inside a batch.
---
## Batch ingest
Batch size never changes who writes the pages: the active agent in the current conversation reads each paper and writes each source and wiki page. Do not spawn another model CLI or model API from an ingest script. Local scripts do deterministic extraction, metadata checks, validation and repair, logging, and index rebuilds — nothing else.
1. Fix the total count first. Identify title, authors, DOI, and document type from a one-page extraction.
2. Check for duplicates by DOI across all of `sources/` and `wiki/`.
3. Filter out correction notices and excluded venues, then copy the survivors into `papers/` under the naming convention.
4. Extract full text and write complete, original-grounded source and wiki pairs.
5. Add synthesis links and context sentences for every paper, and run the supersede check.
6. Run the closing sweeps and the validator. Extraction failures, empty text, and timeouts count as unchecked, not as passing.
7. Report so that completed plus excluded plus unchecked equals the starting count, and append it to the day's log.
8. Rebuild the generated Markdown catalogs. Leave routine BM25 freshness to the next scheduled local rebuild; run an immediate rebuild only when the user needs the new pages to be searchable now.
Closing sweeps must return zero matches before the batch is done — a title-based filter in step 3 leaks, because a preprint DOI passes a publisher-DOI check even when the paper was published in an excluded venue. Scan the body text as well as the DOI.
---
## What the extractor gets wrong
Every one of these reports success:
- Glyphs substituted rather than dropped, especially Greek letters in proper nouns.
- Minus signs lost. Undetectable from the text alone. If a sign matters, look at the PDF.
- Table columns reordered or merged, binding numbers to the wrong label.
- Figure reading order scrambled, binding values to the wrong axis label. Color is never extracted at all.
- The body silently emptied on some font runs. Corruption is per font run, not per file — a spot check on one page proves nothing about another.
- Tables that are images: invisible even to a missing-content check.
- Zero-width spaces triggering false alarms in completeness checks.
- Superscripts flattened. A layout failure, so font tools do not warn.
- Another paper appended to the end of the PDF, silently merged into one.
Two working rules: check which tool actually produced the text, because a failing extractor may fall back silently; and never run a corruption-repair substitution table in reverse, which corrupts clean text at scale and is far harder to detect than the original problem.
---
## Logs
There are no commit messages here, so the log is the only narrative record of what changed and why. One Markdown file per day, per agent, per machine:
```
logs/{YYYY-MM-DD}-{agent}-{host}.md
```
`{agent}` is `claude` or `codex`. `{host}` distinguishes machines that share this folder — resolve it by running a command, never from memory. Several files may coexist for one day; never write into another agent's or machine's file.
Frontmatter, then the title, then append-only entries, oldest first:
```markdown
---
date: 2026-07-29
agent: claude
host: laptop-a
model: [which model did the work]
---
# 2026-07-29 — Claude work log (laptop-a)
```
Entry headings carry a kind and a topic slug so one campaign can be followed across days, agents, and machines:
```
## [YYYY-MM-DD] ingest | {topic} | Paper or batch title
## [YYYY-MM-DD] maintenance | {topic} | Task
## [YYYY-MM-DD] query-to-wiki | {topic} | Result page
## [YYYY-MM-DD] semantic-lint | {topic} | Scope
## [YYYY-MM-DD] supersede | {topic} | Claim or page
```
A topic slug is lowercase letters, digits, and hyphens. Read the existing topic table before inventing one — a near-duplicate slug splits a campaign in two and neither half looks incomplete.
Directly under the heading, add whichever index lines apply, then a blank line, then the narrative. Write only what the entry actually did:
- `targets:` records or paths this entry changed
- `scripts:` scripts this entry ran
- `report:` the report page this entry produced
- `model:` only when it differs from the file's frontmatter
Generate `logs/README.md` from the headings; never hand-edit it. Keep reports in `logs/reports/` — a dated report sitting beside dated daily logs gets read back as a log.
---
## Query-to-wiki
When a question produces a reusable comparison, analysis, decision framework, or handoff-quality synthesis, file it back instead of leaving it in chat:
- `wiki/overviews/` — research-facing literature synthesis.
- `wiki/concepts/` — stable definitions, methods, mechanisms.
- `wiki/questions/` — an open question with the evidence currently available.
- `agenda/{project-or-context}/` — execution plans, handoffs, operational notes.
Save only durable outputs a future reader would search for, not every casual answer.
---
## Search and health checks
- `rg` for exact strings across `wiki/`, `sources/`, `agenda/`, `scripts/`.
- Once the wiki passes ~500 pages, use a pinned, memory-mapped `bm25s` index built directly from `wiki/**/*.md` as the default candidate generator. QMD or another vector index is an optional semantic fallback, not an upstream dependency of the default index.
- Exclude `wiki/questions/` from retrieval results by default. Search evidence-bearing paper, overview, and concept pages instead of returning the query-shaped page derived from the same question.
- Use a complete English research question as the BM25 query. For a non-English question against an English corpus, preserve the original question and supply a complete English rewrite; do not reduce it to three keywords.
- Retrieval scores identify candidates only. Read the page and, when necessary, its source summary or canonical PDF before making a claim. Abstain when the retrieved text does not explicitly support the requested relationship.
- Run the validator before and after any large ingest or reorganization. It should check YAML frontmatter, PDF path existence, source-to-wiki references, broken wikilinks, category drift, duplicate titles, forbidden ingest metadata, empty required sections, and placeholder prose.
Two validator findings need the right fix. A section containing only an instruction to consult the original, and a page describing its own processing status, both mean the record was never properly built: re-ingest it from the PDF or drop it. Deleting the offending line is the wrong fix — it leaves an empty section that reads as a complete one and is invisible to every text check afterward.
### Retrieval index policy
Pin the runtime (`bm25s==0.3.10`, `PyStemmer==3.1.0`) and expose one sanctioned interface:
```bash
# One-time on each machine: runtime, first index, local scheduled rebuild
bash scripts/setup_llm_wiki_retrieval.sh
# Default search
bash scripts/search_llm_wiki.sh "full English research question" --json
# Non-English question against an English corpus
bash scripts/search_llm_wiki.sh "original question" \
--english-query "complete English research-question rewrite" --json
# Immediate freshness only
bash scripts/reindex_bm25s.sh
```
Rules:
1. Every machine keeps its runtime, index versions, build lock, and success stamp outside the synced repository. Never put a live retrieval index or lock in cloud storage.
2. A scheduled rebuild reads the current `wiki/**/*.md` tree but writes only machine-local state, so every machine may run its own job independently. Do not rebuild after every ingest.
3. Build a complete version first; validate the document count, deterministic corpus hash, memory-mapped load, and a non-question smoke query; then atomically switch `current`. Retain the previous complete version for rollback. Never edit index files in place.
4. Semantic fallback processes must be serialized locally and must treat an unexpectedly empty result as a health failure. Do not run concurrent vector or hybrid processes against a backend known to initialize shared accelerator state.
5. Use semantic fallback for ambiguous lexical retrieval, source/agenda collection navigation, or decomposed multi-document questions. It does not control BM25 freshness.
---
## Wikilinks and Obsidian
The Obsidian vault root is `wiki/`, not the repository root. Every `[[...]]` target is vault-root-relative with no `wiki/` prefix:
- Correct: `[[overviews/some-page]]`, `[[concepts/some-method]]`
- Wrong: `[[wiki/overviews/some-page]]`
Inside a `wiki/`-rooted vault, the prefixed form resolves to `wiki/wiki/...`, which does not exist, and Obsidian creates an empty ghost page there every time the link is opened. Link-checking scripts accept both forms, so they will not catch a reintroduced prefix — keep it out by hand.
Do not create an `.obsidian/` directory at the repository root. Its presence is what makes a tool assume the wrong vault root; the only vault config lives at `wiki/.obsidian/`.
---
## Housekeeping
- No scratch folders inside this repository. Temporary artifacts, text dumps, conversion output, and debug files go to `~/tmp`. Delete tool caches that appear.
- Supplements never go in `papers/`, which holds exactly one canonical PDF per paper. They go in `papers-supplementary/` with a manifest recording what each opaque publisher filename actually contains. Do not keep peer-review files, decision letters, or blank checklists — but do keep an author-written reporting summary, which often holds the power analysis, exclusion criteria, blinding, software versions, and demographics found nowhere else. Stop the extraction page range before it so its blank boilerplate never reaches a page.
- Never use `path` as a shell variable name. In zsh it is a special array tied to `PATH`, and assigning to it can break command lookup mid-script. The same goes for `HOME`, `IFS`, `SHELL`, `PWD`, and `RANDOM`.
- If you use git, keep the `.git` directory outside any cloud-synced folder. A sync conflict on the git index can stage hundreds of healthy files as deletions.
---
## Design principles
- Four tiers: raw PDF (immutable), `sources/`, `wiki/{category}/`, synthesis layer.
- English only in wiki content, for writing and retrieval.
- Obsidian compatible: `[[wikilinks]]`, plain Markdown.
- Consistent YAML in every paper record, extraction provenance included.
- No web search, no placeholders, no unconnected pages.
When in doubt, follow rule 1.

LLM Wiki: Building a Personal Knowledge Base for Academic Papers with AI Agents

A methodology for using Claude Code or OpenAI Codex to build and maintain a structured, searchable wiki from academic PDFs — for researchers who read dozens of papers a month and want the reading to compound instead of evaporate.

This is a starter template. Fork the structure, swap in your own categories. The wiki only becomes useful once it reflects your domain, not someone else's.

Revised 2026-08. The first version of this gist described a three-tier pipeline and four rules. Running the system to more than 15,000 PDFs and 16,000 wiki pages taught a set of lessons that were not in it — chiefly that a paper summary nobody linked to anything is not knowledge, it is a cache entry. The July revision added the synthesis requirement, the extraction-failure catalog, the log convention that replaced commit messages, and the two folders that hold everything which is not a paper. This August revision replaces QMD-first retrieval with a memory-mapped, full-question BM25 index built directly from the Markdown corpus. See "What changed from the original version" at the bottom for the full list.

The rules

The point of this wiki is to prevent hallucination by forcing every answer to be traceable to a paper you actually have. Without rules the wiki degrades into a dressed-up web search with citations that look right.

The four original rules

  1. No web search. Forbid WebSearch and WebFetch outright in the rule file. Turn them on only when you explicitly ask for them, for one task, and say so.
  2. Answer from the wiki first. sources/ and wiki/ are the only sources of truth.
  3. If the wiki is insufficient, re-read the original PDF in papers/. Then update the wiki so the next question does not need the PDF.
  4. If no paper exists on the topic, say so. "I don't have a paper on this — give me the PDF." Do not improvise, do not look it up online.

These apply to every response, overview pages included: cite only papers that exist in the wiki.

Five rules the first two years added

  1. A paper is not ingested until it is connected. A source and wiki page with no link into the synthesis layer is a synthesis-orphan. See "The synthesis requirement" below — this is the single most important change in this revision.
  2. No tiers, no placeholders. Every admitted paper meets the same standard: read from the original, with concrete methods, results, and limitations. Never create a searchable page from an abstract, a reference-list mention, a filename, or a metadata record. A placeholder page is worse than a missing one, because retrieval will find it and treat it as evidence. If the original cannot support a complete page, leave the paper out and write down why.
  3. Exhaustive means exhaustive. When you ask for "all of them" or "check every one", the agent finishes the whole set before reporting, states the denominator and the numerator, and counts files it could not evaluate — extraction failures, empty text, timeouts — as unchecked rather than as passing. Silence is not success. Sampling thirty of four thousand and reporting it as a survey is the worst failure mode available, because it looks exactly like the real thing.
  4. Confidential material stays out of the knowledge layer. Unpublished manuscripts, drafts under review, anything under embargo: not in wiki/, not even the title, not as a tag, not as a "relevance to our manuscript" section bolted onto a published paper's page. Those notes belong in agenda/ (below). Published work with a DOI is ordinary wiki content.
  5. PDFs only, never HTML captures. The canonical full text of a paper is an exact PDF in papers/. A publisher page saved as HTML, or a browser text snapshot, is not a substitute — it silently drops figures, tables, superscripts, and half the Methods. If you cannot get the PDF, keep the paper on a missing list; do not ingest a degraded copy.

One more filter that is pure housekeeping: correction, erratum, and retraction notices are not papers. Their titles start with Author Correction:, Publisher Correction:, Correction:, Erratum, or Retraction, and they must never get a page. Enforce it with a pattern that requires the correction word to be followed by : or to, so that real papers about correcting something — cognitive deficits, batch effects, motion artifacts — survive.

The concept

Inspired by Karpathy's LLM Wiki pattern, with one tier added:

Original PDF  →  sources/*.md  →  wiki/{category}/*.md  →  wiki/overviews/ + wiki/concepts/
   immutable      LLM summary       structured page          the synthesis layer
  1. papers/ — the original PDF, immutable archive.
  2. sources/ — an LLM-generated structured summary with a fixed section list.
  3. wiki/{category}/ — the structured page with cross-references as [[wikilinks]].
  4. wiki/overviews/ and wiki/concepts/ — where papers meet each other.

The fourth tier is the whole point. Tiers 1 through 3 are a well-organized pile. Tier 4 is the part that answers a question you have not asked yet.

What is in this gist

File What it is
llm-wiki-gist.md This essay, the methodology
AGENTS.md.template A ready-to-fill rule file. Drop it into your project root as AGENTS.md, then symlink CLAUDE.md to it

The earlier CLAUDE.md.template is gone. The rule file is now named AGENTS.md because two different agents read it — see "Two agents, one folder" below.

Repository structure

your-llm-wiki/
├── AGENTS.md               # The rulebook. CLAUDE.md is a symlink to it
├── docs/                   # Rule detail split out of AGENTS.md once it got long
│   ├── INGEST.md           #   add-a-paper flow, PDF management, extraction traps
│   └── WIKI-SCHEMA.md      #   frontmatter and section schemas, category list
├── index.md                # Generated slim catalog: categories, counts, coverage
├── indexes/                # Generated per-category page listings
├── logs/                   # The narrative record. See "Logs, not commit messages"
│   ├── {YYYY-MM-DD}-{agent}-{host}.md
│   ├── README.md           #   generated folder guide and topic index
│   └── reports/            #   audit reports written for a person to read
├── scripts/                # Extraction, validation, index and log builders
├── papers/                 # Original PDFs, canonical storage
│   └── {author}-{year}-{title-5-tokens}.pdf
├── papers-supplementary/   # Publisher supplements, kept out of papers/
│   └── manifest/           #   what each opaque publisher filename actually holds
├── sources/                # PDF summaries, English
├── wiki/                   # The knowledge layer, English. Obsidian vault root
│   ├── {category}/
│   ├── concepts/           #   methods, mechanisms, reusable definitions
│   ├── overviews/          #   encyclopedic topic pages, declarative titles
│   └── questions/          #   paper-anchored research questions, question titles
├── agenda/                 # Project execution notes, plans, handoffs, unpublished work
└── materials/              # Non-paper reference material

The last four items did not exist in the first version and are the most-used parts of the system now.

wiki/overviews/ vs wiki/concepts/ vs wiki/questions/

Three synthesis layers, deliberately kept apart:

  • overviews/ is encyclopedic. Declarative noun-phrase titles: "Astrocyte heterogeneity across brain regions". It states what the field currently holds.
  • concepts/ is the dictionary. A stable definition, method, or mechanism that many papers lean on, written once so that paper pages can point at it instead of re-explaining it.
  • questions/ is explicitly a question layer. The title is a question, and the body follows a fixed shape: ## Question, ## Sharper follow-up, ## What the knowledge base holds, ## Tentative answer. It is where an open problem lives while the evidence is still thin.

Keeping them separate matters more than it sounds. Question-shaped pages filed into overviews/ make the encyclopedia read like a lab notebook, and the encyclopedia is the part you hand to somebody else.

agenda/ — the working layer

agenda/{project-or-context}/ holds everything that is about doing research rather than about published knowledge: execution plans, handoff documents, analysis to-do lists, decision records, journal-selection notes, meeting follow-ups, training material for new lab members, and notes on unpublished manuscripts. It is indexed for search alongside wiki/ and sources/, so questions can draw on it, but it is a separate namespace with no schema requirements.

Two conventions make it work. First, a handoff document has a fixed shape: why the method is needed and what it enables, the analyses to run, the expected results, a decision section saying when to continue or pause or stop if the results are weak, and the concrete deliverables. Second, agenda/ is where confidential material goes — rule 8 sends unpublished manuscript notes here precisely because it is outside the knowledge layer you might one day share.

Give the operating decisions about the wiki itself their own subfolder (agenda/llm-wiki-ops/). Short ADR-style notes on why a rule exists survive far better than the rule alone.

materials/ — reference material that is not a paper

Not everything worth keeping is a paper, and forcing non-papers into the paper schema corrupts both. materials/ holds those, with no filename convention, no category, no PDF frontmatter requirement:

  • journals/ — scope statements, author guidelines, and editorial-board snapshots for the journals you might submit to. Treat every file as a dated snapshot and re-verify before an actual submission.
  • writing-style/ — profiles of how specific scientists write, built from verbatim quotes of their papers, for learning sentence construction and argument flow rather than findings.
  • lectures/ — notes from talks and courses.
  • meetings/ — curated meeting and journal-club notes. Keep the raw audio and raw transcripts outside the wiki entirely; only the curated note comes in.
  • professional-activity/ — your own talks and service record.

The boundary is a rule, not a suggestion: do not link materials/ from wiki/, sources/, or index.md. If a style profile or a journal snapshot starts appearing in the paper knowledge graph, the graph stops being a graph of papers.

Naming convention

All three tiers share one stem:

{first-author-lastname}-{year}-{first-5-title-tokens}.{ext}

Tokens, not words. Lowercase the title, then take runs of alphanumerics: every non-alphanumeric character is a separator, not something to delete. self-report is two tokens, not one; digits count as tokens. So Direct haplotype-resolved 5-base HiFi sequencing becomes direct-haplotype-resolved-5-base. Hyphenated titles hit five tokens faster than you expect, which is fine.

Two things learned the hard way:

  • Let a script generate the stem. Hand-built stems drift, and stop-word handling ("of", "the", "and") is exactly the kind of decision two scripts written a year apart will make differently. Ours did, and now both forms exist on disk.
  • Do not bulk-rename legacy stems to match a newer rule. Every [[wikilink]] and index entry has to follow, and you have to decide which form is canonical first. Fix only the cases where a paper's own three tiers disagree with each other.

Consortium papers use the consortium name. Author parsing is the one place worth a real test suite: a naive splitter that breaks on ; and and but not on commas reads "Cristina Bacon, Gudrun A. Rappold" as one author and files the paper under the wrong name.

Adding a paper

Step 0 — find the PDF

Standard workflow is that you download papers yourself and hand them over. Make ~/Downloads the default inbound location so "ingest the papers I just downloaded" needs no path. The agent identifies each file from its first page — title, authors, DOI — and asks when a mapping is ambiguous instead of guessing. It never moves or deletes anything in your directories.

Step 1 — copy to papers/ and extract text

Copy, never symlink. pdf_path always points inside papers/, and pdf_filename is its basename.

For extraction, opendataloader-pdf is the default and it is a real upgrade over the simpler options. In a head-to-head on two-column papers, PyMuPDF interleaved figure-panel scaffolding — axis ticks, stray gene labels — into the body text and made whole passages unreadable, while opendataloader reconstructed paragraphs as continuous prose, separated figure captions, tidied references, and stripped running headers. It needs Java, so wrap it in a shell helper that locates a working Python and Java on whatever machine you are on, with a timeout, falling back to pypdf and then pdftotext:

bash scripts/extract_pdf_text.sh "/path/to/paper.pdf" 1-25 400000

The two arguments are a page range and a character cap, and confusing them is the most expensive small mistake in this system. A generous character cap costs nothing; a small one truncates the paper mid-Methods and reports success. Check the page count with pdfinfo first and widen the range for long papers. The old advice in the first version of this gist — 15 pages, 12,000 characters — is enough for an abstract-grade summary and not enough for a real one. That was a stale default and it is why this section exists.

For a narrower job the answer flips. When the goal is pulling gene and relation lists out for a knowledge graph, PyMuPDF keeps columns pure, matches on recall, needs no Java, and runs six to twenty-four times faster. Pick the extractor for the task.

Record what you used. Three frontmatter fields, mandatory on every new or re-extracted record:

source_format: pdf
text_extractor: opendataloader-pdf-2.x
text_extracted_date: 2026-07-29

This is what makes re-extraction possible later: when a better tool arrives, you can select exactly the pages built with the worse one instead of redoing everything. Do not invent values for old records that lack them; that is a backlog, not a formatting problem.

Raw extractor output needs a bounded cleanup before it becomes a page. Write down exactly what may be removed — page furniture, running headers, hyphenation at line breaks, empty pseudo-tables — and remove nothing else. An unbounded "clean this up" pass will eat content.

Step 2 — write sources/{stem}.md

Fixed frontmatter and a fixed section list: One-line Summary, Document Information, Key Contributions, Methodology and Architecture, Key Results and Benchmarks, Limitations and Future Work, Related Work, Glossary.

Do not add a section holding the raw extraction dump. The canonical full text is the PDF; copying fifteen pages of it into the summary duplicates the original and drags publisher boilerplate — line numbers, copyright lines, "OPEN ACCESS" — into your search index.

Keep processing metadata out of paper frontmatter entirely. No tier, no curation_level, no batch names, no agent names, no dated run identifiers. Batch processing is an execution strategy, and it should be invisible in the knowledge layer. Provenance about a run belongs in the log.

Step 3 — write wiki/{category}/{stem}.md

Same frontmatter plus source: and tags:. Sections: Summary, Key Contributions, Methodology and Architecture, Results, Related Papers.

Step 4 — connect it to the synthesis layer

This is not optional and it is the step that gets skipped. It has its own section below.

Step 5 — log it, then rebuild the indexes

Append one entry to the day's log file, then regenerate the catalog and the log index. Do not put ingest narrative into index.md.

The agent does all of this in one go when you say "add this paper to the wiki".

The synthesis requirement

A source and wiki page with no link into overviews/ or concepts/ is a synthesis-orphan. It behaves like a RAG chunk: retrievable, but connected to nothing, contributing nothing to any question you did not phrase in its own vocabulary. Accumulate enough of them and you have built a search index with extra steps.

So every ingest, routine ones included, ends with four things:

  1. At least one bidirectional link between the new paper page and the single most relevant existing overview or concept page. Bidirectional matters: Related Papers on the paper, Related Pages on the synthesis page. A one-way link is invisible from the side that would have used it.
  2. One sentence in that synthesis page's body placing the paper in context. Not a bullet in a list — a sentence in the prose. If the paper genuinely only restates what the page already says, a link is enough, but say so in the log.
  3. A supersede check. Does this paper strengthen, narrow, contradict, or replace a claim already written down? If it contradicts or replaces one, record that explicitly in the day's log as a supersede entry. This is the difference between a wiki and a pile of summaries: the pile keeps both claims and lets you find whichever one you searched for.
  4. If no overview or concept covers the topic, write the gap down rather than skipping silently. Recurring gaps become the next anchor page.

Audit the drift with a read-only script that lists orphans. The orphan list is your backlog, and it should trend to zero rather than growing with the corpus.

For a paper that can change an active argument, go further: read the existing synthesis before writing, update the page bodies rather than just the links, create the concept anchor if one is missing, and make the claim delta explicit. Process those one at a time. The value of this system is concentrated in maybe two percent of the papers, and batching them is how you lose it.

What the extractor gets wrong

Every failure below was found in real papers. They share one property: the extractor reports success.

  • Glyphs are substituted, not dropped. A missing character is easy to catch; a character silently replaced by a different one is not. Greek letters inside proper nouns are a common victim.
  • Minus signs vanish. Undetectable in principle from the text alone — a negative effect size becomes a positive one and nothing marks it. If a sign matters, look at the PDF.
  • Table column order flips, and adjacent columns merge. Numbers end up attached to the wrong label.
  • Figure reading order scrambles. Values get bound to the wrong axis label. If a paper's conclusion lives in a heatmap direction or an effect-size plot, render the figure to PNG and read it as an image. Text extraction cannot see color, and no warning tells you it could not.
  • The body silently disappears. Whole sections come back empty on some font runs. Corruption is per font run, not per file, so a spot check on page 2 proves nothing about page 14.
  • A table that is an image is invisible even to a missing-content check, because nothing is missing from the extractor's point of view.
  • Zero-width spaces produce false alarms in completeness checks.
  • Superscript flattening is a layout failure, not an encoding one, so font tools do not warn about it.
  • Sometimes another paper is appended to the end of the PDF. The extraction succeeds and now two papers are one.

Two working rules follow. First, when the extractor dies it may fall back silently — check which tool actually produced the text, which is one more reason for text_extractor in the frontmatter. Second, when you build a substitution table to repair a known corruption, never run it in reverse "to be safe": that corrupts clean text at scale, and it is much harder to detect than the original problem.

Logs, not commit messages

The original setup kept the wiki in git. That ended, for a reason worth passing on: the repository lived inside a cloud-synced folder, and one day the sync produced a conflicted copy of the git index, which staged 980 healthy files as deletions. If you want git, keep the .git directory outside the synced folder. Or drop git, let the sync tool own versioning, and accept that per-file version history is your rollback.

Dropping git removes the one thing git was quietly providing: a written record of what changed and why. So the log takes over, and it needs more discipline than a commit message did.

One Markdown file per day, per agent, per machine:

logs/{YYYY-MM-DD}-{agent}-{host}.md

The host token matters as soon as two machines share the folder — two agents working the same day would otherwise collide on one file. Resolve the host by running a command, never from memory. The file opens with frontmatter recording date, agent, host, and which model did the work, then append-only entries, oldest first.

Entry headings carry a kind and a topic slug, so one campaign can be followed across days, agents, and machines:

## [2026-07-29] ingest | canonical-pdf-contamination | Paper or batch title
## [2026-07-29] maintenance | canonical-pdf-contamination | Re-extraction sweep
## [2026-07-29] supersede | astrocyte-reactivity | Claim replaced

Kinds in use: ingest, maintenance, query-to-wiki, semantic-lint, supersede. Under the heading go optional index lines — which records changed, which scripts ran, which report was produced — then the narrative.

Read the existing topic list before inventing a slug. A near-duplicate slug splits one campaign into two and neither half looks incomplete.

Generate logs/README.md from the headings rather than maintaining it. A generated index cannot claim work the logs do not record, and it surfaces entries missing a slug as a backlog. Keep reports in a subfolder: a dated report sitting beside dated daily logs gets read back as a log.

Two agents, one folder

Claude Code and Codex both work in this folder, which is why the rule file is AGENTS.md with CLAUDE.md as a symlink to it. One rulebook, agent-neutral wording, no per-tool dialects unless a rule genuinely applies to one tool only.

The friction is not the rules, it is concurrency. The default BM25 index is deliberately not shared: each machine keeps its runtime, versioned indexes, build lock, and success stamp outside the synced wiki. A rebuild writes a complete new version, validates it, and only then atomically repoints current; readers never see a half-built index. Semantic fallback has a stricter rule: allow one local vector process at a time and fail loudly on an empty result, because concurrent Metal-backed searches can return empty rankings without reporting an error.

A startup checklist in the rule file earns its place here too: re-read the rulebook, re-read the registries that change often, and do not rely on memory from a previous session for anything that moves.

The knowledge tree method

This is how the wiki actually grows. Not "ingest a thousand papers, then search" — branching outward from real questions.

Root question ("what is known about non-cortical brain cell types?")
├── 1st wave: direct overview pages
│   ├── Thalamic molecular architecture
│   └── Cerebellar cell diversity
├── 2nd wave: branches from what the first wave surfaced
│   ├── Dopaminergic neuron diversity
│   └── Region-specific disease vulnerability
└── 3rd wave: cross-cutting themes
    └── Circadian regulation in brain evolution

In practice:

  1. Ask a question. The agent searches the wiki and answers from what exists.
  2. If the wiki is insufficient, it re-reads the original PDFs and updates the pages.
  3. If no paper covers it, it says so and you supply the PDF.
  4. Save the good answers. A durable comparison, decision framework, or synthesis goes to overviews/ or concepts/; an open problem goes to questions/; an execution plan goes to agenda/.

Do not save every casual answer. Save the ones a future reader would search for.

Scaling

Hold off until you need to.

  • A category past ~500 files should split. Pick the split axis by asking what you would deliberately exclude when reading about the topic.
  • index.md past a few hundred pages should stop being one file. Ours is now a slim root catalog — one row per category with a page count and a synthesis-coverage number — linking to a generated per-category listing. The coverage column is the useful part: it makes orphan accumulation visible on the front page instead of buried in an audit.
  • The total wiki past ~500 pages wants a real search index. The default is a pinned, memory-mapped bm25s index built directly from wiki/**/*.md. QMD remains useful for source/agenda collection navigation and serialized semantic fallback, but it is not an upstream dependency of the production index. Below that threshold, index.md plus the agent's built-in search is fine.

The production retrieval path

Pin the runtime (bm25s==0.3.10, PyStemmer==3.1.0), use Lucene-style BM25 with English stemming and stopwords, and exclude wiki/questions/ from results by default. Question pages are useful synthesis, but their query-shaped titles and prose can dominate a search for the question they were derived from. Return the evidence-bearing paper, overview, and concept pages instead.

Expose the workflow through project-local scripts with this interface:

# One-time per machine: install the runtime, build the first index, add a local schedule
bash scripts/setup_llm_wiki_retrieval.sh

# Default read path: use the complete English research question
bash scripts/search_llm_wiki.sh "Which mechanisms connect tandem repeat mutations to autism?" --json

# A non-English question against an English corpus keeps the original and supplies a
# complete English rewrite, not a three-keyword reduction
bash scripts/search_llm_wiki.sh "자폐증과 탠덤반복 변이를 연결하는 기전은?" \
  --english-query "Which mechanisms connect tandem repeat mutations to autism?" --json

# Rebuild now only when immediate freshness matters
bash scripts/reindex_bm25s.sh

Keep the runtime and index under machine-local application/cache directories, never inside the cloud-synced wiki. A scheduled rebuild may run independently on every machine because it reads the synced Markdown but writes only local state. Build into a new version directory, verify document count, corpus hash, memory-mapped load, and a non-question smoke query, then atomically switch current and retain the previous complete version for rollback. Do not rebuild after every ingest; let the next scheduled run pick up routine changes.

When lexical candidates are ambiguous or insufficient, use a serialized semantic fallback, then decompose multi-document or conflicting-evidence questions into narrower searches. A retrieval score is never an answerability threshold: read the retrieved pages and abstain when they do not explicitly support the relationship in the question.

The switch was empirical. On 120 real stored research questions, full-question BM25 found the primary paper in 118/120 searches and some valid evidence in 120/120 at top 10, with a 0.68 ms median. The QMD vector baseline found the primary paper in 113/120 and some evidence in 119/120, with a 678 ms median. BM25 also supported concurrent memory-mapped reads without changing rankings. These numbers justify BM25 as the candidate generator; they do not justify treating lexical similarity as evidence.

Run a validator before and after any large ingest or reorganization. Ours checks YAML frontmatter, PDF paths, source-to-wiki references, broken wikilinks, category drift, duplicate titles, forbidden ingest metadata, and two body-level patterns worth calling out: sections containing only an instruction to consult the original, and pages stating their own processing status. The fix for either is to re-ingest from the PDF or drop the record. Deleting the placeholder line is the wrong fix, because it leaves an empty section that reads as a complete one and is invisible to every text check afterward.

Browsing with Obsidian

The agent handles ingest and questions; Obsidian is the best way to read and navigate. Free, local, native [[wikilinks]], graph view, full-text search. Open the wiki folder as a vault. It only reads files, so it never interferes with what the agent maintains.

One gotcha that cost a full revert. The vault root is wiki/, not the repository root, so wikilinks resolve relative to wiki/ and must never carry a wiki/ prefix. [[overviews/some-page]] is correct; [[wiki/overviews/some-page]] resolves to wiki/wiki/overviews/..., which does not exist, and Obsidian creates an empty ghost page there every time somebody clicks it. A well-meaning pass that added the prefix to tens of thousands of links broke the vault, and link-checking scripts accept both forms, so they do not catch it. Related: do not create an .obsidian/ directory at the repository root — its presence is what makes a tool assume the wrong vault root.

Housekeeping that turns out to matter

  • Keep scratch work outside the wiki. Temporary folders inside the repository get synced to every other machine and pollute the knowledge base. Send intermediate artifacts to ~/tmp and clean up the caches tools leave behind.
  • Never name a shell loop variable path. In zsh it is a special array tied to PATH, and read -r key path can make basic commands disappear mid-script. The same applies to HOME, IFS, SHELL, and friends.
  • Supplements do not go in papers/, which holds exactly one canonical PDF per paper. They get their own folder and, because publishers name them things like 41586_2024_1234_MOESM3_ESM.pdf, a manifest saying what each file actually contains. Peer-review files and blank checklists are not kept — with one exception: a reporting-summary form is author-written and often holds the power analysis, exclusion criteria, blinding, software versions, and demographics that appear nowhere else. Keep it, and stop your page range before it so its blank boilerplate never reaches a page.
  • Do not spawn another model CLI or API from an ingest script. Paper interpretation stays in the active session; scripts do deterministic extraction, validation, logging, and index maintenance. An ingest pipeline that calls out to a second model produces pages nobody read.

Getting started

You do not need to install Python, Java, or Node yourself. Claude Code and Codex have native installers for Mac, Linux, and Windows, with no WSL2 needed. Let the agent bootstrap.

  1. Install Claude Code or Codex.

  2. Open the agent in an empty folder. This becomes your wiki root.

  3. Paste this prompt:

    Set up an LLM Wiki for me, following this gist: https://gist.github.com/joonan30/cbce305684d079dbe9a3fbaefe4e3959

    Read all files in the gist, ask me about my research field and 5–10 starting categories, then:

    • Create papers/, sources/, wiki/{my-categories}/, wiki/overviews/, wiki/concepts/, wiki/questions/, agenda/, materials/, logs/, scripts/
    • Write AGENTS.md from the template in this gist, filled in for my domain, and symlink CLAUDE.md to it
    • Write scripts/extract_pdf_text.sh per the extraction section, with opendataloader-pdf first and pypdf/pdftotext as fallbacks
    • Implement the retrieval interface described above, including setup, pinned-runtime installation, atomic reindexing, search, and any helper scripts it needs; keep the runtime and indexes outside the wiki
    • Apply all the rules from this gist verbatim, including the synthesis requirement. Never use web search.
  4. Drop in your first 5–10 PDFs and say "add these papers to the wiki".

  5. Ask questions. Save the good answers as overview, concept, or question pages.

  6. When you cross ~500 pages, run bash scripts/setup_llm_wiki_retrieval.sh once on each machine that will search the wiki.

A note on scale

The wiki this came from currently holds 15,259 PDFs, 14,872 source summaries, and 16,294 wiki pages, of which 459 are overviews, 368 are concepts, and 550 are question pages, spread across 52 categories.

The paper count is the least interesting number there. The synthesis pages grow faster than the corpus over time and they are where the value sits — they are what makes the thing answer a question rather than retrieve a document. If you take one operational rule from this revision, take rule 5: a paper you did not connect to anything is a paper you will not benefit from having read.

You need none of this on day one. Start with the four rules and five papers.

What changed from the original version

  • Replaced QMD-first retrieval with full-question bm25s as the default candidate generator, backed by a 120-question benchmark.
  • Made retrieval state machine-local, memory-mapped, versioned, validated before an atomic switch, and independently schedulable on every synced machine.
  • Kept QMD as an optional serialized semantic and collection-navigation fallback rather than an upstream dependency.
  • Added the synthesis requirement and the synthesis-orphan concept, plus wiki/concepts/ and wiki/questions/ as distinct layers.
  • Replaced the pypdf snippet with opendataloader-pdf as the default extractor, the A/B result behind that choice, the page-range vs character-cap trap, and mandatory extraction-provenance frontmatter.
  • Added the extraction-failure catalog. Every entry is a real silent failure.
  • Added the log convention that replaced commit messages, and the reason git left the folder.
  • Added agenda/ and materials/ as first-class folders.
  • Added rules on placeholders and tiers, exhaustive reporting, confidential material, PDF-only acquisition, and correction notices.
  • Made the naming convention token-based, with the warning against bulk-renaming legacy stems.
  • Split index.md into a slim catalog plus per-category listings, with a synthesis-coverage column.
  • Added the Obsidian vault-root gotcha, the concurrency lock for shared search indexes, and the two-agent setup.
  • Renamed CLAUDE.md.template to AGENTS.md.template.

Built with Claude Code (Anthropic) and Codex (OpenAI). Browsing with Obsidian. Default retrieval with bm25s; semantic fallback with QMD. Extraction with opendataloader-pdf. For Karpathy's original idea: @karpathy/1dd0294ef9567971c1e4348a90d69285.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment