A methodology for using Claude Code or OpenAI Codex to build and maintain a structured, searchable wiki from academic PDFs — for researchers who read dozens of papers a month and want the reading to compound instead of evaporate.
This is a starter template. Fork the structure, swap in your own categories. The wiki only becomes useful once it reflects your domain, not someone else's.
Revised 2026-08. The first version of this gist described a three-tier pipeline and four rules. Running the system to more than 15,000 PDFs and 16,000 wiki pages taught a set of lessons that were not in it — chiefly that a paper summary nobody linked to anything is not knowledge, it is a cache entry. The July revision added the synthesis requirement, the extraction-failure catalog, the log convention that replaced commit messages, and the two folders that hold everything which is not a paper. This August revision replaces QMD-first retrieval with a memory-mapped, full-question BM25 index built directly from the Markdown corpus. See "What changed from the original version" at the bottom for the full list.
The point of this wiki is to prevent hallucination by forcing every answer to be traceable to a paper you actually have. Without rules the wiki degrades into a dressed-up web search with citations that look right.
- No web search. Forbid
WebSearchandWebFetchoutright in the rule file. Turn them on only when you explicitly ask for them, for one task, and say so. - Answer from the wiki first.
sources/andwiki/are the only sources of truth. - If the wiki is insufficient, re-read the original PDF in
papers/. Then update the wiki so the next question does not need the PDF. - If no paper exists on the topic, say so. "I don't have a paper on this — give me the PDF." Do not improvise, do not look it up online.
These apply to every response, overview pages included: cite only papers that exist in the wiki.
- A paper is not ingested until it is connected. A source and wiki page with no link into the synthesis layer is a synthesis-orphan. See "The synthesis requirement" below — this is the single most important change in this revision.
- No tiers, no placeholders. Every admitted paper meets the same standard: read from the original, with concrete methods, results, and limitations. Never create a searchable page from an abstract, a reference-list mention, a filename, or a metadata record. A placeholder page is worse than a missing one, because retrieval will find it and treat it as evidence. If the original cannot support a complete page, leave the paper out and write down why.
- Exhaustive means exhaustive. When you ask for "all of them" or "check every one", the agent finishes the whole set before reporting, states the denominator and the numerator, and counts files it could not evaluate — extraction failures, empty text, timeouts — as unchecked rather than as passing. Silence is not success. Sampling thirty of four thousand and reporting it as a survey is the worst failure mode available, because it looks exactly like the real thing.
- Confidential material stays out of the knowledge layer. Unpublished manuscripts, drafts under review, anything under embargo: not in
wiki/, not even the title, not as a tag, not as a "relevance to our manuscript" section bolted onto a published paper's page. Those notes belong inagenda/(below). Published work with a DOI is ordinary wiki content. - PDFs only, never HTML captures. The canonical full text of a paper is an exact PDF in
papers/. A publisher page saved as HTML, or a browser text snapshot, is not a substitute — it silently drops figures, tables, superscripts, and half the Methods. If you cannot get the PDF, keep the paper on a missing list; do not ingest a degraded copy.
One more filter that is pure housekeeping: correction, erratum, and retraction notices are not papers. Their titles start with Author Correction:, Publisher Correction:, Correction:, Erratum, or Retraction, and they must never get a page. Enforce it with a pattern that requires the correction word to be followed by : or to, so that real papers about correcting something — cognitive deficits, batch effects, motion artifacts — survive.
Inspired by Karpathy's LLM Wiki pattern, with one tier added:
Original PDF → sources/*.md → wiki/{category}/*.md → wiki/overviews/ + wiki/concepts/
immutable LLM summary structured page the synthesis layer
papers/— the original PDF, immutable archive.sources/— an LLM-generated structured summary with a fixed section list.wiki/{category}/— the structured page with cross-references as[[wikilinks]].wiki/overviews/andwiki/concepts/— where papers meet each other.
The fourth tier is the whole point. Tiers 1 through 3 are a well-organized pile. Tier 4 is the part that answers a question you have not asked yet.
| File | What it is |
|---|---|
llm-wiki-gist.md |
This essay, the methodology |
AGENTS.md.template |
A ready-to-fill rule file. Drop it into your project root as AGENTS.md, then symlink CLAUDE.md to it |
The earlier CLAUDE.md.template is gone. The rule file is now named AGENTS.md because two different agents read it — see "Two agents, one folder" below.
your-llm-wiki/
├── AGENTS.md # The rulebook. CLAUDE.md is a symlink to it
├── docs/ # Rule detail split out of AGENTS.md once it got long
│ ├── INGEST.md # add-a-paper flow, PDF management, extraction traps
│ └── WIKI-SCHEMA.md # frontmatter and section schemas, category list
├── index.md # Generated slim catalog: categories, counts, coverage
├── indexes/ # Generated per-category page listings
├── logs/ # The narrative record. See "Logs, not commit messages"
│ ├── {YYYY-MM-DD}-{agent}-{host}.md
│ ├── README.md # generated folder guide and topic index
│ └── reports/ # audit reports written for a person to read
├── scripts/ # Extraction, validation, index and log builders
├── papers/ # Original PDFs, canonical storage
│ └── {author}-{year}-{title-5-tokens}.pdf
├── papers-supplementary/ # Publisher supplements, kept out of papers/
│ └── manifest/ # what each opaque publisher filename actually holds
├── sources/ # PDF summaries, English
├── wiki/ # The knowledge layer, English. Obsidian vault root
│ ├── {category}/
│ ├── concepts/ # methods, mechanisms, reusable definitions
│ ├── overviews/ # encyclopedic topic pages, declarative titles
│ └── questions/ # paper-anchored research questions, question titles
├── agenda/ # Project execution notes, plans, handoffs, unpublished work
└── materials/ # Non-paper reference material
The last four items did not exist in the first version and are the most-used parts of the system now.
Three synthesis layers, deliberately kept apart:
overviews/is encyclopedic. Declarative noun-phrase titles: "Astrocyte heterogeneity across brain regions". It states what the field currently holds.concepts/is the dictionary. A stable definition, method, or mechanism that many papers lean on, written once so that paper pages can point at it instead of re-explaining it.questions/is explicitly a question layer. The title is a question, and the body follows a fixed shape:## Question,## Sharper follow-up,## What the knowledge base holds,## Tentative answer. It is where an open problem lives while the evidence is still thin.
Keeping them separate matters more than it sounds. Question-shaped pages filed into overviews/ make the encyclopedia read like a lab notebook, and the encyclopedia is the part you hand to somebody else.
agenda/{project-or-context}/ holds everything that is about doing research rather than about published knowledge: execution plans, handoff documents, analysis to-do lists, decision records, journal-selection notes, meeting follow-ups, training material for new lab members, and notes on unpublished manuscripts. It is indexed for search alongside wiki/ and sources/, so questions can draw on it, but it is a separate namespace with no schema requirements.
Two conventions make it work. First, a handoff document has a fixed shape: why the method is needed and what it enables, the analyses to run, the expected results, a decision section saying when to continue or pause or stop if the results are weak, and the concrete deliverables. Second, agenda/ is where confidential material goes — rule 8 sends unpublished manuscript notes here precisely because it is outside the knowledge layer you might one day share.
Give the operating decisions about the wiki itself their own subfolder (agenda/llm-wiki-ops/). Short ADR-style notes on why a rule exists survive far better than the rule alone.
Not everything worth keeping is a paper, and forcing non-papers into the paper schema corrupts both. materials/ holds those, with no filename convention, no category, no PDF frontmatter requirement:
journals/— scope statements, author guidelines, and editorial-board snapshots for the journals you might submit to. Treat every file as a dated snapshot and re-verify before an actual submission.writing-style/— profiles of how specific scientists write, built from verbatim quotes of their papers, for learning sentence construction and argument flow rather than findings.lectures/— notes from talks and courses.meetings/— curated meeting and journal-club notes. Keep the raw audio and raw transcripts outside the wiki entirely; only the curated note comes in.professional-activity/— your own talks and service record.
The boundary is a rule, not a suggestion: do not link materials/ from wiki/, sources/, or index.md. If a style profile or a journal snapshot starts appearing in the paper knowledge graph, the graph stops being a graph of papers.
All three tiers share one stem:
{first-author-lastname}-{year}-{first-5-title-tokens}.{ext}
Tokens, not words. Lowercase the title, then take runs of alphanumerics: every non-alphanumeric character is a separator, not something to delete. self-report is two tokens, not one; digits count as tokens. So Direct haplotype-resolved 5-base HiFi sequencing becomes direct-haplotype-resolved-5-base. Hyphenated titles hit five tokens faster than you expect, which is fine.
Two things learned the hard way:
- Let a script generate the stem. Hand-built stems drift, and stop-word handling ("of", "the", "and") is exactly the kind of decision two scripts written a year apart will make differently. Ours did, and now both forms exist on disk.
- Do not bulk-rename legacy stems to match a newer rule. Every
[[wikilink]]and index entry has to follow, and you have to decide which form is canonical first. Fix only the cases where a paper's own three tiers disagree with each other.
Consortium papers use the consortium name. Author parsing is the one place worth a real test suite: a naive splitter that breaks on ; and and but not on commas reads "Cristina Bacon, Gudrun A. Rappold" as one author and files the paper under the wrong name.
Standard workflow is that you download papers yourself and hand them over. Make ~/Downloads the default inbound location so "ingest the papers I just downloaded" needs no path. The agent identifies each file from its first page — title, authors, DOI — and asks when a mapping is ambiguous instead of guessing. It never moves or deletes anything in your directories.
Copy, never symlink. pdf_path always points inside papers/, and pdf_filename is its basename.
For extraction, opendataloader-pdf is the default and it is a real upgrade over the simpler options. In a head-to-head on two-column papers, PyMuPDF interleaved figure-panel scaffolding — axis ticks, stray gene labels — into the body text and made whole passages unreadable, while opendataloader reconstructed paragraphs as continuous prose, separated figure captions, tidied references, and stripped running headers. It needs Java, so wrap it in a shell helper that locates a working Python and Java on whatever machine you are on, with a timeout, falling back to pypdf and then pdftotext:
bash scripts/extract_pdf_text.sh "/path/to/paper.pdf" 1-25 400000The two arguments are a page range and a character cap, and confusing them is the most expensive small mistake in this system. A generous character cap costs nothing; a small one truncates the paper mid-Methods and reports success. Check the page count with pdfinfo first and widen the range for long papers. The old advice in the first version of this gist — 15 pages, 12,000 characters — is enough for an abstract-grade summary and not enough for a real one. That was a stale default and it is why this section exists.
For a narrower job the answer flips. When the goal is pulling gene and relation lists out for a knowledge graph, PyMuPDF keeps columns pure, matches on recall, needs no Java, and runs six to twenty-four times faster. Pick the extractor for the task.
Record what you used. Three frontmatter fields, mandatory on every new or re-extracted record:
source_format: pdf
text_extractor: opendataloader-pdf-2.x
text_extracted_date: 2026-07-29This is what makes re-extraction possible later: when a better tool arrives, you can select exactly the pages built with the worse one instead of redoing everything. Do not invent values for old records that lack them; that is a backlog, not a formatting problem.
Raw extractor output needs a bounded cleanup before it becomes a page. Write down exactly what may be removed — page furniture, running headers, hyphenation at line breaks, empty pseudo-tables — and remove nothing else. An unbounded "clean this up" pass will eat content.
Fixed frontmatter and a fixed section list: One-line Summary, Document Information, Key Contributions, Methodology and Architecture, Key Results and Benchmarks, Limitations and Future Work, Related Work, Glossary.
Do not add a section holding the raw extraction dump. The canonical full text is the PDF; copying fifteen pages of it into the summary duplicates the original and drags publisher boilerplate — line numbers, copyright lines, "OPEN ACCESS" — into your search index.
Keep processing metadata out of paper frontmatter entirely. No tier, no curation_level, no batch names, no agent names, no dated run identifiers. Batch processing is an execution strategy, and it should be invisible in the knowledge layer. Provenance about a run belongs in the log.
Same frontmatter plus source: and tags:. Sections: Summary, Key Contributions, Methodology and Architecture, Results, Related Papers.
This is not optional and it is the step that gets skipped. It has its own section below.
Append one entry to the day's log file, then regenerate the catalog and the log index. Do not put ingest narrative into index.md.
The agent does all of this in one go when you say "add this paper to the wiki".
A source and wiki page with no link into overviews/ or concepts/ is a synthesis-orphan. It behaves like a RAG chunk: retrievable, but connected to nothing, contributing nothing to any question you did not phrase in its own vocabulary. Accumulate enough of them and you have built a search index with extra steps.
So every ingest, routine ones included, ends with four things:
- At least one bidirectional link between the new paper page and the single most relevant existing overview or concept page. Bidirectional matters:
Related Paperson the paper,Related Pageson the synthesis page. A one-way link is invisible from the side that would have used it. - One sentence in that synthesis page's body placing the paper in context. Not a bullet in a list — a sentence in the prose. If the paper genuinely only restates what the page already says, a link is enough, but say so in the log.
- A supersede check. Does this paper strengthen, narrow, contradict, or replace a claim already written down? If it contradicts or replaces one, record that explicitly in the day's log as a supersede entry. This is the difference between a wiki and a pile of summaries: the pile keeps both claims and lets you find whichever one you searched for.
- If no overview or concept covers the topic, write the gap down rather than skipping silently. Recurring gaps become the next anchor page.
Audit the drift with a read-only script that lists orphans. The orphan list is your backlog, and it should trend to zero rather than growing with the corpus.
For a paper that can change an active argument, go further: read the existing synthesis before writing, update the page bodies rather than just the links, create the concept anchor if one is missing, and make the claim delta explicit. Process those one at a time. The value of this system is concentrated in maybe two percent of the papers, and batching them is how you lose it.
Every failure below was found in real papers. They share one property: the extractor reports success.
- Glyphs are substituted, not dropped. A missing character is easy to catch; a character silently replaced by a different one is not. Greek letters inside proper nouns are a common victim.
- Minus signs vanish. Undetectable in principle from the text alone — a negative effect size becomes a positive one and nothing marks it. If a sign matters, look at the PDF.
- Table column order flips, and adjacent columns merge. Numbers end up attached to the wrong label.
- Figure reading order scrambles. Values get bound to the wrong axis label. If a paper's conclusion lives in a heatmap direction or an effect-size plot, render the figure to PNG and read it as an image. Text extraction cannot see color, and no warning tells you it could not.
- The body silently disappears. Whole sections come back empty on some font runs. Corruption is per font run, not per file, so a spot check on page 2 proves nothing about page 14.
- A table that is an image is invisible even to a missing-content check, because nothing is missing from the extractor's point of view.
- Zero-width spaces produce false alarms in completeness checks.
- Superscript flattening is a layout failure, not an encoding one, so font tools do not warn about it.
- Sometimes another paper is appended to the end of the PDF. The extraction succeeds and now two papers are one.
Two working rules follow. First, when the extractor dies it may fall back silently — check which tool actually produced the text, which is one more reason for text_extractor in the frontmatter. Second, when you build a substitution table to repair a known corruption, never run it in reverse "to be safe": that corrupts clean text at scale, and it is much harder to detect than the original problem.
The original setup kept the wiki in git. That ended, for a reason worth passing on: the repository lived inside a cloud-synced folder, and one day the sync produced a conflicted copy of the git index, which staged 980 healthy files as deletions. If you want git, keep the .git directory outside the synced folder. Or drop git, let the sync tool own versioning, and accept that per-file version history is your rollback.
Dropping git removes the one thing git was quietly providing: a written record of what changed and why. So the log takes over, and it needs more discipline than a commit message did.
One Markdown file per day, per agent, per machine:
logs/{YYYY-MM-DD}-{agent}-{host}.md
The host token matters as soon as two machines share the folder — two agents working the same day would otherwise collide on one file. Resolve the host by running a command, never from memory. The file opens with frontmatter recording date, agent, host, and which model did the work, then append-only entries, oldest first.
Entry headings carry a kind and a topic slug, so one campaign can be followed across days, agents, and machines:
## [2026-07-29] ingest | canonical-pdf-contamination | Paper or batch title
## [2026-07-29] maintenance | canonical-pdf-contamination | Re-extraction sweep
## [2026-07-29] supersede | astrocyte-reactivity | Claim replaced
Kinds in use: ingest, maintenance, query-to-wiki, semantic-lint, supersede. Under the heading go optional index lines — which records changed, which scripts ran, which report was produced — then the narrative.
Read the existing topic list before inventing a slug. A near-duplicate slug splits one campaign into two and neither half looks incomplete.
Generate logs/README.md from the headings rather than maintaining it. A generated index cannot claim work the logs do not record, and it surfaces entries missing a slug as a backlog. Keep reports in a subfolder: a dated report sitting beside dated daily logs gets read back as a log.
Claude Code and Codex both work in this folder, which is why the rule file is AGENTS.md with CLAUDE.md as a symlink to it. One rulebook, agent-neutral wording, no per-tool dialects unless a rule genuinely applies to one tool only.
The friction is not the rules, it is concurrency. The default BM25 index is deliberately not shared: each machine keeps its runtime, versioned indexes, build lock, and success stamp outside the synced wiki. A rebuild writes a complete new version, validates it, and only then atomically repoints current; readers never see a half-built index. Semantic fallback has a stricter rule: allow one local vector process at a time and fail loudly on an empty result, because concurrent Metal-backed searches can return empty rankings without reporting an error.
A startup checklist in the rule file earns its place here too: re-read the rulebook, re-read the registries that change often, and do not rely on memory from a previous session for anything that moves.
This is how the wiki actually grows. Not "ingest a thousand papers, then search" — branching outward from real questions.
Root question ("what is known about non-cortical brain cell types?")
├── 1st wave: direct overview pages
│ ├── Thalamic molecular architecture
│ └── Cerebellar cell diversity
├── 2nd wave: branches from what the first wave surfaced
│ ├── Dopaminergic neuron diversity
│ └── Region-specific disease vulnerability
└── 3rd wave: cross-cutting themes
└── Circadian regulation in brain evolution
In practice:
- Ask a question. The agent searches the wiki and answers from what exists.
- If the wiki is insufficient, it re-reads the original PDFs and updates the pages.
- If no paper covers it, it says so and you supply the PDF.
- Save the good answers. A durable comparison, decision framework, or synthesis goes to
overviews/orconcepts/; an open problem goes toquestions/; an execution plan goes toagenda/.
Do not save every casual answer. Save the ones a future reader would search for.
Hold off until you need to.
- A category past ~500 files should split. Pick the split axis by asking what you would deliberately exclude when reading about the topic.
index.mdpast a few hundred pages should stop being one file. Ours is now a slim root catalog — one row per category with a page count and a synthesis-coverage number — linking to a generated per-category listing. The coverage column is the useful part: it makes orphan accumulation visible on the front page instead of buried in an audit.- The total wiki past ~500 pages wants a real search index. The default is a pinned, memory-mapped
bm25sindex built directly fromwiki/**/*.md. QMD remains useful for source/agenda collection navigation and serialized semantic fallback, but it is not an upstream dependency of the production index. Below that threshold,index.mdplus the agent's built-in search is fine.
Pin the runtime (bm25s==0.3.10, PyStemmer==3.1.0), use Lucene-style BM25 with English stemming and stopwords, and exclude wiki/questions/ from results by default. Question pages are useful synthesis, but their query-shaped titles and prose can dominate a search for the question they were derived from. Return the evidence-bearing paper, overview, and concept pages instead.
Expose the workflow through project-local scripts with this interface:
# One-time per machine: install the runtime, build the first index, add a local schedule
bash scripts/setup_llm_wiki_retrieval.sh
# Default read path: use the complete English research question
bash scripts/search_llm_wiki.sh "Which mechanisms connect tandem repeat mutations to autism?" --json
# A non-English question against an English corpus keeps the original and supplies a
# complete English rewrite, not a three-keyword reduction
bash scripts/search_llm_wiki.sh "자폐증과 탠덤반복 변이를 연결하는 기전은?" \
--english-query "Which mechanisms connect tandem repeat mutations to autism?" --json
# Rebuild now only when immediate freshness matters
bash scripts/reindex_bm25s.shKeep the runtime and index under machine-local application/cache directories, never inside the cloud-synced wiki. A scheduled rebuild may run independently on every machine because it reads the synced Markdown but writes only local state. Build into a new version directory, verify document count, corpus hash, memory-mapped load, and a non-question smoke query, then atomically switch current and retain the previous complete version for rollback. Do not rebuild after every ingest; let the next scheduled run pick up routine changes.
When lexical candidates are ambiguous or insufficient, use a serialized semantic fallback, then decompose multi-document or conflicting-evidence questions into narrower searches. A retrieval score is never an answerability threshold: read the retrieved pages and abstain when they do not explicitly support the relationship in the question.
The switch was empirical. On 120 real stored research questions, full-question BM25 found the primary paper in 118/120 searches and some valid evidence in 120/120 at top 10, with a 0.68 ms median. The QMD vector baseline found the primary paper in 113/120 and some evidence in 119/120, with a 678 ms median. BM25 also supported concurrent memory-mapped reads without changing rankings. These numbers justify BM25 as the candidate generator; they do not justify treating lexical similarity as evidence.
Run a validator before and after any large ingest or reorganization. Ours checks YAML frontmatter, PDF paths, source-to-wiki references, broken wikilinks, category drift, duplicate titles, forbidden ingest metadata, and two body-level patterns worth calling out: sections containing only an instruction to consult the original, and pages stating their own processing status. The fix for either is to re-ingest from the PDF or drop the record. Deleting the placeholder line is the wrong fix, because it leaves an empty section that reads as a complete one and is invisible to every text check afterward.
The agent handles ingest and questions; Obsidian is the best way to read and navigate. Free, local, native [[wikilinks]], graph view, full-text search. Open the wiki folder as a vault. It only reads files, so it never interferes with what the agent maintains.
One gotcha that cost a full revert. The vault root is wiki/, not the repository root, so wikilinks resolve relative to wiki/ and must never carry a wiki/ prefix. [[overviews/some-page]] is correct; [[wiki/overviews/some-page]] resolves to wiki/wiki/overviews/..., which does not exist, and Obsidian creates an empty ghost page there every time somebody clicks it. A well-meaning pass that added the prefix to tens of thousands of links broke the vault, and link-checking scripts accept both forms, so they do not catch it. Related: do not create an .obsidian/ directory at the repository root — its presence is what makes a tool assume the wrong vault root.
- Keep scratch work outside the wiki. Temporary folders inside the repository get synced to every other machine and pollute the knowledge base. Send intermediate artifacts to
~/tmpand clean up the caches tools leave behind. - Never name a shell loop variable
path. In zsh it is a special array tied toPATH, andread -r key pathcan make basic commands disappear mid-script. The same applies toHOME,IFS,SHELL, and friends. - Supplements do not go in
papers/, which holds exactly one canonical PDF per paper. They get their own folder and, because publishers name them things like41586_2024_1234_MOESM3_ESM.pdf, a manifest saying what each file actually contains. Peer-review files and blank checklists are not kept — with one exception: a reporting-summary form is author-written and often holds the power analysis, exclusion criteria, blinding, software versions, and demographics that appear nowhere else. Keep it, and stop your page range before it so its blank boilerplate never reaches a page. - Do not spawn another model CLI or API from an ingest script. Paper interpretation stays in the active session; scripts do deterministic extraction, validation, logging, and index maintenance. An ingest pipeline that calls out to a second model produces pages nobody read.
You do not need to install Python, Java, or Node yourself. Claude Code and Codex have native installers for Mac, Linux, and Windows, with no WSL2 needed. Let the agent bootstrap.
-
Install Claude Code or Codex.
-
Open the agent in an empty folder. This becomes your wiki root.
-
Paste this prompt:
Set up an LLM Wiki for me, following this gist:
https://gist.github.com/joonan30/cbce305684d079dbe9a3fbaefe4e3959Read all files in the gist, ask me about my research field and 5–10 starting categories, then:
- Create
papers/,sources/,wiki/{my-categories}/,wiki/overviews/,wiki/concepts/,wiki/questions/,agenda/,materials/,logs/,scripts/ - Write
AGENTS.mdfrom the template in this gist, filled in for my domain, and symlinkCLAUDE.mdto it - Write
scripts/extract_pdf_text.shper the extraction section, with opendataloader-pdf first and pypdf/pdftotext as fallbacks - Implement the retrieval interface described above, including setup, pinned-runtime installation, atomic reindexing, search, and any helper scripts it needs; keep the runtime and indexes outside the wiki
- Apply all the rules from this gist verbatim, including the synthesis requirement. Never use web search.
- Create
-
Drop in your first 5–10 PDFs and say "add these papers to the wiki".
-
Ask questions. Save the good answers as overview, concept, or question pages.
-
When you cross ~500 pages, run
bash scripts/setup_llm_wiki_retrieval.shonce on each machine that will search the wiki.
The wiki this came from currently holds 15,259 PDFs, 14,872 source summaries, and 16,294 wiki pages, of which 459 are overviews, 368 are concepts, and 550 are question pages, spread across 52 categories.
The paper count is the least interesting number there. The synthesis pages grow faster than the corpus over time and they are where the value sits — they are what makes the thing answer a question rather than retrieve a document. If you take one operational rule from this revision, take rule 5: a paper you did not connect to anything is a paper you will not benefit from having read.
You need none of this on day one. Start with the four rules and five papers.
- Replaced QMD-first retrieval with full-question
bm25sas the default candidate generator, backed by a 120-question benchmark. - Made retrieval state machine-local, memory-mapped, versioned, validated before an atomic switch, and independently schedulable on every synced machine.
- Kept QMD as an optional serialized semantic and collection-navigation fallback rather than an upstream dependency.
- Added the synthesis requirement and the synthesis-orphan concept, plus
wiki/concepts/andwiki/questions/as distinct layers. - Replaced the pypdf snippet with opendataloader-pdf as the default extractor, the A/B result behind that choice, the page-range vs character-cap trap, and mandatory extraction-provenance frontmatter.
- Added the extraction-failure catalog. Every entry is a real silent failure.
- Added the log convention that replaced commit messages, and the reason git left the folder.
- Added
agenda/andmaterials/as first-class folders. - Added rules on placeholders and tiers, exhaustive reporting, confidential material, PDF-only acquisition, and correction notices.
- Made the naming convention token-based, with the warning against bulk-renaming legacy stems.
- Split
index.mdinto a slim catalog plus per-category listings, with a synthesis-coverage column. - Added the Obsidian vault-root gotcha, the concurrency lock for shared search indexes, and the two-agent setup.
- Renamed
CLAUDE.md.templatetoAGENTS.md.template.
Built with Claude Code (Anthropic) and Codex (OpenAI). Browsing with Obsidian. Default retrieval with bm25s; semantic fallback with QMD. Extraction with opendataloader-pdf. For Karpathy's original idea: @karpathy/1dd0294ef9567971c1e4348a90d69285.