Obsidian RAG with Claude integration

setup-llm-wiki.sh

#!/usr/bin/env bash
#
# setup-llm-wiki.sh — scaffold a git-backed LLM Wiki (Karpathy pattern).
#
# Safe to re-run: existing files are never clobbered (only missing ones are created).
# Override the location with: LLM_WIKI_DIR=/path/to/wiki ./setup-llm-wiki.sh
#
set -euo pipefail

WIKI_DIR="${LLM_WIKI_DIR:-$HOME/llm-wiki}"
DATE="$(date +%F)"

# --- guards --------------------------------------------------------------------

command -v git >/dev/null 2>&1 || { echo "error: git not found in PATH" >&2; exit 1; }

case "$WIKI_DIR" in
  *"Mobile Documents"*|*"com~apple~CloudDocs"*|*"iCloud~"*)
    echo "WARNING: '$WIKI_DIR' is inside iCloud. A git repo + iCloud sync produces" >&2
    echo "         conflict copies and a corrupted .git. This is exactly what we" >&2
    echo "         agreed not to do. Set LLM_WIKI_DIR to a non-synced path." >&2
    echo "         Continuing anyway in 3s... (Ctrl-C to abort)" >&2
    sleep 3 ;;
esac

echo "==> Scaffolding LLM Wiki at: $WIKI_DIR"

# --- helper: write only if the file is absent ----------------------------------

write_if_absent() {
  local path="$1"
  if [[ -e "$path" ]]; then
    cat >/dev/null            # drain heredoc, leave file untouched
    echo "  skip (exists): ${path#"$WIKI_DIR"/}"
  else
    cat >"$path"
    echo "  created:       ${path#"$WIKI_DIR"/}"
  fi
}

# --- directory tree ------------------------------------------------------------

mkdir -p \
  "$WIKI_DIR/raw/assets" \
  "$WIKI_DIR/wiki/sources" \
  "$WIKI_DIR/wiki/entities" \
  "$WIKI_DIR/wiki/concepts"

# keep otherwise-empty dirs in git
for d in raw/assets wiki/sources wiki/entities wiki/concepts; do
  : >"$WIKI_DIR/$d/.gitkeep" 2>/dev/null || true
done

# --- CLAUDE.md (the operating contract) ----------------------------------------
# Quoted heredoc: nothing inside is expanded (preserves backticks/brackets).

write_if_absent "$WIKI_DIR/CLAUDE.md" <<'EOF'
# LLM Wiki — Schema & Operating Contract

You are the maintainer of this vault, run as an LLM Wiki (Karpathy pattern).
This is NOT a RAG store. You do not embed-and-retrieve. You read sources, compile
knowledge into interlinked markdown, and keep it current. You own `wiki/` entirely.
The human curates sources and asks questions. You do all bookkeeping.

## Architecture (three layers)
- `raw/`   — immutable source documents (clipped articles, papers, notes, transcripts).
             READ-ONLY. Never edit or delete. This is the source of truth.
- `wiki/`  — your output. LLM-generated, interlinked `.md`. You create/update freely.
- this file — the contract. We co-evolve it as conventions emerge. Don't bloat it.

## Layout
```
raw/                 # sources in, never touched
raw/assets/          # downloaded images
wiki/
  index.md           # catalog of every page (you maintain)
  log.md             # append-only event log (you maintain)
  overview.md        # the living synthesis / thesis
  sources/           # one summary page per ingested source
  entities/          # people, orgs, products, systems
  concepts/          # ideas, methods, patterns
```

## Page conventions
- Every page starts with YAML frontmatter:
  ```
  ---
  type: source | entity | concept | overview
  created: YYYY-MM-DD
  updated: YYYY-MM-DD
  sources: ["raw/<file>", ...]   # citation = stable identity, see Idempotency
  tags: [...]
  ---
  ```
- Cross-link aggressively with [[wikilinks]]. A page with no inbound links is a bug.
- Source pages cite their `raw/` origin in frontmatter `sources`. Always.
- Write claims so they trace to a source. No orphan assertions.

## Operations

### Ingest  (human drops a file in raw/, says "ingest X")
1. Read the source fully.
2. Surface 3-5 key takeaways to me before writing. Wait if I want to steer.
3. Write/update `wiki/sources/<slug>.md` (summary + frontmatter citation).
4. Update affected `entities/` and `concepts/` pages — create them if missing.
5. If new data contradicts an existing claim: do NOT silently overwrite.
   Note both, flag the contradiction inline, surface it to me.
6. Update `index.md`. Append one line to `log.md`.
   A single source typically touches 5-15 pages. That's expected.

### Query  ("ask the wiki ...")
1. Read `index.md` first to locate relevant pages, then drill in.
2. Answer with citations to wiki pages / raw sources.
3. If the answer is reusable (a comparison, an analysis, a synthesis),
   ASK whether to file it back as a new page. Good answers shouldn't die in chat.

### Lint  ("lint the wiki" — run periodically)
Health-check and report (don't auto-fix structural stuff without confirming):
- contradictions between pages
- stale claims superseded by newer sources
- orphan pages (no inbound links)
- concepts mentioned but lacking their own page
- missing cross-references
- gaps worth a web search
Suggest next sources/questions worth pursuing.

## Idempotency (the one rule that prevents rot)
Before writing any claim, check whether its source citation already exists in the
target page (grep the `sources` frontmatter / inline cites). If the source was already
ingested, UPDATE in place — never append a second near-duplicate. Re-ingesting the same
source in a different order must converge to the same state. Identity = citation, not text.

## index.md format
Catalog, grouped by type. Each entry: `[[page]] — one-line summary (n sources)`.
Update on every ingest.

## log.md format
Append-only, newest at bottom. One line per event, greppable prefix:
`## [YYYY-MM-DD] ingest | <Title>`  /  `query`  /  `lint`
So `grep "^## \[" wiki/log.md | tail -5` gives recent activity.

## Hard rules
- Never modify `raw/`.
- Never fabricate. If unsure, mark it and flag for review.
- Don't reconcile contradictions into a single "clean" account silently — surface them.
- Keep this schema lean. A new convention earns its place only after it's hurt twice.
EOF

# --- wiki stubs (unquoted heredocs: expand ${DATE}, no backticks inside) --------

write_if_absent "$WIKI_DIR/wiki/index.md" <<EOF
---
type: index
created: ${DATE}
updated: ${DATE}
---

# Index

Catalog of every wiki page. Maintained by the LLM on each ingest.

## Sources

## Entities

## Concepts
EOF

write_if_absent "$WIKI_DIR/wiki/log.md" <<EOF
# Log

Append-only. Newest at bottom. Prefix every entry with: ## [YYYY-MM-DD] <op> | <Title>

## [${DATE}] init | wiki scaffolded
EOF

write_if_absent "$WIKI_DIR/wiki/overview.md" <<EOF
---
type: overview
created: ${DATE}
updated: ${DATE}
---

# Overview

Living synthesis. The LLM maintains this. Empty until the first source is ingested.
EOF

# --- .gitignore (quoted: literal) ----------------------------------------------

write_if_absent "$WIKI_DIR/.gitignore" <<'EOF'
# macOS
.DS_Store

# iCloud placeholder stubs (should never be here, but just in case)
*.icloud

# sync conflict copies (iCloud / Dropbox / etc.)
* [0-9].md
*conflicted copy*

# Obsidian: keep config, drop per-machine workspace/cache
.obsidian/workspace*
.obsidian/cache
.trash/
EOF

# --- README (quoted: literal) --------------------------------------------------

write_if_absent "$WIKI_DIR/README.md" <<'EOF'
# LLM Wiki

A git-backed personal knowledge base on the Karpathy LLM-Wiki pattern.
Not RAG: an agent compiles sources into interlinked markdown and keeps it current.

- `raw/`  — immutable sources you drop in
- `wiki/` — agent-owned, interlinked markdown
- `CLAUDE.md` — the operating contract the agent follows

Open this folder as an Obsidian vault for the graph view. Run an agent
(Claude Code / Codex) from this directory to ingest, query, and lint.
EOF

# --- git -----------------------------------------------------------------------

if [[ ! -d "$WIKI_DIR/.git" ]]; then
  git -C "$WIKI_DIR" init -q
  echo "  git init"
fi

git -C "$WIKI_DIR" add -A
if git -C "$WIKI_DIR" diff --cached --quiet; then
  echo "  git: nothing to commit"
else
  if git -C "$WIKI_DIR" commit -q -m "chore: scaffold llm-wiki (Karpathy pattern)"; then
    echo "  git commit"
  else
    echo "  git: commit failed — set your identity, then commit manually:" >&2
    echo "       git -C \"$WIKI_DIR\" config user.name  'you'" >&2
    echo "       git -C \"$WIKI_DIR\" config user.email 'you@example.com'" >&2
  fi
fi

echo
echo "==> Done. Next:"
echo "    1. Obsidian → Open folder as vault → $WIKI_DIR"
echo "    2. cd \"$WIKI_DIR\" and start your agent there."
echo "    3. Drop a file in raw/, tell the agent: \"ingest <file>\"."
echo "    4. Add a remote when ready:  git -C \"$WIKI_DIR\" remote add origin <url>"