Paxel Discovery Rewrite Spec

Summary

The current host-side discovery path is slow because it repeatedly reconstructs the same repo identity information from scratch using Bash loops and many short-lived subprocesses. The slow part is not the Docker analysis pipeline itself. The slow part is the pre-analysis scanner that tries to answer these questions on every run:

Which local transcript/session directories exist?
Which working directory did each one come from?
Which repo/worktree does that path belong to?
What is the canonical remote for that repo?
If the original path is gone, can we recover the remote from nearby live worktrees?

This spec proposes replacing the current shell-based discovery layer with a compiled local scanner plus a persistent local index. The goal is to preserve the same external behavior while making the common path fast and making the worst path bounded.

Goals

Preserve existing product behavior and upload semantics.
Keep the Docker analysis container and auth flow intact.
Preserve support for Claude Code, Codex CLI, Cursor, opencode, and Gemini inputs.
Preserve current remote normalization semantics.
Preserve current orphan/deleted-worktree recovery behavior where practical.
Make repeated runs incremental instead of mostly full rescans.
Reduce startup time for large transcript histories and large worktree farms.

Non-goals

Rewriting the analysis pipeline inside the Docker container.
Changing what data is uploaded.
Changing project grouping semantics.
Removing support for orphan recovery.
Making the scanner depend on GitHub APIs or any network access.

Current Problems

1. Process-heavy hot path

The current script repeatedly shells out to git, find, stat, jq, grep, sed, awk, and sqlite3. The cost is dominated by process startup and repeated filesystem walks, not just raw CPU.

2. Per-workspace remote lookup

A large number of transcript buckets eventually funnel into the same repo remote, but the script often resolves them independently. This is especially wasteful for multiple worktrees belonging to one underlying repo.

3. Repeated metadata extraction

For Claude projects, the script must recover the original working directory from sessions-index.json or by reading JSONL content. This is done repeatedly rather than being treated as indexed metadata.

4. Expensive orphan recovery

Deleted worktrees trigger recovery heuristics:

ancestor walks
sibling stem guesses
git worktree list --porcelain
jj workspace list

This logic is correct-minded but expensive when repeated across many dead paths.

5. Weak run-local memory

The current cache is a TSV file and is useful, but the shell implementation still replays too much work:

inferred entries are revalidated frequently
resolver state does not persist in-memory cleanly through subshell boundaries
the scanner still walks broad directory sets to decide what changed

6. Discovery work is interleaved with shell orchestration

The shell script currently mixes:

auth and user interaction
Docker setup
local discovery
cache mutation
log and replay logic

That makes the slow path harder to optimize independently.

Proposed Architecture

Split the current script into two layers:

Layer 1: Thin shell wrapper

Keep Bash only for:

Docker checks
auth and token handling
replay/pending-upload logic
user-facing prompts
invoking the scanner binary
translating scanner output into Docker mount args

Layer 2: Compiled local scanner

Introduce a single local binary, preferably Go, responsible for:

session source discovery
metadata extraction
cwd and repo-root resolution
remote normalization
orphan/worktree recovery
persistent indexing
incremental invalidation
emitting one manifest describing the run

The Docker container should consume the scanner manifest instead of inferring as much from raw host layout.

Implementation Handoff

This section defines what an implementation agent should build first, what can be deferred, and what interfaces must stay stable.

Deliverable

Ship a new local binary named paxel-discover that can fully replace the current host-side discovery path while preserving current grouping and scoping behavior.

Required first deliverable

The first shippable milestone must include:

a Go binary paxel-discover
a SQLite-backed local index
Claude project discovery
live repo root and normalized remote resolution
Codex discovery
manifest JSON output
shell integration behind a feature flag
shadow-mode comparison against the current Bash path

Deferred to later milestones

The following can land after the first shippable milestone:

Cursor direct-SQLite reader
opencode direct-SQLite reader
Gemini direct reader
warm background watcher
full replacement of shell-generated sidecars

Stable interface boundaries

The implementation agent should preserve these boundaries:

Bash remains the entrypoint used by users.
paxel-discover owns host-side discovery and cache/index mutation.
Docker container continues to own analysis and upload.
The shell wrapper only translates scanner output into mounts, env vars, and user-facing messages.

Why Go

Go is the recommended choice for this rewrite.

Reasons:

fast enough for filesystem-heavy local scanning
easy bounded concurrency
straightforward static binary distribution
lower maintenance cost than Rust for this category of tool
good SQLite support
good cross-platform behavior for macOS, Linux, and Windows/Git Bash environments

Rust would also work, but the likely bottlenecks here are subprocess count, repeated I/O, and missing incremental indexing. Go is enough to solve those directly.

Scanner Responsibilities

The scanner should produce a deterministic manifest for the current run. Conceptually:

Enumerate source session locations.
Read only the minimum metadata required to attribute each session bucket.
Resolve each bucket to a repo identity.
Reuse cached answers whenever inputs have not changed.
Recover orphaned buckets using indexed heuristics.
Emit:
- selected transcript directories
- extracted sidecar metadata
- per-tool bucket mapping
- per-repo git metadata collection plan
- recovery telemetry

Manifest Output

The scanner should emit a JSON manifest under ~/.paxel/cache or a temp run directory. Suggested top-level structure:

{
  "version": 1,
  "run_scope": {
    "mode": "single-project",
    "selected_remote": "github.com/org/repo",
    "selected_label": "repo"
  },
  "projects": [
    {
      "source": "claude",
      "dir_name": "-Users-me-code-repo",
      "cwd": "/Users/me/code/repo",
      "repo_root": "/Users/me/code/repo",
      "normalized_remote": "github.com/org/repo",
      "session_count": 42,
      "recovery_source": null
    }
  ],
  "tool_buckets": {
    "codex": [],
    "cursor": [],
    "opencode": [],
    "gemini": []
  },
  "git_targets": [
    {
      "repo_root": "/Users/me/code/repo",
      "normalized_remote": "github.com/org/repo"
    }
  ],
  "telemetry": {
    "orphan_recovery_count": 3,
    "recovery_breakdown": {
      "ancestor": 1,
      "worktree_list": 2
    }
  }
}

The shell wrapper can then mount exactly what the manifest says rather than redoing discovery logic inline.

Manifest requirements

The manifest must be:

deterministic for the same filesystem state
self-contained for the current run
versioned
safe to consume from shell without further discovery work
explicit about which entries were recovered versus directly resolved

Minimum required fields

At minimum, the shell integration must be able to answer these questions from the manifest alone:

What mode is this run in: single-project, all-projects, or child-repo-selection?
What is the selected normalized remote, if any?
Which Claude directories should be mounted?
Which extracted per-tool buckets should be mounted?
Which git targets should have host-side git metadata collected?
What label should be shown to the user for the current run?
How many sessions per tool were included?
How many orphan recoveries happened, and by which source?

Suggested manifest file layout

The implementation agent should write all scanner outputs under one run directory, for example:

~/.paxel/cache/discovery-runs/<run-id>/
  manifest.json
  sidecar/
    _metadata.json
    _git/
  extracted/
    codex/
    cursor/
    opencode/
    gemini/

The shell wrapper should receive the manifest path and derive all runtime mounts from it.

CLI Contract

The scanner should expose one primary command:

paxel-discover scan [options]

Suggested options:

--claude-dir <path>
--codex-dir <path>
--cursor-dir <path>
--cursor-global-db <path>
--opencode-dir <path>
--opencode-db <path>
--gemini-dir <path>
--cwd <path>
--project <name>
--all
--since-epoch <unix-seconds>
--mode auto|single|all
--output-dir <path>
--shadow-compare-with-shell
--json

CLI behavior

Exit 0 on success, even if no sessions were found.
Emit a manifest path on stdout in --json mode.
Emit structured diagnostics on stderr.
Avoid interactive prompts entirely.
Never perform network access.

Suggested stdout contract

In --json mode, print exactly one JSON object:

{
  "ok": true,
  "manifest_path": "/Users/me/.paxel/cache/discovery-runs/abc123/manifest.json",
  "stats": {
    "claude_projects_scanned": 100,
    "repos_resolved": 14,
    "orphan_recoveries": 2
  }
}

This lets the shell wrapper avoid scraping plain text.

Persistent Index

Use SQLite rather than TSV files. Suggested tables:

`claude_projects`

dir_name
project_dir
latest_mtime
session_count
original_cwd
metadata_hash
last_scanned_at

`repo_resolution`

cwd
repo_root
normalized_remote
resolution_kind
resolution_source
resolved_from_path
verified_exists
last_verified_at

`repo_identity`

repo_root
git_common_dir
normalized_remote
git_dir_fingerprint
last_verified_at

`tool_sessions`

tool
session_id
source_path
cwd
normalized_remote
originator
mtime
last_scanned_at

`worktree_inventory`

repo_root
candidate_path
normalized_remote
inventory_source
last_seen_at

`scanner_state`

schema version
source roots
feature flags
last full scan timestamp

Schema notes

The implementation agent should enforce:

a schema version integer
migrations at process start
WAL mode for concurrent reads during future watcher support
indexes on dir_name, cwd, repo_root, normalized_remote, and source_path

Suggested indexes

claude_projects(dir_name)
claude_projects(latest_mtime)
repo_resolution(cwd)
repo_resolution(repo_root)
repo_resolution(normalized_remote)
tool_sessions(tool, source_path)
tool_sessions(normalized_remote)
worktree_inventory(candidate_path)

Cache invalidation rules

The implementation agent should codify these rules in code, not prose:

if source file or directory is missing, mark prior entry stale
if mtime or size changed, re-parse
if scanner version changed, invalidate derived rows
if schema version changed, migrate or rebuild
if repo identity lookup fails for a previously live repo, preserve the old row as historical but mark it unverified

Discovery Algorithm

Step 1: Fast source inventory

For each source root:

Claude: enumerate top-level project dirs only
Codex: enumerate JSONL files
Cursor: enumerate workspace DBs and global DB
opencode: enumerate opencode*.db
Gemini: enumerate chats/session files

At this stage, only collect path and mtime information.

Step 2: Incremental invalidation

For each discovered source item:

if mtime and size match an indexed entry, reuse the indexed parse result
otherwise re-parse just that source item

This alone removes most repeated parsing work on steady-state runs.

Step 3: Metadata extraction

Replace shell parsing with direct readers:

Claude:
- read sessions-index.json directly
- if absent, read only enough of the first relevant JSONL to obtain cwd
Codex:
- parse the first JSON object from each session file
Cursor:
- query SQLite directly from Go, not through the sqlite3 CLI
opencode:
- query SQLite directly
Gemini:
- infer from .project_root and chat files directly

No jq, grep, or sed should be needed.

Step 4: Repo identity resolution

For every distinct live cwd:

Find repo root once.
Resolve git common dir once.
Read remote config once.
Normalize once.
Reuse that answer for all related worktrees and session buckets.

Important change: the unit of resolution should be repo identity, not transcript directory.

Repo identity resolution contract

For each live candidate cwd, the scanner must emit:

cwd
repo_root
git_common_dir if available
normalized_remote
resolution_kind
- git-origin
- jj-origin
- local-only
- recovered
resolution_source
- direct
- indexed
- ancestor
- worktree-list
- jj-workspace-list
- project-name

Step 5: Orphan recovery

Do not recover each orphan independently from scratch.

Instead:

Build an in-memory inventory of all live repos and worktrees first.
Group orphaned paths by likely parent/project key.
Attempt recovery using the inventory.
Only if that fails, run targeted fallback probes.

Fallback strategy order:

indexed exact prior resolution
indexed repo-root match from historical inventory
ancestor repo match
worktree membership check
jj workspace membership check
project-name fallback

This preserves current behavior but cuts repeated expensive probes.

Orphan recovery constraints

The implementation agent should treat these as hard constraints:

Never use network APIs for recovery.
Never guess a normalized remote if there is no evidence chain.
Emit recovery source explicitly for every recovered entry.
Preserve current “unresolvable orphan” behavior when recovery fails.
Keep recovery telemetry queryable from the manifest and index.

Step 6: Scope selection

The scanner should own the project-scoping decision for:

current repo auto-detect
--project
--all
child repo selection
“Claude-less but Codex/Cursor/opencode/Gemini present” cases

The shell wrapper should consume the scanner’s answer, not replicate it.

Performance Techniques

1. Bounded concurrency

Use worker pools for:

directory metadata reads
JSON metadata parsing
live repo remote resolution
orphan grouping and candidate evaluation

Do not parallelize Git blindly without bounds. Start with a configurable pool size such as min(16, 2*cpu_count).

2. Repo-level deduplication

Many worktrees share one common Git config and one origin. Cache by:

git common dir
repo root
normalized remote

This prevents repeated remote get-url lookups for the same underlying repo.

3. In-process SQLite access

Cursor and opencode extraction should stop spawning the sqlite3 CLI. Use a SQLite driver directly and stream only the columns needed for attribution during discovery.

4. Minimal reads

Do not read full transcripts during discovery. Read only:

session header metadata
first line or first record where possible
index files
DB metadata queries

5. Structured cache invalidation

Do not invalidate by broad directory rescans alone. Invalidate on:

mtime change
file size change
missing path
source schema version bump
scanner version bump

6. Background warm index

Optional second phase:

run a background watcher using fsnotify or Watchman
update the SQLite index as new sessions land

Then the foreground upload command becomes mostly:

read current index
reconcile a small delta
emit manifest

This is the only path that will make the startup experience feel near-instant for heavy users.

Internal Module Layout

The implementation agent should keep the scanner code modular. Suggested package layout:

cmd/paxel-discover/
internal/config/
internal/manifest/
internal/index/
internal/inventory/
internal/sources/claude/
internal/sources/codex/
internal/sources/cursor/
internal/sources/opencode/
internal/sources/gemini/
internal/gitresolve/
internal/jjresolve/
internal/recovery/
internal/normalize/
internal/scoping/
internal/shadow/
internal/logging/

Module responsibilities

config: CLI flags and environment mapping
manifest: manifest structs and serialization
index: SQLite schema, migrations, read/write helpers
inventory: filesystem enumeration and source invalidation
sources/*: per-provider metadata extraction
gitresolve: repo root, common dir, origin lookup
jjresolve: jj workspace and remote lookup
recovery: orphan recovery planner and executor
normalize: canonical remote normalization parity with current behavior
scoping: --all, current cwd auto-detect, project selection, child repo logic
shadow: comparison against shell outputs during rollout
logging: structured diagnostics and counters

Shell Integration Contract

The implementation agent should update the Bash wrapper to do exactly this:

Parse user flags as it does today.
Call paxel-discover scan.
Read manifest.json.
Build Docker mounts and env vars from the manifest.
Continue into the existing Docker analysis path.

The shell wrapper should stop doing:

per-project cwd extraction
per-project remote lookup
orphan recovery
per-tool discovery walks
project-remote cache reads and writes

Feature flag

Gate the new path behind an env var first, for example:

PAXEL_DISCOVERY_V2=1

Recommended rollout:

default off in initial integration
default on for dev and internal users
shadow compare in production before default flip

Compatibility Behavior To Preserve

The rewrite should keep:

existing remote normalization semantics
git then jj fallback behavior
support for deleted worktrees
current per-tool grouping semantics
current distinction between standalone Codex and Claude-launched Codex
current handling of repo-less/local-only sessions
current --all, --project, --since, and --no-repo semantics

Migration Strategy

Phase 1: Scanner in shadow mode

Keep the Bash flow unchanged.
Add a scanner binary that runs alongside the shell logic.
Compare:
- selected remote
- per-project grouping
- recovery counts
- session counts
Log mismatches only.

Phase 1 acceptance criteria

scanner result can be generated without affecting current uploads
mismatch logs are structured and attributable to a run id
at least 95 percent of internal test fixtures match current shell output exactly

Phase 2: Scanner owns discovery, shell owns orchestration

Shell delegates discovery and cache loading entirely to the scanner.
Existing Docker invocation remains unchanged except it reads manifest output.

Phase 2 acceptance criteria

shell path no longer performs direct remote discovery
scanner path is measurably faster on warm runs
all existing upload modes still complete successfully

Phase 3: Direct manifest-driven mounts

Container inputs come from scanner outputs rather than ad hoc shell-generated temp dirs where possible.
Legacy shell fallback remains behind a flag for one release window.

Phase 3 acceptance criteria

no duplicate discovery logic remains in shell
all mount paths originate from the manifest
fallback path remains usable for emergency rollback

Phase 4: Optional warm daemon

Add a background index updater for power users with large histories.
Foreground upload becomes a manifest read plus small reconciliation pass.

Phase 4 acceptance criteria

watcher failures do not break foreground scanning
index can be rebuilt from scratch at any time
warm path is faster than non-watcher warm path for large histories

Validation

The rewrite is correct only if it preserves attribution semantics.

Required validation coverage:

single repo, live cwd
many worktrees, one remote
deleted worktree with recoverable sibling
deleted worktree with ancestor repo recovery
unrecoverable orphan
conductor workspaces
jj workspaces
mixed ssh/https remote forms
Cursor-only repo
opencode-only repo
Gemini-only repo
Claude-less single-project auto-detect
--all mode
child repo menu mode
Windows/Git Bash path handling
WSL Cursor path handling

Required measurements:

cold scan wall time
warm scan wall time
number of subprocesses spawned
number of filesystem stats
number of git invocations
index size over time
correctness delta versus current implementation

Test harness expectations

The implementation agent should add:

golden fixtures for each supported provider
golden fixtures for mixed-provider project grouping
orphan recovery fixtures
cross-platform path fixtures
integration tests for manifest generation
shadow diff tests comparing scanner output to current shell-derived output

Performance gates

The implementation agent should not flip the default until all of the following are true on representative internal datasets:

warm scan time is at least 3x faster than the current path
cold scan time is materially improved
git subprocess count is reduced by an order of magnitude on large worktree setups
correctness mismatches are understood and explicitly approved

Expected Results

If implemented correctly, the biggest gains should come from:

replacing many tiny subprocesses with in-process parsing
deduplicating remote resolution at the repo level
making steady-state runs incremental
making orphan recovery inventory-based instead of repeated ad hoc probing

The likely outcome is:

first run: noticeably faster, but still bounded by real filesystem and DB reads
subsequent runs: dramatically faster
large worktree setups: biggest absolute improvement
orphan-heavy setups: much more stable worst-case behavior

Recommended Implementation Order

Build the Go scanner with SQLite index and Claude project support.
Move repo-root and remote resolution into the scanner.
Add Codex support.
Add Cursor and opencode direct-SQLite readers.
Add Gemini support.
Port orphan recovery into indexed inventory-based logic.
Integrate scanner manifest into the existing shell wrapper.
Add shadow-mode comparison and mismatch logging.
Flip the default to scanner-driven discovery.
Add optional watcher-based warm indexing.

Immediate Task Breakdown

If this doc is handed to an implementation agent today, the first sequence of tasks should be:

Scaffold cmd/paxel-discover and internal/index with SQLite migrations.
Implement manifest writing and a minimal scan --json path.
Implement Claude inventory plus sessions-index.json and JSONL header parsing.
Implement normalized remote resolution for live git repos.
Implement repo-level deduplication keyed by repo root and common dir.
Integrate shell feature flag path that prints manifest-driven debug output without changing Docker mounts.
Add shadow comparison against the current shell grouping.
Add Codex support.
Replace shell single-project remote discovery with scanner results.
Expand provider coverage and orphan recovery until the shell path is removable.

Open Decisions

These are the only design choices that may need explicit confirmation during implementation:

whether the scanner writes extracted sidecar directories itself in phase 1, or only emits manifest metadata
whether the watcher ships in the same binary or as a separate subcommand later
whether recovery should preserve the exact current project-name fallback ordering or allow a stricter evidence-only mode behind a flag

Unless product requirements change, the implementation agent should not reopen broader architectural questions.

Practical Recommendation

Do not start with “rewrite the Bash script in Rust.” Start with “replace the discovery engine with a Go scanner and a real index.” That captures almost all of the upside while minimizing migration risk.

The product boundary should become:

Bash: interaction and launch
Go scanner: local discovery and attribution
Docker container: analysis and upload

That split is the cleanest way to keep behavior the same while making the slow part fast.

amxv/paxel-discovery-rewrite-spec-2026-06-19.md