A synthesized digest of all experiment threads, findings, decisions, open questions, and vocabulary conventions. Each thread is referenced to the primary source file(s).
Dates: 2026-04-17 (single session)
Sources: docs/session_20260417_notes.md, data/results/SUMMARY-20260417.md
Core hypothesis: Which individual recall parameters (limit, time_query, auto_decompose, expand_relations, tags) actually drive recall quality, and do they compound or substitute?
Setup: 78-memory synthetic corpus (v1), 3 fake projects + preferences, 10 session_start_v1 scenarios, 12 rulesets run in an N×M matrix on a local Docker AutoMem stack.
Headline result:
bare_tag_1m_v2recovers 44/44 expected hits vs baseline's 24/44 — a 2.5× improvement.- The gain is emergent from
limit + time_querycompounding, not any single knob.limitalone: +2 hits.time_queryalone: net zero (redistributes, doesn't add). Both together: +10 hits. auto_decomposeadds 0 on focused queries.expand_relationsis entirely blocked by the tag gate —expanded_count: 0whenevertags: [<slug>]is set. A 50-line client-side walker over.results[].relations[]achieves what server-side expansion cannot.- Dropping the tag gate is a -5 hit disaster. The project-slug gate is load-bearing.
Decision: Adopted. bare_tag_1m_v2 enshrined as the recommended default. expand_relations explicitly documented as a no-op under tag gating. Client-side expansion prototype created at runners/client_side_expand.py.
Natural next steps recorded: Agent-matrix dimension, negative controls, propose client-expand as mcp-automem feature, server-side bug report for tag-gate blocking expansion.
Dates: 2026-04-18 (single session)
Sources: data/results/SUMMARY-20260418-v2.md
Core hypothesis: Does the v1 knob-isolation finding (bare_tag_1m_v2 dominates) hold when the corpus scales to 370 memories, 6 projects, and Voyage voyage-4 embeddings?
Setup: New parameterized 370-memory corpus (corpus_v2.jsonl) with 50 preferences, 66 noise memories, 110 scenario IDs, a deliberate slug-collision (video) project, and memories aged up to 91+ days. 15 new session_start_v2 scenarios.
Headline result:
- Confirmed: 8 wins, 7 ties, 0 losses for
bare_tag_1m_v2. Same ~2.5× improvement: baseline 11 hits vs v2 27 hits in Phase 2+3. - Slug collision (
videoproject) is a real but manageable risk — specific semantic queries outscore noise even with a generic slug. SS-OS-SUNSET-V2: 0/4 under both schemes — dormant/old memories (60–180+ days) are missed even by the 90-day window. Confirmed need for a "stale project" variant.- Negative control (Kubernetes/Istio query on absent corpus): both schemes exhibit over-retrieval (5 tensor-pipeline memories returned); neither has a relevance floor.
Decision: Adopted. V2 corpus shipped as corpus_v2.embedded.jsonl. Stale-project variant and query relevance floor noted as follow-ups. compare_rulesets.py and snapshot_corpus.py extended with --manifest and --corpus flags.
Dates: 2026-04-21 → 2026-04-22
Sources: docs/session_20260421_beam_setup.md, docs/session_20260421_beam_smoke.md, docs/session_20260422_beam_fullbucket.md
Core hypothesis: Can AutoMem produce a BEAM score by impersonating mem0-OSS's REST wire contract through a shim, and where does it compare?
Setup: V1 shim (runners/beam_shim.py) does raw dialogue pass-through (no fact extraction). Run against official BEAM scorer via vendored third_party/memory-benchmarks. Multiple runs: gpt-5 smoke (2 conv, $5.44), Ollama Llama/Qwen experiments (failures), gpt-5-mini full bucket (20 conv, 400 questions, $8–10, 2h 54m).
Headline result (full bucket, gpt-5-mini): 76.25% pass rate (305/400), avg score 0.677.
Key category findings vs mem0's published 1M numbers:
contradiction_resolution: 90% vs mem0's 25–48% — standout win, graph-layer doing work.information_extraction: 97.5% — unambiguous retrieval strength.knowledge_update: 57.5% vs mem0's 65–80% — clear weakness; V1 shim can't fireINVALIDATED_BYedges on raw dialogue blobs.abstention: 55% — answerer confabulates from thematic context.
Cannot claim "AutoMem beats mem0" — different tiers (100K vs 1M/10M), different ingest pipelines, possible judge-model wedge. A Qwen vs gpt-5 rejudge study: 97.5% PASS/FAIL agreement — judge noise is ≤3pp at n=400.
Bug discovered: shim DELETE handler uses recall-by-tag (top-k, not exhaustive) → leaves thousands of memories un-cleaned. Workaround documented; real fix needs pagination endpoint.
Decision: Adopted as V1 baseline. Archived at data/results/beam/20260421-234814-100K-0_19/. Three options evaluated for next steps; recommendation was A (regression harness) then B (V2 extraction shim). Phase 4 (1M apples-to-apples) deferred.
Dates: 2026-04-23 (overnight multi-phase)
Sources: docs/session_20260423_overnight_strategy.md, docs/session_20260423_failure_modes.md, docs/session_20260423_v2_smoke.md, docs/session_20260423_v1_vs_v2_fullbucket.md, docs/session_20260423_overnight_insights.md
Core hypothesis: Replicating mem0-OSS's gpt-4o-mini fact extraction before AutoMem storage (using the verbatim pinned FACT_RETRIEVAL_PROMPT at SHA daa4495) should close the knowledge_update gap and produce an apples-to-apples comparison.
Five-phase plan: Phase 1 (failure forensics) → Phase 2 (V2 shim build + smoke) → Phase 3 (V2 full bucket 400Q) → Phase 4 (1M subset, skipped) → Phase 5 (insights).
Phase 1 — Failure forensics (n=95 V1 failures classified by gpt-5-mini, $0.20):
- 49.5% retrieval-miss (A), 8.4% chronology-confusion (B), 26.3% answerer-ignored (C), 15.8% hallucination (E).
knowledge_update94% A+B → proceed to V2. Gate triggered: ≥60% A+B.- Forensics framework correctly predicted V2 would help some categories; critically failed to predict V2's losses.
Phase 3 — V2 full bucket (400Q, ~$9, ~7h): Overall V1 76.25% vs V2 73.75% (−2.50pp).
Per-category delta (selected):
abstention: +15pp (surprise win — sparse atomic facts reduce thematic confabulation)knowledge_update: +7.5pp (predicted ✓, magnitude smaller than expected)event_ordering: −20pp (catastrophic — temporal adjacency signal destroyed by extraction)information_extraction: −12.5pp (extraction strips assistant-turn content)instruction_following: −10pp (code/format artifacts lost)
The mechanism: V2 gives up raw signal (ordering, structured content, assistant recommendations) in exchange for sharper user-fact retrieval. At 100K, the trade is slightly net-negative.
Noise floor quantified: Two V2 runs on the same 40 questions differ by +10pp (72.5% vs 82.5%). Per-category error bars ≈ ±5pp at n=40, ±3pp at n=400. Judge: Qwen at temp=0 > gpt-5-mini for regression tracking.
Decision: V2 NOT promoted. V1 remains default shim. V3 hybrid proposed: store atomic facts AND source dialogue blob — complementary retrieval pools. Phase 4 (1M) skipped due to time. DELETE handler bug documented.
Dates: 2026-04-28
Sources: docs/session_2026-04-28_hook_replay.md
Core hypothesis: A fixture-driven replay harness can deterministically test mcp-automem production hook code paths (Claude Code hooks → AutoMem) locally without real API calls or docker-volume resets.
Setup: 8 PostToolUse + 1 Stop + 1 negative-control fixtures. HOME=<temp_sandbox> override for queue isolation. eval_run_id UUID4 tag per run. Stdlib-only Python. --cleanup wired from day one.
Key implementation findings (non-obvious behaviors):
/recallis GET withX-Api-Key, not POST + Bearer.- Response shape:
.results[].memorynested,idat top level. - No
/memory/by-tagendpoint — cleanup is per-id only. - Deploy hook excludes
'unknown'from tags but leaves it in content (NER then hallucinates entity tags around'on unknown'). process-session-memory.pyemitstags: null, type: null— different field shape than thecapture-*.shhooks.
Decision: Adopted. Layer 2 (recall probes with Haiku + caching) scoped to a follow-up PR. Planned variants: fix-v2-sanitize-content, fix-v3-add-fields, fix-v4-relations.
Dates: 2026-05-01
Sources: data/results/20260501-124737-prod-stage1-execute.md, data/sweep_runs/20260501-095153-mirror8011-stage1/, multiple data/results/20260501-* files
Core hypothesis: Production AutoMem has accumulated noise memories (LoCoMo fixtures, build results, test results, engagement pings, engagement summaries) that can be swept safely using a tag-based filter with preserve-probe regression gates.
Setup: runners/compare_recall_endpoints.py with scenarios/recall_cleanup_v1.json. Dry-run → diff → execute pattern. Pre/post health checks. Per-filter backup JSONLs.
Headline result: 3,158 memories deleted from 11,046 (28.6% noise). Zero preserve regressions. Recall noise probes decreased as expected. Recall of "mixed" probes improved slightly (less noise competition).
| Filter | Deleted |
|---|---|
| SWEEP-LOCOMO-BENCHMARK | 953 |
| SWEEP-BUILD-RESULTS | 526 |
| SWEEP-TEST-RESULTS | 542 |
| SWEEP-MOLTBOOK-ENGAGEMENT-PINGS | 328 |
| SWEEP-TWITTER-ENGAGEMENT-PINGS | 150 |
| SWEEP-STARTED-SESSION | 132 |
| SWEEP-ENGAGEMENT-WORKFLOW-SUMMARY | 527 |
Decision: Executed and complete. The compare_recall_endpoints.py + recall_cleanup_v1.json pattern is now the standard production cleanup instrument.
Dates: 2026-05-02
Sources: data/results/matrix/20260502T*.md (multiple runs)
Core hypothesis: A multi-endpoint matrix runner can provide baseline vs candidate PR comparisons across multiple rulesets and scenario sets simultaneously.
Setup: runners/run_matrix.py. Two seeded stacks (baseline on :8031, candidate on :8041). Same corpus v1, same session_start_v1 scenarios, two rulesets (bare_tag_1m_v2, baseline_v1).
Headline result: Identical results between baseline and candidate endpoints in the initial validation run (42/44 and 24/44 for v2 and v1 respectively — as expected from prior experiments). Infrastructure confirmed working.
Decision: Adopted as the standard PR regression matrix gate.
Dates: 2026-05-22
Sources: data/results/SUMMARY-20260522-pr170-current-state.md
Core hypothesis: PR #170 (fix/recall-current-state) correctly suppresses stale, future, invalidated (INVALIDATED_BY), and evolved (EVOLVED_INTO) memories from default recall, while preserving CONTRADICTS pairs, without regressing broad recall quality.
Setup: Targeted runner (runners/run_current_state_recall_eval.py) seeds isolated current/stale/future/superseded memories under a unique run tag. Two separate stacks (base on :8051, candidate on :8061). 8 runs total across PR commits.
Headline result: 6/6 targeted current-state probes pass on all PR commits. Broad recall: identical between base and PR on all 10 session_start_v1 scenarios for both rulesets. Server tests: 15 passed, 84 deselected.
Decision: PR accepted. Broad recall runner established as a collateral-damage smoke test (sufficient for presence/rank, not sufficient for stale-hit absence detection).
Dates: 2026-05 (ongoing, results in data/results/writ/)
Sources: docs/writ_integration.md
Core hypothesis: WRIT's write-integrity scenario categories (drift, temporal, lifecycle, provenance) probe AutoMem's update/invalidation behaviors complementarily to recall-quality experiments.
Setup: runners/run_writ.py + TypeScript adapter (runners/writ/automem-adapter/automem.ts). Adapter maintains a local observation index (generic extractors for drift facts, money, addresses, dates, people, task states, etc.). factId resolved via normalized token overlap. AutoMem /memory for storage, /recall for probe retrieval. reset() uses per-id DELETE to avoid docker resets.
Headline result (drift category):
recall_accuracy: 100% (AutoMem) vs 0% (baseline)update_fidelity: 20% — caveat: WRIT's evaluator does a literal substring check against narrative ground truth, not present in any user message. Not an AutoMem retrieval failure.detectability: 100%,drift_rate: 0%,abstention_quality: 100%
Decision: Experimental / adopted for diagnostic use. Lifecycle and provenance categories not yet adapted. update_fidelity gap documented as a WRIT evaluator quirk, not an AutoMem bug. Not used for benchmark claims.
Dates: 2026-06-10 → 2026-06-11
Sources: docs/session_2026-06-10_prod_entity_repair.md, data/sweep_runs/prod-rollout-20260610/
Core hypothesis: Production entity tags are heavily noisy (false-positive person names like jack, vendor noise, context-hint over-rejection) and can be repaired in stages without harming recall.
Setup: Two AutoMem PRs (#176 tooling, PR #8 harness). Three dry-run iterations (Dry-run v1: 7,677 rejections; PR #178 fix → v2: 7,494; PR #179 fix → v3: 6,106 byte-identical to reviewed expectation → execute). Stages: 3A reject-only, 3B canonicalize-safe. Rollback JSONLs kept. Pre/post snapshots.
Headline result:
- Stage 3A: 6,106 noise entity tags removed across 7,912 memories. Zero failures. Re-audit
changed=0. - Stage 3B: 121 canonical rewrites + 1,584 ambiguous single-name people suppressed. Re-audit
changed=0. - Post-verify:
/entities/auditrejections 7,5xx → 0. Qdrant payload parity 5/5. - Surprise discovery: Pre-repair, many Qdrant payloads carried entity tags not in the graph (payload drift). Repair's
sync_qdrant_payloadhealed them → recall probes improved (WP-FUSION probe: top hit went from absent to ranks 1–2 at score 0.73). - Entity-node migration (Gate 5): DEFERRED — 80% of single-reference people are word-pair noise.
Lesson documented: Review-bot suggestions applied at the last minute can silently regress the exact path under repair. The dry-run-diff-against-reviewed-expectation gate caught two such regressions.
Decision: Stages 3A and 3B executed and complete. Entity-node migration parked pending better noise filtering (--min-references or --categories gate).
Dates: 2026-06-11
Sources: docs/session_2026-06-11_negative_control_probes.md
Core hypothesis: The existing recall cleanup harness only measures positive recall (expected hits present). Adding "nothing relevant" probes will surface over-retrieval regressions.
Setup: scenarios/recall_cleanup_v2.json — strict superset of v1 (22 preserve/mixed/noise probes byte-identical) + 5 new "group": "negative" probes. Runner updated: classify_status for negative probes = REGRESSION when candidate returns more results or higher top-1 final_score than baseline.
Five negative probes:
NEG-SCOPED-OFFTOPIC-130— on-corpus topic gated to wrong tag (automem#130 replica)NEG-PREFERENCE-TOPIC-ABSENT— topic with no matching preference memoriesNEG-OFFDOMAIN-COOKING— off-domain query, no tag gateNEG-NONEXISTENT-ENTITY— synthetic person/project absent from corpusNEG-ABSENT-STACK— plausible-but-absent tech stack (embedding near-neighbor leakage)
Decision: Adopted as recall_cleanup_v2.json. Negative regressions are report-only (not exit-code gated). --fail-on-negative-regression flag identified as a cheap follow-up for the #130 rollout.
Dates: 2026-06-13
Sources: docs/session_20260613_beam_retrieval_proxy_100k.md, docs/eval/beam.md
Core hypothesis: A deterministic, no-judge, no-answerer retrieval proxy can cheaply detect recall regressions in BEAM by scoring source-chat hit rate and rubric token overlap.
Setup: runners/beam_retrieval_eval.py. 29,902 memories seeded (100K tier, 20 conversations). 29,882 OCCURRED_BEFORE graph edges. Per-question: scores source_chat_hit, rubric_overlap, abstention_evidence_absent. Cleanup and health check before/after.
Headline result:
- Proxy pass rate: 75.5%, source-chat hit rate: 85.1%, mean rubric overlap: 0.43.
- Abstention category: 0.0% (deliberately harsh — overlapping snippets count as evidence presence).
knowledge_update97.5%,multi_session_reasoning97.5% in this proxy.- Run fully validated (manifest vs result cross-checks, per-item assertions).
Key property confirmed: The proxy is insensitive to ranking flags like recency_bias — it measures raw retrieval presence, not end-to-end quality. This validates its use as a cheap smoke test but means it cannot catch ranking regressions; those require the judged pipeline.
Decision: Adopted as the cheap regression check (beam_eval.py --cleanup-after). Not a substitute for the judged harness.
Dates: 2026-06-15 → 2026-06-16
Sources: docs/session_20260616_beam_judged_100k.md, docs/eval/beam.md, data/results/beam-judged/ (multiple runs)
Core hypothesis: AutoMem's native chunking + /recall + official BEAM rubric-nugget judge gives a leaderboard-comparable score that can detect ranking and ingest changes.
Setup: runners/beam_judged_eval.py (wrapper: scripts/beam_judged.py). Imports upstream answer/judge prompts from vendored submodule. Adds per-turn time_anchor → memory timestamp mapping. Multiple runs: failed runs (quota exhaustion → 43.5% poisoned; gpt-5 4096-token truncation → 67.5%), then final clean run.
Headline result — official (gpt-5/gpt-5 judge): 70.25% (281/400), avg_score 0.649
| System | Judge | 100K accuracy |
|---|---|---|
| Graphonomous | undisclosed | 95.0% |
| Hindsight | undisclosed | 73.4–75.0% |
| AutoMem mem0-shim | gpt-5-mini/gpt-5-mini | 76.25% |
| AutoMem native (this work) | gpt-5/gpt-5 | 70.25% |
| AutoMem native | gpt-5-mini/gpt-5-mini | 82.0% |
Key findings:
- gpt-5 truncation: 4096 default → 23/400 empty answers → 67.5%. Fix:
--answer-max-tokens 8192→ 70.25%.--answer-max-tokensexposed as a flag. - Quota exhaustion guard: Mid-run quota → 140 empty answers, misleading 43.5%. Added
--max-empty-rate(default 0.30) fail-fast guard + start-of-run preflight. time_anchor → timestampmatters: Dropping it costs −12.5pp onevent_ordering(77.5% → 65.0%). Other abilities unaffected.recency_bias=auto: −0.5pp overall — doesn't help BEAM 100K.- Native beats mem0-shim at matched judge: 82.0% vs 76.25% (both gpt-5-mini) — AutoMem's own chunking +
/recalloutscores the mem0 wire-contract path by ~6pp. - Judge model dominates absolute score: ~12pp swing between gpt-5 and gpt-5-mini judges. Cross-system comparison is only meaningful at a fixed judge.
Efficiency (objective axes, judge-independent): AutoMem ~1.6s recall / ~11k context tokens vs Hindsight's 6.4s / 17.7k — ~4× faster recall, ~35% leaner context.
Decision: Adopted as canonical BEAM 100K native baseline. Headline: "70.25% (gpt-5 judge, official)". docs/BENCHMARK_JUDGE_POLICY.md created, pinning the internal judge to gpt-5.4-mini-2026-03-17 for routine regression tracking. Result promoted to automem/benchmarks/EXPERIMENT_LOG.md.
Dates: 2026-06-16 (spec/doc, triggered by BEAM 100K findings)
Sources: docs/eval/maturation.md
Core hypothesis: BEAM/LoCoMo/LongMemEval, as normally run (ingest → immediate recall), test AutoMem's cold retrieval substrate, not its cognitive layer (enrichment, consolidation, decay). Cold scores are floors, not ceilings.
Evidence: BEAM 100K: 94% retrieval recall. Only 9/400 are retrieval misses. The one clearly AutoMem-actionable miss (knowledge_update stale value) is an INVALIDATED_BY/consolidation case that cold ingest never exercises.
Framework defined:
| Variant | Maintenance | Reported as |
|---|---|---|
cold |
None (current default) | Leaderboard comparator |
matured |
Enrichment drain + consolidation on narrative clock (decay OFF) | "warm memory" realistic |
matured-full |
+ decay/forget ON | Most realistic; may score < cold |
Standing rule: Every curated result must report BOTH cold and matured numbers and compare them. The decay caveat: matured-full can score below cold because benchmarks reward total recall of un-decayed history; that is a finding, not a regression.
Implementation status: cold only is fully wired. matured/matured-full are blocked on AutoMem /consolidate gaining a reference_time (narrative-clock) parameter. Every result now explicitly labeled maturation: cold.
Decision: Framework adopted and documented. Blocked on AutoMem dependency. Every run now carries explicit maturation: cold label.
Dates: 2026-06-19 → 2026-06-22 (in-flight)
Sources: docs/superpowers/specs/2026-06-19-amb-throughput-runbook-design.md, docs/superpowers/plans/2026-06-20-amb-throughput-runbook-plan.md
Core hypothesis: AutoMem can produce full AMB results (Core-3: LoCoMo 1,540q, LongMemEval 500q, PersonaMem 589q + BEAM 100K/500K/1M/10M) crash-safely, with statistical credibility, using Gemini-3.1-pro-preview answerer + Gemini-2.5-flash-lite judge against a self-spinning Docker stack.
Key discovered facts (measured, not estimated):
- Previous run crashed PersonaMem at 247/589 — batch mode saved only at run end.
- Real question counts differ from docs: LoCoMo=1,540 (not 152), others as expected.
- One AutoMem stack saturates the whole host (18 cores at ~1781% CPU via FastEmbed) — parallel stacks are infeasible; must run serially.
- Repeat strategy revised from 3× all datasets to 1× per dataset with within-run CIs + beam-100k ×3 as explicit reproducibility check.
Four moves in the design: (A) Parallel fan-out (revised to serial), (B) 3× repeats for CI (revised), (C) Calibrate-then-commit (20q burst), (D) Harden checkpointing (incremental save + resume in batch mode).
Task 1 (batch checkpoint fix): Identified: runner.py batch branch saves only at end and ignores --skip-ingested. Fix: append every 10 queries + resume-filter at batch-branch entry. TDD approach — failing tests written first.
Decision: In-flight. Task 1 complete. Tasks 2–5 (calibration burst → serial small suite → big BEAM tiers → triplet aggregation report) pending execution. Publish/PR to vectorize-io is human-gated and explicitly out of scope.
| # | Topic | Status | Source |
|---|---|---|---|
| 1 | V3 hybrid shim — store both atomic facts (V2) AND raw dialogue (V1) per BEAM conversation | Proposed, not built | docs/session_20260423_overnight_insights.md |
| 2 | 1M BEAM apples-to-apples — V2 shim at 1M tier with gpt-5 judge vs mem0's published 1M numbers | Phase 4 skipped in overnight session | docs/session_20260423_overnight_strategy.md |
| 3 | Maturation harness — matured/matured-full variants blocked on AutoMem /consolidate reference_time parameter |
Blocked on AutoMem dependency | docs/eval/maturation.md |
| 4 | Entity-node migration — Gate 5 deferred; needs --min-references or --categories gate; residual verb-person fragments remain |
Parked | docs/session_2026-06-10_prod_entity_repair.md |
| 5 | --fail-on-negative-regression exit-code gate — trivial addition to the #130 rollout harness |
Planned but not built | docs/session_2026-06-11_negative_control_probes.md |
| 6 | Agent-matrix dimension — wrap runner so a "ruleset" includes an agent adapter (Claude Code, Codex, Cursor) | Mentioned every session since 2026-04-17, not started | docs/session_20260417_notes.md, data/results/SUMMARY-20260417.md |
| 7 | Client-side expansion as mcp-automem feature — 60-line prototype validated; needs strength-weighted scoring and phase-awareness before shipping |
Prototype only | data/results/SUMMARY-20260417.md |
| 8 | AMB full submission runs — Tasks 2–5 of 2026-06-20 plan (calibration → small suite → big BEAM tiers → triplet report) | In-flight | docs/superpowers/plans/2026-06-20-amb-throughput-runbook-plan.md |
| 9 | BEAM regression harness — beam_diff.py + --regression-preset + baselines/current symlink; planned after V2 work |
Partially built (beam_diff.py exists); not wired as a preset |
docs/session_20260422_beam_fullbucket.md |
| 10 | Server-side expand_relations bypass — expansion targets should bypass the tag filter; filed as a recommendation for AutoMem server |
Recommendation only; server not changed | data/results/SUMMARY-20260417.md |
| 11 | BEAM 10M judged run — scheduled as part of AMB submission; heavy ingest (~984 MB) | Pending | docs/superpowers/specs/2026-06-19-amb-throughput-runbook-design.md |
| 12 | Stale-project recall variant — Phase 2 with 180-day (or no) time window for dormant projects like old-service |
Documented as needed; not built | data/results/SUMMARY-20260418-v2.md |
| 13 | WRIT lifecycle/provenance categories — adapter doesn't yet implement full source-authority, association chains, or conflict resolution | Known gaps | docs/writ_integration.md |
| 14 | Layer-2 hook-replay — recall probes with Haiku + caching (scoped to a separate PR from Layer-1) | Not started | docs/session_2026-04-28_hook_replay.md |
| 15 | FAMA / Memora experiments — data/results/fama/, data/results/memora/, data/results/memora-fama/ directories exist but are undocumented in any session notes read |
Unknown / undocumented | data/results/ directory listing |
| 16 | relchain experiments — data/results/relchain_{null,pass1-5}_20260611 artifacts exist; not covered in any session doc |
Unknown / undocumented | data/results/ directory listing |
| Term | Meaning |
|---|---|
| Ruleset | A rulesets/*.json file defining phase_1/phase_2/phase_3 recall parameters |
| Scenario | A scenarios/*.json entry: {id, phase, query?, project_slug?, expected_hit_tags} |
| Phase 1 | Tag-only recall, no query — surfaces session-start preferences |
| Phase 2 | Semantic + project-tag-gated + time-windowed recall |
| Phase 3 | Semantic + bugfix/solution-tag-gated recall |
| Corpus | The seed memory set loaded into AutoMem (v1=78, v2=370 memories) |
| Manifest | *.manifest.json — bidirectional mapping between scenario IDs and memory_ids |
| Snapshot | *.embedded.jsonl — corpus with pre-computed Voyage vectors for $0 reseeding |
| hit / expected hit | Whether an expected memory_id was returned in /recall results |
| probe | A specific recall call in the comparison harness |
| iso_a..iso_j | Naming pattern for knob-isolation rulesets (one parameter varied at a time) |
| Term | Meaning |
|---|---|
| Proxy score / proxy pass | Deterministic retrieval-only metric (no judge); official_beam_score: false |
| Official BEAM score | LLM rubric-nugget judge (0/0.5/1.0 per nugget, pass ≥ 0.5); official_beam_score: true |
| Tier (BEAM) | 100K / 500K / 1M / 10M — conversation pool size |
| Triplet | accuracy ± CI / recall latency (ms) / context tokens — public AMB reporting format |
| cold | No maturation — ingest then immediately recall. The leaderboard comparator |
| matured | Drain enrichment + run consolidation on narrative clock, decay OFF |
| matured-full | matured + decay/forget ON. May score below cold — expected |
| V1 shim | Raw dialogue pass-through BEAM shim (runners/beam_shim.py) |
| V2 shim | Fact-extraction shim using mem0's pinned FACT_RETRIEVAL_PROMPT (runners/beam_shim_v2.py) |
| run_id | Stable BEAM run identifier; BEAM memories tagged beam-run-<run_id> |
| eval_run_id | UUID4 tag for hook-replay run isolation |
| flip | A question that changed from pass→fail or fail→pass between two runs |
| CI | 95% confidence interval; t-distribution (n=3: t=4.303; large n: within-run 1.96·sd/√n) |
| Term | Meaning |
|---|---|
| preserve | Probe that should survive any change; exit-code gated |
| mixed | Probe with expected and noise memories interleaved; observed |
| noise | Probe whose count should decrease under cleanup; observed |
| negative | Probe where correct answer is "nothing relevant"; REGRESSION = candidate returns more or higher-scoring results |
| REGRESSION | Candidate is worse than baseline on a preserve probe |
| improved | Candidate is better (fewer noise results, or higher score on a positive probe) |
| ok | No change between baseline and candidate |
| Prefix | Domain |
|---|---|
SS-* |
Session-start scenarios |
PREF-* |
Preference-scope scenarios |
TP-* |
tensor-pipeline project |
DA-* |
dashboard-app project |
PS-* |
payment-service project |
SE-* |
search-engine project |
OS-* |
old-service project |
VID-* |
video project |
DEBUG-* |
Phase-3 debugging scenarios |
NEG-* |
Negative-control probes |
SWEEP-* |
Production memory classes identified for cleanup |
| Term | Meaning |
|---|---|
| Recall | Fraction of expected memory_ids returned in top-k |
| P@5 | Precision at rank 5 |
| rank₁ | Rank of the first expected hit in the returned list |
| source_chat_hit | Retrieved memory metadata intersects BEAM source_chat_ids |
| rubric_overlap | Token overlap between BEAM rubric nuggets and retrieved memory text |
| agent adapter | Code implementing the MemoryProvider interface for AutoMem in the omb (AMB) harness |
time_anchor |
BEAM per-turn date field; mapped to memory timestamp in the judged harness |
| dry-run / execute | Sweep pattern: generate plan → diff against reviewed expectation → execute → re-audit |