jack-arturo

---

Experiment History — automem-evals

A synthesized digest of all experiment threads, findings, decisions, open questions, and vocabulary conventions. Each thread is referenced to the primary source file(s).

---

1. Experiment Threads

jack-arturo

AutoHub Offline Voice Stack — Snapshot (2026-06-15)

The fully-local realtime voice pipeline, with hybrid escalation to cloud for tool/agentic turns. Everything below runs on-device (Apple Silicon) except the cloud-escalation tier.

Pipeline (local-first)

Stage	Component	Notes
VAD	Silero	local voice-activity detection
STT	Parakeet (:8178)	local speech-to-text; Deepgram is the cloud alt

jack-arturo

offline-voice-stack-june-2026

AutoHub Offline Voice Stack — Snapshot (2026-06-15)

The fully-local realtime voice pipeline, with hybrid escalation to cloud for tool/agentic turns. Everything below runs on-device (Apple Silicon) except the cloud-escalation tier.

Pipeline (local-first)

Stage Component

jack-arturo

Jack and the Devil

---

[Verse 1]
The devil came knockin' on a Tuesday night
Said "boy, I got a daughter — she's a real delight"
"Her name is Isabella and she burns like flame

jack-arturo

Russian Straw

---

[Verse 1]
Sippin' slow through a golden straw
Russian summer, breaking every law
You lean in close, I feel the heat
Milk and honey dripping, oh so sweet

jack-arturo

Plan: Diagnose "x/y scroll options discarded" warning attributed to error-tracking.js:31

Context

A customer reports console noise on their site:

error-tracking.js?ver=1.14.2:31 The following options are discarded due to invalidity:
{ "x": "scroll", "y": "scroll" }

jack-arturo

<?php
public function handle_pause_notification() {

    if ( ! isset( $_REQUEST['fen_action'] ) ) {
        return;
    }

    $type = isset( $_REQUEST['type'] ) ? sanitize_text_field( $_REQUEST['type'] ) : 'all';
    $file = sanitize_text_field( urldecode( $_REQUEST['file'] ) );
    $line = isset( $_REQUEST['line'] ) ? intval( $_REQUEST['line'] ) : 0;

jack-arturo

clustering-graph-diagnostics.md

My take: PR #163 adds something useful, but I would not merge the default retune as-is.

The valuable part is the config surface. Exposing CONSOLIDATION_CLUSTER_SIMILARITY_THRESHOLD and CONSOLIDATION_MIN_CLUSTER_SIZE is clearly right because embedding geometry varies by provider/model. That part removes the need to fork/subclass just to calibrate clustering. The PR does exactly that across config.py, runtime_helpers.py, runtime_bindings.py, and app.py in the files changed.

The default changes are where I’d be more conservative:

• 0.75 -> 0.65: our local diagnostics do not show 0.75 as dead. On 8001, 0.75 still returned 5,878 sampled top-k neighbor hits; 0.65 returned 7,764. On 8011, 0.75 returned 4,112; 0.65 returned 6,617. So lowering to 0.65 broadens recall materially, but we have not proven those extra edges are good clusters.
• min_cluster_size 3 -> 2: good as an env var, quest

jack-arturo

Session 2026-04-23 — Overnight insights: V1 vs V2 on BEAM 100K

Consolidates failure-mode forensics (Phase 1), V2 shim design (Phase 2), smoke (Phase 2c), and full-bucket V1 vs V2 at n=400 (Phase 3). See session_20260422_beam_fullbucket.md for the V1 baseline context.

TL;DR — the honest result

V2 is not a net win over V1 at 100K. Overall: V1 76.25% vs V2 73.75% (-2.50pp). But the category-level picture is richer and more useful than the headline.

Noise-floor caveat (important): two V2 runs on the same 40 questions (convs 0-1) differ by +10pp (smoke 72.5% vs Phase 3 subset 82.5%). Same shim, same models, same prompts, different times. That puts our per-category error bars at roughly ±5pp at n=40 and ±3pp at n=400. Don't over-interpret small deltas. The signal that does survive noise: event_ordering -20pp and information_extraction -12.5pp are real structural losses; abstention +15pp and knowledge_update +7.5pp are real gains. The -2.5pp overall is within one standard deviation

jack-arturo

Session 2026-04-22 — BEAM full bucket + AutoMem vs mem0 meta-summary

Headline

gpt-5-mini answerer + judge, 400 questions (20 conversations × 100K tier), clean run, 2h 54m wall-clock:

76.25% overall pass (305/400), avg score 0.677, zero errors.

Results at data/results/beam/20260421-234814-100K-0_19/.