The fully-local realtime voice pipeline, with hybrid escalation to cloud for tool/agentic turns. Everything below runs on-device (Apple Silicon) except the cloud-escalation tier.
| Stage | Component | Notes |
|---|---|---|
| VAD | Silero | local voice-activity detection |
| STT | Parakeet (:8178) |
local speech-to-text; Deepgram is the cloud alt |
| LLM (local) | Qwen3.6-35B-A3B 4-bit | MoE, ~3B active params, ~20GB; served two ways (below) |
| TTS | Chatterbox (hub-managed) | local; falls back to ElevenLabs on failure/60s cooldown |
| Escalation | Claude Sonnet | tool / research / safety / agentic turns route to cloud |
MLX — PRIMARY. mlx_lm.server on :8081, model mlx-community/Qwen3.6-35B-A3B-4bit.
- Faster: ~1.6× TTFT, ~1.4× generation vs Ollama (see benchmark).
- Has an active degeneration guard (cancels repeat-loops mid-stream).
- No tool calling (mlx_lm.server function-calling is unreliable → tool turns escalate to cloud).
num_predict96 (tight, voice-tuned).
Ollama — FALLBACK. :11434, model huihui_ai/qwen3.6-abliterated:35b-a3b (abliterated/uncensored variant of the same Qwen).
- More robust daemon (stays resident,
keep_alive=-1) — the right thing for a fallback. - Has a native tool loop MLX lacks → unlocks experimental read-only local tools.
- Tuned 2026-06-15 for parity:
NUM_PREDICT256→128,NUM_CTX8192→16384. - Still missing the active degeneration guard (relies on
repeat_penalty=1.3).
Routing — src/voice/local-turn-router.js (hybrid mode):
- Casual chat + personal-memory recall → stay local.
- Tool / messaging / research / safety intent → escalate to cloud (Sonnet).
- Both
VOICE_LOCAL_MLX_BYPASSandVOICE_LOCAL_OLLAMA_BYPASSaretrue; MLX takes precedence, Ollama runs only when MLX isn't ready.
Same model class, 4-bit, 128 max-tok / temp 0.7 / top-p 0.9 / rep 1.1, via the production stream clients:
| Metric (warm avg) | MLX | Ollama |
|---|---|---|
| TTFT (first token) | 186 ms | 295 ms |
| Generation | ~79 tok/s | ~58 tok/s |
| Total (short reply) | 389 ms | 644 ms |
| Cold load | (pre-warm) | ~4.0 s |
| Quality @ matched decoding | identical | identical |
MLX leads on speed; answer quality is a wash (abliteration aside). Gotcha: the two clients' raw tps fields aren't comparable — MLX divides by total wall-clock (incl. prefill), Ollama by eval_duration (decode only). Normalize before comparing.
Spec: docs/superpowers/specs/2026-06-15-voice-local-cloud-escalation-design.md (branch feat/voice-local-cloud-escalation). Covers: widen tool-intent rules so messaging/contact turns escalate (the voice_communication group already holds the WhatsApp tools — just no rule enabled them); pre-load the group on escalation; confabulation guard in the per-path local prompts; brief spoken "one sec" transition; unify the local history window to ~20 turns; bring Ollama to MLX parity.
- STT / VAD tuning — the garbled-transcript problem; biggest quality lever, needs live mic iteration (
VOICE_VAD_THRESHOLD,VOICE_VAD_SILENCE_MS, Parakeet). - Local TTS fallback — Chatterbox currently falls back to ElevenLabs (cloud); add a local fallback (
say/Kokoro) for true offline. Also fix the ChatterboxErrno 48port-bind crash. - Mouth-to-ear latency — measure full pipeline (STT-finalize + LLM TTFT + TTS first-audio) and tune the TTS lead-in/chunking.
- MLX warm-on-boot + survive-restart — avoid the cold first turn and ~20 GB cold-reload on restart.
- Ollama
repeat_penalty1.3→~1.1 — once the degeneration guard is wired. - Offline-completeness audit — AutoMem is cloud (Railway); confirm the local memory overlay is enough offline.
worktree MCP-config ENOENT (#702), stale tool_use 400s (#703), MCP SSE auto-reconnect (#704), voice audio-route fallback (#705), Slack messages_tab_disabled cascade (#706), agent_run reconciliation (#707), Chatterbox TTS retry-before-cooldown (#714), restart-churn EPIPE guard (#715), voice runtime hardening — crash guard / divergence / barge-in tool preservation (#716).