pulam-web / Dock Agent-State Desync — 7 Root Cause Bugs (with evidence)

pulam-web-dock-desync-bugs.md

pulam-web / Dock Agent-State Desync — Root Cause Analysis

When PULAM_WEB_HOSTS=localhost, pulam-web spawns pulam as a separate child process (no ssh for localhost — host.ts:203 returns the binary path directly, and hostSession.ts:567 spawns it via spawn(command, args, { stdio: [...] })). That pulam calls startAwareness (daemon.ts:245), starting a full sensor set. Meanwhile, kolu-server's LocalTerminalEndpoint also calls startAwareness (local.ts:696) for the same terminal — the same function, starting the same full sensor set.

Both kolu-server and pulam are clients of the same kaval daemon (ptyHost/index.ts:5-8: "the server is a client of a kaval daemon it spawns"; daemon.ts:67-69: "Default: the running kaval, discovered"). They subscribe to the same kaval tap streams and observe the same terminals, but each runs its own independent sensor instances.

Two independent derivations of the same observation, no shared state, no reconciliation.

All 7 divergence mechanisms below stem from this single architectural fact.

---

Bug 1 — Dual sensor instances

What: kolu-server and pulam each independently call startAwareness() for the same local terminal, each writing to their own collection (terminalMetadata vs awareness).

Evidence:

• packages/server/src/terminalEndpoint/local.ts:696 — const stopAwareness = startAwareness(record, id, signals, sink, log);
• packages/pulam/src/daemon.ts:245 — const stopAwareness = startAwareness(record, id, signals, sink, log);
• packages/terminal-workspace/src/sensors.ts:827 — export function startAwareness(...) starts 7 independent sensors per terminal: agent-command, git, PR, Claude, Codex, OpenCode, foreground
• packages/surface-nix-host/src/host.ts:203 — isLocalHost(opts.host) → direct spawn, no ssh wrapper

Impact: Every downstream bug (2–7) is a consequence of two sensor instances racing on the same data source. Even if all timing were perfect, two independent derivations can still disagree on any transient state.

Status: NOT fixed. startAwareness is still called from both sites independently.

---

Bug 2 — matchesAgent gate startup race (brief, not persistent)

What: OpenCode (and all agent) detection requires state.lastAgentCommandName === agentName || state.readForegroundBasename() === agentName. Each process tracks currentForeground from its own kaval foreground tap subscription, seeded empty. Until the first foreground sample arrives, the gate fails → session not resolved → shows idle.

Evidence:

• packages/integrations/anyagent/src/agent-adapter.ts:193-201:

  export function matchesAgent(
    state: AgentTerminalState,
    agentName: string,
  ): boolean {
    return (
      state.lastAgentCommandName === agentName ||
      state.readForegroundBasename() === agentName
    );
  }

• packages/integrations/opencode/src/agent-adapter.ts:22-24 — resolveSession calls matchesAgent first, then findSessionByDirectory(state.cwd, log)
• packages/terminal-workspace/src/sensors.ts:569-572 — currentForeground is local state seeded empty and updated only via the signals.foreground channel:

  let currentForeground: ForegroundSample = {
    process: "",
    foregroundPid: undefined,
  };

Correction — this is a brief startup race, NOT a persistent divergence. The foreground tap does replay a snapshot on subscribe (local.ts:278: "surviving foreground tap (which replays a snapshot on subscribe)"). So after subscribing, both processes receive the current foreground state within milliseconds, matchesAgent passes, and the session is resolved. Note: cwd and title taps do NOT replay (local.ts:301), but cwd is seeded from the kaval terminal.list entry (daemon.ts:215, local.ts:316), and title is not needed for agent detection.

The only persistent scenario where this gate stays failed: a process restarts (kolu-server redeploy) and adopts a terminal where an agent is already running, but the foreground tap snapshot delivery is delayed or the foreground is genuinely idle (shell prompt, not the agent). In that case, the gate depends on lastAgentCommandName (from commandRun), which also doesn't replay — but commandRun events are not needed if the foreground tap delivers the agent process name.

Impact: Brief startup race (milliseconds). NOT the primary mechanism behind persistent desync. The persistent desync for OpenCode is Bug 3 (missed WAL events).

Status: NOT fixed (but low severity — self-resolves once the foreground snapshot arrives).

---

Bug 3 — Independent WAL debounce (OpenCode)

What: Each process has its own 150ms trailing-edge debounce on fs.watch WAL events. fs.watch is unreliable on Linux (events coalesce and drop). A missed event leaves one process stuck on its last-derived state indefinitely.

Evidence:

• packages/integrations/opencode/src/session-watcher.ts:38:

  const WAL_DEBOUNCE_MS = 150;

• packages/integrations/opencode/src/session-watcher.ts:55-59 — createOpenCodeWatcher sets up fs.watch on the SQLite WAL file with the debounce
• The debounce timer is per-watcher-instance (local setTimeout in the watcher closure), not shared

How it diverges: Process A's fs.watch fires → debounce starts → 150ms later it re-reads the DB → derives thinking. Process B's fs.watch on the same WAL file drops the event (Linux inotify can coalesce/drop under load) → B's debounce never starts → B keeps its last-derived state (idle from before the missed event) until the next event that happens to get through. Since OpenCode has no timers or screen-scrape to self-heal, the stale state persists indefinitely.

Impact: Intermittent "stuck on last state" for OpenCode agents. No self-healing unless a new WAL event arrives.

Status: NOT fixed.

---

Bug 4 — decayTransientState timers (Claude Code)

What: Each process arms its own setTimeout recheck for stale transient states (tool_use, thinking). If one process decays tool_use → waiting (→ urgency "idle" after R-dock-unify) while the other hasn't fired yet (→ still working), they diverge.

Evidence:

• packages/integrations/claude-code/src/core.ts:1242-1277 — decayTransientState is a pure function: given a state, quietMs, and probes, it returns either the same state (not yet stale) or waiting (stale + idle). The recheck deadline is now + (staleMs - quietMs).
• packages/integrations/claude-code/src/session-watcher.ts:217-230 — scheduleStaleRecheck arms a per-watcher setTimeout:

  function scheduleStaleRecheck(deadline: number | null) {
    if (staleDeadlineTimer) {
      clearTimeout(staleDeadlineTimer);
      staleDeadlineTimer = null;
    }
    if (destroyed || deadline === null) return;
    const delay = Math.max(0, deadline - Date.now()) + 1;
    staleDeadlineTimer = setTimeout(() => {
      staleDeadlineTimer = null;
      onTranscriptMaybeChanged();
    }, delay);
  }

• packages/integrations/claude-code/src/session-watcher.ts:401-422 — each onTranscriptMaybeChanged call recomputes staleDeadline and re-arms the timer

How it diverges: Both processes read the same JSONL transcript file and compute the same quietMs and staleMs. But Date.now() at the moment each process's onTranscriptMaybeChanged fires differs by milliseconds. Process A's timer fires at deadline_A = now_A + (staleMs - quietMs) and process B's at deadline_B = now_B + (staleMs - quietMs). If now_A < now_B by even 1ms, A decays to waiting first. After R-dock-unify (#1541), waiting → agentUrgency = "idle" — so A's Dock shows idle while B's pulam-web still shows working until B's timer fires. The window is small (the +1ms guard means they differ by at most a few ms) but it exists, and for tool_use the subtree probe (isClaudeSubtreeIdle) can differ if one process's ps call races differently.

Impact: Brief flicker of disagreement on the working → idle edge for Claude Code agents. Usually self-heals within seconds but is visibly wrong during the window.

Status: NOT fixed.

---

Bug 5 — Screen-scrape poll (Claude Code)

What: Each process has its own 1000ms screen-scrape poll that can promote/demote pollable states independently. The poll reads the terminal screen and can promote waiting → awaiting_user (prompt detected) or demote awaiting_user → waiting (prompt cleared).

Evidence:

• packages/terminal-workspace/src/sensors.ts:470:

  const SCREEN_SCRAPE_POLL_MS = 1000;

• packages/terminal-workspace/src/sensors.ts:696-760 — startScreenScrapePoll arms a recursive setTimeout(tick, SCREEN_SCRAPE_POLL_MS) that:
  1. Checks scrape.isPollable(info) against the latest watcher info
  2. Reads the screen via sink.readScreenText
  3. Calls scrape.promote(info, text) — returns a new info if the screen warrants promotion, same ref otherwise
  4. Republishes via publishAgentField only on structural divergence from the published agent
• packages/terminal-workspace/src/sensors.ts:658-665 — the watcher callback suppresses raw publishes when a promotion is live AND the state is still pollable, deferring to the poll

How it diverges: The poll reads the rendered screen (sink.readScreenText), which is the PTY's pixel content. Two processes reading the same PTY at different times in their 1000ms cycles can see different screen content — e.g. process A reads the screen at t=0 and sees a prompt (promotes to awaiting_user), process B reads at t=500ms and the prompt has already scrolled past (no promotion → stays waiting). The promote/demote edge is inherently racy because it depends on rendered screen content at a specific moment.

Impact: Flicker on the waiting ↔ awaiting_user edge for Claude Code agents. The poll is designed to be the single writer for this edge within one process, but two processes each have their own poll.

Status: NOT fixed.

---

Bug 6 — Activity stream divergence

What: The green-dot "moving bytes" signal (activity) is also derived independently by each process's startForegroundSensor.

Evidence:

• packages/terminal-workspace/src/sensors.ts:879-885 — startForegroundSensor is the 7th sensor started by startAwareness
• packages/pulam-web/src/server/reserve.ts:306-312 — pulam-web's re-serve maintains its own activityLatest live-set, independent of kolu-server's
• packages/pulam-web/src/server/hostEntry.ts:122 — onLinkDown: () => reServe.resetRemoteFold() clears pulam-web's activity on link death (partial fix from #1550)

How it diverges: kolu-server's activity sensor writes to kolu's in-process terminalMetadata. pulam's activity sensor writes to pulam's awareness collection, which pulam-web's re-serve mirrors over the reconnect link. Two independent derivations of the same foreground samples, with independent timing. Additionally, #1550 fixed the reconnect ghost (a dead link's last activity frame persisting), but the underlying dual-sensor derivation is still independent.

Impact: The green dot can disagree between Dock and pulam-web. Partially mitigated by #1550 (reconnect ghost fixed) but the root dual-sensor issue remains.

Status: PARTIALLY fixed. #1550 fixes the reconnect ghost for activity. Dual-sensor derivation still independent.

---

Bug 7 — agentInfoEqual locks in divergence

What: Once two processes diverge (via any of bugs 2–6), the dedup gate publishAgentField blocks a re-publish until the info structurally changes. A missed event means the stale state persists until the next differing event — which could be never.

Evidence:

• packages/terminal-workspace/src/sensors.ts:494:

  if (agentInfoEqual(record.meta.agent, nextAgent)) return;

• packages/integrations/anyagent/src/agent-adapter.ts:209-223 — agentInfoEqual compares kind, state, sessionId, model, summary, contextTokens, startedAt, and taskProgress. If all match, it returns true and the publish is skipped.

How it diverges: Suppose process A missed a WAL event (Bug 3) and is stuck on { state: "idle" }. Process B saw the event and derived { state: "thinking" }. Now the next WAL event arrives and both processes re-read the DB. If the new state is also thinking, process B's agentInfoEqual returns true (already published thinking, new is thinking) → no publish. Process A's agentInfoEqual compares its published idle to the new thinking → they differ → publishes thinking. So A self-heals on the next change — but if the agent stays thinking for a long time and no new state-changing event arrives, A is stuck on idle until the agent transitions again.

Worse: if the agent goes thinking → idle and A was stuck on idle, A's agentInfoEqual returns true (both idle) → A never re-publishes. A's stale idle is now correct by accident, but only because the agent happened to return to the state A was stuck on. If A was stuck on thinking and the agent went idle, A stays thinking until the next state change.

Impact: Amplifies all other bugs. Once a divergence occurs, the equality gate prevents self-healing until a different state arrives. For agents with long-lived states (e.g. OpenCode thinking for 30s), the desync window is the full duration of that state.

Status: PARTIALLY fixed. #1550 fixes the reconnect variant (stale state pinned across a link death). Stale state from missed events (bugs 2–5) still persists.

---

Summary

#	Bug	Root cause	Status
1	Dual sensor instances	startAwareness called from both local.ts:696 and daemon.ts:245	NOT fixed
2	matchesAgent gate startup race	Brief — foreground tap replays snapshot (local.ts:278); self-resolves in ms	NOT fixed (low severity)
3	Independent WAL debounce	Per-process fs.watch with 150ms debounce, events can drop	NOT fixed
4	decayTransientState timers	Per-process setTimeout recheck, Date.now() timing differs	NOT fixed
5	Screen-scrape poll	Per-process 1000ms poll reading PTY screen at different moments	NOT fixed
6	Activity stream divergence	Independent startForegroundSensor per process	PARTIALLY fixed (#1550 reconnect ghost)
7	agentInfoEqual locks in divergence	Dedup gate blocks re-publish of structurally-equal info	PARTIALLY fixed (#1550 reconnect ghost)

Persistent desync mechanisms are bugs 1, 3, 4, 5, and 7. Bug 2 is a brief startup race that self-resolves. Bug 6 is partially mitigated by #1550.

The fix

R-compose-1 (the awarenessFor seam, R8) makes kolu-server's sensors write to ONE terminalWorkspaceSurface.awareness collection, and kolu reads via awarenessFor. But that alone leaves pulam-web's localhost pulam running its own sensors.

The real fix for the localhost case: pulam-web should not spawn its own pulam for localhost — it should read from kolu-server's awareness surface directly. That's the same awarenessFor backing-swap R9 does for remote, applied locally: one sensor set, one collection, two readers.

All 7 bugs collapse to zero once there is one sensor instance per terminal — the premise R-compose-1 and R9 are built on.