Smithers Feedback

smithersfeedback.md

Smithers: Biggest Issues, Opportunities, and Hardening Plan

Date: 2026-04-16
GeneratedBy: Chat GPT Pro Audience: Smithers / JJHub maintainers
Basis: full read of smithers.sh/llms-full.txt, current codeplaneapp/smithers repo structure, JJHub docs, the uploaded field report, and the uploaded Live Run DevTools UI spec.

---

Executive summary

Smithers does not primarily have a workflow-engine problem. It has an operator trust problem.

The core design is strong: React/JSX workflows, SQLite-backed persistence, re-render-driven state, schema-validated task outputs, parallel fan-out, VCS pointers, resumability, event history, and time-travel/devtools primitives. The field report is also clear: users still cannot reliably answer the operational questions that matter most during real autonomous work:

• Is the run alive?
• What is it doing right now?
• Is the UI telling the truth?
• Is it safe to resume, retry, hijack, or rewind?
• Which surface is authoritative when reality and the dashboard disagree?

That gap is why the user who wrote the report ended up building a custom dashboard, a stuck-task reset script, a cron batch wrapper, a docs fetcher, and a bespoke sysbox environment just to make Smithers usable for unattended work.

The main conclusion is this:

Do not re-architect the Smithers DAG engine first. Re-architect and harden the control plane, runtime contract, and inspection surfaces around it.

The second conclusion is equally important:

JJHub should become the blessed Smithers runtime for durable autonomous work, not just an adjacent integration.

JJHub already points in the right direction: jj-native repos, cloud sandboxes, microVM isolation, suspend/resume, streaming, and an upcoming web UI. That is a materially better foundation for long-running Smithers work than asking users to assemble their own bwrap / sysbox / docker proxy stack.

---

What changed since the report

The uploaded report reflects a 0.12.x operating experience. The current repo is now at 0.15.1 and the public docs describe a larger operational surface than the one the report author used. That matters, because some pain points are now at least partially addressed in concept or code, even if they are not yet fully hardened.

The docs and repo now show that Smithers already has or intends to have:

• a Gateway-based remote control plane,
• cron scheduling and persisted cron rows,
• HTTP/SSE observability surfaces and /metrics,
• run/event/frame inspection APIs,
• a TUI with monitoring, approvals, telemetry, and run inspection concepts,
• event types for auto-resume and heartbeat timeouts,
• dedicated devtools, time-travel, scheduler, sandbox, server, and observability packages.

That is good news. It means the path forward is not “invent all of this from scratch.” The path forward is to make these surfaces authoritative, unified, and test-verified.

---

Action items extracted from the talk

These are the clearest product and engineering actions implied by the report.

1. Make liveness truthful. Never show idle unless the system can prove it. Prefer unknown or stale over false calm.
2. Expose progress that humans actually trust. Show current node, current attempt, last tool call, last output, recent token activity, and reason blocked.
3. Turn recovery into a first-class product surface. Stale-run handling, orphan detection, resume, retry, hijack, and rewind need explicit semantics and tests.
4. Bless a real sandbox story. Stop making users discover bwrap vs sysbox vs docker-proxy by trial and error. Make JJHub workspaces the reference runtime.
5. Build the new TUI as a debugger, not just a prettier dashboard. The uploaded DevTools-style tree/inspector spec is the right direction.
6. Standardize on one inspection API. Every UI surface should consume the same snapshot / delta / event model.
7. Productize remote Smithers. Gateway + auth + approvals + signals + cron + streaming should be the standard remote story.
8. Treat docs as contracts. If the docs promise a command, inspector, or metric surface, CI should verify it.
9. Add fault-injection E2E tests for every known failure mode. Every crash-recovery fix needs a reproducible test.
10. Reduce papercuts in the CLI. Resume ergonomics, short-ID handling, diagnostics, and repair flows matter more than they look.
11. Promote backpressure and guardrails. Testing, diff review, approvals, scoring, and policy checks should be default workflow patterns.
12. Separate “implemented” from “hardened.” A feature only counts as done when it survives restart, reconnect, stale ownership, and UI desync.

---

The biggest issues and opportunities

1. The operator trust gap is the real product gap

The strongest signal in the field report is not a specific crash. It is repeated operator confusion: “Why is smithers dead?”, “Why is the dashboard showing idle?”, and “Why can’t I get feedback on progress?” Those questions show up again and again across many conversations. The report author explicitly says dashboards that do not reflect reality are worse than nothing.

This is the top issue because all other autonomous-work ambitions depend on it. If users cannot trust liveness or progress, they will either avoid running jobs unattended or build sidecar tools to work around the product.

Recommendation

Make “truthful observation” a first-class architectural concern:

• Introduce an authoritative run-state model with running, waiting-approval, waiting-event, waiting-timer, recovering, stale, orphaned, failed, cancelled, and unknown.
• Add a first-class notion of reason blocked and reason unhealthy.
• Surface two heartbeat domains everywhere: engine heartbeat and UI heartbeat.
• Add recent-activity signals: last event time, last tool call, last token usage, last stdout/stderr, last file change, last heartbeat payload size.
• Never infer “idle” from an absence of matching process names or stale heuristics alone.
• Make the control plane the source of truth. UIs should not directly infer status by ad hoc SQL queries plus ps scraping.

Opportunity

If Smithers becomes the system that operators trust to tell the truth about long-running AI work, that is a major competitive advantage. The engine is already differentiated; the trust layer is where product value compounds.

Definition of done

A run can always be placed into a truthful state, and every UI surface shows the same answer from the same backend data.

---

2. Durability exists, but ownership and recovery semantics are still the hard part

The report shows a durable core with messy operational recovery: stuck in_progress rows, manual reset scripts, sandbox crashes, auth failures, and the recurring question of whether a run is actually dead or merely detached. The current docs and event model show that Smithers now has stale-task recovery concepts, task heartbeats, auto-resume events, and a supervisor model. That is the correct direction. The problem is that these need to be treated as a system contract, not as best-effort behavior.

Recommendation

Define explicit ownership and recovery semantics:

• Persist a run lease / owner epoch so a run has exactly one active owner at a time.
• Distinguish between owner alive, owner stale, owner lost, and owner superseded.
• Make resume, hijack, auto-resume, and rewind auditable state transitions, not informal operational actions.
• Add explicit state transitions for recovering and replaying.
• Make supervisor behavior a product surface with clear policies: stale thresholds, backoff, max concurrent recoveries, retry windows, and idempotency expectations.
• Treat child runs, approvals, timers, and continue-as-new handoffs as part of the same recovery model.
• Require tool side effects to participate in this model through idempotency metadata and replay warnings.

Opportunity

Smithers can become unusually strong at “crash and continue” operation because the underlying re-render/persistence model is already a good fit for it. Most orchestration systems bolt recovery on later; Smithers can make it foundational.

Definition of done

Every recovery path is deterministic, typed, observable, and covered by a reproducible end-to-end test.

---

3. The sandbox story should be simplified and centered on JJHub

The report’s most expensive detour was not writing workflows; it was discovering an environment that could keep credentials, preserve filesystem state, support browser automation, and survive repeated execution. The author tried bubblewrap, docker socket proxy, sysbox, and a VM before landing on a bespoke sysbox-based setup.

That is exactly the kind of problem JJHub is positioned to solve. JJHub’s docs already describe cloud sandboxes, microVM isolation, suspend/resume, streaming, and repository-scoped workspaces. The roadmap explicitly calls out web UI, local workflow execution, and first-class Smithers support.

Recommendation

Make JJHub the blessed Smithers runtime for unattended and remote workflows:

• Define a runtime capability contract: filesystem persistence, VCS access, tool execution, browser support, secret injection, auth persistence, suspend/resume, snapshotting, network policy, and log streaming.
• Implement that contract first against JJHub workspaces / sandboxes.
• Treat local host, bubblewrap, docker, and sysbox as secondary adapters for dev and debugging rather than the main story.
• Ship a reference “Smithers on JJHub” architecture with opinionated defaults for agent auth, browser automation, secret management, and workspace lifecycle.
• Make the failure domains explicit: engine, workspace, sandbox transport, provider auth, and external tools.

Opportunity

This is not only a reliability fix. It is also product alignment. JJHub’s philosophy is already “AI agents are first-class citizens” and “backpressure and guardrails are the product.” Smithers provides the workflow kernel; JJHub can provide the operational substrate.

Definition of done

A user who wants durable autonomous work no longer has to invent a sandbox architecture before Smithers becomes usable.

---

4. The new TUI should be an inspector-first debugger

The report’s criticism of the old TUI is straightforward: it was too shallow to answer real operational questions. The uploaded Live Run DevTools UI spec points in the right direction. It reframes Smithers honestly as a React tree and proposes a DevTools-style tree + inspector + logs + diff + output + time travel surface with heartbeats and ghost state.

That spec should not be treated as cosmetic UI work. It should be treated as the front-end expression of the correct control-plane contract.

Recommendation

Adopt the spec’s model, but with one refinement:

• Prioritize the inspection model and backend contracts before polishing the chat shell or broad TUI surface area.
• Standardize on a shared DevToolsSnapshot / delta stream contract for every inspector surface: TUI, JJHub web UI, and any desktop GUI.
• Keep the React-tree mental model; it is one of Smithers’ biggest conceptual strengths.
• Implement dual heartbeats, error bubble-up, ghost state for unmounted nodes, explicit tabs for output/diff/logs, and time-travel viewing.
• Make “current activity” legible even for long-running tasks: tool calls, file writes, last stdout chunk, tokens in the last minute.
• Support reconnect and replay from a sequence cursor instead of requiring the UI to rediscover state from scratch.

Opportunity

A real run debugger would directly address the report’s top pain. It would also create a reusable inspection substrate for JJHub’s future web UI and session replay features.

Definition of done

A user can attach to any run and answer, within seconds, whether it is healthy, what it is doing, what changed, and what to do next.

---

5. Remote orchestration is already part of Smithers’ intended shape; productize it

The report author wanted remote Smithers managed via Hermes/OpenClaw-style infrastructure but never got there. That is not a fringe request. The current Gateway docs explicitly position Smithers for long-lived remote control, including approvals, signals, cron, bots, dashboards, and OpenClaw-style systems.

This is a strategic opportunity: Smithers should not merely permit remote operation. It should have a clean, opinionated remote control plane.

Recommendation

Make Gateway the standard remote contract:

• Publish a reference deployment pattern for Smithers Gateway behind auth.
• Define stable RPC contracts for run launch, resume, cancel, approvals, signals, events, frames, outputs, diffs, and cron.
• Treat WebSocket + HTTP + SSE reconnect semantics as part of the product, not an implementation detail.
• Provide clear auth and scope models for bots, humans, and automation.
• Make JJHub the preferred operator UI and runtime for this remote mode.

Opportunity

This unlocks exactly what the field report originally wanted: background work, scheduled batches, remote agents, and durable workflows reachable from anywhere.

Definition of done

Running Smithers remotely is a documented, supported path, not a custom architecture assembled from side projects.

---

6. Docs/runtime drift is itself a reliability bug

Right now there is an important asymmetry:

• the report shows what 0.12.x felt like in practice,
• the current docs describe a much broader operational surface,
• the repo structure shows many of the right packages and primitives,
• and the user explicitly says the new TUI is still in progress.

That combination creates a predictable risk: users will read the docs, trust the behavior, and then run into partial implementations, edge-case failures, or WIP surfaces.

Recommendation

Treat documentation as an executable contract:

• Add maturity labels: stable, beta, experimental, design target.
• Every CLI command and documented route gets at least one CI smoke test.
• Every example workflow in docs gets executed in CI.
• Every inspector surface described in docs gets a snapshot or E2E test.
• If a feature is aspirational, mark it clearly and point to the design spec or ticket instead of describing it as present-tense product behavior.

Opportunity

This reduces user confusion and creates a self-reinforcing loop between docs, tests, and implementation quality.

Definition of done

The docs never overstate runtime reality.

---

7. Testing is the highest-leverage investment from here

The user’s instruction is correct: even if a solution already exists, the team needs tests that make regressions obvious and recovery trustworthy. The uploaded UI spec’s quality bar is also correct: typed errors, observability, integration tests with real dependencies, E2E happy paths, failure-mode tests, performance budgets, and soak tests.

That standard should be adopted beyond the UI work and applied across the control plane and runtime.

Recommendation

Create a Smithers hardening matrix with five mandatory test classes for every operational feature:

1. Contract tests: typed errors, payload bounds, auth scopes, RPC invariants.
2. Integration tests: real DB, real VCS, real gateway/server, real sandbox adapter.
3. Fault-injection E2E: kill process, kill sandbox, lock DB, drop WebSocket, stale owner, auth expiry, forced resume, concurrent subscribers, duplicate approvals, partial output, hot reload mid-run.
4. Soak tests: long-lived subscriptions, repeated cron triggers, repeated resume cycles, memory growth checks.
5. Golden-state inspector tests: frames, event streams, diff/output/log views, time-travel snapshots, ghost state after node unmount.

Opportunity

This is the difference between “we think we fixed it” and “we know it survives production reality.”

Definition of done

Every reliability claim in the docs is backed by a repeatable test that would fail if the behavior regressed.

---

8. CLI and operator papercuts matter because Smithers is operational software

The report’s resume ID confusion, direct DB inspection, JSON hand edits, and repeated “why is this blocked?” loops are not cosmetic problems. They are evidence that the operator surface is under-designed.

Recommendation

Improve CLI ergonomics deliberately:

• Accept unique run-ID prefixes and return an ambiguity error when needed.
• Add smithers why <runId> and smithers doctor run <runId> as first-class diagnostics.
• Add smithers repair <runId> for safe, auditable recovery actions instead of forcing JSON edits.
• Add --json to all inspection commands with stable machine-readable schemas.
• Add lineage-aware inspection so resumed / continued runs are easier to reason about.
• Add better migration/bootstrap diagnostics for missing DB tables or schema drift.

Opportunity

Better operator CLI flows reduce the need for custom glue and make Smithers easier to embed into other systems.

Definition of done

An operator can diagnose and recover common problems from the official CLI without touching the database manually.

---

9. Smithers should lean harder into guardrails, not just autonomy

JJHub’s philosophy is right: backpressure and guardrails are the product. Smithers already has useful primitives here: explicit approvals, side-effect annotations, idempotency warnings, schema validation, metrics, diff review events, and scorer surfaces.

These should move from “available features” to “default workflow architecture.”

Recommendation

Make strong workflow patterns the default:

• plan → implement → verify → review → report,
• explicit approval nodes for irreversible actions,
• diff review before patch application in sandboxed modes,
• policy/scorer gates before merge or deploy,
• CI/test/lint hooks treated as normal workflow backpressure,
• clear side-effect classification for tools and tasks.

Opportunity

This is how Smithers/JJHub can support more autonomy without becoming operationally reckless.

Definition of done

The safest and most reliable workflow shapes are the easiest ones to author.

---

Recommended architecture changes

1. Keep the Smithers execution kernel stable

Do not start by rewriting the DAG/re-render engine. Preserve the core strengths:

• React/JSX workflow model,
• persisted outputs keyed by run / node / iteration,
• re-render as the control plane,
• JJ/Git VCS pointers,
• schema-validated outputs,
• explicit approval / wait / continue primitives.

This is the part users still praise, even in the negative report.

2. Make the control plane explicit

Define Smithers as four layers:

1. Execution kernel: engine, reconciler, DB, VCS, scheduler internals.
2. Control plane: Gateway, serve mode, supervisor, cron, approvals, signals, events, run ownership, recovery.
3. Runtime adapters: JJHub workspace/microVM, local host, bubblewrap, docker/sysbox.
4. Inspection surfaces: CLI, TUI, JJHub web UI, possible desktop GUI.

This separation matters because the report’s pain came almost entirely from layers 2–4.

3. Standardize inspection on one data model

Use one canonical backend model for inspection:

• event stream,
• run snapshot,
• tree snapshot/delta,
• node output,
• node diff,
• time-travel frame access,
• typed run health and block reason.

Every inspection UI should consume this, not build its own truth from SQLite internals.

4. Make JJHub the reference runtime

The recommended runtime hierarchy should be:

• Primary: JJHub workspaces / sandboxes for durable autonomous work.
• Secondary: local execution for development, debugging, and quick iteration.
• Fallback: specialty local adapters where needed.

This gives Smithers a real deployment story instead of a grab bag of sandbox recipes.

5. Treat recovery as a designed state machine

Resume, retry, hijack, continue-as-new, rewind, auto-resume, and approval unblock should all be designed against one state machine and one audit trail.

---

E2E and reliability test plan

Below is the minimum test matrix I would treat as required.

A. Crash / resume / stale ownership

1. Kill the Smithers process mid-task; verify the run becomes stale, supervisor detects it, and resume continues exactly once.
2. Kill the sandbox but keep the orchestrator alive; verify heartbeat timeout and retry / failure semantics.
3. Restart during waiting-approval; verify the approval remains visible and the decision resumes the correct node.
4. Restart during waiting-event; verify correlation and eventual resume still work.
5. Restart during waiting-timer; verify timer persistence and no duplicate wakeups.
6. Force concurrent resume / hijack; verify single active owner.
7. Continue-as-new handoff; verify lineage, carried state, and no duplicate downstream tasks.

B. Inspector truthfulness

8. Active run with frequent tool calls; verify TUI/JJHub UI never reports false idle.
9. UI disconnect and reconnect from sequence cursor; verify state catches up without manual refresh.
10. Node unmount after selection; verify ghost state preserves last-known details.
11. Frame scrub + view-only time travel; verify the engine is untouched.
12. Rewind action; verify VCS state and subsequent frames reflect the rewound run.
13. Error bubble-up in collapsed trees; verify ancestry shows failure markers.

C. Remote control plane

14. Launch, approve, signal, cancel, and resume via Gateway over authenticated RPC.
15. Drop WebSocket mid-stream and reconnect with afterSeq; verify no event gap.
16. Multiple subscribers attached to the same run; verify consistent state and bounded memory growth.
17. Webhook-triggered signal with invalid signature; verify rejection and audit trail.
18. Cron-triggered run plus manual trigger; verify expected dedupe / concurrency semantics.

D. Sandbox / JJHub runtime

19. Auth persistence across workspace suspend/resume.
20. Browser automation task inside the reference runtime.
21. File persistence and VCS pointer integrity across repeated runs.
22. Secret injection and redaction in logs.
23. Network-denied task and policy-allowed task both behave correctly.

E. Safety / side effects / guardrails

24. Retry a task that previously executed a sideEffect: true, idempotent: false tool; verify warning and policy behavior.
25. Approval restriction by user/scope; verify denial and audit logging.
26. Diff-review-required sandbox flow; verify accept / reject semantics.
27. Scorer or policy failure blocks downstream destructive action.

F. Soak and memory

28. 10+ minute live event stream with a busy run; verify no monotonic RSS growth.
29. Repeated cron runs over hours; verify no stuck scheduler state.
30. Long-lived JJHub workspace attached to repeated Smithers runs; verify stable operation.

---

Phased roadmap

Phase 0: trust and recovery first (next 2–4 weeks)

• Freeze the authoritative run-state model.
• Add engine/UI dual heartbeat semantics.
• Add unknown, stale, recovering, and orphaned states.
• Build why, doctor run, and repair CLI surfaces.
• Land supervisor hardening and stale ownership tests.
• Add doc maturity labels and basic docs-to-runtime smoke tests.

Goal: eliminate the “why is smithers dead?” class of confusion.

Phase 1: shared inspector backend + JJHub runtime path (next 4–8 weeks)

• Implement the uploaded DevTools-style snapshot/delta backend.
• Standardize node output/diff APIs.
• Build the inspector-first TUI view on top of that backend.
• Stand up the JJHub reference runtime for Smithers runs.
• Publish a reference remote/Gateway deployment pattern.

Goal: make observation and remote operation first-class.

Phase 2: unified UI and time-travel operations (next 8–12 weeks)

• Bring the same inspector into JJHub’s web UI.
• Add time-travel viewing and carefully gated rewind.
• Add session replay / run replay integration.
• Add policy, scorer, and diff-review defaults to workflow templates.

Goal: make Smithers + JJHub feel like one coherent product.

Phase 3: docs as contracts, not marketing (ongoing)

• Every public command, route, and inspector claim gets CI coverage.
• Every example in docs runs in automation.
• Every major bugfix comes with a fault-injection regression test.

Goal: never let the documented surface drift far from reality again.

---

What I would not change

These are Smithers strengths and should remain central:

• JSX/React as the workflow language.
• Re-render-driven control plane semantics.
• SQLite as the durable local source of truth.
• Explicit workflow primitives like approvals, waits, loops, parallelism, and continue-as-new.
• Zod/schema-validated boundaries.
• JJ-aware VCS integration and pointer capture.
• Fan-out across multiple agents/providers.

The mistake would be to tear these up because the operational surface is weak. The right move is to build a better control plane and better runtime beneath them.

---

Bottom line

Smithers already has the bones of a very strong orchestration system. The current opportunity is not to add more clever workflow features. It is to make the system operationally trustworthy:

• truthful liveness,
• explicit ownership,
• reliable recovery,
• a real remote control plane,
• a debugger-grade inspector,
• a blessed sandbox/runtime story,
• and tests that prove all of it under failure.

If that happens, JJHub becomes the natural home for Smithers: the place where durable autonomous workflows actually live, run, recover, and get inspected.

That is a much stronger future than “Smithers has a cool engine but everyone ends up writing a dashboard and a stale-task reaper.”

---

Source appendix

Primary inputs used for this memo:

1. Smithers consolidated docs: https://smithers.sh/llms-full.txt
2. Smithers repo: https://github.com/codeplaneapp/smithers
3. JJHub docs: https://docs.jjhub.tech/
4. Uploaded report: My smithers experience: a report
5. Uploaded spec: Live Run DevTools UI — Design Spec