OpenClaw Sandbox Startup Observability Guide

Goal

Design OpenClaw so a parent process that spawns it inside a sandbox can observe startup deterministically from process launch through gateway bind, sidecar startup, channel/account startup, and final operational readiness.

This guide is intentionally high-level. It describes the product and architecture we should build, not a file-by-file implementation checklist.

Problem Statement

Today, a sandbox supervisor can observe parts of startup, but not the whole lifecycle through one clear contract.

What exists now:

The child process can be observed at the OS level.
The gateway can be probed with /healthz, /ready, and /readyz.
WebSocket clients can observe steady-state events such as health, tick, and session events.
Channel-specific probes exist through channels.status with probe: true.

What is missing:

A canonical startup state model.
A replayable startup snapshot for late observers.
Incremental startup progress events.
A clean distinction between transport live, startup in progress, and operationally ready.
A parent-process-friendly machine interface for standard sandbox launches.

The current system is good at answering "is the process up?" and "is the gateway ready right now?" but weak at answering:

What phase is startup currently in?
Which channel/account is still blocking readiness?
Has startup completed but degraded?
Is the process still making progress, or is it stuck?

Design Principles

1. Logs are not the control plane

Human logs are useful for debugging, but they should not be the primary machine contract for a parent process.

2. Use layered observability

A parent process should be able to observe startup through multiple layers:

process state
structured startup stream
startup snapshot
readiness probe
health and channel detail

Each layer answers a different question.

3. Separate startup from readiness

Startup answers "what phase of boot are we in?"

Readiness answers "can this instance safely accept real work now?"

Those are related, but not the same.

4. Make state replayable

If a sandbox attaches late, it should still be able to fetch the current startup state instead of reconstructing it from missed events.

5. Keep the normal UX intact

Standard human-facing runs should remain human-readable. Machine-readable startup should be explicit and stable.

The State Model We Should Add

OpenClaw should expose a first-class startup state object owned by the gateway runtime.

Recommended top-level shape:

type StartupPhase =
  | "booting"
  | "http_listening"
  | "early_runtime_started"
  | "subscriptions_started"
  | "runtime_services_started"
  | "post_attach_started"
  | "channels_starting"
  | "sidecars_starting"
  | "startup_complete"
  | "startup_degraded"
  | "startup_failed";

type StartupSnapshot = {
  phase: StartupPhase;
  startedAt: number;
  updatedAt: number;
  complete: boolean;
  failed: boolean;
  degraded: boolean;
  live: boolean;
  ready: boolean;
  pendingMethods: string[];
  blockers: string[];
  channels: Record<string, {
    status: "pending" | "starting" | "running" | "healthy" | "failed" | "disabled" | "unconfigured";
    accounts: Record<string, {
      status: "pending" | "starting" | "running" | "healthy" | "failed" | "disabled" | "unconfigured";
      lastError?: string | null;
      lastStartAt?: number | null;
      lastProbeAt?: number | null;
    }>;
  }>;
};

This object should be the source of truth for sandbox startup observation.

The Observation Surfaces We Should Support

1. Structured parent-process stream

OpenClaw should support an explicit machine mode for sandbox launches, for example:

openclaw gateway run --startup-json

In that mode:

structured startup events are emitted as JSON lines on stdout
human logs are routed to stderr
the output contract is stable and documented

Example lines:

{"type":"startup","phase":"http_listening","ts":1713139200000,"port":18789}
{"type":"startup","phase":"channels_starting","ts":1713139200100}
{"type":"startup","phase":"channel_account","channel":"telegram","accountId":"default","status":"starting","ts":1713139200200}
{"type":"startup","phase":"channel_account","channel":"telegram","accountId":"default","status":"healthy","ts":1713139201200}
{"type":"startup","phase":"startup_complete","ready":true,"ts":1713139201300}

This is the best surface for a direct parent process supervising a child.

2. Snapshot endpoint

Add a startup snapshot surface that a sandbox can poll after the port is known.

Preferred options:

HTTP: /startup
RPC: startup.status

The snapshot should return the current canonical startup state object.

3. Event stream

Add a new gateway event such as:

startup.changed

This should be broadcast over the existing WebSocket event mechanism whenever startup state changes.

This gives a live UI or supervisor incremental progress without polling.

4. Existing readiness probe

Keep /ready and /readyz as the coarse final readiness gate.

These should continue to answer a narrow question:

ready now, yes or no

They should not become the only startup protocol.

5. Detailed health and channel inspection

Keep health and channels.status for deeper diagnosis once the caller needs details.

These are good secondary surfaces, not the primary startup contract.

The Semantic Layers We Should Expose

The parent process needs three independent answers.

Live

The process is alive and the gateway transport is listening.

This is the point where:

the parent knows the bind succeeded
the child is reachable
but channels and sidecars may still be starting

Starting

The process is still progressing through startup work.

This includes:

sidecar initialization
plugin service start
channel/account startup
deferred warmups

Ready

The gateway is operationally ready for real work.

This should be true only when managed channels that are expected to be online are healthy under the existing readiness policy.

Degraded

Startup may complete even when readiness is false.

Examples:

HTTP is up
startup flow has finished
one required channel failed

That is not "starting" anymore. It is "complete but degraded."

Channel and Account Semantics

Channel startup needs finer granularity than "running."

We should treat these states distinctly:

pending: account has not started yet
starting: startup initiated
running: task is launched
healthy: connectivity/readiness confirmed
failed: startup failed
disabled: account intentionally disabled
unconfigured: account configured surface exists but cannot start

Important rule:

running must not mean ready.

For sandbox observability, healthy is the first state that should count as online.

How Health and Startup Should Interact

The current readiness calculation is useful and should remain the source of truth for the final ready verdict.

The new startup state should incorporate that readiness verdict, but not replace it.

Recommended relationship:

startup state owns phase and progress
readiness checker owns final healthy/not-healthy judgment
startup snapshot embeds the current readiness result

This avoids duplicating policy while still giving sandboxes a unified state object.

Recommended Behavior for a Sandbox Supervisor

The ideal sandbox orchestration flow should be:

Spawn openclaw gateway run --startup-json.
Watch stdout for structured startup events.
Mark the instance as live once http_listening is emitted.
Poll /startup or subscribe to startup.changed for replayable current state.
Use /ready as the final coarse admission gate.
If startup completes but readiness remains false, inspect blockers, health, and channels.status.

This gives a parent process:

deterministic startup progress
no need to scrape human logs
a stable final gate
actionable failure detail

Failure Handling

Startup observability should describe failure explicitly.

The system should emit:

startup_failed when startup aborts
per-channel/account failure records with last error
startup_complete plus ready=false when startup finishes in degraded mode

This distinction matters because a parent process may want different behavior for:

transient startup in progress
startup hard failure
startup completed but with unavailable channels

Minimum Viable Build

If we want the smallest useful version, build this first:

Add a startup state store in the gateway runtime.
Add startup.changed to the gateway event list.
Add /startup or startup.status.
Emit phase transitions for:
- process boot
- HTTP listening
- post-attach start
- channels starting
- sidecars starting
- startup complete
- startup failed
Add --startup-json to emit the same transitions to stdout as JSON lines.
Route logs to stderr in --startup-json mode.

That would already give sandbox users a strong and predictable contract.

Recommended Next Layer

After the minimum version, add:

per-channel/account startup transitions
readiness blockers in the startup snapshot
initial startup snapshot replay on WebSocket connect
explicit degraded terminal state

That second layer is what makes the system excellent rather than merely usable.

What We Should Avoid

Do not tell sandbox users to scrape human log lines.
Do not overload health as the startup protocol.
Do not redefine running to mean healthy.
Do not make late observers reconstruct state from missed events.
Do not make stdout machine-readable in default mode unless we are willing to freeze that contract.

Success Criteria

We should consider this work complete when a sandbox parent process can reliably answer:

Has the child process started?
Is the gateway listening yet?
What startup phase is it in?
Which channels/accounts are still starting?
Which channels/accounts failed?
Is startup complete?
Is the instance operationally ready?
If not, what is blocking readiness?

Without scraping human logs.

johnlindquist/sandbox-startup-observability-guide.md

Select an option

No results found