Hybrid persistence — touch points (Sandy × Overeasy)

This note only maps where layers meet. Code lives in the linked repos.

Think of a running agent as three things that can be saved separately:

Layer	Plain English	What it saves	Survives kill/reboot?	Who provides it today
Disk	The agent’s files under `/workspace` (scripts, artifacts, `context_N`, the dill file on disk)	Bytes on a filesystem	Yes, if mounted via an Overeasy-like session	Overeasy (`oe mount` / session id) + Sandy paths that already use `/workspace`
Mind	The agent’s live Python state (variables, imports, REPL locals)	Process / dill namespace	Only while the sandbox stays warm (Tier A), or after a future VM checkpoint (Tier B)	Sandy dill in `agentd/repl_runner.py`; keep-alive via `rlm_env.py` + `reaper.py`
Fan-out	Parallel copies of a warm sandbox to try several paths at once	CoW child VMs/processes from one checkpoint	N/A — this is branching, not durability	Sandy `ForkdSpawnBackend` / `forkd_client.py` (not Overeasy’s FS `branch`)

Why “hybrid”: Overeasy is strong at Disk, weak at Mind. Sandy Tier A is strong at Mind (while warm), weak at Disk-after-kill. Fan-out needs Sandy/forkd either way — an Overeasy file-branch is not the same as N live children.

Analogy: Disk = the notebook on the desk. Mind = what’s in your head mid-sentence. Fan-out = cloning yourself to try two answers in parallel.

Overeasy (disk)

Public surface is CLI + Modal examples (implementation is the shipped oe binary, not open-sourced in-tree).

Concern	Where
Mental model / CLI	README.md — `oe session new`, `oe mount`, `oe session branch`, `oe checkpoint`
Kill → remount same files	example/main.py
Short-TTL sandbox loop	example/persistent_sandbox.py
Install into image	install.sh, example/README.md

Hybrid mapping: treat Overeasy’s mountpoint as Sandy’s /workspace (not /).

Sandy (mind + control plane + fan-out)

Concern	Where
Dill load/save across execute	`agentd/repl_runner.py` — `load_state` / `save_state` / `execute_repl`
State file path	`agentd/main.py` — `WORKSPACE / ".rlm_state.dill"`
Files API under `/workspace`	`docs/api.md` — `/v1/sandboxes/{id}/files/*`
Tier A keep-alive + `SupportsPersistence`	`sdk/python/sandy/rlm_env.py` — `persistent=True`, `add_context` / `add_history`
Persistent idle / max-age reaper	`runtime-api/reaper.py` — `is_persistent_sandbox`, TTLs
CoW child spawn	`sdk/python/sandy/spawn_backends.py` — `ForkdSpawnBackend.branch_child`
Shim `POST /v1/branch`	`sdk/python/sandy/forkd_client.py`; host shim under `infra/forkd/shim/`
Provenance `checkpoint_id`	`sdk/python/sandy/call_tree.py`
Tier B plan (live checkpoint / in-guest snapshot)	`docs/milestones/milestone-persistence.md` §§4–5 Phases 3–4 — not implemented
Parallel netns prerequisite	PR #17

Flow (pointers only)

create
  rlm_env.py (persistent label)
  → runtime-api spawn
  → oe mount  (README / example/main.py) on /workspace

turn
  files  → FUSE session (oe)
  mind   → repl_runner.py load/save .rlm_state.dill
  optional fence → oe checkpoint

hibernate
  oe checkpoint
  → [planned] forkd live checkpoint  (milestone-persistence Phase 3)
  → runtime-api delete / reaper.py path

resume
  oe mount <fs_session_id>     (example/main.py pattern)
  → [planned] forkd restore
  → else Tier A: dill file already on remounted /workspace

branch (files)   → oe session branch [--to ts]   (README)
branch (warm N)  → ForkdSpawnBackend.branch_child  (spawn_backends.py)
                 → record checkpoint_id in call_tree.py

Glue still missing (by design)

Glue	Notes
`fs_session_id` on sandbox row	Not in runtime-api schema today — would sit beside `labels.persistent`
Live `forkd.checkpoint` / `restore`	Beyond current `branch` in `forkd_client.py`
In-guest `snapshot()` / `restore()` / `branch()`	Phase 4 in milestone-persistence; peer of `llm_query` in agentd
Mount policy	Enforce `/workspace` only; Overeasy cannot mount `/`

Restore order: FS mount → mind restore (or Tier A dill-only after remount).

Persistence for agent sandboxes: Overeasy vs Sandy

A plain-English comparison of two approaches to pause / resume / branch agent state — and a hybrid that can take advantages of both.

Project	Repo
Overeasy (Modal Labs)	https://github.com/modal-labs/overeasy
Sandy (Decision Labs)	https://github.com/decision-labs/sandy
Sandy persistence plan	https://github.com/decision-labs/sandy/blob/main/docs/milestones/milestone-persistence.md
Sandy Tier A (shipped)	https://github.com/decision-labs/sandy/pull/21
Sandy Tier A follow-ups	https://github.com/decision-labs/sandy/pull/22
Sandy forkd parallel (Tier B prerequisite)	https://github.com/decision-labs/sandy/pull/17

One-line difference

Overeasy = durable filesystem overlay (files survive kill/reboot; process memory does not).
Sandy = REPL / sandbox mind today (warm box + dill), later VM memory + disk checkpoints with in-guest branch/restore (Phase 3–4 — not built yet).

They are different layers of the stack, so a hybrid is possible (see below).

Feature matrix

Capability	Overeasy	Sandy today (Tier A)	Sandy planned (Tier B, Phases 3–4)
Persist files across sandbox kill	Yes (FUSE session → S2 log + S3)	Partial (sandbox must stay alive)	Yes (forkd checkpoint of workspace + more)
Persist Python / REPL locals	No	Yes (dill `.rlm_state`)	Yes (memory + disk checkpoint)
Pause / resume across hosts	Yes (remount same session id)	No (warm container on one host)	Yes (hibernate / restore)
Branch cheaply	Yes (`oe session branch`)	No	Yes (`branch(checkpoint, n)` via forkd CoW)
Revert / time-travel	Yes (branch `--to <timestamp>`)	No	Yes (`restore(checkpoint_id)`)
Parallel live children from one warm state	No (FS branch ≠ N running VMs)	Fan-out via `rlm_query` (new children)	Yes — CoW fork-from-warm (~ms)
In-guest API (`snapshot` / `restore` / `branch`)	No (CLI / host: `oe …`)	No	Planned (next to `rlm_query`)
Ties into call-tree / provenance	No	Call tree for LLM/RLM calls	Planned: real `checkpoint_id` in JSONL
Substrate	FUSE + S2.dev + S3 (Modal-first)	Docker or forkd control plane	forkd / KVM required for full Tier B
Maturity	Experimental OSS (Aug 2026)	Tier A in main; Tier B no PR yet	Spec only

Overeasy

What it is

A virtual filesystem served as a FUSE mount over a base directory. Changes are copy-on-write under a session id. An append-only log (S2) + object storage (S3) make the session durable so you can pause, resume on another host, branch, or revert to a prior timestamp.

Features

oe session new / oe mount <session_id> / oe unmount
oe session branch and branch --to <timestamp> (revert = new branch from past event)
oe checkpoint ≈ fsync to durable storage (~150ms eager upload typical)
Designed for Modal Sandboxes; works on any Linux with /dev/fuse
Example: kill sandbox → new sandbox → remount → same files

Pros

Simple mental model: “Dropbox with undo + copy-this-folder”
Portable; no hypervisor required
Cheap FS branching and time-travel
Strong for coding agents whose state lives mainly on disk
Survives short sandbox TTLs / reboots well

Cons

Does not save RAM / REPL / imported libs
Weak for recursive “continue thinking from this exact mental state”
Branch ≠ N parallel warm sandboxes
Needs S3 + S2 (or Modal-provided creds); FUSE constraints
Explicitly experimental; authors ask to chat before production use

Sandy

What it is

Agent sandbox control plane (runtime-api + in-guest agentd + optional forkd). Persistence is staged:

Already shipped: dill locals across /repl/execute; SandyIsolatedREPL(persistent=True) keep-alive + RLM SupportsPersistence (add_context / add_history, …).
Planned (Phases 3–4): forkd checkpoints of memory + disk; in-guest snapshot() / restore() / branch() for Monte-Carlo / RLM tree-search.

Features (today)

Dill-backed REPL namespace across executes
persistent=True → sandbox stays alive across turns; longer idle TTL
RLM protocol: versioned context_N / history_N, handler address updates
Call-tree provenance for LLM / RLM fan-out
forkd parallel children (host reachability) — prerequisite for Tier B, not checkpoint API itself

Features (planned Tier B)

Incremental CoW checkpoint (memory + configurable disk scope)
Survive kill / eviction; hibernate idle sessions to storage
Real checkpoint_id (not a fake ckpt-{sandbox_id})
In-sandbox snapshot / restore / branch(n) for backtracking rollouts
Deterministic RL episodes from one warmed checkpoint

Pros

Keeps the “mind” of the agent (locals / warm context), not only files
Aligns with recursive RLM + call trees + parallel rollouts
Can become zero-cost idle (hibernate) instead of pinning a warm box
Differentiator vs backends that only keep a container alive or only sync files

Cons

Tier B is hard (forkd/KVM) and not implemented yet
Heavier ops than remounting a FUSE session
Full Tier B won’t land on Docker-only hosts
More moving parts (control plane, reaper, storage upload, readiness)

Hybrid: advantages of both

Yes — a hybrid is possible, because the two systems solve different layers. They are complementary, not mutually exclusive.

Layer split

Layer	Job	Tool
Disk	Files durable across kill/reboot; cheap FS branch/revert	Overeasy-like FUSE overlay on `/workspace`
Mind	Python locals, warm imports, RLM `context_N`	Dill today → forkd memory checkpoint later
Fan-out	N parallel live children from one warm point	forkd CoW (Overeasy branch alone is not enough)

Story: the agent writes under a mounted FS session → files survive host death. Separately, Sandy checkpoints the guest (or at least dill) so “where the REPL was” also comes back. Use Overeasy-style branch when you only need file divergence; use forkd branch when you need live parallel sandboxes with shared warm memory.

What you get

Overeasy’s pause/resume for files and Sandy’s warm mind
Idle hibernate without losing workspace artifacts
FS durability alone on Docker hosts where forkd isn’t available

What you don’t magically fix

Overeasy still doesn’t give parallel warm VMs — you still need forkd for that
Two systems to operate (FUSE + S2/S3 + forkd)
Restore order matters: remount FS, then restore memory (or memory won’t see the files it expects)
More latency/complexity if every turn touches both layers

Phased hybrid (practical)

Now: Sandy Tier A (warm + dill); optional FS overlay only for /workspace artifacts
Soon: Overeasy-style durability under /workspace for hibernate / kill-resume of files
Later: forkd memory+disk checkpoint; FS layer becomes backup / cross-host file sync, not the only source of truth

Bottom line for hybrid: files on an event-log overlay, mind on VM/REPL checkpoint, fan-out on CoW forks. Overeasy alone or Sandy Tier A alone each miss half; together they cover both if you accept the ops cost.

When to pick which

If you need…	Prefer
Files durable across kill/reboot, simple branch/revert	Overeasy (or Overeasy-like FS layer)
Multi-turn Python state without cold restart	Sandy Tier A (available now)
Branch / backtrack reasoning state, warm fan-out, RLM tree-search	Sandy Tier B (planned)
Both durable files and warm mind	Hybrid (FS overlay + REPL/VM checkpoint)

Bottom line

Same words — pause, resume, branch — different layer:

Overeasy = filesystem time machine
Sandy = sandbox mind time machine (partially shipped; full VM branch still ahead)
Hybrid = filesystem time machine under a mind time machine

Refs: overeasy · sandy · persistence plan · hybrid touch points

Glossary

Term	Plain English
FUSE	Filesystem in Userspace. A normal program acts as a filesystem: apps write to a mount; the kernel forwards those ops to the program. Overeasy uses this so every write can be logged and uploaded. Needs `/dev/fuse` (and usually `fuse3`) on Linux. Analogy: a proxy for the filesystem.
CoW	Copy-on-Write. Share one copy of data until someone changes it; only then copy the changed bits for that branch. Why forkd can spawn children in tens of ms and Overeasy can branch sessions cheaply — you don’t clone the whole world up front. Analogy: two people reading one doc until someone edits and gets their own changed pages.
REPL	Interactive Python environment inside the sandbox (locals, imports, variables). Sandy persists this with dill / planned VM checkpoints; Overeasy does not.
dill	Python serialization library (like pickle, but handles more object types). Sandy saves REPL locals to `.rlm_state.dill` across executes.
Tier A	Sandy’s shipped keep-alive persistence: sandbox stays warm; dill + RLM `SupportsPersistence` protocol. “Don’t hang up the phone.”
Tier B	Sandy’s planned forkd checkpoint/restore/branch of memory + disk. “Hang up, come back, or open 8 parallel worlds from a saved moment.” Not built yet.
forkd	Forking microVM / snapshot stack Sandy uses for fast CoW children (KVM). Prerequisite for full Tier B.
checkpoint	A saved point in time you can restore or branch from. Overeasy: durable FS stream position (`oe checkpoint`). Sandy Tier B: forkd `checkpoint_id` for memory+disk.
branch	Create a divergent path from a prior state. Overeasy: new FS session from log tail or timestamp. Sandy Tier B: N live CoW sandbox children from one warm checkpoint.
session id	Overeasy’s durable identity for a filesystem overlay across mounts / hosts.
RLM	Recursive Language Model pattern: agents spawn child sandboxes (`rlm_query`), keep call trees, often need warm context across turns.
call tree	Provenance graph of who called whom (parent/child sandboxes, depth, cost). Sandy records this; planned to bind real checkpoint ids to nodes.
hibernate	Stop paying for a live sandbox but keep enough state to resume later (files and/or mind).
warm	Sandbox (and often REPL) already running with state loaded — no cold start / re-import.
S2 / S3	Overeasy durability backends: append-only event stream (S2.dev) + object blobs (AWS S3).

sabman/sandy-hybrid-persistence-pseudocode.md

Select an option

No results found

Select an option

No results found

Hybrid persistence — touch points (Sandy × Overeasy)

Overeasy (disk)

Sandy (mind + control plane + fan-out)

Flow (pointers only)

Glue still missing (by design)

Persistence for agent sandboxes: Overeasy vs Sandy

One-line difference

Feature matrix

Overeasy

What it is

Features

Pros

Cons

Sandy

What it is

Features (today)

Features (planned Tier B)

Pros

Cons

Hybrid: advantages of both

Layer split

What you get

What you don’t magically fix

Phased hybrid (practical)

When to pick which

Bottom line

Glossary