Your startup has an AI agent. It lives on Slack. Customers talk to it, and sometimes — usually — it talks back. It manages their ad campaigns, writes their email copy, generates their performance reports. You built it on an open-source agent framework called Hermes, because Hermes existed when you needed it and had a nice README.
Your CTO, Jess Jessington, has been muttering about "the framework" for weeks. She keeps finding things she wants to change and discovering that "change" means "fork the core and lose upstream updates forever." One morning she walks into standup and announces:
"We're moving to Mastra. I found it Thursday night browsing my Slop AI feed on Twitter."
By Friday she'd migrated the core agent loop. By Monday she was demoing dynamic workflows where the agent writes three poems in parallel, scores them, and picks the best one. Max Maxington, your co-founder, who has been trying to add a single cron job for three weeks, puts down his coffee.
"How is the new thing different?"
"Let me draw it on the whiteboard," says Jess.
An agent is a while loop around a language model.
You send the model a message. It responds with either text (the answer) or a list of tool calls (things it wants to do before answering). If it responds with tool calls, you execute them, feed the results back, and ask again. When it finally responds with just text, the turn is over.
while True:
response = call_model(messages)
if response.tool_calls:
results = execute_tools(response.tool_calls)
messages.append(results)
continue
else:
return response.text
That's the whole thing. Seven lines. Everything else in an agent framework exists to answer the question: what happens around this loop?
What happens when the process crashes mid-loop? What happens when someone sends a Slack message while the agent is already thinking? What happens when you want the agent to run a background job that takes twenty minutes? What happens when you deploy new code and the process restarts?
These are systems design questions, not AI questions. And the differences between Hermes and Mastra are almost entirely about how they answer them.
Jess draws the full picture on the whiteboard:
┌──────────┐
│ Customer │
└────┬─────┘
│ message
┌────▼─────┐
│ Adapter │ Slack, Web, API
└────┬─────┘
│
┌─────────────▼──────────────┐
│ Gateway │
│ │
│ ┌───────────────────────┐ │
│ │ Session Manager │ │ Who is this?
│ │ │ │ What were we
│ │ state · history · │ │ talking about?
│ │ resume on restart │ │
│ └───────────┬───────────┘ │
│ │ │
│ ┌───────────▼───────────┐ │
│ │ Turn Loop │ │ The while loop.
│ │ │ │
│ │ ┌─────┐ ┌──────┐ │ │
│ │ │Model│◄──►│Tools │ │ │
│ │ └─────┘ └──────┘ │ │
│ │ │ │
│ └───┬───────────┬───────┘ │
│ │ │ │
└──────┼───────────┼──────────┘
│ │
┌─────────────────┼───────────┼─────────────────┐
│ │ │ │
┌────▼─────┐ ┌─────▼────┐ ┌────▼──────┐ ┌──────▼──────┐
│ Pub/Sub │ │ Storage │ │ Workflows │ │ Background │
│ │ │ │ │ & Tasks │ │ Tasks │
│ external │ │ sessions │ │ multi-step│ │ long-running│
│ signals │ │ memory │ │ agent work│ │ jobs │
│ arrive │ │ spans │ │ with deps │ │ that outlive │
│ here │ │ threads │ │ │ │ a turn │
└──────────┘ └──────────┘ └───────────┘ └─────────────┘
"Everything above the line is the request path — a message comes in, finds a session, runs the loop," says Jess. "Everything below the line is infrastructure. And the difference between Hermes and Mastra is almost entirely about what's below the line."
"Start at the center," says Jess, tapping the Turn Loop box. "The turn loop is the core of every agent framework. It's the seven lines I just described, plus everything that goes wrong when you run them in production."
In Hermes, the turn loop lives in a single function: conversation_loop.py, starting at line 498. It is four thousand five hundred lines long.
Let that register. One function. It handles message processing, tool dispatch, budget tracking, interrupts, compression, system prompt management, response streaming, and error recovery. The actual tool-call check that constitutes the "while loop" part is buried around line 4,132. The loop termination condition, when you finally excavate it:
while (api_call_count < agent.max_iterations
and agent.iteration_budget.remaining > 0) \
or agent._budget_grace_call:
if agent._interrupt_requested:
interrupted = True
break
api_call_count += 1
...This works. It has clearly been battle-tested. But if you want to change how tools are dispatched, or add a step between the model response and tool execution, or insert logging at the boundary between "model thinking" and "tools running" — you're editing a 4,500-line function.
In Mastra, the turn loop is a workflow. The entire thing:
createWorkflow({ id: 'agentic-loop' })
.dowhile(agenticExecutionWorkflow, async ({ inputData }) => {
return inputData.stepResult?.isContinued ?? false;
})
.commit();And the loop body — the single iteration — is another workflow:
createWorkflow({ id: 'agentic-execution' })
.then(llmExecutionStep)
.foreach(toolCallStep)
.then(backgroundTaskCheckStep)
.then(signalDrainStep)
.then(isTaskCompleteStep)
.commit();Call model. Run each tool call. Check for background tasks. Drain signals. Check if we're done. Each step is an independent, testable, replaceable unit.
"But here's the thing that matters," says Jess, underlining something on the whiteboard. "The agent loop uses the same workflow primitive you'd use to build any multi-step business logic. It's workflows all the way down."
We'll come back to why that matters. First, we need to talk about what happens when you restart.
You need to deploy new code. Your Hermes gateway handles this politely: it drains active connections, waits for ongoing turns to finish, shuts down. But if the drain timeout fires before a turn completes — or if the process gets OOM-killed, or your cloud provider decides your instance is someone else's problem now — you have a different situation.
Hermes persists session state to SQLite. But it writes once per turn, at the very end, in a function called finalize_turn(). Everything that happened during the turn — every tool call, every intermediate result, every message — lives only in memory until that final write.
┌──────────────────────────────────────────────────────┐
│ Hermes turn lifecycle │
│ │
│ Tool call 1 ──── result in memory │
│ Tool call 2 ──── result in memory │
│ Tool call 3 ──── result in memory │
│ Tool call 4 ──── 💥 process dies │
│ │
│ finalize_turn() ──── never reached │
│ │
│ On restart: the entire turn is gone. │
│ The agent doesn't know it happened. │
│ The customer watches their report disappear. │
└──────────────────────────────────────────────────────┘
The Hermes codebase is refreshingly honest about this. A code comment in the gateway:
An agent forcibly interrupted by the drain-timeout escalation may never reach
finalize_turn... Its in-flight tool rounds live only in the in-memory_session_messages... never written to SQLite mid-turn, so the immediate pre-restart turn is silently dropped fromload_transcript()on resume.
Silently dropped. The customer is the last to find out.
Mastra takes the opposite approach. Because the agent loop is a workflow, and workflows are durable, every step boundary is a persistence point. When a workflow suspends — whether intentionally or because the process died — it saves a snapshot. When it resumes, it replays from the snapshot:
┌──────────────────────────────────────────────────────┐
│ Mastra turn lifecycle │
│ │
│ Step 1: LLM call ────── persisted ✓ │
│ Step 2: Tool call A ──── persisted ✓ │
│ Step 3: Tool call B ──── persisted ✓ │
│ Step 4: Tool call C ──── 💥 process dies │
│ │
│ On restart: │
│ Steps 1-3: replayed from snapshot (no re-execution) │
│ Step 4: re-executes from its saved input │
└──────────────────────────────────────────────────────┘
You can even call run.timeTravel() on a completed run — not a crashed one, a finished one — to rewind to any step and re-execute from there with different inputs. It's not just crash recovery. It's a debugging primitive.
"This is what I mean about workflows all the way down," says Jess. "Durability isn't a feature they bolted on. It's a property of the workflow engine. The agent loop inherits it for free."
Max raises his hand. "What about the Slack thing? Right now, when someone sends a message while the agent is already processing, we just... drop it."
Jess draws a longer diagram.
"This is the real problem. An LLM has two roles: user and assistant. There's no concept of a third party arriving while it's thinking. So if you want your agent to react to external signals — a Slack message, a cron job completing, a webhook firing — you need infrastructure."
In Hermes, you need to build this infrastructure yourself. Jess walks through the list on the whiteboard:
To handle an external signal in Hermes, you need:
1. A pub/sub mechanism (signals arrive independently of turns)
2. A durable message store (signals survive restarts)
3. Session state tracking (active? idle? mid-turn?)
4. A routing layer (which session gets this signal?)
5. An injection strategy (user msg? assistant msg? system?)
6. Thread binding (where does the reply go?)
7. Prompt injection defense (what if the signal says "ignore
all previous instructions"?)
"You're starting to build this fucking giant list of shit," says Jess. "And each of these is non-trivial. Pub/sub alone — are messages fire-and-forget or durable? At-most-once or at-least-once? Do you use Redis? Roll your own? And then you realize every item on that list has the same sub-problem: make it durable. And then make that durable."
Hermes does have hook systems — two of them, in fact, built independently by different contributors. PluginManager.invoke_hook() is a simple callback registry with fail-open try/except per handler. HookRegistry in gateway/hooks.py is a filesystem-discovered hook system in a separate module. They share no code, no event taxonomy, and no documentation explaining when you'd use one vs. the other.
Mastra has one answer: PubSub. An abstract class with four methods:
abstract class PubSub {
abstract publish(topic: string, data: any): Promise<void>;
abstract subscribe(topic: string, handler: Function, opts?): Promise<void>;
abstract unsubscribe(topic: string, handler: Function): Promise<void>;
abstract flush(): Promise<void>;
}You swap implementations by passing one to the constructor:
new Mastra({
pubsub: new RedisStreamsPubSub({ url: 'redis://localhost:6379' })
})The default is an in-memory EventEmitter — fine for development. Redis Streams gives you consumer groups and crash recovery via XAUTOCLAIM. Google Cloud Pub/Sub gives you managed infrastructure with enableExactlyOnceDelivery as a subscription flag.
Signals reach running sessions through AgentThreadStreamRuntime. The thread ID becomes the pub/sub topic name. If the agent is mid-turn, the signal queues and gets drained at the next step boundary. If the agent is idle, exactly one process acquires a lease, wakes the thread, and the signal becomes the input to the next turn.
External event arrives
│
▼
Publish to thread topic
│
├── Agent mid-turn? → Queue signal, drain at step boundary
│
└── Agent idle? → Race for lease
└── Winner wakes thread
└── Signal becomes next turn input
Max, who has been taking notes, looks up. "So the cron job thing I've been trying to build..."
"Just publish a signal," says Jess. "The infrastructure already exists."
Speaking of things Max has been trying to build.
Hermes has three separate mechanisms for running work in the background, each with different durability guarantees:
┌──────────────────┬─────────────────────────┬──────────────────┐
│ Mechanism │ How it runs │ Survives restart │
├──────────────────┼─────────────────────────┼──────────────────┤
│ Cron jobs │ In-process scheduler, │ Yes (jobs.json) │
│ │ 60s tick, daemon thread │ │
├──────────────────┼─────────────────────────┼──────────────────┤
│ Background │ ThreadPoolExecutor, │ No │
│ delegation │ daemon thread │ │
├──────────────────┼─────────────────────────┼──────────────────┤
│ Kanban tasks │ asyncio loop, 60s poll, │ Yes (SQLite) │
│ │ subprocess per task │ │
└──────────────────┴─────────────────────────┴──────────────────┘
The background delegation system — the one you'd naturally reach for when an agent needs to kick off a long-running task — is explicitly not durable. From its own docstring:
Background delegations are NOT durable: if the parent session is closed... or the process exits before a subagent finishes, that subagent's work is discarded.
Fire and forget. If the process restarts, the work vanishes.
Mastra's background tasks are built on pub/sub:
const TOPIC_DISPATCH = 'background-tasks';
const TOPIC_RESULT = 'background-tasks-result';
const WORKER_GROUP = 'background-task-workers';
// Exactly one worker claims the task (consumer group)
await pubsub.subscribe(TOPIC_DISPATCH, workerCallback,
{ group: WORKER_GROUP });
// Every subscriber sees lifecycle events (fan-out)
await pubsub.subscribe(TOPIC_RESULT, resultCallback);Every state transition — dispatched, running, suspended, completed, failed — is a published event. The parent can await task.waitForCompletion() or stream progress via manager.stream(), which is literally pubsub.subscribe wrapped as a ReadableStream. If you're using Redis-backed pub/sub, the tasks survive process restarts because the messages survive in the stream.
Same primitive. Same infrastructure. Different application.
"Okay," says Jess, drawing a bigger box on the whiteboard. "This is the part I actually care about."
In Hermes, a "workflow" is a Kanban board. Literally. There's a SQLite database with a tasks table, a dispatcher that polls it every 60 seconds, and when a task enters the "ready" state — meaning all parent tasks are complete — it spawns a new OS process:
subprocess.Popen([
"hermes", "-p", profile,
"chat", "-q", f"work kanban task {task_id}"
])One operating system process per task. The whole cycle looks like this:
┌─────────────────────────────────────────────────────────────┐
│ Kanban Dispatcher │
│ (polls every 60s) │
│ │
│ ┌──── SQLite ────┐ │
│ │ │ │
│ │ tasks table │ │
│ │ ┌──────────┐ │ │
│ LLM │ │ task A │──┼─── all parents done? │
│ decomposes ──▶│ │ task B │ │ │ │
│ goal into │ │ task C │ │ ┌───▼────┐ │
│ task DAG │ └──────────┘ │ │ claim │ CAS lock │
│ │ │ │ task │ │
│ │ task_links │ └───┬────┘ │
│ │ (parent → │ │ │
│ │ child deps) │ ▼ │
│ └────────────────┘ ┌──────────────┐ │
│ │ subprocess │ │
│ │ .Popen( │ │
│ ┌── result ◄──────────────│ "hermes │ │
│ │ │ chat -q │ │
│ ▼ │ work task" │ │
│ mark complete │ ) │ │
│ in SQLite └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Dependencies are tracked as parent-child links in the database. It works — it's durable, it handles dependencies — but it's a task queue, not a workflow engine. There's no step typing, no retries-per-step, no compensation if step three fails after steps one and two succeeded. The database schema even has columns for a future workflow system (workflow_template_id, current_step_key) that are commented as "v2, not yet routed on."
"V2, not yet routed on" is the saddest phrase in systems design.
In Mastra, a workflow is a flat array of step definitions with a fluent builder:
const processOrder = createWorkflow({ id: 'process-order' })
.then(validateStep)
.then(chargeStep)
.parallel([shipStep, notifyStep])
.commit();.then() is literally array.push. .commit() freezes the array into an execution graph. A pluggable ExecutionEngine interprets the graph — you can swap in Temporal or Inngest as the execution backend without changing your workflow definition. Inngest does this by overriding a single hook function. Temporal does it by Babel-parsing your source and rewriting it into proxyActivities calls. Same DSL, opposite strategies, both real and shipped.
The operations that a Kanban board can't give you:
Rewind. run.timeTravel() can target any step in a completed workflow, override its input, and re-execute from there. The implementation walks the execution graph, replays every prior step's output from the snapshot, and marks only the target step as running. You can't rewind a subprocess that already exited.
Typed boundaries. Mastra threads a type parameter through every .then() call — if step N's output type doesn't match step N+1's input type, TypeScript rejects at compile time. Hermes passes unstructured text between processes via the Kanban's description field.
Composition. A Mastra Workflow implements the Step interface, so you can nest a workflow inside another workflow by passing it to .then(). The dispatcher detects this via a component === 'WORKFLOW' tag and handles it recursively. You can compose good workflows into bigger workflows, and the nesting is real — child workflows get their own execution context, their own snapshots, their own logging.
And here's the punchline. The agent's own turn loop is a workflow built with the exact same primitives:
┌───────────────────────────────────────────────────────┐
│ YOUR CODE │
│ │
│ createWorkflow({ id: 'process-order' }) │
│ .then(validateStep) │
│ .then(chargeStep) │
│ .parallel([shipStep, notifyStep]) │
│ .commit() │
│ │
├───────────────────────────────────────────────────────┤
│ THE AGENT LOOP ITSELF │
│ │
│ createWorkflow({ id: 'agentic-loop' }) │
│ .dowhile( │
│ createWorkflow({ id: 'agentic-execution' }) │
│ .then(llmExecutionStep) │
│ .foreach(toolCallStep) │
│ .then(signalDrainStep) │
│ .then(isTaskCompleteStep) │
│ .commit(), │
│ isContinued │
│ ).commit() │
│ │
├───────────────────────────────────────────────────────┤
│ SAME WORKFLOW ENGINE │
│ ↓ │
│ Durability, logging, replay, typed boundaries │
│ — implemented once, inherited by everything │
└───────────────────────────────────────────────────────┘
When Mastra's team improves their workflow engine — better logging, better snapshot performance, better debugging tools — the agent loop improves automatically. Not as a separate feature. Not as a backport. The agent loop is a workflow, so it gets workflow improvements for free.
"One more thing," says Jess. "This is the one that originally made me start looking."
Hermes's extensibility story has a split personality. Some things are beautifully externalized. The agent's identity loads from a markdown file on disk — SOUL.md in your Hermes home directory. Skills are plain markdown files, read fresh on every invocation. Plugins have a real register(ctx) contract with four discovery paths (bundled, user home, project directory, pip entry points). The architecture doc explicitly states the philosophy: "The core is a narrow waist; capability lives at the edges."
But then you hit the guidance layer. The operational rules that actually shape agent behavior — how it uses tools, how it manages memory, how it decides a task is complete — are hardcoded as Python string constants:
# prompt_builder.py, lines 136-438
HERMES_AGENT_HELP_GUIDANCE = "When the user asks for help..."
TASK_COMPLETION_GUIDANCE = "Before marking a task complete..."
PARALLEL_TOOL_CALL_GUIDANCE = "When multiple tools can run..."Three hundred lines of behavioral instructions, baked into Python. Boolean on/off gates, but no content override. You can turn TASK_COMPLETION_GUIDANCE on or off. You cannot change what it says. To change what it says, you edit core Python. Which means you fork. Which means you lose upstream updates.
"It was either you fork it and then you lose all the pieces from their updates," says Jess, "or you kind of just use it and accept what they give you."
Mastra takes constructor injection to its logical conclusion. Every subsystem is a field on a configuration object:
new Mastra({
agents: { myAgent: new Agent({ instructions, tools }) },
storage: new PostgresStore({ connectionString }),
pubsub: new RedisStreamsPubSub({ url }),
memory: new Memory({ storage, vector, embedder }),
})No field is required. Every field has a safe default — in-memory store, EventEmitter pub/sub, no memory. If you use a default that isn't production-safe, the framework tells you:
storage = new InMemoryStore();
this.#logger?.warn(
'No `storage` configured... not safe for production...'
);You don't fork Mastra. You compose it. The difference between "extend at the edges" and "swap anything in the middle."
Hermes has four independent SQLite databases: state.db for sessions and transcripts, projects.db for project metadata, kanban.db for the Kanban board, and response_store.db for gateway API responses. Plus a flat-file memory store (memories/MEMORY.md). Plus eight pluggable memory backends that each store data in their own format.
These databases share a path convention — they all live under get_hermes_home() — but no storage abstraction, no common schema, and no unified query interface. The state database's own docstring disclaims responsibility: "Batch runner and RL trajectories are NOT stored here (separate systems)."
Imagine a company where everybody keeps their ad performance data in Excel files on their own desktop. That's roughly what's happening here, except the desktops are SQLite files and the company is your agent framework.
Mastra defines 38 table names in one file (storage/constants.ts) with a shared schema registry. A generic createTable() function renders real DDL from that registry — the same schema, targeting Postgres with jsonb columns or LibSQL with TEXT columns, depending on your backend. MastraCompositeStore lets you route different data domains to different backends:
new MastraCompositeStore({
default: pgStore,
domains: { memory: libsqlStore.stores.memory }
})One schema. Many backends. Composable by domain. Observability data and application data live in the same storage layer, which means you get correlated tracing without building a separate logging pipeline.
This will be brief, because the architectural point is simple even if the implementation isn't.
Mastra defines roughly 28 span types in a single enum: AGENT_RUN, WORKFLOW_RUN, WORKFLOW_STEP, TOOL_CALL, MODEL_GENERATION, MEMORY_OPERATION, and so on. Every workflow step opens a span before execution and closes it after. This is automatic — part of the same executeWithContext function that runs the step's code.
An OpenTelemetry bridge maps every Mastra span onto a real OTel span, parent-child relationships preserved. A DualLogger wraps every logger.info() call to also forward into the span-correlated logger with traceId and spanId, requiring zero changes at call sites.
The result: you get structured traces at every boundary — model thinking, tool executing, workflow stepping — without manual instrumentation. Because the agent loop is a workflow, every agent turn automatically gets workflow-level observability.
In Hermes, observability is what you get when you grep.
The Slack integration is two layers stacked on top of each other, and the lower one isn't even Mastra's code:
┌──────────────────────────────────────────────────────┐
│ Vercel's chat SDK (open source) │
│ interface Adapter<TThreadId, TRawMessage> │
│ postMessage / parseMessage / handleWebhook │
│ → Message protocol. Parsing. Thread management. │
├──────────────────────────────────────────────────────┤
│ Mastra's channel layer │
│ interface ChannelProvider │
│ getRoutes / connect / configure │
│ → OAuth. Account lifecycle. Agent wiring. │
└──────────────────────────────────────────────────────┘
The hard part — parsing Slack's message format, managing threads, handling webhooks — comes from Vercel's @chat-adapter/slack package. Mastra adopted it rather than rebuilding it. Mastra's own ChannelProvider interface requires only id and getRoutes(); everything else is optional.
This is why Jess implemented a working Slack adapter in an afternoon. The protocol layer was already solved by someone else. She just had to wire it to the agent.
Max has been quiet for a while. "Why does any of this matter? The agent still talks to people on Slack. The customer doesn't care if the turn loop is 4,500 lines or a .dowhile()."
"The customer cares when we deploy and their conversation disappears," says Jess. "The customer cares when background tasks evaporate on restart. The customer cares when the agent loses context because our session recovery strategy is 'load the last five messages and hope for the best.'"
She draws a final summary:
Hermes Mastra
Agent loop: 4,500-line function workflow
Sessions: write once per turn snapshot per step
Pub/sub: two separate hook one pluggable
systems abstract class
Background tasks: fire-and-forget durable, on pub/sub
ThreadPool
Workflows: kanban board + compiled step graph
subprocess per task with rewind/replay
Storage: 4 SQLite databases 1 composable schema
Observability: grep structured spans
Extension model: fork the core swap a constructor arg
"The insight isn't that Mastra has more features," says Jess. "It's that everything is built on three primitives — workflows, pub/sub, and pluggable storage." She draws one more diagram:
Three primitives What they power
───────────────── ─────────────────────────────────
┌──── Agent turn loop (.dowhile)
│
┌────────────┐ ├──── Business workflows (.then/.parallel)
│ Workflows │────────────┤
└────────────┘ ├──── Dynamic agent-authored workflows
│
└──── Typed, rewindable step graphs
┌──── External signal delivery (Slack, webhooks)
│
┌────────────┐ ├──── Background task dispatch + lifecycle
│ Pub/Sub │────────────┤
└────────────┘ ├──── Session wake-up (lease-based)
│
└──── Cross-process coordination
┌──── Sessions, threads, messages
│
┌────────────┐ ├──── Workflow snapshots (durability)
│ Storage │────────────┤
└────────────┘ ├──── Observability spans + traces
│
└──── Memory (vector + contextual)
"When you improve one primitive, everything that uses it gets better."
She caps the marker.
"In Hermes, each of these is a separate system, built separately, with separate durability guarantees. To add durability to background tasks, you'd have to build durability for background tasks specifically. In Mastra, background tasks are already durable because they already use pub/sub, and pub/sub is already backed by durable storage."
Max nods slowly. "And the poem tournament demo?"
"That was me showing off. But yes — the agent can construct a workflow at runtime and execute it. Because workflows are just arrays of step definitions. If you can build an array, you can build a workflow. The agent writes three poems in parallel, evaluates each one, picks the winner. Each step has typed inputs and outputs. Each boundary is logged and replayable. And the agent didn't need special 'dynamic workflow' machinery to do it — it used the same createWorkflow and .then() calls that everything else uses."
"Can Hermes do that?"
Jess pulls up the Kanban database schema and points to a column.
"workflow_template_id. Commented: 'v2, not yet routed on.'"
"So no."
"So no."