Lion lispy smithering

lionlispysmithering.md

Here’s the useful answer: yes, this is technically plausible, and Lion should target Smithers’ Effect API first, not JSX text generation as the long-term path. Smithers now has a documented React-free Effect authoring surface specifically suitable for generated workflow definitions: Smithers.createWorkflow(...).build(($) => ...), $.step, $.sequence, $.parallel, $.match, $.loop, $.approval, and $.component. The docs explicitly say this API uses the same Smithers runtime as JSX and is appropriate when you “need a React-free API for generated workflow definitions.” (Smithers)

Lion is also a good fit because its programs are valid JSON, arrays are Lisp-style function-call expressions, objects evaluate recursively, and custom environments can inject ordinary JS values/functions/classes. (Lionlang) That means a Lion front-end can be a real AST compiler rather than another string DSL. The old TOON research is the strongest precedent: TOON compiled declarative workflows into Smithers’ builder graph, inferred dependencies from references, supported schemas/components/imports/agents/loops/approvals/cache/retry, and its main warts were exactly the things Lion improves: string schemas, string interpolation, JavaScript new Function, and manual component ID hygiene.

Message I’d send to Andrue / the Lion dev

I dug into this and I think Lion is actually a very good fit for a Smithers front-end.

The important thing is: don’t think of this as “Lion instead of JSX” by generating JSX strings. The cleaner path is Lion → Smithers Effect API / builder graph. Smithers already exposes a React-free API with Smithers.createWorkflow({ name, input }).build(($) => ...), where you construct steps, sequences, parallels, matches, loops, approvals, and components directly. That API is explicitly meant for generated workflow definitions.

There was also an older Smithers .toon declarative format that did basically this: parse declarative workflow syntax, compile it to the same internal graph, then reuse the existing durable runtime. TOON got retired because its surface had a bunch of string-DSL problems: schemas like "string[]", JS expressions in strings, brace interpolation, manual {id} component expansion, etc. Lion solves most of those by making schemas, refs, prompts, conditions, and component params first-class data.

The MVP I’d propose:

1. Define a small smithers/* Lion module: workflow, step, sequence, parallel, approval, loop, match, schema/*, ref, input, template.
2. Compile Lion AST to Smithers.createWorkflow(...).build(($) => ...).
3. Walk the AST to infer dependencies from ["ref", "research", "summary"] or similar. That replaces TOON’s {research.summary} string scanning.
4. Compile Lion schema forms to Effect Schema: ["schema/struct", {...}], ["schema/array", ...], ["schema/optional", ...], ["schema/literal", ...], etc.
5. For agent prompt steps, either reuse/expose Smithers’ existing structured-output prompt/extraction path or implement a small wrapper that appends JSON-schema instructions, calls agent.generate, extracts JSON, and lets Smithers validate the result.
6. Keep node IDs stable. That’s load-bearing for resume.

I’d start with a codegen-less prototype:

import { Smithers } from "smithers-orchestrator";
import { Schema } from "effect";
import { run as runLion } from "@lionlang/core/evaluation/evaluate";
import { stdlib } from "@lionlang/core/modules";

export function compileLionWorkflow(program, env) {
  const wf = parseWorkflow(program);
  const input = compileSchema(wf.input);

  return Smithers.createWorkflow({ name: wf.name, input }).build(($) => {
    const ctx = {
      $,
      handles: new Map(),
      schemas: new Map(),
      agents: env.agents ?? {},
      services: env.services ?? {},
      baseDir: env.baseDir,
    };

    return compileNode($, wf.body, ctx);
  });
}

A Lion workflow could look roughly like:

["smithers/workflow",
  {"name": "research-report",
   "input": ["schema/struct", {
     "topic": "schema/string"
   }]},

  ["smithers/sequence",
    ["smithers/step", "research",
      {"agent": "researcher",
       "prompt": ["template",
         "Research the topic.\nTopic: ",
         ["input", "topic"]],
       "output": ["schema/struct", {
         "summary": "schema/string",
         "keyPoints": ["schema/array", "schema/string"]
       }]}],

    ["smithers/step", "report",
      {"agent": "writer",
       "prompt": ["template",
         "Summary: ", ["ref", "research", "summary"],
         "\nKey points: ", ["ref", "research", "keyPoints"]],
       "output": ["schema/struct", {
         "title": "schema/string",
         "body": "schema/string",
         "wordCount": "schema/number"
       }]}]]]

The compiler walks the prompt AST, sees ["ref", "research", ...], registers needs: { research }, and creates a Smithers step. No string parsing. No new Function. No "string[]" grammar. That’s the actual win.

The first integration could be @smithers/lion or smithers-orchestrator/lion with:

import { Smithers } from "smithers-orchestrator/lion";

const workflow = Smithers.loadLion("./research.lion.json", {
  agents: { researcher, writer },
});

Then later the CLI can learn .lion.json, but the MVP can just export a normal Smithers workflow from a .ts wrapper and run with the existing CLI.

Concrete direction / implementation plan

1. Target the Effect API first

Use this as the central compile target:

const workflow = Smithers.createWorkflow({
  name,
  input: inputSchema,
}).build(($) => {
  return compileLionNode($, rootNode, env);
});

Smithers’ Effect API already covers the mechanical graph operations: steps, dependencies, sequence, parallel, match/branching, loops, approvals, components, retry, cache, timeout, and SQLite persistence. (Smithers) This is a better fit than emitting JSX because Lion is already data and the Effect API is already a graph builder.

2. Use Lion AST references for dependency inference

Replace TOON-style string interpolation:

prompt: "Summary: {research.summary}"

with structured Lion references:

["template", "Summary: ", ["ref", "research", "summary"]]

Dependency collection becomes a pure AST walk:

function collectDeps(expr: unknown, deps = new Set<string>()) {
  if (Array.isArray(expr)) {
    const [tag, ...rest] = expr;

    if (tag === "ref" && typeof rest[0] === "string") {
      deps.add(rest[0]);
      return deps;
    }

    if (tag === "quote") return deps;

    for (const item of rest) collectDeps(item, deps);
  } else if (expr && typeof expr === "object") {
    for (const value of Object.values(expr)) collectDeps(value, deps);
  }

  return deps;
}

That preserves TOON’s best feature, implicit workflow edges from references, without TOON’s brittle expression parsing. The old TOON docs identify implicit dependency inference as the “killer feature” to preserve.

3. Compile schemas as data, not strings

Use Lion forms like:

["schema/struct", {
  "summary": "schema/string",
  "severity": ["schema/literal", "low", "medium", "high"],
  "tags": ["schema/array", "schema/string"],
  "assignee": ["schema/optional", "schema/string"]
}]

Compile to Effect Schema:

function compileSchema(form: unknown): Schema.Schema<any> {
  if (form === "schema/string") return Schema.String;
  if (form === "schema/number") return Schema.Number;
  if (form === "schema/boolean") return Schema.Boolean;

  if (Array.isArray(form)) {
    const [tag, ...args] = form;

    if (tag === "schema/array") {
      return Schema.Array(compileSchema(args[0]));
    }

    if (tag === "schema/optional") {
      return Schema.optional(compileSchema(args[0]));
    }

    if (tag === "schema/literal") {
      return Schema.Literal(...args as [string, ...string[]]);
    }

    if (tag === "schema/struct") {
      const fields = args[0] as Record<string, unknown>;
      return Schema.Struct(
        Object.fromEntries(
          Object.entries(fields).map(([k, v]) => [k, compileSchema(v)])
        )
      );
    }
  }

  throw new Error(`Unsupported schema form: ${JSON.stringify(form)}`);
}

This maps cleanly to Smithers’ Effect API, which uses Effect.Schema in Smithers.createWorkflow and step outputs. (Smithers) It also aligns with Lion’s own design, since Lion is built with Effect and Schema for validation/error handling. (Lionlang)

4. Implement three step modes

Smithers JSX <Task> has three modes: agent, compute, and static. (Smithers) Mirror that in Lion:

["smithers/step", "analyze",
  {"agent": "coder",
   "prompt": ["template", "Analyze ", ["input", "repo"]],
   "output": ["schema/struct", {"summary": "schema/string"}]}]

["smithers/step", "count-files",
  {"run": ["fs/count-files", ["input", "repo"]],
   "output": ["schema/struct", {"count": "schema/number"}]}]

["smithers/step", "static-config",
  {"value": {"region": "us-east-1"},
   "output": ["schema/struct", {"region": "schema/string"}]}]

For compute steps, evaluate the Lion expression inside the Smithers step context:

const handle = $.step(id, {
  output,
  needs,
  run: async (stepCtx) => {
    const lionEnv = {
      ...stdlib,
      input: stepCtx.input,
      ...stepCtx,          // dependency outputs by step id
      services: env.services,
      ...env.services,
    };

    return await Effect.runPromise(runLion(runExpr, lionEnv));
  },
});

For prompt steps, either generate JSX first or implement a run wrapper that calls agent.generate. The wrapper should preserve Smithers’ structured-output behavior: append schema instructions, parse JSON, and let Smithers validate. Smithers’ current JSX path injects JSON-schema instructions into agent prompts, parses the response, validates it, and persists it. (Smithers)

5. Branching: use $.match, or synthesize a guard step

Smithers Effect API exposes $.match(source, { when, then, else }) for branches driven by a completed step. (Smithers) For arbitrary Lion conditions, create a small guard step:

["smithers/branch",
  ["boolean/and",
    ["ref", "tests", "passed"],
    ["string/equals?", ["input", "env"], "prod"]],
  ["smithers/approval", "deploy-gate", {...}],
  ["smithers/step", "skip-deploy", {...}]]

Compile as:

1. Collect dependencies in the condition.
2. Create a hidden step like branch-id:condition with output { ok: boolean }.
3. Run the Lion condition in that step.
4. Use $.match(conditionStep, { when: ({ ok }) => ok, then, else }).

That gives a general branch primitive without needing a custom Smithers branch API.

6. Components: fix the TOON hygiene wart

TOON had awkward expanded IDs like tech-review-revise.content. The Lion version should expose component outputs structurally:

["smithers/component", "tech-review", "ReviewCycle",
  {"content": ["ref", "draft", "content"],
   "reviewer": "senior engineer"}]

Downstream references should look like:

["ref", "tech-review", "revise", "content"]

Internally, the compiler can still prefix generated step IDs with tech-review/* or tech-review-revise, but the source language should not leak that. Smithers’ Effect API components already prefix instance step IDs to avoid collisions. (Smithers)

7. MVP test sequence

Port the old TOON test ideas in this order:

1. Static step and run step.
2. Two run steps with inferred needs.
3. Prompt step with imported fake agent.
4. Parallel group.
5. Retry and timeout.
6. Cache key.
7. Approval.
8. Loop with latest/iteration state.
9. Component with stable prefixed IDs.
10. Cross-file imports.

The old TOON reference already has fixtures for almost all of these behaviors, and those fixtures are a good blueprint for parity tests.

The strategic recommendation

Start with @smithers/lion as a compiler/adapter, not a change to Smithers core.

First release:

import { loadLion } from "@smithers/lion";

export default loadLion("./workflow.lion.json", {
  agents: { researcher, writer },
  services: { github, linear },
});

Then the generated/default export is just a normal SmithersWorkflow, which already matches the public type shape: a workflow has a build(ctx) function, options, and an optional schema registry. (Smithers)

Later, add CLI support:

bunx smithers-orchestrator up workflow.lion.json --input '{"topic":"Zig"}'

But don’t block the MVP on CLI integration. The first milestone is: Lion JSON in, Smithers durable workflow out.