Skip to content

Instantly share code, notes, and snippets.

@BLamy
Last active August 7, 2026 16:41
Show Gist options
  • Select an option

  • Save BLamy/63a2dd4f7f676c7c769284f21fc8fedb to your computer and use it in GitHub Desktop.

Select an option

Save BLamy/63a2dd4f7f676c7c769284f21fc8fedb to your computer and use it in GitHub Desktop.
ReplayQA-local-loop-building.md

ReplayQA Loop

You are going to help the user create an application through elicitation. You will ask the user what kind of application they want then when they explain it you could come up with the names and descriptions and then spawn a background agent who scaffolds out all the additional boiler plate. While that is running in the background you should continually ask the user about how they want their UI to look. If you have tools for generating photos feel free to create mockups using that tool and present them to the suer in the chat and ask them if they like them or if they want to improve them (remember we are using shadcn on react-aria so the image generator should know that). Once the project is setup and we feel like we have a pretty good understanding of what the user wants to build we should create a ROADMAP.md file which exists in side of our .replay folder. This should have multiple epics and multiple tickets in each epic with a capstone at the end of each epic proving that everything works.

Then after you create the ROADMAP.md file spawn subagents using the smartest model you can on the highest setting you can to expand from a single roadmap file to fully expanded roadmap. There should be a .replay/tasks folder and inside of it we should have a folder for each epic and inside of each epic we should have folders for each ticket. Each ticket should have .md file with its name and that folder is where we will commit things like /evidence and any ephemeral scripts we wrote in order to validate that ticket.

When creating tickets please make epic-0 just creating components in storybook. We should try to plan out every component we need build it, storybook it, Then paste screenshots of the storybook back to the chat so the user can give direction on updating the UI components. When you are validating each ticket you must look at the durable stream if their are backend changes and create replayqa journeys to validate the changes for that ticket.

All tickets will be organized by a priority queue. They must all have a header which looks like this. Make sure there is a pre-commit hook which validates tickets so no invalid tickets ever get committed. 

--- id: E2-T01 epic: 2 title: "Versioned agent configuration schema without embedded secrets" priority: 201 status: pending depends_on: [E1] estimate: M capstone: false ticketUpdateTimes: [2026-08-03T18:07:57] —

When the user ask you to start working you should grab the next item off the queue, build it, validate it with a subagent and then mark it as verified and commit everything back. Only stop looping if you hit invalid_loop or the user explicitly ask you to stop

Create a nx mono repo npx create-nx-workspace@latest

Make sure git is setup. Initialize a repo locally. Try to also push it as a new repo to GitHub but don’t stress too hard if you cant.

Inside of that repo create a new package in /packages for react components

Then install storybooks npm create storybook@latest

Please write all storybooks using https://storybook.js.org/docs/api/csf/csf-next

And test the components using playwright https://storybook.js.org/docs/api/portable-stories/portable-stories-playwright

Create a new react project using
pnpm dlx shadcn@latest init --preset b0 --base aria --template vite
 Clone blamy/emulate a /vendors folder (not as a submodule just a clone) https://github.com/blamy/emulate

Please stick with this structure if the user ask you to create a new app just create a new one in the /apps folder if they ask you to create something that could be reused across multiple apps make it a package and if you need to make changes to a 3rd party library clone it into vendor. Please note you can still use just normal pnpm for anything you don’t need to edit. Vendor is only for libraries you need to modify.

All apps use the auth0 emulator from emulate by default. All application have a login page we should create several different users and validate that the emulated login page works.

You will use durable streams and netlify functions as the backend for your application. All data is added to an append only ledger with its own indexes and caches that are also in their own durable streams. All of these streams should be heavily relied on when we are validated if our application works and we should pay close attention to who is able to access which stream and advasarialy verify all potential angles for unauthorized or unauthenticated access to streams that should be private. https://github.com/durable-streams/durable-streams 
If the user ask you to use Postgres please use neon as the sdk but pglite in dev with the emulator from pg emulate. I’d also like it if we could just check in our pglite database that we would use for our testing fixtures.

When you finish all of this please us create a new project with npx replayqa. After each ticket you implement you should create a new journey in replay QA and are the FRPC tunnel to create a recording. You should fix any bugs that come up and when you have no bugs on your ticket you can consider it done and move on to the next one. At the end of each epic we should do a full replay qa test run. Please actually start the app and the tunnel and validate that replay qa is actually connected and running.

While that is running update the gents.md file to have the following charter

  AGENTS.md — how agents build Stream apps

This is the operating manual for every human or agent working in this repository.
`ROADMAP.md` defines the product and epic order. `.replay/tasks/QUEUE.md` is the
generated source for what may start next. `.replay/loop.md` defines the
builder/critic loop. This file turns those contracts into day-to-day rules.

The target is a multi-tennate workspace where humans and agents are first-class members.
Durable Streams are authoritative for workspace facts and run
history. Query stores, caches, and search indexes are disposable projections.

## The one rule

A builder saying a task works is a claim. Reproducible evidence is proof material. A
task becomes `verified` only when a fresh critic fails to refute the claim against the
task specification, the exact diff, and the cited evidence.

For server work, the primary evidence is a deterministic stream dump, offsets,
canonical state digests, and cold-start command output. For browser work, it is the same
stream evidence plus one final Replay Chromium session that produces both an uploaded
Replay recording and its same-session MP4. A screenshot or a passing test summary is
supporting context, never a replacement for an interrogable run.

## Role

**Builder** — implements exactly one eligible ticket. The builder may self-test freely,
but finishes by recording a final evidence run and writing a falsifiable claim.

**Critic** — a fresh agent/session that did not implement the ticket. The critic does not
fix product code. It tries to falsify every acceptance criterion, maps evidence to the
diff, reruns the task's attacks with independent inputs, and proves the verification
apparatus can go red.

The same agent may fill both roles on different tickets, never on the same ticket.

## Task lifecycle

```text
pending → in-progress → implemented → verified
                              ↘ refuted → in-progress

Status lives in each task's readme.md frontmatter. Only the critic sets verified. After any status or dependency change, run:

python3 tools/build_queue.py

Commit the task readme and regenerated queue together. There is one active queue gate at a time. Every dependency must be verified before a task starts; a bare epic dependency means that epic's capstone must be verified.

Project state lives in .replay/project.json:

  • building — the queue may advance;
  • paused — only a human may resume it;
  • invalid_loop — verification cannot progress honestly; record the reason and stop;
  • complete — every required ticket and final capstone is verified.

Never route around paused or invalid_loop.

Builder protocol

  1. Read .replay/project.json, .replay/tasks/QUEUE.md, and the complete top eligible task. Confirm no other task is active.

  2. Inspect git status --short --branch. Existing changes belong to the user or another worker unless proven otherwise. Do not overwrite, clean, stage, or reformat them.

  3. Set only the selected task to in-progress, regenerate the queue, and commit that transition when the workflow calls for publication.

  4. Keep scratch scripts, provider responses, logs, and exploratory artifacts inside the task folder's work/ directory. It is gitignored. Do not use the repository root as a scratchpad.

  5. Implement the smallest coherent task. Respect the task's declared write scope and adapter boundaries.

  6. Run gates from cheapest to most expensive. A fix after any failure restarts the sequence:

    pnpm format:check
    pnpm lint
    pnpm typecheck
    pnpm test
    pnpm build

    The current prototype predates some gates. A command applies once its defining task lands it. Before then, pnpm test plus task-local commands are the baseline; a missing gate is never reported as a pass.

  7. For stream/server work, record a cold-start final run, source offsets, canonical digest(s), replay/rebuild result, and sensitivity proof in evidence/.

  8. For browser-impacting work, the final run must:

    • drive the real UI with pointer/keyboard events;
    • assert zero console errors and no unhandled request failures;
    • expose and compare relevant stream offsets/digests in the DOM;
    • use Replay Chromium once to produce both the uploaded Replay recording and the same-session MP4;
    • exercise error, cancellation, or removal paths changed by the diff.
  9. Append a builder entry to ## Verification log: exact commit, commands, evidence paths, offsets/digests, recording URL and MP4 path when applicable, plus the claim. Server-only claims state Replay: N/A (<reason>) + mitigation explicitly.

  10. Set status: implemented, regenerate the queue, and hand the exact diff and evidence to a fresh critic. Evidence boundaries in this repository

pnpm test is the routine local baseline. pnpm record:replay performs external uploads, writes recording metadata/media, and may clear test artifact directories. Run it only when the selected task requires browser evidence and external recording is authorized; it is not a harmless substitute for the normal test command.

Durable evidence belongs in evidence/ and is committed: event fixtures, digest files, redacted conformance output, and promoted traces. Raw secrets, provider tokens, session cookies, unredacted HTTP captures, and customer content are forbidden.

Every browser claim cites the uploaded Replay URL as durable proof and names the local MP4 for quick visual review. If Replay upload fails, report the failure; never invent a URL or silently present the MP4 as equivalent.

Critic protocol

  1. Orient. Read the task, its dependency contracts, the exact diff, and the builder's evidence manifest before running anything.
  2. Predict first. For every acceptance criterion, write a falsifiable prediction and the narrow observation that would refute it.
  3. Reproduce. Replay cited stream dumps, compare claimed digests, and verify the evidence came from the claimed commit and cold configuration.
  4. Attack independently. Run every adversarial item using new IDs, timing, seeds, and canaries. Add at least one plausible attack not listed by the builder.
  5. Audit coverage. Classify each changed behavior as executed, explicitly waived (types/config/logging with reason), dead, or requiring new evidence.
  6. Prove sensitivity. In a disposable worktree, introduce a targeted defect and show the claimed task verifier fails. A detector that cannot go red is refuted even when the happy path is green.
  7. Interrogate browser evidence. Use the uploaded Replay recording for browser claims: console, network, exceptions, interaction timeline, source execution, and stream correlation. Do not replace that recording with a fresh unrelated rerun.
  8. Issue a verdict. First line is VERDICT: verified | refuted | needs-evidence. Every finding cites a diff hunk, command result, stream offset/digest, or Replay point. Append it to the Verification log, update status, regenerate the queue, and commit.

Queue, branches, and concurrent work

  • One queue gate is active. Parallel agents may investigate or criticize independently, but may not implement later dependency tasks.
  • One ticket owns one focused branch and one task folder. If scope grows beyond one coherent session/day, split the ticket rather than weakening acceptance criteria.
  • Hardcoded prototype ports collide. Ticket work that starts services must allocate task-local ports and record them in work/.
  • Never write task artifacts into another ticket's work/ or evidence/.

Mock data

All mock data must be provided through the blamy/emulate project. Do NOT add any mock data that could later show up in prod. We should always just be able to drop in our prod keys and have the project run. Emulate should always emulate production as closely as possible. If the user adds a 3rd party service you should update the vendor emulate so that that service is properly emulated and that 3rd party data comes from our emulate config file when testing and running locally.

Stacked PRs and merge authority

Please use the $GH_STACKS skill from GitHub to use the new stacked PRs feature on GitHub.

Creating a stack with GitHub CLI

  1. Initialize a stack. This creates and checks out the first branch on top of your trunk branch.
gh stack init auth-layer
  2. Write code for the first layer, then stage and commit your changes.
git add .
  3. git commit -m "helpful-commit-message"
  4. Add a branch for the next logical unit of work. The new branch is created on top of the current one.
gh stack add BRANCH-NAME
  5. Write code and commit on the new branch. Repeat for each additional layer.
  6. Push all branches to the remote repository and create the stacked pull requests on GitHub.
gh stack submit
  7. Each pull request is created with the correct base branch, so reviewers see only the diff for that layer, and the pull requests are automatically linked together as a stack.

Definition of done

A ticket is done only when:

  • its deliverables and every acceptance criterion are implemented;
  • required gates pass from a cold, documented state;
  • evidence is redacted, committed where appropriate, and tied to the exact diff;
  • a fresh critic has failed to refute correctness, evidence sufficiency, and detector sensitivity;
  • status is verified and the regenerated queue is committed;
  • any publication requested by the workflow is complete, without inferring merge authority.
```

Also create a file called .replay/tools/build_queue.py
```
#!/usr/bin/env python3 """Regenerate .eforest/tasks/QUEUE.md from task readme frontmatter.

Stdlib only. The accepted flat-YAML subset is documented in .eforest/tasks/README.md. """

import re import sys from pathlib import Path

ROOT = Path(file).resolve().parent.parent TASKS = ROOT / ".eforest" / "tasks" QUEUE = TASKS / "QUEUE.md"

STATUS_ICON = { "pending": " ", "in-progress": "", "in_progress": "", "implemented": "?", "refuted": "!", "verified": "x", "cancelled": "-", }

def parse_frontmatter(path: Path) -> dict | None: text = path.read_text(encoding="utf-8") match = re.match(r"\A---\n(.*?)\n---\n", text, re.DOTALL) if not match: return None

frontmatter: dict = {"_path": path}
for line in match.group(1).splitlines():
    if ":" not in line:
        continue
    key, _, raw_value = line.partition(":")
    key = key.strip()
    value = raw_value.split("#", 1)[0].strip()
    if key == "depends_on":
        frontmatter[key] = [
            dependency.strip()
            for dependency in value.strip("[]").split(",")
            if dependency.strip()
        ]
    elif key == "priority":
        frontmatter[key] = float(value) if "." in value else int(value)
    elif key == "epic":
        frontmatter[key] = int(value) if value.isdigit() else float(value)
    elif key == "capstone":
        frontmatter[key] = value.lower() == "true"
    else:
        frontmatter[key] = value.strip('"')
return frontmatter

def main() -> int: tasks: list[dict] = [] for path in sorted(TASKS.glob("epic-/E-T*/readme.md")): task = parse_frontmatter(path) if task is None or "id" not in task: print(f"error: invalid or missing frontmatter: {path}", file=sys.stderr) return 1

    folder_id = re.match(r"(E[0-9.]+-T[0-9]+[ab]?)", path.parent.name)
    if not folder_id or folder_id.group(1) != task["id"]:
        print(
            f"error: folder {path.parent.name!r} disagrees with id {task['id']!r}",
            file=sys.stderr,
        )
        return 1
    tasks.append(task)

if not tasks:
    print("error: no task readmes found", file=sys.stderr)
    return 1

tasks.sort(key=lambda task: task.get("priority", 999999))
task_ids = [task["id"] for task in tasks]
if len(task_ids) != len(set(task_ids)):
    print("error: duplicate task id", file=sys.stderr)
    return 1

known_refs = set(task_ids)
capstones: dict[str, dict] = {}
for task in tasks:
    if task.get("capstone"):
        epic_ref = f"E{task['epic']}"
        if epic_ref in capstones:
            print(f"error: multiple capstones for {epic_ref}", file=sys.stderr)
            return 1
        capstones[epic_ref] = task
known_refs.update(capstones)

for task in tasks:
    for dependency in task.get("depends_on", []):
        if dependency not in known_refs:
            print(
                f"error: {task['id']} depends on unknown reference {dependency}",
                file=sys.stderr,
            )
            return 1

verified = {task["id"] for task in tasks if task.get("status") == "verified"}
verified_epics = {
    epic_ref
    for epic_ref, task in capstones.items()
    if task.get("status") == "verified"
}
satisfied = verified | verified_epics

def eligible(task: dict) -> bool:
    return task.get("status") in ("pending", "refuted") and all(
        dependency in satisfied for dependency in task.get("depends_on", [])
    )

active = [
    task
    for task in tasks
    if task.get("status")
    in ("in-progress", "in_progress", "implemented", "refuted")
]
if len(active) > 1:
    print(
        "error: multiple active tasks violate the one-gate rule: "
        + ", ".join(task["id"] for task in active),
        file=sys.stderr,
    )
    return 1

for task in tasks:
    status = task.get("status", "pending")
    if status not in STATUS_ICON:
        print(f"error: {task['id']} has unknown status {status!r}", file=sys.stderr)
        return 1

current_gate = active[0] if active else None
next_up = [
    task
    for task in tasks
    if eligible(task) and task.get("status") == "pending"
][:10]

unlocks: list[dict] = []
if current_gate is not None:
    hypothetical = satisfied | {current_gate["id"]}
    if current_gate.get("capstone"):
        hypothetical.add(f"E{current_gate['epic']}")
    unlocks = [
        task
        for task in tasks
        if task.get("status") == "pending"
        and all(dep in hypothetical for dep in task.get("depends_on", []))
        and task not in next_up
    ][:10]

lines = [
    "# Stream Slack Priority Queue",
    "",
    "*Generated by `tools/build_queue.py` — do not edit by hand.*",
    "",
    f"**{len(verified)} / {len(tasks)} tasks verified.**",
    "",
    "Legend: `[ ]` pending · `[~]` in-progress · `[?]` implemented "
    "(awaiting adversarial verification) · `[!]` refuted · `[x]` verified · "
    "`[-]` cancelled",
    "",
    "## Current gate",
    "",
]

if current_gate is None:
    lines.append("No task is currently in progress, awaiting verification, or refuted.")
else:
    action = {
        "in-progress": "builder working",
        "in_progress": "builder working",
        "implemented": "awaiting independent critic",
        "refuted": "builder rework required",
    }[current_gate["status"]]
    lines.append(
        f"1. **{current_gate['id']}** — {current_gate.get('title', '?')} "
        f"*({action})*"
    )

lines.extend(["", "## Next up (dependencies satisfied)", ""])
if next_up:
    for task in next_up:
        lines.append(f"1. **{task['id']}** — {task.get('title', '?')}")
elif current_gate is not None:
    lines.append(
        f"No new task may start until **{current_gate['id']}** clears the gate."
    )
else:
    lines.append("No pending task currently has all dependencies verified.")

if current_gate is not None:
    lines.extend(["", f"## Unlocks when {current_gate['id']} verifies", ""])
    if unlocks:
        for task in unlocks:
            lines.append(f"1. **{task['id']}** — {task.get('title', '?')}")
    else:
        lines.append("No task unlocks directly.")

current_epic = None
for task in tasks:
    if task.get("epic") != current_epic:
        current_epic = task.get("epic")
        epic_dir = task["_path"].parent.parent.name
        lines.extend(["", f"## Epic {current_epic} — `{epic_dir}`", ""])

    icon = STATUS_ICON[task.get("status", "pending")]
    relative_path = task["_path"].relative_to(TASKS).as_posix()
    dependencies = ", ".join(task.get("depends_on", [])) or "—"
    capstone = " **[CAPSTONE]**" if task.get("capstone") else ""
    lines.append(
        f"- [{icon}] `{task.get('priority', '?'):>4}` "
        f"[{task['id']}]({relative_path}) — {task.get('title', '?')}"
        f"{capstone} *(deps: {dependencies})*"
    )

QUEUE.write_text("\n".join(lines) + "\n", encoding="utf-8")
print(
    f"wrote {QUEUE.relative_to(ROOT)}: {len(tasks)} tasks, "
    f"{len(verified)} verified, {len(capstones)} epics"
)
return 0

if name == "main": sys.exit(main())

</backgroundAgent>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment