You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Status: Draft
Date: 2026-03-29
See also: ai-agent-integration-02-landscape.md (See next document)
Background
AI coding agents (Claude Code, Cursor, Copilot, etc.) are becoming primary consumers of test
runners. The dev ecosystem is converging on patterns for making tools agent-friendly.
Vitest already has a foot in the door with AgentReporter (PR #9779, Mar 2026).
This proposal outlines three independent pillars, each actionable on its own.
What's Already Shipped
--reporter=agent (PR #9779) — only prints failing tests, token-efficient
llms-full.txt — full docs as LLM feed
CLI — agents can already run, filter, watch
isAgent detection via std-env — used to auto-switch defaults when an agent is detected:
watch defaults to false (run mode instead of watch mode)
Rationale: Agents can already run vitest via CLI, but MCP is self-documenting — agents
don't need to guess flags, parse unstructured output, or know where artifacts are written.
The key value-add is structured, queryable access to test results and coverage data without
re-running.
Playwright example: Ships test_list, test_run, test_debug as MCP tools inside the
playwright package. test_debug is the key one — runs a single test with pauseOnError=true
so the agent can attach browser tools to the live paused state. The healer agent is built on
top: run → debug → inspect → edit → repeat. MCP is what makes that loop tight.
TBD: Which operations actually need MCP vs. CLI being sufficient. What the right home is
(vitest-community/mcp vs. main package). What coverage output format agents actually need.
Pillar 2: Docs → Skills
Rationale:llms-full.txt is everything — too broad. A skill is a curated, prioritized
view of what agents actually need to be effective. antfu's community skill already has 10K
installs, proving demand. An official skill can stay accurate across releases and cover
Vitest-specific knowledge that community skills miss (artifact locations, raw data formats,
new features like --reporter=agent).
Two sub-actions with different natures:
2a. New agent-specific docs — humans use UI (HTML coverage report, --ui); agents need
the raw data path and format explained. These docs don't exist yet because there was no human
need for them. Working through this also reveals whether a doc is sufficient or whether the
raw format is too unwieldy — in which case the real answer is a dedicated feature (Pillar 3).
2a is the discovery mechanism for Pillar 3.
2b. Curation of existing docs — surface existing docs that enable high-value agent workflows.
The curation lens isn't just "what's useful to read" but "what tasks are well-suited for agents
to execute autonomously, and do the docs provide a good enough spec for that?"
Jest → Vitest migration is the clearest example: mechanical, well-defined, existing migration
doc is essentially a step-by-step spec an agent can follow end-to-end with high reliability.
Highlighting it in a skill makes the workflow discoverable. Other candidates likely exist —
needs investigation.
Playwright example: Ships SKILL.md + references/ inside the npm package itself, installed
via playwright-cli install --skills. The skill is lean; topic docs are loaded on demand.
The playwright-tests.md reference is the clearest parallel — it describes how to debug a
failing test via CLI (PWPAUSE=cli), something that doesn't exist in any human-facing doc
because humans just use the interactive debugger. The skill teaches the agent the agent-native
path. Separately, playwright trace CLI + its own SKILL.md replace the human-facing
show-trace browser UI — same data, different consumer.
TBD: Which existing docs are actually high-value for agents vs. assumed. What raw formats
(coverage.json, etc.) look like and whether docs suffice. Skill structure and install mechanism.
Pillar 3: New Dedicated Workflows
Rationale: Some agent needs have no existing human-facing equivalent and can't be solved
with docs alone. AgentReporter is the first example — it's not a doc, it's a new output
mode that only makes sense for agents. The pattern from Playwright: every human surface
(HTML reporter, trace viewer, interactive debug) gets a parallel agent surface that is
token-efficient and scriptable.
Pillar 3 candidates emerge from two sources: community requests (issue #8411 asked for compact
coverage output), and the doc-writing process in 2a revealing that the underlying data is too
raw to document usefully.
Playwright example:AgentReporter equivalent already exists in Vitest (PR #9779, same
pattern). Beyond that, Playwright built playwright trace CLI — humans use show-trace in a
browser, agents use playwright trace actions/errors/snapshot in the terminal. The same data,
a new surface. They also built PWPAUSE=cli: when a test fails, instead of opening an
interactive debugger for humans, it pauses and exposes a CLI session the agent can attach to.
Each of these started from the observation that the human tool wasn't usable by an agent.
For Vitest the equivalent question: humans open the HTML coverage report — what's the agent
surface for the same data? Humans run --ui to browse test results — what's the agent surface?
TBD: Concrete candidates to be determined as 2a progresses. Compact coverage output is the
strongest known candidate. Others need investigation before committing.
Appendix: Eval System
Not a blocker — ship the pillars regardless. But worth keeping in mind as the work matures.
Without reproducible evals, improvements are unmeasurable. The ecosystem is converging on this:
Storybook ships an eval harness in storybookjs/mcp/eval/ (actively developed, separate from
the storybook ai init tracking issue) — fresh project per trial, real agent CLI, graded on
build/test/coverage/cost/turns, variant comparison. Vercel runs nextjs.org/evals as a public
leaderboard. Vercel's key finding: AGENTS.md hit 100% vs 79% for skills — evals are what proved that.
For Vitest the equivalent: tasks like "write tests for this file", "fix this failing test",
"fill coverage to X%" — run against real agents, graded on pass rate, coverage delta, turns,
cost. Variants isolate what each pillar contributes (with skill vs. without, with MCP vs. CLI,
with --reporter=agent vs. default).
TBD: Task set, success metrics, model tiers, CI integration.
Status: Landscape survey in progress
Date: 2026-03-29
Goal
Explore what "AI agent integration" could mean for Vitest.
Understand the landscape first — what others are shipping, what problems they're solving —
before forming opinions on what Vitest should do.
Two packages: @storybook/mcp (standalone doc server) + @storybook/addon-mcp (dev server addon)
MCP tools in addon-mcp:
get-storybook-story-instructions — returns bundled markdown instructions for writing/testing
stories. Agent must call this before touching any story file. Dynamically includes/excludes
test guidance based on whether @storybook/addon-vitest is installed.
run-story-tests — triggers Vitest runs via @storybook/addon-vitest internal channel,
not by calling Vitest CLI directly. Requires Storybook dev server to be running.
Returns pass/fail per story + a11y violations.
preview-stories — render stories in a headless browser, return screenshot/snapshot
is-manifest-available — check if component manifest is available
storybook ai CLI subcommand for outputting agent guidance (tracking issue #34295, in progress)
Full eval harness (storybookjs/mcp/eval/, actively developed separately from the tracking issue):
Spins up a fresh Vite + React + Storybook project per trial
Runs a real agent (Claude Code CLI or GitHub Copilot CLI) against a task prompt
Grades on: build success, typecheck, lint, test pass/fail, a11y, coverage %, token cost, turns
Supports variants — same task, different context (with MCP / without) to isolate what each change contributes
Results tracked in Google Sheets across runs
Svelte — two-pronged: remote MCP server (doc search) + local skills (code writing guidance)
Nuxt/vite-plugin-mcp — lightweight: exposes Vite internals (config, module graph) as MCP tools
skills.sh — marketplace with 90K+ installs; single npx skillsadd command; supports Claude Code,
Cursor, Copilot, Cline, etc.
vitest-community/mcp — stub repo created by sheremet-va, basically empty today
Vercel — deepest investment in eval methodology and the skills vs. AGENTS.md question:
nextjs.org/evals — public leaderboard: agents run against Next.js tasks (router migration,
new APIs), scored on build/lint/test pass rate, with/without AGENTS.md delta shown per model
Key finding: AGENTS.md (compressed 8KB docs index always in context) hit 100% pass rate
vs. 79% for skills and 53% baseline — passive context outperforms active skill retrieval
because it eliminates the activation decision entirely
"Eval-driven development" — treat evals as the feedback loop for AI tooling, same role
unit tests play for code. Catch regressions, measure improvements, iterate with confidence.
What the community actually wants (from issue #8411)
Test results without re-running (read last run output)
Coverage: which lines/branches are uncovered, in a compact format
Self-documenting: agent shouldn't need to guess CLI flags/config options
"Vitest already running" awareness (watch mode / IDE extension integration)
A simpler/token-efficient output format for LLMs
How to Proceed
1. Team alignment first
Share proposal internally before any public move. Align on pillar priority and ownership.
2. A few cheap, high-signal actions in parallel
Don't close #8411 yet — community anchor point, reference it from whatever comes next
Try writing the coverage.json doc (2a) — one person, one day. Either it's documentable
and you ship a doc, or it's too painful and that becomes a Pillar 3 ticket. Fast signal,
no infrastructure needed.
Add 2-3 MCP tools to vitest-community/mcp — test_list + test_run are trivial.
Gets the stub non-empty, lets community react, low risk since it's not the main package yet.
Consider AGENTS.md at repo root — Vercel's finding (100% vs 79% for skills) suggests
always-in-context beats active retrieval. llms-full.txt exists but AGENTS.md serves a
different role: always-in-context for agents working on Vitest itself.
3. Open a community discussion
Once there's something concrete to show, open a GitHub Discussion with the framing.
Easier to discuss around something real. Link #8411 from it.
What to avoid
Don't open a big RFC before shipping anything — the work will clarify direction better than upfront discussion
Don't move MCP into the main vitest package until vitest-community/mcp has validated the shape
Maintainer position (hi-ogawa in issue #8411)
"Dedicated MCP doesn't seem necessary — letting LLM run vitest run CLI seems powerful and flexible enough."
"More robust to feed llms-full.txt to LLM than maintain a drifting MCP server."
Coverage output format might be worth improving for LLMs.
Playwright as Test Runner — Deep Dive
Playwright has been aggressively building AI-native tooling. Three distinct surfaces, all
shipped or actively landing now.
Surface 1: MCP server (test runner side)
Defined in packages/playwright/src/mcp/test/. Designed for persistent agentic loops.
Tool
What it does
test_list
List all tests with IDs
test_run
Run tests by file / location / project, returns full output
test_debug
Run single test with pauseOnError=true, workers=1 — pauses on failure so agent can inspect live browser state
generator_*
Record live browser interactions → generate test code
planner_*
Explore UI, save test plan
Agent definitions bundled in lib/agents/ inside the npm package:
playwright-test-healer — run tests, debug failures, edit code, repeat. The most transferable: uses only test_run + test_debug + edit (+ browser tools for inspection).
playwright-test-planner / playwright-test-generator — browser-exploration-heavy, not relevant to Vitest.
Simple .prompt.md entry points: "Run all my tests and fix the failing ones."
Surface 2: @playwright/cli + bundled skills
A separate package (@playwright/cli, old repo microsoft/playwright-cli, recently repositioned
as the agent-first CLI). Install skills via playwright-cli install --skills.
Skills are bundled inside the npm packages themselves. Full inventory:
Skill 1: playwright-cli (browser automation)
Two near-identical copies shipped in two packages:
playwright/lib/skill/SKILL.md — distributed with test runner package; narrower allowed-tools
Both use progressive disclosure via references/ subdirectory — topic docs loaded on demand:
Reference file
Topic
references/playwright-tests.md
Running + debugging tests via CLI (only in playwright-core version)
references/test-generation.md
Generating test code from interactions
references/tracing.md
Trace recording workflow
references/request-mocking.md
Network mocking
references/running-code.md
run-code for arbitrary Playwright snippets
references/session-management.md
Named sessions, persistent profiles
references/storage-state.md
Cookies, localStorage
references/video-recording.md
Video capture
The playwright-tests.md reference is particularly notable — it describes how to debug a failing
test by running it with PWPAUSE=cli (pauses at failure, attaches playwright-cli to live session):
PLAYWRIGHT_HTML_OPEN=never PWPAUSE=cli npx playwright test# pauses on failure
playwright-cli --session=test-worker-abcdef snapshot # agent inspects live page
This is the CLI equivalent of test_debug from the MCP surface.
Skill 2: playwright-trace
playwright-core/src/tools/trace/SKILL.md — terminal trace inspection without browser UI.
Direct replacement for human-facing playwright show-trace. All commands support --json.
npx playwright trace open trace.zip
npx playwright trace actions --errors-only
npx playwright trace snapshot <id># DOM accessibility tree at that moment
npx playwright trace requests --failed
Skill 3: playwright-dev (internal)
.claude/skills/playwright-dev/SKILL.md — for Playwright contributors, not end users.
Covers how to add APIs, MCP tools, CLI commands. Not shipped in the npm package.
The pattern across all surfaces
Playwright is building parallel consumption modes for the same underlying data:
Data
Human surface
Agent surface
Test run output
terminal reporter
--reporter=agent (token-efficient)
Trace files
show-trace (browser UI)
playwright trace CLI
Browser state
DevTools, headed mode
playwright-cli snapshot
Test debugging
interactive --debug
PWPAUSE=cli + playwright-cli
Skills are the connective tissue — they teach the agent which surface to use and how.
What's transferable to Vitest (test runner angle only)
Playwright pattern
Vitest equivalent
test_list / test_run MCP tools
same — straightforward
playwright-tests.md reference
vitest-tests.md — how to run, filter, debug via CLI
playwright trace CLI
vitest coverage CLI? structured failure output?
PWPAUSE=cli debug bridge
no browser, but richer structured error output from tool
Skills bundled in npm package
vitest package ships its own SKILL.md
Three Independent Pillars
1. CLI → MCP
Expose existing CLI operations as MCP tools. test_list, test_run, coverage, etc.
Infrastructure-heavy. vitest-community/mcp stub exists but empty.
2. Docs → Skills
Wrap existing docs into an installable skill. Mostly a curation/packaging problem —
what to include, how to structure for progressive disclosure.
Vitest already has strong docs (llms-full.txt, vitest.dev) so the skill is a filtered view.
Prerequisite: the underlying docs need to be in good shape first.
3. New dedicated workflows
Net-new things designed from scratch for agent consumption.
AgentReporter (just landed) is the first example.
Others: structured failure output, coverage gap format, debug bridge, etc.
Highest value, most open-ended.
These are independent — any one can be pursued without the others.
Open Questions
What's the actual problem? CLI is already usable by agents. What can't they do today?
Coverage gaps are underserved (no compact uncovered-lines format)
Storybook uses Vitest as its healing verifier → does Vitest need to support that pattern better?
Skills vs MCP? Different surfaces, different users:
Skills = procedural knowledge (how to use Vitest well)
Where does Vitest fit in the chain?
Storybook's agentic workflow: storybook ai init → agent writes stories → Vitest runs tests → healer fixes
Vitest is already a tool in the chain. Do we just make that experience better?
Ownership: official Vitest repo vs. vitest-community/mcp vs. skills.sh listing?
Already in Vitest
AgentReporter — packages/vitest/src/node/reporters/agent.ts
Added Mar 9 2026 by Christoph Nakazawa (Jest creator), PR #9779.
Usage: --reporter=agent
Extends DefaultReporter with:
Only prints failing tests/modules (skips passed output entirely)
No summary footer
Designed to reduce token usage for AI agents
This is Vitest already moving in this direction — token-efficient output is the first solved piece.
Existing Vitest Skills on skills.sh
Already published by the community — not official:
Skill
Repo
Installs
Notes
vitest
antfu/skills
10,326
Best practice guidance, Vitest 3.x, links to GitHub docs
vitest-skill
LambdaTest/agent-skills
?
Core patterns, mocking, in-source testing
vitest
majiayu000/claude-skill-registry
?
Vitest 4.x, config patterns, updated 2026-02-06
These are best-practices / "how to write Vitest tests" skills — static guidance, no MCP tools.
antfu's is the most authoritative (10K installs, linked from skills.sh leaderboard).
Implication: the skills space is already covered informally. An official one would need to be
meaningfully better or authoritative (e.g. maintained alongside releases).
Further Angles to Explore
What does storybookjs/mcprun-story-tests actually do with Vitest? (already using @storybook/addon-vitest)
Are there existing Vitest skills on skills.sh?
What does Playwright's healer agent look like in practice — can it be adapted for unit test healing?
How does storybook ai init structured output work? Could vitest ai / vitest instructions be a thing?
