ACCEPTANCE_TESTING_FRAMEWORK_SPEC.md

gistfile1.txt

# Markdown-Driven Integration Testing with Claude + Playwright MCP

## 1. Goals and Objectives

### Primary Goal
Create a **markdown-driven integration test framework** where test scenarios are authored as human-readable markdown files and executed by an AI agent using the Playwright MCP (Model Context Protocol) browser automation tools.

### Objectives
1. **Human-readable test definitions** -- Tests are written in plain English as structured markdown, not code. Anyone on the team can author, review, and understand tests without programming knowledge.
2. **AI-powered test execution** -- Claude reads each test markdown file, interprets the steps, drives the browser via Playwright MCP, and determines pass/fail outcomes by observing actual page state.
3. **No traditional test runner** -- There is no `jest`, `mocha`, or `playwright test` runner. Claude *is* the test runner. This eliminates boilerplate and lets tests focus on intent.
4. **Visual and semantic validation** -- Claude can use snapshots (accessibility trees), screenshots, console messages, and network requests to validate outcomes -- going beyond simple DOM assertions.
5. **Incremental coverage** -- Start with critical user journeys and expand over time.

---

## 2. How It Works

### Architecture Overview

```
tests/
├── spec/
│   └── FRAMEWORK_SPEC.md        ← Framework design and conventions
├── integration/
│   ├── 01-public-pages.md       ← Test scenario files (numbered for ordering)
│   ├── 02-auth-flow.md
│   ├── 03-dashboard.md
│   └── ...
├── fixtures/
│   ├── sample-upload.pdf         ← Upload fixtures, sample data
│   └── test-credentials.env     ← Test account credentials (gitignored)
└── results/
    └── YYYY-MM-DD-HH-MM/        ← Timestamped run results (gitignored)
        ├── summary.md
        └── screenshots/
```

### Execution Flow

1. **User prompts Claude** with a request like: *"Run the integration test in `tests/integration/01-public-pages.md`"*
2. **Claude reads the markdown file** to understand the test scenario, preconditions, steps, and expected outcomes.
3. **Claude launches a browser** via Playwright MCP (`browser_navigate`) and executes each step:
   - Navigates to URLs
   - Takes snapshots (`browser_snapshot`) to read page structure
   - Clicks elements, fills forms, presses keys
   - Waits for content to appear or disappear
   - Takes screenshots as evidence
4. **Claude evaluates outcomes** by comparing observed page state against the expected outcomes defined in the markdown.
5. **Claude reports results** -- a pass/fail summary with evidence (screenshots, observed values, error messages).

### Why This Approach

| Traditional E2E Tests | Markdown + Claude + Playwright MCP |
|---|---|
| Brittle CSS selectors | Claude uses accessibility snapshots -- resilient to UI changes |
| Complex setup (Node, config, CI) | Zero setup -- just markdown files and Claude |
| Binary pass/fail | Nuanced judgment -- Claude can flag "technically works but looks broken" |
| Hard to write for non-devs | Anyone can write a test in English |
| Debugging = reading stack traces | Debugging = reading Claude's narration + screenshots |

---

## 3. Test File Format

Each integration test is a markdown file in `tests/integration/` following this structure:

```markdown
# Test: [Descriptive Name]

## Metadata
- **Priority**: critical | high | medium | low
- **Area**: auth | dashboard | settings | public | api
- **Requires Auth**: yes | no
- **Estimated Duration**: fast (<30s) | medium (<2min) | slow (>2min)

## Preconditions
Describe what must be true before this test starts.
- Application is running at `http://localhost:3000`
- User is logged out (if testing auth flow)
- Required test data exists (if testing CRUD operations)

## Steps

### Step 1: [Action description]
**Action**: Navigate to the home page at `/`
**Expected**: The page loads with the application branding visible. A navigation bar is present with links to key sections and a Sign In button.

### Step 2: [Action description]
**Action**: Click the "Sign In" button in the navigation bar.
**Expected**: The browser navigates to `/login`. A login form is displayed with email and password fields.

### Step 3: ...
(Continue with numbered steps)

## Success Criteria
Summarize what constitutes a full pass for this test scenario.
- All pages loaded without errors
- Navigation between pages worked correctly
- No console errors were logged

## Notes
Any additional context, known issues, or things Claude should watch for.
```

### Format Conventions

- **Steps are numbered and sequential.** Claude executes them in order.
- **Each step has an Action and an Expected outcome.** The Action tells Claude what to do. The Expected tells Claude what to verify.
- **Steps should be atomic.** One action per step. If a step does two things, split it.
- **Use semantic descriptions, not selectors.** Say *"Click the Sign In button"* not *"Click `#btn-signin`"*. Claude will find elements via accessibility snapshots.
- **Be explicit about what "success" looks like.** Don't say *"the page loads correctly"* -- say *"the page displays a heading that reads 'Dashboard' and a list of recent items"*.

---

## 4. Execution Instructions for Claude

When asked to run a test, Claude should follow this protocol:

### Before Running
1. Read the test markdown file completely before starting.
2. Check preconditions. If a precondition is not met (e.g., app not running), report it and stop.
3. Announce which test is being run.

### During Execution
1. For each step:
   a. State which step is being executed.
   b. Perform the **Action** using Playwright MCP tools.
   c. Use `browser_snapshot` to capture the page's accessibility tree after each action.
   d. Compare observed state against the **Expected** outcome.
   e. Record **PASS** or **FAIL** for the step, with evidence.
   f. If a step fails, continue to the next step unless the failure is blocking (e.g., navigation failed).
2. Take a screenshot (`browser_take_screenshot`) at key moments:
   - After page navigations
   - On step failures
   - At the end of the test
3. Check `browser_console_messages` for errors after critical steps.

### After Running
1. Summarize results: total steps, passed, failed, skipped.
2. For any failures, provide:
   - What was expected
   - What was observed
   - Screenshot evidence
   - Suggested cause (if identifiable)
3. Save results to `tests/results/` with a timestamped folder.

---

## 5. Prioritizing Test Coverage

### Critical (must always pass)
- Public pages load correctly
- Auth flow (sign up, sign in, sign out, password reset)
- Core CRUD operations (create, view, edit, delete primary resources)

### High
- Key user workflows end-to-end
- Role-based access control
- Account/profile settings

### Medium
- Admin panel functionality
- Secondary features and pages
- Search and filtering

### Low
- Edge cases (404 page, invalid URLs, deep link redirects)
- Legacy route redirects

---

## 6. Environment Configuration

### Target URLs
- **Local development**: `http://localhost:3000` (adjust port to your setup)
- **Staging/Production**: Configure in `tests/fixtures/test-credentials.env`

### Test Credentials
Store in `tests/fixtures/test-credentials.env` (gitignored):
```
TEST_USER_EMAIL=test@example.com
TEST_USER_PASSWORD=...
TEST_ADMIN_EMAIL=admin@example.com
TEST_ADMIN_PASSWORD=...
```

---

## 7. Guidelines for Writing New Tests

1. **One scenario per file.** Each markdown file tests one cohesive user journey.
2. **Name files with a numeric prefix** for ordering: `01-`, `02-`, etc.
3. **Keep steps focused.** 5-15 steps per test is ideal. If a test exceeds 20 steps, consider splitting it.
4. **Describe what a human would see**, not implementation details. Claude interprets the page like a user would.
5. **Include negative cases** where appropriate (wrong password, missing fields, unauthorized access).
6. **Reference fixtures explicitly** when uploads or test data are needed.
7. **Tag metadata correctly** so tests can be filtered by area or priority.

---

## 8. Limitations and Considerations

- **No parallel execution.** Claude runs one test at a time in a single browser context.
- **State carries over.** If test A logs in, test B may start logged in. Each test should declare and establish its own preconditions.
- **Timing sensitivity.** Use `browser_wait_for` for async operations rather than assuming instant responses.
- **External dependencies.** Tests that rely on third-party APIs or services may need mocking or dedicated test environments.
- **Screenshots are point-in-time.** They capture what the page looked like but won't catch transient flickers or animations.

---

## 9. Prerequisites

To use this framework you need:

1. **[Claude Code](https://docs.anthropic.com/en/docs/claude-code)** -- Anthropic's CLI agent for Claude.
2. **[Playwright MCP server](https://github.com/anthropics/playwright-mcp)** -- Gives Claude browser automation capabilities. Add it to your `.mcp.json`:
   ```json
   {
     "mcpServers": {
       "playwright": {
         "command": "npx",
         "args": ["@anthropic-ai/playwright-mcp@latest"]
       }
     }
   }
   ```
3. **Your application running locally** (or a deployed URL to test against).

That's it. No test dependencies to install, no configuration files to maintain, no CI pipeline to set up. Write a markdown file, ask Claude to run it.