Skip to content

Instantly share code, notes, and snippets.

@hi-ogawa

hi-ogawa/merge-reports-cross-platform.md

Last active March 23, 2026 10:19
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save hi-ogawa/0d11864e5dbf54d152135bf8ce61ea06 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save hi-ogawa/0d11864e5dbf54d152135bf8ce61ea06 to your computer and use it in GitHub Desktop.
Download ZIP
feat: support merging reports for non-sharded multi-environment runs
Raw
merge-reports-cross-platform.md

Cross-Platform Merge Reports

Issue Title

feat: support merging reports for non-sharded multi-environment runs

Feature Request

Clear and concise description of the problem

As a developer using Vitest in CI with a matrix strategy (e.g., linux/macos/windows, or node 20/22), I want to merge test reports from multiple platform runs into a single report, so that I can get a unified view of test results across all environments.

Currently --merge-reports only supports merging sharded runs — where test files are split across shards with no overlap. When the same tests are run on multiple platforms (a common CI pattern), merging those blob reports produces duplicate unlabeled entries with no way to tell which result came from which platform.

Playwright supports this via their merge-reports + tag config, and it's a frequently needed workflow.

Suggested solution

Use explicit tags to distinguish blobs from different environments. The blob reporter would accept a tag option that gets stored in the blob metadata and injected into each test's tags array during merge. This builds on Vitest's existing tag infrastructure (filtering, reporter display).

Blob reporter config:

// vitest.config.ts
export default defineConfig({
  test: {
    reporter: [['blob', {
      tag: process.env.CI_PLATFORM, // e.g., "linux", "windows", "node-22"
    }]],
  },
})

Merge behavior:

  • Each blob carries its tag in metadata
  • On merge, the tag is prepended to every test's tags array (e.g., ["linux", ...])
  • Same test file from different blobs becomes distinct entries, each labeled by platform
  • Reporters already understand tags, so results are immediately distinguishable

Example CI workflow:

jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
    steps:
      - run: vitest run --reporter=blob
        env:
          CI_PLATFORM: ${{ matrix.os }}
      - uses: actions/upload-artifact@v4
        with:
          name: blob-${{ matrix.os }}
          path: .vitest-reports/

  merge:
    needs: test
    steps:
      - uses: actions/download-artifact@v4
        with:
          path: .vitest-reports/
          merge-multiple: true
      - run: vitest --merge-reports --reporter=default --reporter=json

Alternative

  • Auto-detect platform (process.platform, process.arch) and inject as tags without explicit config. Simpler for users but less flexible — doesn't cover non-OS dimensions like Node version or custom environments.
  • Playwright's approach: automatic ID deconfliction (salt duplicate IDs) + separate explicit tag config for labeling. We could start with the simpler combined approach (tag = deconflict + label) and add automatic deconfliction later if needed.

Implementation Notes

Playwright's Approach (Reference)

Playwright has mature support for this via their merge-reports CLI + blob reporter.

Architecture

  • Blob format: ZIP files containing report.jsonl (line-delimited JSON events) + resources/ dir for attachments
  • Metadata per blob: version, userAgent, name (environment label), shard info, pathSeparator
  • CLI: npx playwright merge-reports [dir] with --reporter and --config options

How Deconfliction Actually Works

Two separate concerns:

1. ID Deconfliction (automatic, "dumb")

  • IdsPatcher (merge.ts#L406-483) maintains a global set of test IDs
  • Any duplicate ID across blobs gets salted with the blob's index → becomes a unique entry
  • This is purely mechanical — it doesn't know why duplicates exist
  • For sharded runs, duplicates don't normally occur (files are split, not duplicated), so salting is a no-op
  • For cross-platform runs, same tests run everywhere → all duplicates get salted → separate entries

2. Labeling (requires explicit user config)

  • Without config, duplicated tests appear as separate entries but are unlabeled — you can't tell which came from which platform
  • Users must set TestConfig.tag (e.g., tag: '@linux') to distinguish environments
  • Legacy: PWTEST_BOT_NAME env var sets the blob's name field (being deprecated, blob.ts line 55: "TODO: remove after some time, recommend config.tag instead")
  • Tags are prepended to test tags on merge, e.g., @linux, @windows
  • GlobalErrorPatcher also tags unhandled errors with the environment name

From Playwright docs on merging across environments:

If you want to run the same tests in multiple environments... you need to differentiate these environments. Use TestConfig.tag property to tag all tests with the environment name.

Other Cross-Platform Mechanisms

  1. Path Separator Patching: PathSeparatorPatcher converts \ ↔ / based on blob's pathSeparator metadata vs current platform

  2. Root Directory Validation: All blobs must share the same rootDir, or user provides -c config with explicit testDir. Clear error message when cross-OS rootDirs differ.

Playwright's Merge Flow

blob-linux.zip  blob-win.zip  blob-mac.zip
       |              |              |
       v              v              v
  extract JSONL, parse events
       |              |              |
  IdsPatcher: salt duplicate test IDs (automatic)
  PathSeparatorPatcher: normalize paths
  AttachmentPathPatcher: fix resource refs
  GlobalErrorPatcher: tag errors with env name
  WorkerIndexPatcher: adjust worker indices
       |
       v
  merged event stream → reporters (html, json, etc.)

Test ID Generation

Test IDs are SHA1 of [project=${projectId}]${filePath}\x1e${titleHierarchy} (suiteUtils.ts#L59-61). Same test on different platforms gets the same base ID → that's why salting is needed for cross-platform merges.

Vitest's Current Implementation

Blob Format

Tuple-based JSON (not JSONL):

type MergeReport = [
  vitestVersion: string,
  files: File[],
  errors: unknown[],
  coverage: unknown,
  executionTime: number,
  environmentModules: MergeReportEnvironmentModules,
]

Key files:

How Merge Works Today

  1. Read all blob JSON files from .vitest-reports/
  2. Validate ALL blobs have identical Vitest version (and match current)
  3. Restore module graphs per project/environment
  4. Flatten all File[] arrays, sort by startTime
  5. Report merged files to reporters
  6. Merge errors + delegate coverage merge to provider

Current Limitations for Cross-Platform Use

Limitation Detail
No duplicate handling If same test file appears in multiple blobs, it shows up multiple times with no distinction
No environment metadata Blobs don't store platform/environment info
No path normalization No \ ↔ / conversion for cross-OS
Strict version match All blobs must be exact same Vitest version
Naming assumes sharding Files named blob-{shardIndex}-{shardCount}.json
No rootDir validation Silently merges blobs from different project roots
File-level dedup only No mechanism to distinguish same file run on different platforms

Design Considerations

Core Question: How to Present Cross-Platform Results?

Option A: Playwright-style (separate entries, tagged)

  • Same test from different platforms → separate test entries with platform tag
  • Pro: Simple, each result is independent, familiar to Playwright users
  • Con: Test count inflates (3 platforms x 100 tests = 300 entries)

Option B: Grouped results (single entry, multiple results)

  • Same test → single entry with results array per platform
  • Pro: Natural "matrix" view, clear pass/fail per platform
  • Con: More complex data model, reporter changes needed

Option C: Hybrid (configurable)

  • Default to separate entries (simpler), offer grouped view in HTML reporter
  • Pro: Flexibility
  • Con: Complexity

Proposed Changes

1. Extend Blob Metadata

Add environment/platform metadata to the blob format:

type MergeReport = [
  vitestVersion: string,
  files: File[],
  errors: unknown[],
  coverage: unknown,
  executionTime: number,
  environmentModules: MergeReportEnvironmentModules,
  metadata?: BlobMetadata,  // NEW - optional for backward compat
]

interface BlobMetadata {
  name?: string           // user-defined label, e.g. "linux", "node-22"
  platform?: string       // auto-detected: process.platform
  arch?: string           // auto-detected: process.arch
  nodeVersion?: string    // auto-detected: process.version
  pathSeparator?: string  // auto-detected: path.sep
  shard?: { index: number; count: number }
}

2. Configuration

// vitest.config.ts
export default defineConfig({
  test: {
    reporter: [['blob', {
      name: process.env.CI_PLATFORM || 'default',  // environment label
    }]],
  },
})

For merge:

// or CLI: vitest --merge-reports --merge-reports.testDir=/path
export default defineConfig({
  test: {
    mergeReports: {
      path: '.vitest-reports',
      testDir: '/normalized/path',  // override for cross-OS rootDir mismatch
    },
  },
})

3. Merge Logic Changes

  • Duplicate file handling: When same test file appears in multiple blobs:
    • Automatic: just works (files appear multiple times, each tagged)
    • Tag each File and its tasks with the blob's environment name via task.meta
    • Optionally prefix project name: "my-project [linux]", "my-project [windows]"
  • Path normalization: Convert path separators based on blob metadata
  • Relaxed version check: Consider allowing patch version differences, or make it a warning

4. Reporter Enhancements

  • Default/list reporter: Show platform tag next to test name for cross-platform merges
  • JSON reporter: Include platform metadata in output
  • HTML reporter (vitest-ui): Platform filter/grouping in the UI
  • JUnit reporter: Use <testsuite> attributes or <properties> for platform info

Blob File Naming

Current: blob-{shardIndex}-{shardCount}.json — assumes sharding.

For cross-platform, need a naming scheme that doesn't collide:

  • Option: blob-{name}.json when name is set (e.g., blob-linux.json)
  • Option: blob-{name}-{shardIndex}-{shardCount}.json for sharded cross-platform
  • Or just let users set outputFile explicitly per platform

Example CI Workflow

jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
    steps:
      - run: vitest run --reporter=blob
        env:
          VITEST_BLOB_NAME: ${{ matrix.os }}
      - uses: actions/upload-artifact@v4
        with:
          name: blob-${{ matrix.os }}
          path: .vitest-reports/

  merge:
    needs: test
    steps:
      - uses: actions/download-artifact@v4
        with:
          path: .vitest-reports/
          merge-multiple: true
      - run: vitest --merge-reports --reporter=default --reporter=json

Implementation Priority

  1. Phase 1: Blob metadata (platform info + name) — backward compatible
  2. Phase 2: Path normalization in merge — needed for Windows blobs
  3. Phase 3: Duplicate test handling with platform tagging (like Playwright: automatic deconflict + explicit label config)
  4. Phase 4: Reporter enhancements (especially HTML/UI grouping)

Open Questions

  • Should the blob format change from JSON tuple to something more extensible (object with named fields)?
    • Tuple is fragile for adding fields; an object { version, files, errors, ... } would be cleaner
    • Could support both formats during transition
  • How to handle coverage merge across platforms? (same file, different coverage data)
    • Istanbul/v8 providers would need to merge coverage from identical source files
  • Should --merge-reports accept a glob pattern instead of just a directory?
    • Would make it easier to merge artifacts from different CI jobs
  • How strict should version matching be? Playwright doesn't seem to enforce version match at all
  • Should we support merging blobs from different Vitest projects (different configs)?
  • Blob file naming: how to handle cross-platform + sharded combos?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment