Visual Regression Testing with Storybook Test Runner

.gitlab-ci.yml

storybook:test-runner8:
  image: mcr.microsoft.com/playwright:v1.39.0-jammy
  stage: build-and-test
  artifacts:
    expire_in: 2 weeks
    when: always
    paths:
      - .storybook-images/__diff_output__/
      - .storybook-images/__received_output__/
  before_script:
    - npm install -g pnpm@$PNPM_VERSION
    - pnpm config set store-dir .pnpm-store
    - pnpm install --frozen-lockfile
  script:
    - STORYBOOK_TEST_RUNNER_CI=true pnpm test-storybook:build-and-run

.storybook main.ts

import fs from 'fs';
import path from 'path';
import { StorybookConfig } from '@storybook/react-vite';

// custom stuff...

const config: StorybookConfig = {
  stories: process.env.GENERATE_IMAGES_LOCALLY
    ? ['../stories/**/components/**/*.stories.@(js|jsx|ts|tsx)']
    : [
        '../stories/**/*.stories.mdx',
        '../stories/**/*.stories.@(js|jsx|ts|tsx)',
      ],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-interactions',
    '@storybook/addon-styling',
    '@storybook/addon-a11y',
    '@storybook/test-runner',
  ],
  viteFinal: async (config) => {
    config.resolve = config.resolve ?? {};
    config.resolve.alias = {
      ...config.resolve.alias,
      stories: path.resolve(__dirname, '../stories'),
      '.storybook': path.resolve(__dirname),
    };

    if (process.env.STORYBOOK_NOT_MINIFIED === 'true') {
      config.build = config.build ?? {};
      config.build.minify = false;
    }

    // custom stuff...

    return config;
  },
  // custom stuff...
};

export default config;

.storybook test-runner.js

const { getStoryContext } = require('@storybook/test-runner');
const { toMatchImageSnapshot } = require('jest-image-snapshot');

module.exports = {
  setup() {
    expect.extend({ toMatchImageSnapshot });
  },
  // https://github.com/storybookjs/test-runner#prepare
  // https://github.com/storybookjs/test-runner/blob/next/src/setup-page.ts#L12
  async prepare({ page, browserContext, testRunnerConfig }) {
    // this line is customized!
    const targetURL = process.env.STORYBOOK_TEST_RUNNER_CI
      ? 'http://127.0.0.1:58414'
      : 'http://host.docker.internal:58414';
    const iframeURL = new URL('iframe.html', targetURL).toString();

    if (testRunnerConfig?.getHttpHeaders) {
      const headers = await testRunnerConfig.getHttpHeaders(iframeURL);
      await browserContext.setExtraHTTPHeaders(headers);
    }

    await page.goto(iframeURL, { waitUntil: 'load' }).catch((err) => {
      if (err.message?.includes('ERR_CONNECTION_REFUSED')) {
        const errorMessage = `Could not access the Storybook instance at ${targetURL}. Are you sure it's running?\n\n${err.message}`;
        throw new Error(errorMessage);
      }

      throw err;
    });
  },
  // context = { id, title, name }
  async postRender(page, context) {
    if (!context.title.includes('Components/')) return;

    // storyContext.parameters gives you access to the parameters defined in the story and more
    // supported API:
    // {
    //   image: {
    //     waitTime?: number; // wait that long before taking the screenshot
    //     skip?: boolean; // do NOT take a screenshot
    //     snapshotOptions?: MatchImageSnapshotOptions; // override our default options (https://github.com/americanexpress/jest-image-snapshot#%EF%B8%8F-api)
    //   }
    // }
    const storyContext = await getStoryContext(page, context);
    const imageParameters = storyContext.parameters?.image || {};

    if (imageParameters.skip) return;

    // Make sure assets (images, fonts) are loaded and ready
    await page.waitForLoadState('domcontentloaded');
    await page.waitForLoadState('load');
    await page.waitForLoadState('networkidle');
    await page.evaluate(() => document.fonts.ready);
    if (imageParameters.waitTime)
      await new Promise((resolve) =>
        setTimeout(resolve, imageParameters.waitTime)
      );

    const image = await page.screenshot({
      animations: 'disabled',
      fullPage: true,
    });
    expect(image).toMatchImageSnapshot({
      customSnapshotsDir: '.storybook-images',
      customSnapshotIdentifier: context.id,
      storeReceivedOnFailure: true, // sadly this currently doesn't work for new images: https://github.com/americanexpress/jest-image-snapshot/issues/331
      ...imageParameters.snapshotOptions,
    });
  },
};

README.md

Intro

After Loki.js served us well for some years we finally converted our Visual Regression Testing** logic to use the Storybook Test Runner.

The benefits:

• it's way faster
• it has better official support
• it does more (component smoke tests, play tests, extensibility for more like a11y tests)
• it's way more stable

But the setup might be non-trivial and there are some rough edges. The biggest downside: while it is way faster it could be even more faster, if we could just use generate images against the Storybook Dev Server. Sadly this was very flaky and error boundaries would always fail (I assume this is because of Reacts unfortunate decision to re-throw errors in Dev Mode). See also this issue.

What I do now in order to generate images locally is a custom "production" build (without minification and optimization and just the stories that I actually want to test visually).

The other tricky part was the Playwright setup. In order to get the same results across machines we want to run Playwright in a Docker container. For various reasons I decided to use the Network API of Playwright. That means I'll only run Playwright in the Docker container, but I keep my running Storybook instance on the host. (At least in the local setup. Within the Gitlab CI we run everything inside Docker.)

As a general note: We have a lot of stories in Storybook, but we only want to take images of stories that are inside a components/ directory.

package.json

Here we have custom "scripts". The idea is the following:

In case we want to update images during local development we run $ pnpm start-playwright-server to start the Playwright server which will be reachable on ws://127.0.0.1:3000. Then we run $ pnpm test-storybook:generate-images which will create a "small" (not optimized, only relevant stories) production build of Storybook and then runs the Storybook Test Runner which will generate new images.

The Gitlab CI will later run $ test-storybook:build-and-run within the Playwright Docker image (so no need for $ pnpm start-playwright-server). This command will be run on all stories.

test-runner-jest.config.js

Here we say to always use ws://127.0.0.1:3000 in order to connect to Playwright. (Exception: if STORYBOOK_TEST_RUNNER_CI is set in the Gitlab CI.)

.storybook/main.ts

Just shows the "small" production build for the image generation.

.storybook/test-runner-jest.config.js

Sadly we needed to add a custom prepare function here. This is only needed, because of TARGET_URL. Usually this tells the Storybook Test Runner where it can find the Storybook instance (e.g. http://127.0.0.1:58414), but we'd actually need a custom PLAYWRIGHT_TARGET_URL in case the Playwright Server needs to reach the Storybook instance outside of the image (e.g. http://host.docker.internal:58414).

Besides that we have our custom postRender. Here we want to take an image for all stories within a Components directory. Besides that we can add custom settings per story. Super nice! I also use the storeReceivedOnFailure: true option, because I like to check and download images right away from the failed job in the Gitlab CI. Sadly this seems to only work for image conflicts, not new images.

.gitlab-ci.yml

Nothing suprising here. Uses the Playwright Docker image, builds Storybook (all stories) and runs the Storybook Test Runner. Generated images and diffs are persisted as artifacts.

package.json

{
  "scripts": {
    "test-storybook:run": "DEBUG_PRINT_LIMIT=0 test-storybook --index-json",
    "test-storybook:build": "cross-env STORYBOOK_NOT_MINIFIED=true storybook build --quiet --output-dir storybook",
    "test-storybook:server": "pnpm live-server --port=58414 storybook --no-browser",
    "test-storybook:build-and-run": "pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run'",
    "test-storybook:generate-images": "cross-env GENERATE_IMAGES_LOCALLY=true pnpm test-storybook:build && start-server-and-test 'pnpm test-storybook:server' http-get://127.0.0.1:58414 'cross-env TARGET_URL=http://127.0.0.1:58414 pnpm test-storybook:run -u'",
    "start-playwright-server": "docker run -p 3000:3000 --rm --init -it mcr.microsoft.com/playwright:v1.39.0-jammy /bin/sh -c \"cd /home/pwuser && npx -y [email protected] run-server --port 3000\"",
  }
}

test-runner-jest.config.js

const { getJestConfig } = require('@storybook/test-runner');

/**
 * @type {import('@jest/types').Config.InitialOptions}
 */
module.exports = {
  // The default configuration comes from @storybook/test-runner
  ...getJestConfig(),
  /** Add your own overrides below
   * @see https://jestjs.io/docs/configuration
   */
  testEnvironmentOptions: {
    'jest-playwright': process.env.STORYBOOK_TEST_RUNNER_CI
      ? undefined
      : {
          connectOptions: {
            chromium: {
              wsEndpoint: 'ws://127.0.0.1:3000',
            },
          },
        },
  },
};