jest-changedSince-in-github-actions.md

Jest and the --changedSince flag in GitHub Actions

Recently, I've been working a lot more with GitHub Actions - both writing actions and creating CI pipelines for projects.

Last week I picked up a project I started a bit ago: the nodejs/examples repository.

Note: The examples repository is still in early stages. As such, there's still a bunch of WIP work we're doing - we've intentionally not talked a bunch publicly about it yet. That said, if you're interested in helping, feel free to reach out to me on Twitter or the OpenJS Slack ❤️

The goal of this repository is to be home to a bunch of distinct and well-tested examples of real-world Node.js that go beyond "hello, world!". This means there's hopefully going to be a boatload of distinct projects in there.

This structure presents a challenge when trying to be uncombersome to new contributions; specifically, it's a barrier to run a full test suite for many projects when someone submitting a PR only needs to see the results of the one they've worked on.

Jest's Solutions

Jest has a super handy --onlyChanged feature that only tells you what has changed in the current repostiory. This is super duper handy, but the functionality is slightly unclear in one way: does it diff with master or just with the previous commit? It does indeed seem to be the latter (though I could totally be wrong!) which is not particularly helpful in the case of PRs with multiple commits.

As such, I looked through the flags that Jest exposes and found the --changedSince flag which compares the current work with a different branch. Since, in the case of nodejs/examples, master will always be a source of truth this is perfect for the use case of potentially having multiple commits while still wanting to run only the tests relevant to a proposed change.

--changedSince and GitHub Actions CI

Previously, the --onlyChanged flag worked flawlessly with GitHub Actions CI. When trying to simply change from --onlyChanged to --changedSince, the CI build immediately nuked itself with the following command:

  ● Test suite failed to run

    fatal: bad revision '^master'

This was bizarre to me since the test was working completely fine on my machine (shocker, I know). Upon investigating, this is actually a git error and not a Jest error - Jest is simply acting as a courrier for that error.

It turns out that the actions/checkout GitHub Action does not actually checkout your full repository but only the code relevant to the PR. As such, master as a branch did not exist. Further, my specific use case of wanting to have master in the run but have the PR branch checked out is not particularly well supported by actions/checkout at present since it is somewhat of an edge case (though I did open an issue to request it).

While the examples are helpful, they don't really solve my somewhat complex but not over the top use case. Layer on that I'm not super excellent with git and you have a challenging mixture.

I reached out to Shelley Vohr, who's extremely talented with git (amongst many other things) and explained what I was facing. She suggested that I'd need to go one step beyond what the actions/checkout repo recommended:

git fetch --no-tags --prune --depth=1 origin +refs/heads/*:refs/remotes/origin/* # fetches all branches

... and needed to actually checkout master with the following command:

git checkout -b master # -b creates and checks out a new branch

... and then switch back to the PR branch. Luckily, GitHub provides that data in the YAML conifg:

git checkout ${{ github.event.pull_request.head.sha }} # checks out the SHA of the HEAD from the PR

This was all able to be combined as a part of a run property in the YAML for the step, which runs whatever commands are passed to it:

    - uses: actions/checkout@v2
    - run: |
        git fetch --no-tags --prune --depth=1 origin +refs/heads/*:refs/remotes/origin/* # fetches all branches
        git checkout -b master # -b creates and checks out a new branch
        git checkout ${{ github.event.pull_request.head.sha }} # checks out the SHA of the HEAD from the PR

However, that's a rather bulky git fetch that can potentially artificially increase the build times as more branches are added to the repo. As such, I figured I should try to cut it down to just what I needed. After a bit of seaerching around, I found the git fetch <remote> <branch> structure. Since I know I'll always want to use master, this was a pretty easy change (while also ditching --prune since it seems potentially useless in this case):

    - uses: actions/checkout@v2
    - run: |
        git fetch --no-tags --depth=1 origin master
        git checkout -b master
        git checkout ${{ github.event.pull_request.head.sha }}

In addition to all this YAML CI config, I also included a new npm script called test:changedsince:

  "scripts": {
    "test": "jest --coverage",
    "test:onlychanged": "jest --onlyChanged --coverage",
    "test:changedsince": "jest --changedSince=master --coverage",
    "lint": "standard"
  },

This new npm script took the place of the previous test:onlychanged npm script in my final GitHub Actions CI YAML config:

name: tests(push) - install, lint, test:changedsince

on: [push]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
        node-version: [10.x, 12.x]
    steps:
    - uses: actions/checkout@v2
    - run: |
        git fetch --no-tags --depth=1 origin master
        git checkout -b master
        git checkout ${{ github.event.pull_request.head.sha }}
    - name: Use Node.js ${{ matrix.node-version }} on ${{ matrix.os }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}
    - name: npm install
      run: npm install
      env:
        CI: true
    - name: npm run lint
      run: npm run lint
      env:
        CI: true  
    - name: npm run test:changedsince
      run: npm run test:changedsince
      env:
        CI: true

Now, this seems to be working perfectly - it'll diff changes between the current PR's HEAD and master, running only the tests that are different across all commits and not just between the most recent commit and the one prior.

For me this worked well:

      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - run: |
          git checkout -b main origin/main
          git checkout -

This worked for me. It uses a single GH API call to figure out which commits must be fetched. Also it creates a sticky comment on the PR with the coverage summary.

name: CI

on:
  push:
    branches: ["master", "feature/*"]
  pull_request:
    branches: ["master"]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

env:
  HOWDJU_RUNNING_IN_GITHUB_WORKFLOW:

jobs:
  # ... other jobs

  changed-files-coverage-check:
    # Only run on PRs.
    if: github.ref != 'refs/heads/master'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          # During a pre-merge check, Github creates and checks out an temporary merge commit. That
          # commit won't work as the HEAD for yarn --changedSince
          ref: ${{ github.event.pull_request.head.sha }}
      - name: Fetch merge base commits
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Fetch commits to a depth so that head and base reach their merge base.
          comparison=$(gh api\
            repos/Howdju/howdju/compare/${{ github.event.pull_request.base.sha }}...${{github.event.pull_request.head.sha }})
          behind_by=$(echo -E $comparison | jq -r '.behind_by')
          ahead_by=$(echo -E $comparison | jq -r '.ahead_by')
          echo "ahead_by: $ahead_by; behind by: $behind_by"
          # +1 because fetch depth=1 is the commit itself.
          if [[ $behind_by -gt 0 ]]; then
            base_depth=$((behind_by+1))
            echo Fetching base to depth $base_depth
            git -c protocol.version=2 fetch --no-tags --no-recurse-submodules\
              --depth=$base_depth origin ${{github.event.pull_request.base.sha }}
          fi
          if [[ $ahead_by -gt 0 ]]; then
            head_depth=$((ahead_by+1))
            echo Fetching head to depth $head_depth
            git -c protocol.version=2 fetch --no-tags --no-recurse-submodules\
              --depth=$head_depth origin ${{github.event.pull_request.head.sha }}
          fi
      - name: Setup Node.JS
        uses: actions/setup-node@v3
        with:
          node-version: 14
          cache: "yarn"
      - name: Install Yarn
        run: |
          corepack enable
          yarn set version stable
      - name: Install dependencies
        run: yarn install
      - name: Changed-files test coverage
        run: yarn run test:all:coverage:changed ${{github.event.pull_request.base.sha }}
      - name: Output merged changed-files coverage summary
        id: changed-files-coverage-summary
        run: |
          yarn run merge:coverage
          output=$(yarn run nyc report -t coverage --reporter=text-summary)

          delimiter="$(openssl rand -hex 8)"
          echo "coverage_summary<<${delimiter}" >> $GITHUB_OUTPUT
          echo "${output}" >> $GITHUB_OUTPUT
          echo "${delimiter}" >> $GITHUB_OUTPUT
      - name: Create changed-files coverage PR comment
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: Changed-files coverage
          message: |
            # Changed-files coverage summary
            ```
            ${{ steps.changed-files-coverage-summary.outputs.coverage_summary }}
            ```

where

  "test:all:coverage:changed": "yarn run clear:coverage && echo test:all:coverage:changed && time yarn workspaces foreach -Apv run test --ci --runInBand --coverage --changedSince=${0:-origin/master} && echo test:all:coverage:changed succeeded || {echo test:all:coverage:changed failed; exit 1}"

and merge:coverage runs:

#! /bin/bash

set -e

echo Deleting $(pwd)/coverage
rm -rf $(pwd)/coverage
echo Creating coverage/workspaces
mkdir -p coverage/workspaces
echo Copying coverage report from workspaces
yarn workspaces foreach -Apv exec bash -c '[ ! -f coverage/coverage-final.json ] && exit 0 || cp coverage/coverage-final.json '$(pwd)'/coverage/workspaces/$(basename $(pwd))-coverage-final.json'
echo Merging coverage reports
yarn run nyc merge coverage/workspaces coverage/monorepo-coverage.json

I'm not sure how git checkout -b master worked for anyone. I think that the -b flag creates a new branch that is not tracking origin/master. If that were correct, master would point to the checkout commit, and jest --changedSince=master would always detect no changes (since master and HEAD are the same commit.)

Note that if you have configured this action as a pre-merge check, then adding ref: ${{ github.event.pull_request.head.sha }} may invalidate that. Github creates a temporary merge commit behind the scenes and checks it out. But overriding that with ref means that, for example, tests run in the action will not reflect changes to master since the branches diverged. And that is why I created this as a separate job.