Parallelizing Jest in GitHub Actions

ci.yml

name: CI
on: [push]
jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      test-chunks: ${{ steps['set-test-chunks'].outputs['test-chunks'] }}
      test-chunk-ids: ${{ steps['set-test-chunk-ids'].outputs['test-chunk-ids'] }}
    steps:
      - uses: actions/checkout@v2
      - run: npm install
      - id: set-test-chunks
        name: Set Chunks
        run: echo "::set-output name=test-chunks::$(npx jest --listTests --json | jq -cM '[_nwise(length / 2 | ceil)]')"
      - id: set-test-chunk-ids
        name: Set Chunk IDs
        run: echo "::set-output name=test-chunk-ids::$(echo $CHUNKS | jq -cM 'to_entries | map(.key)')"
        env:
          CHUNKS: ${{ steps['set-test-chunks'].outputs['test-chunks'] }}
  test:
    runs-on: ubuntu-latest
    name: test (chunk ${{ matrix.chunk }})
    needs:
      - setup
    strategy:
      matrix:
        chunk: ${{ fromJson(needs.setup.outputs['test-chunk-ids']) }}
    steps:
      - uses: actions/checkout@v2
      - run: npm install
      - name: jest
        run: echo $CHUNKS | jq '.[${{ matrix.chunk }}] | .[] | @text' | xargs npx jest
        env:
          CHUNKS: ${{ needs.setup.outputs['test-chunks'] }}

@imhoffd dude this is awesome

@imhoffd nice, two questions

1. how do you make checks with dynamic names (test (chunk 0)) as a required check?
2. if you know that GHA already has 2 cores, why do you need to calculate the number of runs dynamically?

oh and one more: you need to run npm install at least 3 times, do you think it's faster than jest?

@havenchyk

1. My understanding is the check has to actually run first, then it will appear in Repo Settings -> Branches -> Edit -> Require status checks to pass before merging automatically.
2. I think you may be conflating concurrency with parallelism. Jest will automatically use up the available cores on the VM. This script will run jest tests on separate VMs entirely.
3. You can cache your node_modules directory in Github Actions if you find it's becoming a problem. There will never be a silver bullet for "fastest performance". You have to do your own testing for your own test suite.

Thanks, @imhoffd! I managed to solve those problems with caching, as for required checks, I just made chunks static, so I know how many of those I'm going to use

FYI, Jest is potentially getting a --shard option which makes this way easier: jestjs/jest#12546

Thanks! seems it will be available in jest@28

i got this error when using the above github actions

Error reading JToken from JsonReader. Path '', line 0, position 0.,.github/workflows/chunks.yml (Line: 75, Col: 16): Unexpected type of value '', expected type: Sequence.

any idea how to resolve this?

Do you have to specify the length manually, or is there a way to set up that automatically? For instance, for every 100 tests, create a new test chunk