Microsoft Principal CI Engineer Take Home

ci.yml

# .github/workflows/ci.yml
name: CI
on: [push]
jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      test-chunks: ${{ steps['set-test-chunks'].outputs['test-chunks'] }}
      test-chunk-ids: ${{ steps['set-test-chunk-ids'].outputs['test-chunk-ids'] }}
    steps:
      - uses: actions/checkout@v2
      - run: npm install
      - id: set-test-chunks
        name: Set Chunks
        run: echo "::set-output name=test-chunks::$(node ./bin/get-test-files-for-ci.ts)"
      - id: set-test-chunk-ids
        name: Set Chunk IDs
        run: echo "::set-output name=test-chunk-ids::$(echo $CHUNKS | jq -cM 'to_entries | map(.key)')"
        env:
          CHUNKS: ${{ steps['set-test-chunks'].outputs['test-chunks'] }}
  test:
    runs-on: ubuntu-latest
    name: test (chunk ${{ matrix.chunk }})
    needs:
      - setup
    strategy:
      matrix:
        chunk: ${{ fromJson(needs.setup.outputs['test-chunk-ids']) }}
    steps:
      - uses: actions/checkout@v2
      - run: npm install
      - name: Playwright
        run: echo $CHUNKS | jq '.[${{ matrix.chunk }}] | .[] | @text' | xargs npx playwright test
        timeout-minutes: ${{ secrets.TIMEOUT || 15 }}
        env:
          CHUNKS: ${{ needs.setup.outputs['test-chunks'] }}

description.md

Given the prompt for this assignment I've made some assumptions about softwares we're using. The scripts and test will be in Typescript and the test runner is Playwright. I've decided on Github Actions as my CI since I'm familiar with it. I would generally rely on CircleCI since that's what I have the most experience with and it has some features that Github Actions lacks like automatic test chunking but I wanted to use a Microsoft-approved CI service.

As I understand the task, a "predetermined time frame" should be provided that represents that maximum length of time our entire test suite should take to run. Given this constraint I will endeavor to scale to more CI runners based on the available tests (but no more runners than necessary). If the constraint were instead hardware instead of time I might distribute the tests based on the number of available runners instead of the maximum time. In order to make sure that my code conforms to this constraint I have both the yaml and ts files look for an environment variable called "TIMEOUT" (with a default of 15 minutes). In the ts file this is meant to provide context for how to chunk the test files. In the yaml file this is meant to kill the process if the current test runner takes longer than that predetermined timeframe.

I've set the test to run on "push" such that a new suite of tests will be run each time a pull request is updated. The key here is to use the matrix strategy which Github Actions will interpret as multiple parallel jobs. In the Typescript I fetch and sort the relevant tests into chunks and then in the yaml I use setup.outputs['test-chunk-ids'] to feed the matrix strategy a dynamically generated list of ids derived from the relevant tests which really just serves to tell Github Actions how many runners to spin up. With this setup, Github Actions will always spin up the ideal number of runners for the total run time of all the relevant tests, while still keeping them under the desired threshold.

Further thoughts:

• In some circumstances it might be wise to cache the relevant files locally or determine the relevant files locally such that our CI doesn't rely on an external API. Tools like Jest do something similar by listening for filesystem events but there's no reason we can't just get the list from Git.
• This strategy allocates the test files into chunks linearly. This is not ideal. As a first step approach to improve performance I've sorted the relevant test files by duration, short to long, such that the "most" files should fit in each chunk. In reality this is an example of the bin packing problem and in a real-world implementation I'd want to evaluate strategies that would lead to even more efficient packing.

All in all, we have a rudimentary strategy to:

• Tell our CI how to run our tests.
• Fetch only the specs that are relevant to the files that changed.
• Sort those specs into an arbitrary number of groups based on the total amount of time each spec takes to run.
• Create an arbitrary number of runners that matches the number of groups.
• Set up those runners with our test suite's dependencies.
• Run all the specs.
• Kill a runner if it goes longer than our predetermined time frame.

get-test-files-for-ci.ts

// bin/get-test-files-for-ci.ts

function getChangedFiles() {
  return ["changed_file1.ts","changed_file2.ts"];
}

async function getRelevantTests(changedFiles) {
  const api = "https://tests-selection.azurewebsites.net/api";

  const response = await fetch(api, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify(changedFiles)
  });
  const testFiles = await response.json();
  
  return testFiles;
}

const relevantTests = await getRelevantTests(getChangedFiles());
relevantTests.sort((a, b) => a.duration - b.duration);

const timeout = parseInt(process.env['TIMEOUT'] || 15); // in minutes

function getTotalTime (group, currentFile) {
    const currentTime = group.reduce((acc, curr) => acc + curr.duration, 0);

    return currentTime + currentFile.duration;
}

// create return array
const chunkedRelevantTests = [[]];
for(let i=0; i<relevantTest.length; i++) {
    // if the total minutes in the last group in the return array plus the current file duration is greater than the limit
    if (getTotalTime(chunkedRelevantTests[chunkedRelevantTests.length-1], relevantTest[i]) > timeout) {
        // add a new empty group to the return array
        chunkedRelevantTests.push([]);
    }
    // then push the current file to the return array
    chunkedRelevantTests[chunkedRelevantTests.length-1].push(relevantTest[i].test)
}

// return our chunked file list to where it belongs
process.stdout.write(chunkedRelevantTests);