A Cypress.io test suite with Istanbul-instrumented code took 30 minutes to run instead of 10. For context, we were using https://github.com/cypress-io/code-coverage.
Some overhead is expected with each spec. Collecting coverage will increase the overall run time, surely. But in debug runs (setting DEBUG=code-coverage), we saw an unexpected 31 second overhead running just one spec:
code-coverage NYC file /app/e2e/.nyc_output/out.json has 739 key(s) +13s
code-coverage 1 key /app/frontend/src/config.js file path /app/frontend/src/config.js +1ms
code-coverage 2 key /app/frontend/src/util/logging.js file path /app/frontend/src/util/logging.js +0ms
code-coverage 3 key /app/frontend/src/domains/test-sites/constants.js file path /app/frontend/src/domains/test-sites/constants.js +0ms
code-coverage in file /app/e2e/.nyc_output/out.json all files are not found? false +32ms
code-coverage NYC file /app/e2e/.nyc_output/out.json has 739 key(s) +32ms
code-coverage calling NYC reporter with options { 'report-dir': '/app/e2e/coverage', reporter: [ 'lcov', 'clover', 'json', 'json-summary' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: false, 'temp-dir': '/app/e2e/.nyc_output', tempDir: '/app/e2e/.nyc_output', reportDir: '/app/e2e/coverage' } +0ms
code-coverage current working directory is /app/e2e +0ms
code-coverage after reporting, returning the report folder name /app/e2e/coverage +31s
What was it doing, exactly?
After some digging into the plugin code, we found a task in the code coverage plugin calling nyc.report() after each spec file. But we don't need to generate a report until after all specs are done.
Can it be improved with minimal effort?
The code-coverage plugin code is a project dependency in node_modules, so we couldn't easily edit it. Luckily, though, we saw that there's a custom coverage report hook with npm scripts that we can override, bypassing the report code entirely:
"scripts": {
"coverage:report": "echo 'skipped'"
},
"nyc": {
"excludeAfterRemap": false
}After the workaround, the test suite runs in 15 minutes instead of 30. Progress!
Note: we generated the coverage report manually after the Cypress run via npx nyc report --reporter=html --reporter=text-summary. It took about 10 seconds.