Benchmark tests for Yarn vs NPM speed

yarn-npm-speed-test.md

Yarn vs NPM speed test

Work in progress

I recently made a claim on Twitter that yarn is still significantly faster than npm for installing NodeJS dependencies.

My belief here was based on some light research I did near the end of 2017, not too long after NPM 5 was released claiming huge speed improvements. I found a blog post at the time claiming Yarn was still faster, and then did some light testing which confirmed this for me, by a large margin.

However, my assertion was challenged:

That’s interesting, on my project npm is ~twice as fast. I’ve not done extensive testing but it’s usually pretty clear, and running npm ci instead of npm i is another factor of 2 speed increase. Even with this the fastest install times are 2-3 minutes, our project is big 🙃

(@Syynth on Twitter)

And so I'm now testing my assumption to see how NPM's speed compares to Yarn nowadays.

Headline results

Coming soon

Setup

I'm running this on a Thinkpad T480 20L5, with the following stats:

OS: Ubuntu 18.04.2 LTS (64-bit)
Memory: 31.3 GiB
Processor: Intel® Core™ i7-8650U CPU @ 1.90GHz × 8
Graphics: Intel® UHD Graphics 620 (Kabylake GT2) (onboard)
Disk: 304.1 GB

I installed nodejs version 11.14.0 and npm version 6.7.0 from nodesource as per the official recommendation.

I installed yarn version 1.15.2 from dl.yarnpkg.com as per their official instructions.

I'm on a pretty stable 4G mobile connection. I tested the speed with Ookla Speedtest and got:

Ping: 22
Download: 119.12 Mbps
Upload: 23.11 Mbps

Method

I'll restart my computer, and the only foreground applications I'll run are the terminal and Visual Studio Code, for taking notes.

I'm going to test installing dependencies from the master branch of two of our most significant open-source website projects:

• www.ubuntu.com, currently at 6a3caaa, has a modest 3 dependencies and 13 devDependencies
• snapcraft.io, currently at f73e501, has 55 dependencies and no devDependencies

Tests

I'm going to test the following cases with both yarn and npm. In each case, I'll run the same test 3 times for each tool, alternating tools between each test.

I first run the test on the www.ubuntu.com codebase (package.json), and then with snapcraft.io (package.json).

Install dependencies from scratch without a cache

Before each test:

# Clear out local dependencies, lockfiles and package caches
$ rm -rf node_modules/ yarn.lock package-lock.json && yarn cache clean --force && npm cache clean --force

And then I run either time yarn install or time npm install.

Install dependencies from scratch with a warm cache

Before the first run:

# Install dependencies from scratch with both tools to warm the cache
$ rm -rf node_modules/ yarn.lock package-lock.json && yarn install
$ rm -rf node_modules/ yarn.lock package-lock.json && npm install

Before each run:

# Before each run, delete dependencies and lockfiles
$ rm -rf node_modules/ yarn.lock package-lock.json

Then I run either time yarn install or time npm install.

Install from lockfile

Run install again, with no dependencies, but with a lockfile.

Before first run:

# Generate both lockfiles
$ yarn install && npm install

Before each run:

# Remove node_modules
$ rm -rf node_modules

Then I run either time yarn install or time npm install.

Generate lockfile

Run install again, with all dependencies already installed, but with no lockfile.

Before each run:

# Install dependencies with the relevant tool, and remove lockfile
$ rm -rf node_modules && yarn install && rm yarn.lock
# Or
$ rm -rf node_modules && npm install && rm package-lock.json

Then I run either time yarn install or time npm install.

Nothing to install

Run install again, with all dependencies already installed.

So I first run the install without testing the time, to ensure dependencies are installed exactly as expected, and then run it again, to see how long it takes when no changes are needed.

$ yarn install && time yarn install
# or
$ npm install && time npm install

Results

Install dependencies from scratch without cache

www.ubuntu.com

• time yarn install #1: 28.096 seconds
• time npm install #1: 22.234 seconds
• time yarn install #2: 24.226 seconds
• time npm install #2: 18.710 seconds
• time yarn install #3: 27.061 seconds
• time npm install #3: 19.954 seconds

Summary

• yarn install times: Mean 26.461 seconds, variance 2.6762
• npm install times: Mean 20.2993 seconds, variance 2.1294

snapcraft.io

• time yarn install #1: 46.054 seconds
• time npm install #1: 38.101 seconds
• time yarn install #2: 40.380 seconds
• time npm install #2: 38.131 seconds
• time yarn install #3: 42.527 seconds
• time npm install #3: 37.868 seconds

Summary

• yarn install times: Mean 42.987 seconds, 5.4715 variance
• npm install times: Mean 38.0333 seconds, 0.0138 variance

npm wins. Here's the log.

Install dependencies from scratch with a warm cache

www.ubuntu.com

• time yarn install #1: 19.143 seconds
• time npm install #1: 11.838 seconds
• time yarn install #2: 20.621 seconds
• time npm install #2: 14.741 seconds
• time yarn install #3: 16.729 seconds
• time npm install #3: 10.688 seconds

Summary

• yarn install times: Mean 18.831 seconds, variance 2.5733
• npm install times: Mean 12.4223 seconds, variance 2.9085

snapcraft.io

• time yarn install #1: 23.009 seconds
• time npm install #1: 14.305 seconds
• time yarn install #2: 24.369 seconds
• time npm install #2: 14.666 seconds
• time yarn install #3: 23.506 seconds
• time npm install #3: 14.234 seconds

Summary

• yarn install times: Mean 23.628 seconds, 0.3157 (1.336%) variance
• npm install times: Mean 14.4017 seconds, 0.0357 (0.248%) variance

npm wins. Here's the log.

Install from lockfile

www.ubuntu.com

• time yarn install #1: 6.649 seconds
• time npm install #1: 8.640 seconds
• time yarn install #2: 6.862 seconds
• time npm install #2: 8.497 seconds
• time yarn install #3: 6.953 seconds
• time npm install #3: 8.653 seconds

Summary

• yarn install times: Mean 6.8213, variance 0.0162
• npm install times: Mean 8.5967, variance 0.0050

snapcraft.io

• time yarn install #1: 7.683 seconds
• time npm install #1: 12.098 seconds
• time yarn install #2: 7.645 seconds
• time npm install #2: 12.156 seconds
• time yarn install #3: 7.685 seconds
• time npm install #3: 12.095 seconds

Summary

• yarn install times: Mean 7.671 seconds, 0.0003 (0.004%) variance
• npm install times: Mean 12.1163 seconds, 0.0008 (0.007%) variance

yarn wins. Here's the log.

Generate lockfile

www.ubuntu.com

• time yarn install #1: 13.483 seconds
• time npm install #1: 4.983 seconds
• time yarn install #2: 11.817 seconds
• time npm install #2: 4.660 seconds
• time yarn install #3: 12.767 seconds
• time npm install #3: 4.456 seconds

Summary

• yarn install times: Mean 12.689, variance 0.4656
• npm install times: Mean 4.6997, variance 0.0470

snapcraft.io

• time yarn install #1: 19.142 seconds
• time npm install #1: 6.688 seconds
• time yarn install #2: 17.503 seconds
• time npm install #2: 6.181 seconds
• time yarn install #3: 21.810 seconds
• time npm install #3: 6.263 seconds

Summary

• yarn install times: Mean 19.485 seconds, 3.1505 (16.17%) variance
• npm install times: Mean 6.3773 seconds, 0.0494 (0.77%) variance

npm wins. Here's the log.

Nothing to install

www.ubuntu.com

• yarn install && time yarn install #1: 0.584 seconds
• npm install && time npm install #1: 4.415 seconds
• yarn install && time yarn install #2: 0.547 seconds
• npm install && time npm install #2: 4.816 seconds
• yarn install && time yarn install #3: 0.555 seconds
• npm install && time npm install #3: 4.406 seconds

Summary

• yarn install times: Mean 0.562, variance 0.0003
• npm install times: Mean 4.5457, variance 0.0366

snapcraft.io

• yarn install && time yarn install #1: 0.640 seconds
• npm install && time npm install #1: 6.737 seconds
• yarn install && time yarn install #2: 0.669 seconds
• npm install && time npm install #2: 6.692 seconds
• yarn install && time yarn install #3: 0.660 seconds
• npm install && time npm install #3: 6.599 seconds

Summary

• yarn install times: Mean 0.6563, 0.0001 (0.02%) variance
• npm install times: Mean 6.676, 0.0033 (0.05%) variance

yarn wins. Here's the log.

If you want to know more about where Yarn spends most of its time, you can through the hidden hooks. Just create a js file that contains the following (replace ./yarn.js by the actual path to the Yarn release):

global.experimentalYarnHooks = {
  resolveStep: async fn => {
    console.time(`resolveStep`);
    try {
      return fn();
    } finally {
      console.timeEnd(`resolveStep`);
    }
  }
};

require(`./yarn.js`);

The other enumeration values also work and should print the individual times for each respective step for the install. I'm curious to see where we spend the most time.

In theory the fetch time 0 should be zero with a cache, and the resolve step should be 0 with a lockfile. The link time is a bit weird an not as much optimized as it could (which is why we came up with Plug'n'Play).