Skip to content

Instantly share code, notes, and snippets.

@belgattitude

belgattitude/benchmark-package-managers.md

Last active June 20, 2023 11:19
Show Gist options
  • Save belgattitude/0ecd26155b47e7be1be6163ecfbb0f0b to your computer and use it in GitHub Desktop.
Save belgattitude/0ecd26155b47e7be1be6163ecfbb0f0b to your computer and use it in GitHub Desktop.
Download ZIP
Package managers comparison from the CI time perspective
Raw
benchmark-package-managers.md

Benchmark

TLDR

On a (cached) github action. pnpm7 ±50s (244Mb store cache) / yarn4-nm ±75s (190Mb yarn install cache)

Intro

Tests made with pnpm 7+ and yarn 4.0.0rc (node-modules/pnp) based on https://github.com/belgattitude/nextjs-monorepo-example. Tests have been made in a way to evaluate CI performance. There's other perf bench: pnpm or yarn. Conclusions are pretty similar. This benchmark helped to create this yarn install composite action

With cache.

  • Yarn (pnp-loose): local: 7s / github action: wip)
  • Yarn (node-modules) local: 21s / github action: ±70s, post: +5s (190MB)
  • Pnpm local: 10s / github action: ±40s: post: +10s (244MB)

Without cache.

  • Yarn local: 71s (github action: ±2mins)
  • Pnpm local: 34s (github action: ±1m20s)

Some observations:

  • Yarn is able to rotate the downloaded cache archives. That detail is important on CI, cause it's possible keep the "warm cache" even if the yarn.lock has been changed (only download changed deps). This also reduce cache invalidation on the ci. Example @action/cache with node-module-linker. Works in pnp and node_modules linkers. For example if you have things like renovatebot... pnpm will start with cold cache, yarn not completely. This affects monthly ci time... but carbon emissions too.
  • Yarn in pnp opt-out from nextjs standalone optimization / @vercel/nft important for nextjs/lambdas/serverless/vercel or whatever (to be tested !)
  • Yarn in pnp is not yet totally supported by tools like nx or turbo (to be tested !)
  • Yarn in pnp is the clear winner but it's sometimes difficult to make it work with some deps (it's improving fast).
  • Yarn in pnp will also improve performance/memory of the node apps (runtime / startup time)
  • Pnpm tackles the doppelgangers. Need to see with yarn.
  • Pnpm have some niceties to work with docker in monorepos
  • Yarn performs a bit better with cache compressionLevel: 0 but it affects the lock and at the end action/cache have more work to do (see notes)

Results

Running yarn install --immutable --inline-builds with hyperfine (5 runs).

Scenario Time (seconds) Github Action action/cache
Yarn cold cache (nm/links-local) 71.410 ± 1.835 ±2m
Yarn warm (nm/links-local) 22.765 ± 0.417 ±1m10s ~190Mb (post: ±5s)
Yarn warm (nm/links-local/install-state) 21.576 ± 0.390 ±1m10s ~190 + 3Mb (post: ±5s)
Yarn warm (nm/links-local/install-state/nocomp) 19.182 ± 0.323 ±50s ~?Mb + 3Mb (post: >20s)
Yarn warm (nm/links-global/install-state) 21.657 ± 0.393 ±1m10s
Yarn warm (pnp-loose) 6.822 ± 0.247
Pnpm 7.13.3 cold cache 34.696 ± 1.413 ±1m20s
Pnpm 7.13.3 warm cache 10.780 ± 0.881 ±40s ~244Mb (post: ±10s)

Note

  • all tests assume there's a lock file.
  • the "On Github Action" is not a proper bench, just an estimate. The action/cache(+++) represent an estimate of the action/cache post restore operation. In other words sometimes it's possible to gain on install time with more agressive cache, but at the end the cache will have to be (un-)compressed by the action wich takes additional time. As far as I tried, the best is just to keep the local .yarn/cache (compression of hardlinks-global store or node_modules don't give better results).
  • pnpm benchmark was done with strict-peer-dependencies=false in .npmrc.

Local test machine:

  System:
    OS: Linux 5.15 Ubuntu 22.04.1 LTS 22.04.1 LTS (Jammy Jellyfish)
    CPU: (12) x64 Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
    Memory: 6.02 GB / 15.29 GB
  Binaries:
    Node: 16.17.0 - ~/.nvm/versions/node/v16.17.0/bin/node
    Yarn: 4.0.0-rc.22 - ~/.nvm/versions/node/v16.17.0/bin/yarn
    npm: 8.15.0 - ~/.nvm/versions/node/v16.17.0/bin/npm

Run the benches

Install hyperfine:

wget https://github.com/sharkdp/hyperfine/releases/download/v1.15.0/hyperfine_1.15.0_amd64.deb
sudo dpkg -i hyperfine_1.15.0_amd64.deb

Clone the example repository:

cd /tmp
git clone [email protected]:belgattitude/nextjs-monorepo-example.git
cd /tmp/nextjs-monorepo-example
mkdir .yarn/bench
npm i --global rimraf

Yarn cold cache (nm/hardlinks-local)

Linker Mode Cache Install state Compress
node-modules hardlinks-local no no mixed 
hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_no-cache_no-state_comp-mixed.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/install-state.gz .yarn/cache .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"

Yarn warm cache (nm/hardlinks-local)

Linker Mode Cache Install state Compress Time
node-modules hardlinks-local yes no mixed 
hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_cache_no-state_comp-mixed.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/install-state.gz .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"
Command Mean [s] Min [s] Max [s] Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds 22.765 ± 0.417 22.064 23.115 1.00

Yarn warm cache (nm/hardlinks-local/install-state)

Linker Mode Cache Install state Compress Time
node-modules hardlinks-local yes yes mixed 
hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_cache_state_comp-mixed.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"
Command Mean [s] Min [s] Max [s] Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds 21.576 ± 0.390 21.102 21.948 1.00

Yarn warm cache (nm/hardlinks-local/install-state/no-comp)

Linker Mode Cache Install state Compress
node-modules hardlinks-local yes yes 0 - off 
hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-local_cache_state_no-comp.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=0 PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"
Command Mean [s] Min [s] Max [s] Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=0 PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds 19.182 ± 0.323 18.806 19.482 1.00

Warm cache (nm/hardlinks-global/install-state)

Linker Mode Cache Install state Compress
node-modules hardlinks-global yes yes mixed 
hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/nm_hardlinks-global_cache_state_comp.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/global" \
          "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"
Command Mean [s] Min [s] Max [s] Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NM_MODE=hardlinks-local YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds 21.657 ± 0.393 21.115 22.016 1.00

Yarn PNP Warm cache (loose)

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/yarn_pnp_loose_cache_comp-mixed.md" \
          --prepare "npx --yes rimraf '.yarn/unplugged' .yarn/global" \
         "YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NODE_LINKER=pnp YARN_PNP_MODE=loose YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds"
Command Mean [s] Min [s] Max [s] Relative
YARN_ENABLE_GLOBAL_CACHE=false YARN_GLOBAL_FOLDER=.yarn/global YARN_NODE_LINKER=pnp YARN_PNP_MODE=loose YARN_COMPRESSION_LEVEL=mixed PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 yarn install --immutable --inline-builds 6.822 ± 0.247 6.490 7.175 1.00

PNPM 7 Cold cache

# Content of .npmrc
strict-peer-dependencies=false
store-dir=.yarn/pnpm/global-cache

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/pnpm_no-cache.md" \
          --prepare "npx --yes rimraf '**/node_modules' .yarn/pnpm/global-cache; pnpm prune store" \
          "PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i"
Command Mean [s] Min [s] Max [s] Relative
PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i 34.696 ± 1.413 33.841 37.210 1.00

PNPM 7 Warm cache

hyperfine --runs=5 --warmup=1 --export-markdown ".yarn/bench/pnpm_cache.md" \
          --prepare "npx --yes rimraf '**/node_modules'; pnpm prune store" \
          "PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i"
Command Mean [s] Min [s] Max [s] Relative
PRISMA_SKIP_POSTINSTALL_GENERATE=true HUSKY=0 pnpm i 10.780 ± 0.881 10.190 12.341 1.00

Notes

CI

Tests have been done locally. On the CI we generally want to account for action/cache compression perf as well.

image

nmMode

nmMode: hardlinks-local

image

Compression level

Disabling cache compression might give a small speed-up on some CI (there's trade-offs). It affect the lock file. In other words you can't have it set differently for local and on the CI (at least if using yarn install --immutable) Best is to keep YARN_COMPRESSION_LEVEL at 'mixed' (auto)

With mixed compression level

image

Without compression

image

Why not caching node_modules

The action/cache has more to do in term of compression/decompression

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment