Rollup builds doesn't scale well in large apps. You need to increase Node's memory with --max-old-space-size=4096 to handle all the modules. This is one of Vite's highest-rated issue.
This file documents various findings and attempts to improve this issue.
NOTE: I've only been reading Rollup's source code for a while, so some of these may not be accurate.
High-level flow of Rollup bundling that I find to be performance-sensitive.
- Create a module graph for input entries.
- Each module has an AST (acorn/estree) after plugin transforms.
- Each AST is traversed and re-created as Rollup AST, which nodes are class instances instead of JS objects, each instance hold information for bundling and treeshaking.
- Treeshake in an opt-in fashion by detecting what module/exports are used.
- This interacts with Rollup's AST, calling nodes recursively to mark itself as "included".
- Treeshaking can be done multiple times as "included" nodes can cause more code to be treeshaken. (This operation is cached)
- Generate output files.
- Does many things, but importantly it "renders" each module by "rendering" the AST.
- Each AST node will then rendered itself - the treeshaked output.
-
Solution 1: We can fix this by sharing the AST, or removing/unreferencing one of the AST.
I went with the latter and fixed it at rollup/rollup#4762. It removes the parsed AST (acorn) in favour of re-parsing it when the user requests it again, e.g.
moduleInfo.ast. And also applies a LRU cache to make continuous access less taxing.
Rollup keeps all module's AST in memory at all times. Recently, I sent a PR to Astro docs optimizing the output code of ~750 MDX files to generate smaller AST, which improved Rollup build time from 90s to 30s. (Previously it even needed 6GB of memory space to not OOM)
-
Solution 1: Similar to my PR, collapse the Rollup AST to not keep a huge tree in memory.
Not possible as Rollup applies many optimizations recursively. Collapsing ASTs would lose out on it and may make future optimizations harder.
-
Solution 2: Remove unused AST fields.
Not possible. Rollup uses many fields on the AST and it's not scalable to maintain a list of fields we use. I don't think there's a huge gain either.
-
Solution 3: Create the Rollup AST, do everything we need with it, then quickly tear it down.
Not possible. As shown of how rollup works, the AST is used throughout the build process and they are stateful, so it can't be tear downed. The only thing we can do ahead of time is whether a module has "effects", but that alone doesn't quite help.
Sourcemaps generated for files can be large. However, I haven't looked deep into optimizing how they're handled in Rollup. There are user-reports that disabling sourcemaps in Rollup reduces memory usage though.
The solutions above may not be not possible, but not feasible without a refactor. So maybe there's a way to acheive the proposed solutions.
But besides this, I hope this document prevents duplicate future work, and allows focusing on alternate optimization approaches instead.
For Vite, there is the idea to switch to esbuild or rolldown.
It's hard to switch to esbuild as Vite's plugin API (which inherits Rollup's API) is not compatible with esbuild. We could write a compatibility layer to run Rollup plugins as esbuild plugins, but it will break things as Rollup's plugin API has a wide surface area. It's also unclear if esbuild's plugin API will allow Vite to do all the tricks it needs to build an optimal output.
rolldown is a compelling option, but it's still a WIP. It will need to do more than Rollup OOTB though to minimize JS <-> binary, which Vite could adapt still.
Another approach Vite's been experimenting is optimizing dependencies in builds. Theoretically, if all deps are pre-bundled, it would result in lesser code to be parsed. There's still rough edges to solve though.