Question on git, npm, bower, and CI.

npm-scm.md

Hey, Probably a somewhat n00b question, but on sites of your size that build and release as often as yours do - how do you manage your npm and bower packages? Currently we .gitignore our node_modules and bower_components directories, but I feel we should break that SPOF we have with npm and bower's servers, in the event they're unavailable when our CI is running a build/release.

I had considered checking the packages into git, but;

1. I'm sure this is frequently advised against [citation needed].
2. Some packages such as karma-runner/karma-phantomjs-launcher perform a tailored installation for the OS npm install was run from (eg phantomjs for Windows if installed when on Windows) - so if developer A on Windows runs npm install and checks it into git, CI Server B on Linux will be failing builds.

Is this SPOF something you're even concerned about? And if so, how do you manage it please?

Thanks a lot guys appreciate it.

Jamie.

There's a couple of ways I've used to deal with this, neither are particularly better than the other:

Check in node modules

1. Add node_modules to git
2. Run npm dedupde to save some duplication
3. Run npm rebuild on CI to ensure you have the right build for native modules

This makes it easy to see when things change, but at the cost of noisy diffs with loads of "generated" files in your repo

Run a simple npm mirror

1. Install https://github.com/afbobak/nopar somewhere
2. .gitignore node_modules
3. Ensure package.json is checked in
4. Run npm shrinkwrap after package changes and check in npm-shinkwrap.json
5. Set npm_config_regsitry= environment variable to nopar URL on CI
6. Run npm install on CI

This avoids the noisy git stuff, but means you need to keep the mirror up and running for builds.
It has the added advantage of being able to host private packages.

For various reasons we checkin our bower components, but do not checkin our node_modules because of the OS specific binaries you mentioned above.

We've recently been having npm related networking issues on our CI, so are currently looking for cache solutions such as:

• https://www.npmjs.org/package/sinopia
• http://eng.yammer.com/a-private-npm-cache/

I'm concerned about it. Not just for SPOFs but also regarding build times when fetching dependencies plus the very unnecessary amount of bytes over the wire.

We haven't found a good solution for it. A registry is a decent solution but we don't currently have the resource to set up and maintain one.

Currently, we check in bower deps but don't check in npm. Our bower dependencies don't get updated so often and are pretty light. npm is the opposite and I don't want to bloat our repos by including hundreds of mbs of modules + deal with the constantly changing diffs.

Fwiw I was very close to switching to checking them in at one point but @makeusabrew more or less convinced me not to. I'm still glad we didn't, although I can understand the arguments for both sides.

I put up with it, because having that external dependency is still better than checking the deps into git in my opinion. The justifications in my case are:

1. The CI build doesn't scrap its workspace each time, so "npm install" rarely has to do much as the deps are fairly stable by the time a project nears production (where breakage would matter more). Running "npm prune" beforehand keeps the modules clean
2. If the dependency does fail, at least it's caught at build time
3. Over the lifetime of relevant projects, build failures caused by npm have been a tiny percentage of overall failures. I can live with the odd one until it causes a real problem, at which point I'd investigate a custom mirror as per @mackstar's comment. Even so, you're still not fully protected from failure, you're just shifting the point of it around a bit.

Just to add that nopar is quite simple to set up and run.

The code is pretty easy to follow too.

Hey @glenjamin @makeusabrew @Ianfeather @phamann @mackstar,
Since we had this chat I spoke to @swirlycheetah and came up with https://github.com/JamieMason/shrinkpack. I'd really appreciate your thoughts, do you think something like this could be useful? Any oversights on my part with it?

Thanks a lot,

Jamie

This new shrinkpack approach seems to be a variation on "Check in node_modules", you get a reduction in diff noise, but in exchange you get quite a large increase in repository size? (Binary file changes are less likely to be stored as per-file-diffs when updating version).

I think the earlier comparison still mostly holds here - we have a mirror set up with nopar and I don't recall running into any real issues with this.

in exchange you get quite a large increase in repository size

— @glenjamin

What are the practical considerations of an increase in size? Remember as well that we're talking about predominantly plain text files, which have been tarred and gzipped.