Towards a PureScript Package Manager

(Prompted by discussion in purescript/purescript#631)

There are many things I would like out of a package manager. Separated "installation" and "upgrading" steps, for example, or Elm-like API diffing, Annex-like post-release metadata tagging or whole-registry immutable history for simple lockfiles and repeatable version resolutions, or multiple cross-language dependencies for FFI definitions.

At the moment, though, we have Bower. The Bower registry has worked for us so far as a self-contained universe of PureScript packages. PureScript has (planned? prereleased?) multiple backends, though, and we need to be able to use packages that are "native" to each of those backends. From the available libraries and implementation of the psc-* commands, we've mostly been targeting the JavaScript FFI; in order to use this FFI well, though, we need to be able to bring in parts of what is the de-facto JavaScript ecosystem: npm and its registry.

At the moment there's no way for some purescript-yargs library to state its dependency on the JS yargs library in a way that will result in it being a) installed, b) upgradeable, c) available to compiled code or d) available to bundled code, without npm install-like manual intervention on the part of the user either using purescript-yargs library directly, or indirectly by some other library or app that uses it.

I believe this inability to state and resolve JS dependencies is one of the things that'll hurt adoption if it's not resolved. Every library that wraps some bit of JS needs manual intervention every time it's used directly, or by a dependency, or by a dependency's dependency, etc.

I'd like to see if there's some way to temporarily fix this for the JS-backend users of PureScript, until we figure out some longer-term plan.

Requirements

At the very least, I think any fix needs to allow the following:

PureScript packages should be able to declare their dependencies on npm packages, with version constraints.
A way to initally install those npm packages at the same time as PureScript packages, ie. with the same "command", not a separate script that needs to be run after the fact.
A way to upgrade npm packages to the highest versions allowed by any versioning constraints.
A way for those packages to be found when require()'d from the FFI definitions:
- When built (eg. after a pulp build or psc)
- When bundled (eg. after a pulp build --to ... or psc-bundle)

Ideally, I'd like to:

Have the npm dependencies be "private" per package. When I say "private", I mean in the sense that @hdgarrood uses in this comment; this is, having no "global" resolution of JS FFI packages. For example, having purescript-yargs' yargs JavaScript dependency be:
- Only resolved in terms of that package (treating each npm package.json or other definition as for that package only), and
- Only available to FFI modules in that package for require()ing.
Separate out "install" vs "upgrade" steps, with automatic version locking: like how Ruby's Bundler works, and not how npm works. This is a usability and reliability preference: as someone with ops experience, and someone who's worked in a team that uses npm, reproducible builds are very important to me, and neither Bower nor npm easily gives that to us (Bower doesn't have it, and npm requires that people keep manually running shrinkwrap). Implementing this means a cross-dependency lockfile (ideally), or a bunch of npm shrinkwraps (far less ideal), or some other mechanism.
(See this brief tweetstorm as to why lockfile-like things matter, even in the context of semver guarantees.)
Remove the manual conflict resolution step from our installation of Bower packages; it's meaningless in our case, as both answers presented to the user are wrong (one or the other dependency is broken as a result of the choice).

Proposed Solution

A command-line tool, like Pulp or Bower, that wraps installation and upgrades. Assuming the context is some app or library that may depend on npm packages itself, and depend on :

(It's going to be easier to divvy up responsibility between a package-manager tool and any of the existing build tools if I lay the steps out.)

Using Bower in some way (eg. as a library) to install PureScript packages as it does now.
Using npm in some way (eg. as a library) to install any top-level package.json-defined JS dependencies the app / library may have.
Using npm in the same way to manage node_modules directories inside each bower_components/purescript-... package that has an npm package.json file.
On build, giving each of the modules access to their own package's node_modules bucket, eg. by symlinking an output/ffi/purescript-yargs/node_modules as output/Yargs/node_modules, output/Yargs.Setup/node_modules, etc.
This would mean the output/Yargs/foreign.js file would be require('yargs')ing the version of the library from its local node_modules bucket, without needing to know any of the other buckets exist.
On bundle, inlining the FFI require()s such that the dependency contents are inserted into the bundle itself.

(Bundling is the fuzziest part of this proposal; I'm not sure if it's desired to have a completely independent blob of JavaScript; at the moment any FFI now requires that require() be able to find any dependent libraries, usually by the Node CommonJS loader.)

Current Prototypes

I've created this demonstration repo here to illustrate what the result of a build process would look like. It also includes a script that provides a slapdash implementation of the essential features above, to show how the build output arrived at that state. It's nowhere near how I'd want an implementation to be written; it's just to show the build result hanging together.

Separately, I've started Pallet that ~~provides~~ will shortly provide a proposed console UI. This as a project is intended to meet the proposed solution above including the ideal items. (I admit, I jumped the gun on saying I had this proposal kicking around; I wanted to lay down an initial UI for this first to help make obvious what I'm going for. 😅 )

Help Requested

Where I most need help is figuring out how the "build" parts of the proposed solution above fits into the existing build tools, eg. Pulp or further down. The prototype shellscript brute-forces things just to get it working (with a couple of hardcoded assumptions for things too hard to do quickly in a shellscript), but long-term you wouldn't want anything like this in a package tool.

Edit: And as to whether any of this is actually a good idea.

TL;DR

A tool to handle both Bower and NPM installs. Private-as-in-isolated per-package resolution JS dependencies. Go read https://github.com/damncabbage/purescript-js-proposal/blob/master/install-and-build.sh and see what it's doing, and look at the (committed to that repo) bower_components and output folders.

Anyway. I hope this has provoked some thoughts; please let me know what you think either way. :-)