Pub interop with other package managers

gistfile1.md

I'm starting to think through how pub can interop with bower and other package managers. I'm calling these "foreign" packages. I have a strawman here I want to get feedback on. It's based heavily on earlier work from John Messerly and Justin Fagnani.

High level goals

• Your application can depend on stuff from other package managers. I'm initially targeting bower, but I want this to be open-ended.

• Your application can depend on pub packages which in turn depend on stuff from other package managers.

• You can depend on foreign packages which in turn have their own (foreign) dependencies.

• When dependencies from other package managers are brought in, they are locked in the same way that regular dependencies are. If you check in your pubspec.lock file, that should be enough to pin your foreign dependencies as well as your pub ones.

• Support for additional package managers is done external to pub itself. This lets end users add support for new package managers without having to go through pub and Dart SDK itself. Yay extensibility!

Non-goals:

• Foreign packages that depend on pub packages.

  In other words, your dependency graph starts with pub, moves through multiple layers of pub packages, then transitions to some foreign package manager which may then have layers of dependencies, but never transitions back to pub packages.

Strawman

Your app's pubspec looks like:

# myapp/pubspec.yaml
dependencies:
  awesome_widgets: any
  bower: any
foreign_dependencies:
  bower: clicky-thing

It depends on an awesome_widgets pub package whose pubspec is:

dependencies:
  bower: ">=1.2.3 <2.0.0"
foreign_dependencies:
  bower: snarf

After running pub get, you end up with:

myapp/
  packages/
    awesome_widgets/
    bower/
      libraries of bower package manager pub package...
  bower_packages/
    clicky-thing/
      bower package...
    snarf/
      bower package...
    other bower packages clicky-thing and snarf depend on...
  pubspec.lock
    metadata to pin bower packages...

Your pub package dependencies go under packages as usual. All of the bower packages you (transitively) depend on go under bower_packages.

How this works

You run pub get. That looks at the normal dependencies and does a version resolution. There is a pub package called "bower" that you depend on. Pub picks a version of that, generates a lockfile, etc.

After that, pub looks at the foreign_dependencies sections of of your package and its transitive dependencies. A foreign_dependencies section points to a map. Each key in that map is the name of a foreign package manager (here just "bower").

It collects all of the foreign dependencies for a given package manager across the transitive dependency graph (all of the values associated with "bower" keys in each pubspec). It also looks up the package manager in the existing lockfile, if any. It lumps all that data together and makes a blob of JSON like:

{
  "installDirectory": "/path/to/myapp/bower_packages"
  "packages": {    
    "myapp": "clicky-thing",
    "awesome_widgets": "snarf"
  },
  "lockfile": {
    stuff from lockfile...
  }
}

Pub then effectively runs:

$ pub run bower:install_foreign_dependencies

In other words, it looks for bin/install_foreign_dependencies.dart inside the bower pub package and runs it. It passes in the blob of JSON to that on stdin. That Dart script does a few things:

It does whatever internal magic it needs to do to physically install the foreign dependencies. It is required to put them inside the given "installDirectory". Pub controls the name of this directory. It is always <name of package manager>_packages. It's important that this name is stable because HTML and others assets inside pub packages will contain URLs that walk through that, like ../bower_packages/polymer/some_component.html.

The install_foreign_dependencies script then prints an arbitrary chunk of JSON to stdout. Pub takes that and puts it into the application's lockfile for this package manager. The next time you run pub get, it will take that chunk of JSON and pass it back to the install_foreign_dependencies. This way, the foreign package manager can lock its dependencies too.

Case study: bower

For bower, install_foreign_dependencies would:

• Synthesize a temporary bower.json that contains all of the listed bower packages.
• Add a resolutions map for the locked versions passed from the pubspec.lock file, if any.
• Write the bower.json file to disc.
• Install bower if needed.
• Run bower using the install directory given by pub as the place to install packages.
• Read back the resolutions from the bower.json file.
• Output the resolutions to stdout.
• Discard the temporary bower.json file.

Awesome. Very much looking forward to this feature.

What about expressing version constraints for the bower dependencies?

Maybe it needs to be more like

foreign_dependencies:
    bower:
        snarf: '>=0.1.0 <0.2.0'

Additionally it might be useful to leave the door open to all the usual dart dependency suspects like

foreign_dependencies:
    bower:
        snarf: 
            path: ../snarf
        google-login:
            git: https://.....

Assuming that the bower addon still needs to do something to make the bower packages play well with dart in these cases

What about expressing version constraints for the bower dependencies?

Yes, this is definitely possible, and likely what we'll do for bower. I should have spelled this out more clearly, but the idea is that pub does no interpretation of the chunk of the pubspec that gets passed to install_foreign_dependencies. In the example, it's just the string snarf, but you could have a map, a list, or whatever. It's up to the individual package manager to define what specification it expects.

In the real bower interop implementation, we'll definitely allow version constraints.

all the usual dart dependency suspects like

We probably won't support pub's dependency syntax as-is because that stuff is specific to pub. Instead, it will be what makes the most sense for the specific foreign package manager. Each has its own conventions and semantics, so my goal here is to follow those instead of trying to make everything look as pub-like as possible.

Makes sense

👍

+1 for this proposal.

I have been amazed at all of the activity in the Bower/Grunt/Yeoman/JS world lately. Making Dart compatible with Bower would probably make Dart more appealing to that very active community.

I wonder if this feature couldn't be cover by a more general process : allowing packages to be called when a pub get is done. Thus a bower package could write additional files in packages/bower (or some packages-extra/bower if you don't want to poluate the local pub repo).

The pubspec.yaml would contain the configuration passed to a bin/on_pub_get.dart :

# myapp/pubspec.yaml
dependencies:
  awesome_widgets: any
  bower:
    version: '>=0.13.0 <0.14.0'
    on_get:
      dependencies:
        foo: '>=0.1.0 <0.2.0'
        bar: '>=0.3.0 <0.5.0'

This kind of pub callback could be used for other cases (such as code/asset generation).

What do you think ?

Normally bower resolves dependencies from an existing bower.json. How would package:bower resolve dependencies in order to synthesize a temporary bower.json (which only contains a single version constraint for each dependency) without either:

• implementing bower dependency resolution in dart
• generating some sort of file structure which the entry-point synthesized bower.json references via bower path dependencies to other dart dependencies synthesized bower.json's.

Instead, it will be what makes the most sense for the specific foreign package manager. Each has its own conventions and semantics, so my goal here is to follow those instead of trying to make everything look as pub-like as possible.

For bower, wouldn't that be best accomplished by keeping the bower configuration out of pubspec.yaml and in bower.json instead? Or at least having the bower configuration in the pubspec be some subset of:

https://github.com/bower/bower.json-spec

Pub controls the name of this directory.

Why not allow package managers to specify which directory they put their dependencies in? Any self-respecting package manager should make sure it's default output dir doesn't conflict with existing commonly used package managers. This would make the output a lot less pub-specific.

Sean, I wrote a bower helper package that extracted bower.json files from packages and generated another bower.json file that had them as dependencies, and a specified a directory to install them to - your second bullet point. It worked fine, and allowed Bower's dependency resolver to do the work so you got the same behavior as a normal Bower install.

I almost released it, but it had some usability problems:

• It require node.js and bower to be installed.
• It had to be run manually.
• There was no good way to tell an end user that bower was depended on somewhere down the dependency chain.

install_foreign_dependencies takes care of the last two issues. The first issue is either not worth solving (still require a bower install and generate bower.json files) or is solvable with our bower-in-Dart implementation, in which case we don't need a bower.json file. Any dependency manager that's not reimplemented in Dart will require it to be installed.

I personally like having the foreign dependencies specified in the native format of the foreign dependency manager. It'll be familiar to users of that manager and offers a clear way to implement the integration: find all the configurations, generate a top-level configuration to point to them, run the package manager on that.

The only problem there is that it doesn't lead easily to locking if the manager doesn't support it. It can still be done, but it's not quite as clean or automatic as if pub knows about the dependencies.

This kind of pub callback could be used for other cases (such as code/asset generation).

What do you think ?

This is a good idea, and something we've considered for other use cases. I don't think it's a great fit here though, because I think it's critical that we roll whatever mojo the other package manager does back into pub's own lockfile. We need to ensure repeatable builds of not just pub packages, but foreign ones too.

We could possible just have post-install hooks and then some API to let you put stuff in the lockfile, but I thought it would be better to do something a little more explicitly structured.