Pub interop with other package managers

gistfile1.md

I'm starting to think through how pub can interop with bower and other package managers. I'm calling these "foreign" packages. I have a strawman here I want to get feedback on. It's based heavily on earlier work from John Messerly and Justin Fagnani.

High level goals

• Your application can depend on stuff from other package managers. I'm initially targeting bower, but I want this to be open-ended.

• Your application can depend on pub packages which in turn depend on stuff from other package managers.

• You can depend on foreign packages which in turn have their own (foreign) dependencies.

• When dependencies from other package managers are brought in, they are locked in the same way that regular dependencies are. If you check in your pubspec.lock file, that should be enough to pin your foreign dependencies as well as your pub ones.

• Support for additional package managers is done external to pub itself. This lets end users add support for new package managers without having to go through pub and Dart SDK itself. Yay extensibility!

Non-goals:

• Foreign packages that depend on pub packages.

  In other words, your dependency graph starts with pub, moves through multiple layers of pub packages, then transitions to some foreign package manager which may then have layers of dependencies, but never transitions back to pub packages.

Strawman

Your app's pubspec looks like:

# myapp/pubspec.yaml
dependencies:
  awesome_widgets: any
  bower: any
foreign_dependencies:
  bower: clicky-thing

It depends on an awesome_widgets pub package whose pubspec is:

dependencies:
  bower: ">=1.2.3 <2.0.0"
foreign_dependencies:
  bower: snarf

After running pub get, you end up with:

myapp/
  packages/
    awesome_widgets/
    bower/
      libraries of bower package manager pub package...
  bower_packages/
    clicky-thing/
      bower package...
    snarf/
      bower package...
    other bower packages clicky-thing and snarf depend on...
  pubspec.lock
    metadata to pin bower packages...

Your pub package dependencies go under packages as usual. All of the bower packages you (transitively) depend on go under bower_packages.

How this works

You run pub get. That looks at the normal dependencies and does a version resolution. There is a pub package called "bower" that you depend on. Pub picks a version of that, generates a lockfile, etc.

After that, pub looks at the foreign_dependencies sections of of your package and its transitive dependencies. A foreign_dependencies section points to a map. Each key in that map is the name of a foreign package manager (here just "bower").

It collects all of the foreign dependencies for a given package manager across the transitive dependency graph (all of the values associated with "bower" keys in each pubspec). It also looks up the package manager in the existing lockfile, if any. It lumps all that data together and makes a blob of JSON like:

{
  "installDirectory": "/path/to/myapp/bower_packages"
  "packages": {    
    "myapp": "clicky-thing",
    "awesome_widgets": "snarf"
  },
  "lockfile": {
    stuff from lockfile...
  }
}

Pub then effectively runs:

$ pub run bower:install_foreign_dependencies

In other words, it looks for bin/install_foreign_dependencies.dart inside the bower pub package and runs it. It passes in the blob of JSON to that on stdin. That Dart script does a few things:

It does whatever internal magic it needs to do to physically install the foreign dependencies. It is required to put them inside the given "installDirectory". Pub controls the name of this directory. It is always <name of package manager>_packages. It's important that this name is stable because HTML and others assets inside pub packages will contain URLs that walk through that, like ../bower_packages/polymer/some_component.html.

The install_foreign_dependencies script then prints an arbitrary chunk of JSON to stdout. Pub takes that and puts it into the application's lockfile for this package manager. The next time you run pub get, it will take that chunk of JSON and pass it back to the install_foreign_dependencies. This way, the foreign package manager can lock its dependencies too.

Case study: bower

For bower, install_foreign_dependencies would:

• Synthesize a temporary bower.json that contains all of the listed bower packages.
• Add a resolutions map for the locked versions passed from the pubspec.lock file, if any.
• Write the bower.json file to disc.
• Install bower if needed.
• Run bower using the install directory given by pub as the place to install packages.
• Read back the resolutions from the bower.json file.
• Output the resolutions to stdout.
• Discard the temporary bower.json file.

This kind of pub callback could be used for other cases (such as code/asset generation).

What do you think ?

This is a good idea, and something we've considered for other use cases. I don't think it's a great fit here though, because I think it's critical that we roll whatever mojo the other package manager does back into pub's own lockfile. We need to ensure repeatable builds of not just pub packages, but foreign ones too.

We could possible just have post-install hooks and then some API to let you put stuff in the lockfile, but I thought it would be better to do something a little more explicitly structured.