I spent a bunch of this morning talking with Siggy and Nathan about asset-handling in pub. It's still pretty rough in many places, and we're trying to figure out how to smooth that out. The questions that started this (brought up on this thread) were:
- Can you have non-dart files in a lib/ directory?
- If so, where do they end up when you run pub build?
Since, I love transparency and feedback, and because many of the constraints pub operates under aren't obvious, I thought it might be cool to walk through our thoughts.
When we initially created pub, well before we did any work on assets, the only goal of the system was to be able to re-use Dart code.
At the time, Dart only supported relative and absolute URL imports. "package:" didn't exist. So, say you were writing an application and you wanted to reuse someone else's library. How would you import it?
The only answer was, "copy it physically into your source tree and then do a relative import", like:
import "third_party/somelib/thing.dart";
But what if somelib itself was reusing some code? Would you end up with a directory structure like:
myapp/third_party/somelib/third_party/otherlib/other.dart
This is npm's model, where every dependency contains its dependencies, like some fractal tree of packages. I think this is crazy even in JS, but it's totally infeasible in Dart. The problem is shared dependencies.
If you use two packages, and they both reuse the same package, you can't have two copies of that shared dependency in your app. If you do, the Dart VM will think they are different libraries. That means if you pass an instance of SomeType that was created in one of those copies to a method in the other copy that's annotated to take a SomeType, it will fail in checked mode. somelib/third_party/foo/foo.dart's SomeType is a different type from otherlib/third_part/foo/foo.dart's version of it.
We needed a way to "flatten" import paths so that we could handle shared dependencies. To address that, the language designers added the "package:" URL scheme to imports. All that does is let you specify a URL relative a "packages" directory next to the entrypoint. The fact that it's relative to the entrypoint, regardless of where the import appears lets us flatten out imports and make shared imports work.
The fact that it's relative to the entrypoint is also a pain though. To make this work, you need to have a properly filled in "packages" directory next to every entrypoint in your application. If you have lots of subdirectories containing entrypoints, you need a "packages" directory in each of them. That's why pub sprays them all over your application.
I know what you're thinking. Why not just specify an explcit package root when you run the app?
-
If you fail to do it correctly in a command-line app, it fails in a hard to understand way. It's not user friendly if they have to always pass this confusing argument to Dart when they run an app.
-
There's no usable way to specify this in the browser.
So we had to use the default package root, which is next to every entrypoint. We don't want to copy files all over there, so we create "packages" directories that just symlink to the original one. That's important because you may be simultaneously hacking on those other packages. If we physically copied the files, it wouldn't pick up those changes.
Now we have all of these "packages" directories that are full of symlinks to other packages. Great. But there's a tricky corner. One of the packages in there is your own package. This lets you use "package:" imports to refer to stuff in your own package. We think this is important because it means you can make examples and tests that look exactly like an outside user would import your libraries.
If we aren't careful, this creates a symlink cycle where you have a directory that indirectly contains a symlink pointing back to itself. Pub did used to work this way. It worked fine for pub, but it confused lots of other tools (and users).
To fix that, we said the only things you can import go in a separate "lib" directory, and entrypoints do not go there. The symlink to a package links directly to its "lib" directory, not the root of the package. Since "lib" doesn't contain entrypoints, we don't need a "packages" directory in there, so there's no symlink cycle. Swell. (Credit goes to John for coming up with this.)
This works for running your app, but when it comes time to deploying it, you don't want a bunch of symlinks reaching all over your file system. You need a single self-contained directory with everything your app needs.
Now, imagine your app looks like this:
myapp/
web/
main.dart
feature1/
main.dart
subfeature1/
main.dart
subfeature2/
main.dart
feature2/
main.dart
subfeature3/
main.dart
subfeature4/
main.dart
Every one of those "main.dart" files is an entrypoint. Your package, like many, reuses lots of third party packages. When you go do deploy your app, do you really want to ship seven copies of every single package you depend on?
No! That would be totally gross. Fortunately, we have an out. Since Dart isn't in the browser yet, your shipped app is compiled to JavaScript first. That's one of the things pub build does.
Once that's done, we don't need to copy any of the Dart code in your dependencies. The code for them has already been included in your JS output by dart2js. This is why pub build doesn't include .dart files in its output. They've either been compiled to JS, or imported by other .dart files that were compiled to JS.
(Even when browsers do support Dart natively, we'll still have this out. There is an in-progress dart2dart compiler that will take your app and its dependencies and compile them to a single file in the same way that compilation to JavaScript does.)
This is great, it means we don't need any "packages" directories at all.
After we implemented that, we started working on assets: things in packages that aren't Dart files. Those won't get compiled into the JS, and you may need them. We need to include those files in the output somewhere, but where?
The question hinges on, "How are users referring to those assets?" If you have a .css file in some other package, how are you referring to it in your HTML? What if that HTML is in some subdirectory?
What we came up with was "assets". There would be a single "assets" directory at the top of your built output. To refer to an asset in another package, you use a URL that points inside that.
The important bit is that there's only one "assets" directory in the build output. If you're referring to an asset from some file in a subdirectory, it's your job to either do a root-relative URL ("/assets/..."), or walk out of the directories to get to it ("../../assets/...").
This works correctly now in pub build, serve, etc..
This is where it gets tricky. "asset" and "lib" mostly work the same in that they are both "public" directories exposed by a package. The difference is that in your package, there is only a single "assets" directory where you can reach into to find assets.
With "packages", there is one next to each entrypoint. That's OK since we can eliminate those from the build output assuming they only contain Dart code that's been compiled to JS.
Where it gets nasty is if you put assets inside "lib". When you do that, a "packages" URL to find the asset will work if you just open your app using "file:" in the browser. Since pub puts "packages" symlinks in your source tree, the file system can traverse into those and find everything in there, .dart or not.
But when we build your app, we don't want to have multiple copies of all of those assets. Our initial plan was that users would only put .dart files in "lib" and everything else would go in "asset". That would ensure that your app's behavior would be the same in "file:", through pub serve, and pub build.
Unfortunately, when you have things like web components, that doens't make sense. You end up with a handful of files—.dart, .html, and .css—that are deeply related. They logically should be in the same directory.
Since that Dart file is also imported from other packages, it needs to go in "lib". So we end up really wanting to have non-Dart files in "lib".
What we'd ideally do is only have a single "packages" directory. Much like "assets", if you want to refer to stuff in there from a subdirectory, it's your job to walk up to the right directory to get there. You can't just assume like magic that a "packages" directory will follow you wherever you go.
The problem is that the language semantics of "package:" itself make that exact magic assumption. As has been the case for all of pub's history, the Dart language makes our job harder than it would be in other languages.
So after a bunch of discussion, here's a rough long term plan for this. I'll stress that this is long-term. Our immediate plans are to get/keep things stable and add more features needed to consolidate on a single dev experience. (I.e. things like making pub build and serve work on things outside of "web").
-
We want to move towards a single canonical "packages" directory in the build output.
-
We will allow non-Dart files in there.
-
Since this means it's semantically identical to "asset", we'll probably eventually phase "asset" out. "lib" will be the one directory where you put public stuff that your package exposes, Dart file or not.
-
To deal with "package:" imports, pub build (and serve) will simply rewrite those imports to be relative ones that walk up to the canonical "packages" directory.
This way, when you build your app, you only ever get a single copy of any given file that you depend on.
This is during the build process, so it only affects the output of pub build and pub serve. We don't modify the user's original files.
Assuming the editor is nicely integrated with pub build/serve, it should work fine. I should note that this really only affects pub build in debug mode where you want to include the original files.
Pub serve doesn't need to rewrite the imports since it can resolve the URLs directly. And in normal pub builds, the .dart files are excluded from the output anyway.