Skip to content

Instantly share code, notes, and snippets.

@getify
Last active March 23, 2024 16:16
Show Gist options
  • Save getify/ea3c048c26e95f8797448eac8a82ce1a to your computer and use it in GitHub Desktop.
Save getify/ea3c048c26e95f8797448eac8a82ce1a to your computer and use it in GitHub Desktop.
describing a bundling question in detail

I'm the author of an npm package that comes as a single ESM-format module, let's call it "A". This package is only a client-side (browser) library, it will not work in Node -- it interacts with client-only web platform APIs.

The "A" package doesn't use any typical build tools (typescript, webpack/vite, etc). But it does include a simple "publish-build" script that's used at npm publish time to prepare a dist/ directory for the package.

The package relies on three npm package dependencies (call them "B", "C", and "D"), which I do not own/control. These 3 packages only distribute themselves as plain .js window-global style scripts. They cannot be imported, because (unfortunately) they make assumptions about global-scope this (being window), non-strict mode, etc. That also means they can't just be inlined into the "A" distribution module file.

Moreover, these dependencies are big enough (and potentially useful enough) that a user of "A" might also want to access and use "B", "C", or "D" functionality directly (without just re-including them), so it's actually preferable that they expose themselves (as designed) on the window global.

The "A" publish-build tool copies the relevant distribution files from these "B", "C", and "D" dependencies into its dist/ folder, alongside the "A" distribution module file.

No Bundler Involved?

I do not want to assume (or require!) that someone's client-side web app is using a bundler tool (webpack, vite, etc). So if they are just doing a vanilla web app with no such tooling, they can just copy the 4 dist/* files (A.js, B.js, C.js, and D.js) into their JS assets folder.

To make sure that B.js, C.js, and D.js are loaded, they can do so manually in their HTML:

<script src="/path/to/B.js"></script>
<script src="/path/to/C.js"></script>
<script src="/path/to/D.js"></script>

Alternatively, I provide a "script loader" snippet that will dynamically inject <script> elements for these dependencies.

Either way, then they would import "A" module into their client app code like this:

// if using a client-side import-map:
import { some, thing } from "A";

// otherwise:
import { some, thing } from "/path/to/A.js";

This all works fine and great. But my problem is...

Bundler In Use?

Though I'm not requiring/assuming bundlers for consumers of "A", I do want to be "bundler friendly" if possible, in case they are using such. So let's now assume they will use such tooling.

They import the "A" dependency into their client-side app code like this:

import { some, thing } from "A";

And the package.json file for "A" includes these fields:

{
    "exports": {
        ".": {
            "browser": "./dist/A.js"
        }
    },
    "browser": {
        "A": "./dist/A.js",
        "./A.js": "./dist/A.js",
        "./B.js": "./dist/B.js",
        "./C.js": "./dist/C.js",
        "./D.js": "./dist/D.js"
    },
}

That's enough config that tools (e.g., vite, webpack) seem to find and include the A.js module code into the bundle just fine.

But how to get the tools to include B.js, C.js, and D.js into a bundle as well? Ideally, a bundler sees those entries in package.json and includes them. Perhaps webpack does that (not sure?), but from I can tell, vite does not pick up on those entries. So something else is needed.

Need Split Bundling, Though!

Recall that "A" doesn't use any bundler. So if "A" just had these in it:

import "/path/to/B.js";
import "/path/to/C.js";
import "/path/to/D.js";

...that would be "broken" in that those imports of non-ESM modules actually cause runtime errors, since the code in "B.js", "C.js", and "D.js" is not designed to run inside of ESM. Indeed, vite dutifully tries to inline them into its main app ESM bundle, which seems like it's solving the "discovery" problem (from above), but ultimately is broken when executed.

Vite doesn't seem to have a way to figure out that those dependencies need to be treated as non-ESM for bundling purposes. Obviously, I can't require a client-side web app to contort itself into using vite's "library mode" just so it can output UMD style code for the purposes of fitting in with the "B", "C", and "D" dependencies.

Ideally, vite would just smartly detect and produce a non-ESM bundle (if needed) alongside the main ESM app bundle, but I don't think it can do that, at least not out-of-the-box.

So instead, it looks like just telling authors to include this in their markup is the only reasonable way, and that feels super terrible to me:

<script src="/path/to/B.js"></script>
<script src="/path/to/C.js"></script>
<script src="/path/to/D.js"></script>

Ugh!

I would strongly prefer to suggest a snippet of vite-config they could drop in, that would be like a include directive to tell vite to include the contents from B.js, C.js, and D.js files (from the dist/ directory) in a separate/split non-ESM bundle.

Doesn't seem to be a clean way of doing this. But am I missing something?

I can't just ask the web app authors to change their bundler, or re-design their whole build process, just to use "A".

@bitifet
Copy link

bitifet commented Mar 19, 2024

I'm a little overwhelmed with so much information to process (from our conversation in twitter) since I'm coming from the other side of the problem (serving modules to client without bundling).

Context

I do use bundlers in several projects and I think they are a great tool but, since the arrival of http/2 and http/3, if the only reason is network performance, maybe they are not the way to go. Specially for small or medium projects. (I may be wrong though).

And, as you said you don't want to require using a bundler, I wish to point out that not using a bundler does not imply not using a package manager: Just placing the files at an assets folder condemn them to become outdated soon...

Some people just place an additional package.json for the client code and then (if server is also in JS) "cd" forth and back to (double) execute npm install / upgrade / audit... But I prefer to have a single package.json for the whole project and then serve the needed files from node_modules.

Until now, to do so one needed to either serve the whole node_modules (ok: no secrets there –unless you use private packages–2 but not so polite...) or "manually" hand-wire express (or whatever lib/framework you use) for every JS file you need (if there was a better way to do so, please let me know...). Both approaches also at the risk that future versions may change those files name or path...

That's why as soon I discovered the browser field I was so excited so, after a little unsuccessful research of anything similar, I implemented ESMrouter (and even forked express-generator so that now, when I want to "get the hands dirty" in something, I just need to type a single command and npm install whatever packages I need. But that's another history...).

This way, whenever I install any NPM package which has the browser field defined (or that I specify in a include option to manually add packages that lack it), its browser entry point is transparently served at /node_modules/<package_name> and proper entry is added to the auto-generated importmap.

Notes

Thank you again for pointing me to the defunctzombie's package-browser-field-spec document but, as I've just explained in the fore mentioned twitter thread, looks like it is focused in the usage of the browser field by bundlers but, from my point of view, what NPM Docs says about it is:

  • browser field is mandatory (not optional) for packages expected to work client side.
  • main field must not be provided in client-side only packages.

Original text: If your module is meant to be used client-side the browser field should be used instead of the main field. This is helpful to hint users that it might rely on primitives that aren't available in Node.js modules. (e.g. window)

That being said, defunctzombie's package-browser-field-spec does not look incompatible with that description:

  • In the basic spec, it works seamlessly: If browser field is provided, it is used and, if not, for a bundler it's reasonable to fail back to the main field.
  • In the advanced spec, The package still have the browser field (so it can be recognized as a browser-suitable package) even it makes things a little bit harder to me (for ESMrouter) since I should parse it, add new routes and importmap entries, etc...

The only drawback I see in the advanced spec is that the main field become mandatory which, from my point of view, it shouldn't since it will seem that they're expected to work also server side which not always will be the case.

…Anyway this is not a problem for me (and may be easily fixed specifying an alternate "main" key inide the browser object itself to override -and perhaps omit- the top-level main field). Also maybe I should report this at defunctzombie's package-browser-field-spec and not here. But I think it's worth to mention for the sake of having the whole photo.

Observations

Now, regarding to your question (just in case my humble opinion could be useful...):

  • If your library depend on other package that may be occasionally imported independently client side, that package should be (ideally) already an ESM module so that both bundlers and ESMrouters would bundle / serve by themselves and you don't even need to mention them in the browser map.
  • If they're not already ESM, most times they could be converted through bundlers like rollup. But, of course, it will take a while until, if the author is so gentle to add an ESM build so, in this case, maybe the best solution is to publish our own transitional version (call it "<package_name>_esm") wrapping the original. If we don't want to pollute NPM's registry we may just publish as a github repo and npm install it.
  • «But how to get the tools to include B.js, C.js, and D.js into a bundle as well?»
    • Not sure if I get you here... Bundler's detect when your code requiere/import a file and then include it in the bundle (you already know it) if it's not working with your browser field mappings I'd say that the bundler doesn't support the browser field already... (But I don't know...).
  • How bundlers handle non-ESM files is up to the bundler itself (some support multiple formats, some not... I'm not an expert though...). But, as I said, if they're dependencies of your ESM package (other npm packages your package depend on), from my point of view, they "should" already be ESM packages. Otherwise, if they can be bundled or not will depend on if the bundler that is importing your package support other formats or not. But, anyway, if it doesn't support them, they couldn't be imported independently either....

Not sure if I helped you or worried even more (hope not... 😉).

References

@getify
Copy link
Author

getify commented Mar 23, 2024

Update: I came to the conclusion this wasn't possible to do (with vite or webpack) via only config options. So I wrote plugins for both vite and webpack. :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment