`module.exports =` and ES6 Module Interop in Node.js

interop.md

module.exports = and ES6 Module Interop in Node.js

The question: how can we use ES6 modules in Node.js, where modules-as-functions is very common? That is, given a future in which V8 supports ES6 modules:

• How can authors of function-modules convert to ES6 export syntax, without breaking consumers that do require("function-module")()?
• How can consumers of function-modules use ES6 import syntax, while not demanding that the module author rewrites his code to ES6 export?

@wycats showed me a solution. It involves hooking into the loader API to do some rewriting, and using a distinguished name for the single export.

This is me eating crow for lots of false statements I've made all over Twitter today. Here it goes.

Case 1: require + ES6 export syntax

Problem

Given this on the consumer side:

require("logFoo")();

and this ES5 on the producer side:

module.exports = function () {
  console.log("foo");
};

how can the producer switch to ES6 export syntax, while not breaking the consumer?

Answer

The producer rewrites logFoo's main file, call it logFoo/index.js, to look like this:

export function distinguishedName() {
  console.log("foo");
};

Then, the following hypothetical changes in Node.js make it work:

• require is rewritten to look at logFoo/package.json and sees an "es6": true entry.
• It then switches to loading logFoo/index.js with the ES6 module loader API.
• Once the ES6 module loader API has given the results back, it plucks off the distinguishedName property and returns that to the caller of require.

This means require("logFoo")() will work, since require retrieves the distinguishedName export of logFoo/index.js.

Case 2: import + Node.js module.exports = syntax

Problem

Given this ES5 on the producer side:

module.exports = function () {
  console.log("foo");
};

and this ES5 on the consumer side:

require("logFoo")();

how can the consumer switch to ES6 import syntax, while not demanding that the consumer rewrite his code to accomodate yet-another-module-system?

Solution

The consumer rewrites his code as

import { distinguishedName: logFoo } from "logFoo";
logFoo();

Then, the following hypothetical changes in Node.js make it work:

• The default ES6 module loader API is overriden to intercept any module loads
• It sees the module identifier string "logFoo", goes to look at logFoo/package.json, and sees no entry of the form "es6": true.
• It reads logFoo/index.js into memory, and executes it in a special context.
• Once execution is done, it sees that module.exports now has a value in this context.
• It creates a new module object with a single property, distinguishedName, whose value is filled out by pulling module.exports out of this context.
• It returns this new module object back as the imported module.

This means import { distinguishedName: logFoo } from "logFoo" will work, since the module loader API ensures distinguishedName exists before importing.

Conclusion

Elegant? No. Considerate of Node idioms? No. But does it work? Yes.

With a solution like this, you can interoperably use require on ES6 modules and import on ES5 modules, even in the function-module case. And the burden is entirely on the ES6 user to twist his code into awkward shapes, which is as it should be: updating 22K+ and growing packages is not an acceptable path forward.

If it makes you feel better, replace distinguishedName with _ or $ or module or export throughout.

This is the best point that I really don't think TC39 gets yet:

And the burden is entirely on the ES6 user to twist his code into awkward shapes, which is as it should be: updating 22K+ and growing packages is not an acceptable path forward.

updating 22K+ and growing packages is not an acceptable path forward

… Node is the only place ES6 modules could be used natively in the next ~5-10 years.

@substack hopefully, after today, they do :)

Many of those packages are just compiled or wrapped into Node/CommonJS modules. You know how much JS code is written to work in global scope or a single closure? It's a lot easier to port those to ES6 modules.

I see this ES6 and TC39 stuff really close to this sentence:

being able to speak doesn't mean you are intelligent

How about dropping all this syntax breaking stuff, and let developers decide? 'cause I start thinking this might be the way, as it was for de-facto standards such innerHTML has been, which works everywhere down to IE4!

In that case, standards people adopted it and modules might be really a similar matter, right?

I cannot really wait for modules to be out there, but if this is going to be incompatible with everything we really want, know, need, and use already these days, a big WTF is all I can think about moving forward the Web in these terms: MEH!

They are giving developers choice... You can stick to the existing status quo which you and others have claimed is good enough. Current solutions won't break.

Some of us really want/need the new capabilities provided by ES6 modules. Features which aren't possible without the new spec. Aren't you really saying that you want to limit the choice to only the status quo?

Thanks @domenic this was a really interesting read. It's the final case that truly sucks though. The case where the producer and consumer are both ES6 should be the wonderful case where everyone's happy and instead it's:

export function distinguishedName() {
  console.log("foo");
};

and then

import { distinguishedName: logFoo } from "logFoo";

If ES6 modules are going to be better than node.js modules (which they need to be) then the case of 'single export, single import' needs to be elegant once both parties are on ES6:

export function(){
  console.log("foo");
};

then

let logFoo = import "logFoo";

Don't be so quick to swallow the crow.

Making this work requires several non-trivial changes to a section of node where stability is much more valuable than just about anything else. It's very easy to break every node program by making changes to the module loader. That's why we don't do it. What we have works great. Until and unless very significant improvements can be made to Node (in performance or other radically desirable features), we're not going to make any changes to node-core for any new module system that comes down as part of ES6.

Even when it is implemented in V8, what will be the benefit for us?

It's probably not performance. The module loading step is already rather slow, blocking, and only happens once at program startup. So who cares? But if the speed of the overall program is reduced by wrapping modules in a different sort of boilerplate, well, then that's a very very big deal. (This is why, for example, we cannot use the with(knownObject){$code} trick to provide access to the local activation context -- it prevent V8 from optimizing anything fully.) How long until V8 actually optimizes new ES6 modules effectively? (My guess: at least years.)

As for features, we clearly don't need them, and this can be explored in userland anyway.

So, really what's in it for node to perform this kind of surgery? I don't really see the benefit.

Maybe in a few years, when ES6 actually has modules in some sort of a spec, and they're implemented in V8, and they're already popular enough in the browser world for V8 to optimize them fully, some descendent of node can build something interesting using them. But at least in the near term, none of this is realistic.

If all TC39 did was to write CommonJS or AMD into the spec then people would complain that it "adds" nothing. Those standards exist on their own and are in use today and will remain in use for a long time (I'm betting). The ES6 modules are intended to be a better way forward rather than trying to cobble something together that "works" semi-cleanly accross what developers and implementors have already done.

@ForbesLindesay Exactly. If ES6 modules aren't sufficiently superior to node modules then node will never adopt them. As it stands, ES6 can't even make something as good as what node has even though ES6 can change the language and node can't. That is very sad.

Awesome illustration, @domenic.

+1 @ForbesLindsay and @substack

Fwiw, this is also a problem for the thousands of existing AMD modules that export a function, but I fear a solution for AMD will be even worse since the browser-side loader can't efficiently poke around for package.json files.

[ninja edit] An AMD solution could more efficiently be solved via intermediary modules. volo does something similar for CJS modules at package installation time. Still, this is not really very "efficient" imho.

I posted this to @domenic on twitter, figured I'd share here...

How to test for try/catch support (Jul 23 2005)

...try/catch was introduced into the language in 1999 and 6 years later, developers were still struggling with support. Back then release cycles were slow—often several years between updates to JS runtimes (ie. browsers), but that's not the case anymore, so I remain optimistic and I hope that everyone else will try to do the same.

With respect, @rwldrn, this is different.

With the try-catch situation, the vast majority of devs just waited for IE5 to die out before using try-catch. In this case, we've got to implement work-arounds in order for our current ES5 modules to inter-operate with ES6 modules.

We're entitled to bitch at least a little bit.

@rwldrn I'm optimistic that if we end up with a module system that's better it may eventually catch on even if it has almost zero interoperability with current modules/module systems. The problem is that it it's current state, I hope it doesn't catch on. If it fails completely then at least we still have CommonJS. CommonJS is a far cry from perfect, but it does work, and it's better than the current ES6 spec.

Personally I think on the producer side, CommonJS is pretty much perfect. You can write modules that check for support of CommonJS and export a global if it's not there. This is something I see no way to do with ES6 modules.

On the consumer side, the syntax import * from foo is ugly, and so is import {baz, boz} from foo and so is import foo from foo we have destructuring assignment in ES6:

foo.js:

module.exports = {
  baz: 'baz',
  boz: 'boz'
};

consumer:

let foo = import foo;
let {baz, boz} = import foo;

When for some reason you think import * from ... is a good idea (incidentally it isn't):

with (import foo) {
}

If you don't like that because it doesn't work in strict mode remember why it was taken out. The only valid reason to use with is in templating (EJS, Jade etc.) and the only other place I'd ever consider it is when writing a bit mathematical expression:

with (Math) {
  var res = pow(sin(5), 2) + pow(cos(5), 2)
}
assert.aproximatelyEqual(res, 1);

But that's probably a bad idea on my part. I should probably write those out in full.

@ForbesLindesay the problem with the expression form is that this should not work:

if (false) {
  let foo = import "foo";
}

Quick tip for everyone reading this:
var RSVP = require('rsvp') -> module RSVP from 'rsvp'

I assume this discussion is what led to the export default syntax being added to the ES6 module system.

Im sorry. late to this, but is the new module loading actually better.
I particularly dont care for the import from syntax

import x from a
x = require("a/x");

Split code into files, keep your code size down, and manageable.

Just my late 2 cents

I think there's a mistake.

…while not demanding that the producer rewrite his code…

Hi, and what would replace those lines?:

x = require("a/x");
y = require("b/y")(x);

I am using something like this:

import x from 'a';
import y from 'b';

But when 'b' wants to use 'x' it fails.

Thanks

ES6 specification doesn't seem to support the dynamic loading of modules:

function renderView(viewName) {
    const view = require(path.join('./views', viewName));
    ...
}

Has anything happen within node to support ES modules?

@kruncher dynamic module loads go through System

function renderView (viewName) {
  System.import(path.join('./views', viewName))
    .then(view => ...)
    .catch(error => console.error(error.stack));
}

@balupton See https://nodejs.org/en/docs/es6/ ... tl;dr --es_staging flag turns it on.

@jsg2021 @balupton

V8 has parsing support, but not runtime support. From the few times I've asked the V8 devs themselves, they said they're waiting on the WHATWG Loader spec (which will be the ES6 loader spec) to solidify enough for them to reliably figure out what hooks they need to put in for Chrome/Opera and Node/etc. But as it stands, it isn't even complete enough for a tentative attempt at a polyfill yet.

Also, it'll be relatively difficult to merge that in with the current module loader for Node, as Node's is currently synchronous. It'll require some highly inelegant C++ at first (JS wouldn't work), which will likely have to run largely outside of the event loop just to ensure that require calls remain synchronous, while import ... remains asynchronous, required by the Loader spec.