modules-harmony.md

Modules Harmony Simplification Effort

Disclaimer: This text is exclusively focused on the existing capabilities of the language, and does not weight into additional capabilities like wasm, or virtual modules, or compartments. Those can be discussed later on.

MVP to create a module graph

The bare minimum mechanism to create a module graph that matches ESM requires few main pieces:

1. A portable (serializable) structure that represents the source text of the module. ModuleSource must be portable across processes and across realms.
2. A realm based Module that closes over a ModuleSource.
3. A kicker that can trigger the existing module linkage mechanism to populate the module graph of the corresponding realm. This kicker is the dynamic import() syntax.

With these 3 pieces in place, a module graph can be constructed in user-land.

The gist of the proposal

1. ecma262 to introduce two new intrinsics: ModuleSource and Module.
2. ecma262 to extends the semantics of import() statements so that the first argument can be a Module that can be used to create the corresponding Source Text Module Record.
3. Minor modifications on ecma262's module mechanics so a Source Text Module Record derived from a Module can delegate the resolution of its dependencies and meta object to user-land code associated to the Module itself.
4. No modifications on the behavior of the host.

Interfaces

ModuleSource

interface ModuleSource {
  constructor(source: string);
}

Semantics: A ModuleSource give you no powers. It is a mere representation of a source text with no meta information attached to it.

Note 1: This represents a solution to the eval and CSP, where you have a betted source text available for evaluation without violating the unsafe-eval CSP rules.

Note 2: A ModuleSource can be reused to create multiple Modules associated to it.

Note 3: A ModuleSource could be propagated to other realms and processes via structuredClone or callable boundary wrapping mechanism.

Module Instance

type ImportHook = (specifier: ImportSpecifier, meta: object) => Promise<Module>;

// Module reifies an entangled pair of module environment record
// and module exports namespace from a particular array of bindings
// that correspond to the `import` and `export` declarations of a
// module.
interface Module {
  // Creates a module instance for a module source.
  // The Module to be bound to the realm associated to the Module constructor.
  // The ModuleEnvironmentRecord to be bound to the `Module` constructor.
  constructor(
    source: ModuleSource,
    importHook: ImportHook,
    importMeta: Object,
  );

  readonly source: ModuleSource,

  #namespace: ModuleExportsNamespace;
  #environment: ModuleEnvironmentRecord;
}

Semantics: A Module has a 1-1-1-1 relationship with a Environment Record, a Module Record and a Module Namespace Exotic Object (aka namespace).

Examples

Import Kicker

Any dynamic import function is suitable for initializing, link and evaluate a module instance and all of its transitive dependencies.

const source = new ModuleSource(``);
const instance = new Module(source, importHook, import.meta);
const namespace = await import(instance);

Module Idempotency

Since the Module has a bound module namespace exotic object, importing the same instance should yield the same result:

const source = new ModuleSource(``);
const instance = new Module(source, importHook, import.meta);
const namespace1 = await import(instance);
const namespace2 = await import(instance);
namespace1 === namespace2; // true

Reusing ModuleSource

Any dynamic import function is suitable for initializing a module instance and any of its transitive dependencies that have not yet been initialized.

const source = new ModuleSource(``);
const instance1 = new Module(source, importHook1, import.meta);
const instance2 = new Module(source, importHook2, import.meta);
instance1 === instance2; // false
const namespace1 = await import(instance1);
const namespace2 = await import(instance2);
namespace1 === namespace2; // false

Intersection Semantics with Module Blocks

Proposal: https://github.com/tc39/proposal-js-module-blocks

In relation to module blocks, we can extend the proposal to accommodate both, the concept of a module block instance and module block source:

const instance = module {};
instance instanceof Module;
instance.source instanceof ModuleSource;
const namespace = await import(instance);

To avoid needing a throw-away module-instance in order to get a module source, we can extend the syntax:

const source = static module {};
source instanceof ModuleSource;
const instance = new Module(source, importHook, import.meta);
const namespace = await import(instance);

Intersection Semantics with deferred execution

The possibility to load the source, and create the instance with the default importHook and the import.meta of the importer, that can be imported at any given time, is sufficient:

import instance from 'module.js' deferred execution syntax;
instance instanceof Module;
instance.source instanceof ModuleSource;
const namespace = await import(instance);

If the goal is to also control the importHook and the importMeta of the importer, then a new syntax can be provided to only get the ModuleSource:

import source from 'module.js' static source syntax;
source instanceof ModuleSource;
const instance = new Module(source, importHook, import.meta);
const namespace = await import(instance);

This is important, because it is analogous to block modules, but instead of inline source, it is a source that must be fetched.

Intersection Semantics with import.meta.resolve()

Proposal: whatwg/html#5572

const importHook = (specifier, meta) => {
    const url = meta.resolve(specifier);
    const response = await fetch(url);
    const sourceText = await.response.text();
    return new Module(sourceText, importHook, createCustomImportMeta(url));
}

const source = new ModuleSource(`export foo from './foo.js'`);
const instance = new Module(source, importHook, import.meta);
const namespace = await import(instance);

In the example above, we re-use the ImportHook declaration for two instances, the source, and the corresponding dependency for specifier ./foo.js. When the kicker import(instance) is executed, the importHook will be invoked once with the specifier argument as ./foo.js, and the meta argument with the value of the import.meta associated to the kicker itself. As a result, the specifier can be resolved based on the provided meta to calculate the url, fetch the source, and create a new Module for the new source. This new instance opts to reuse the same importHook function while constructing the meta object. It is important to notice that the meta object has to purposes, to be referenced by syntax in the source text (via import.meta) and to be passed to the importHook for any dependencies of ./foo.js itself.

FAQ

Should importHook be synchronous or asynchronous?

When a source module imports from a module specifier, you might not have the source at hand to create the corresponding Module to be returned. If importHook is synchronous, then you must have the source ready when the importHook is invoked for each dependency.

Since the importHook is only triggered via the kicker (import(instance)), going async there has no implications whatsoever. In prior iterations of this, the user was responsible for loop thru the dependencies, and prepare the instance before kicking the next phase, that's not longer the case here, where the level of control on the different phases is limited to the invocation of the importHook.

Can cycles be represented?

Yes, importHook can return a Module that was either import() already or was returned by an importHook already.

Idempotency of dynamic imports in ModuleSource

Any import() statement inside a module source will result of a possible importHook invocation on the Module, and the decision on whether or not to call the importHook depends on whether or not the Module has already invoked it for the specifier in question. Basically, this means a Module most keep a map for every specifier and its corresponding Module to guarantee the idempotency of those static and dynamic import statements.

Can the subgraph be intertwine with the global module graph?

It certainly can by a) extending the signature of the ImportHook API to allow returning a Module instance of a Module Namespace Exotic Object as described below:

type ImportHook = (specifier: ImportSpecifier, meta: object) => Promise<Module | Namespace>;

This will basically allow developers to load a source, but still delegate to the UA's resolution for its dependencies by using the dynamic import form, e.g.:

const importHook = (specifier, meta) => {
    const url = meta.resolve(specifier);
    return import(url); // or Module.get(await import(url));
}

const source = new ModuleSource(`export foo from './foo.js'`);
const instance = new Module(source, importHook, import.meta);
const namespace = await import(instance);

Or b) implements a reflection mechanism, e.g.: Module.get(ns) that can reify a module instance from a Module Namespace Exotic Object, which is analogous but less ergonomic.

This solves the issue of bundlers trying to create a bundle that contains or defines external dependencies to be loaded by the UA rather than pack them all together in one single bundle.

Note: when delegating to the UA, you not longer have the ability to intercept resolution of dependencies, meaning cycles can't be created where some instances are handled by the UA, and some other are handled in user-land.