Breakdown of How Require Extensions Work

How_Require_Extensions_Work.md

Why

Doing require extensions correctly is essential, because:

1. Users should be able to install multiple extensions in succession, and have them work together.
2. Coverage tools like nyc need it to reliably supply coverage information that takes into account sourcemaps from upstream transforms.
3. Because non-standard, un-predictable behavior causes hard to solve bugs, and major headaches for project maintainers.

What is a require extension anyways?

First, it's worth remembering what default ".js" extension does.

require.extenstions['.js'] = function (module, filename) {
  var content = fs.readFileSync(filename, 'utf8');
  module._compile(internalModule.stripBOM(content), filename);
}

Really simple. It reads the source content from the disk, and calls the module._compile. The default module._compile is a non-trivial piece of code, for simplicity I will just say it is what actually compiles the code.

---

Our first custom extension

Now let's install a transform that just appends the code + "bar" to the end of every file (humor me as I keep things simple).

This is how you would manually create that hook (in what is now widely accepted as the "right" way).

// append-bar.js
var oldHoook = require.extensions['.js'];                 // 1
require.extensions['.js'] = function (module, file) {     // 2
  var oldCompile = module._compile;                       // 3
  module._compile = function (code, file) {               // 4
    code = code + ' + "bar"';                             // 5            
    module._compile = oldCompile;                         // 6
    module._compile(code + ' + "bar"');                   // 7
  };
  oldHook(module, file);                                  // 9
});

Note that this extension never reads from the disk. That is because the first extension in the chain (the system default one) handles loading from disk. If it's not obvious why that's true (it wasn't for me), keep reading.

The really important takeaway here is that you should be implementing require extensions almost exactly as I have above. There are multiple levels of indirection, and it can be confusing. Libraries like pirates can simplify the process.

Breakdown with 1 Custom Extension

Here is what happens when you call require("./foo.js")

// foo.js
module.exports = "foo"

What happens inside require boils down to this:

function pseudoRequire(filename) {
  var ext = path.extname(filename); // ".js"
  var module = new Module();
  require.extensions[ext](module, filename);
}

Now let's step through the sequence of events.

1. The system calls require.extensions['.js'](module, './foo.js').
   This means append-bar is invoked with (module, './foo.js')
2. append-bar stores a reference to module._compile (line 3), an with its own wrapper function (line 4).
   module._compile refers to the append-bar wrapper function. append-bar's reference to originalCompile refers to the actual compile implementation.
3. append-bar calls it's oldHook (the default .js extension) with the modified module and filename (line 9).
4. The default .js extension reads in the source (module.exports = "foo"), and calls module._compile(source, filename). Remember module._compile currently points to the append-bar wrapper function.
5. The append-bar wrapper adds + "bar" to the source (Line 5). The source is now module.exports = "foo" + "bar".
6. The append-bar wrapper now replaces module._compile with it's originalCompile reference (Line 6). module._compile now points to the actual compile implementation
7. module._compile is called again (this time pointing to actual, and the source is evaled and we get our result "foobar".

Breakdown with 2 Custom Extension

Assume we have first added the append-bar extension from above, followed by another called append-quz (which is for all purposes identical, except it appends baz instead.

1. We install the append-bar extension (replacing the original hook) append-bar#originalHook points to the original hook.
2. We install the append-quz extension append-quz#originalHook points to the append-bar hook.
3. We call require('./foo.js');
4. append-quz hook is called with (module, './foo.js'), it replaces module._compile with it's wrapper function. append-quz#originalCompile points to the actual compile module._compile points to the append-quz wrapper.
5. append-quz calls it's originalHook reference, which is append-bar.
6. append-bar replaces module._compile with it's wrapper. append-bar#originalCompile points to append-quz wrapper. module._compile points to the append-bar wrapper.
7. The original extension is called, which loads the source from disk and calls module._compile('module.exports = "foo"', './foo.js')
8. At this point module._compile points to the append-bar wrapper, so the source is appended with + "bar".
9. The append-bar calls the originalCompile reference (which is the append-quz wrapper).
10. The append-quz wrapper does it's appending (so we've now got "foo" + "bar" + "quz")
11. append-quz calls it's originalCompile reference, which is actual and we get "foobarquz"

Hey thanks for the great explanation

module._compile(code + ' + "bar"'); // 7
should be module._compile(code , file); // 7

Deprecated In the past, this list has been used to load non-JavaScript modules into Node.js by compiling them on-demand. However, in practice, there are much better ways to do this, such as loading modules via some other Node.js program, or compiling them to JavaScript ahead of time - https://nodejs.org/api/modules.html#modules_require_extensions

In the Node's documentation it says that there are better ways to achieve on-the-fly compilation. What are the other alternative techniques used to compile modules on runtime that do not depend on Node`s require hook ? Or does all tools use the require hook (e.g. Babel register 7) ?

@myuseringithub - Babel v7 and several other projects use https://github.com/ariporad/pirates, which is a package that uses the same technique as outlined above. For ES modules the documentation describe loader hooks but I can't see an obvious documented replacement for require.extensions.

what is the best way to modify main fields order to consume a package module instead of a package main field?