Better local require() paths for Node.js

better-nodejs-require-paths.md

Better local require() paths for Node.js

Problem

When the directory structure of your Node.js application (not library!) has some depth, you end up with a lot of annoying relative paths in your require calls like:

const Article = require('../../../../app/models/article');

Those suck for maintenance and they're ugly.

Possible solutions

Ideally, I'd like to have the same basepath from which I require() all my modules. Like any other language environment out there. I'd like the require() calls to be first-and-foremost relative to my application entry point file, in my case app.js.

There are only solutions here that work cross-platform, because 42% of Node.js users use Windows as their desktop environment (source).

0. The Alias

1. Install the module-alias package:

   npm i --save module-alias
   

2. Add paths to your package.json like this:

   {
       "_moduleAliases": {
           "@lib": "app/lib",
           "@models": "app/models"
       }
   }

3. In your entry-point file, before any require() calls:

   require('module-alias/register')

4. You can now require files like this:

   const Article = require('@models/article');

1. The Container

1. Learn all about Dependency Injection and Inversion of Control containers. Example implementation using Electrolyte here: github/branneman/nodejs-app-boilerplate

2. Create an entry-point file like this:

   const IoC = require('electrolyte');
   IoC.use(IoC.dir('app'));
   IoC.use(IoC.node_modules());
   IoC.create('server').then(app => app());

3. You can now define your modules like this:

   module.exports = factory;
   module.exports['@require'] = [
       'lib/read',
       'lib/render-view'
   ];
   function factory(read, render) { /* ... */ }

More detailed example module: app/areas/homepage/index.js

2. The Symlink

Stolen from: focusaurus / express_code_structure # the-app-symlink-trick

1. Create a symlink under node_modules to your app directory:
   Linux: ln -nsf node_modules app
   Windows: mklink /D app node_modules

2. Now you can require local modules like this from anywhere:

   const Article = require('models/article');

Note: you can not have a symlink like this inside a Git repo, since Git does not handle symlinks cross-platform. If you can live with a post-clone git-hook and/or the instruction for the next developer to create a symlink, then sure.

Alternatively, you can create the symlink on the npm postinstall hook, as described by scharf in this awesome comment. Put this inside your package.json:

"scripts": {
    "postinstall" : "node -e \"var s='../src',d='node_modules/src',fs=require('fs');fs.exists(d,function(e){e||fs.symlinkSync(s,d,'dir')});\""
  }

3. The Global

1. In your entry-point file, before any require() calls:

   global.__base = __dirname + '/';

2. In your very/far/away/module.js:

   const Article = require(`${__base}app/models/article`);

4. The Module

1. Install some module:

   npm install app-module-path --save

2. In your entry-point file, before any require() calls:

   require('app-module-path').addPath(`${__dirname}/app`);

3. In your very/far/away/module.js:

   const Article = require('models/article');

Naturally, there are a ton of unmaintained 1-star modules available on npm: 0, 1, 2, 3, 4, 5

5. The Environment

Set the NODE_PATH environment variable to the absolute path of your application, ending with the directory you want your modules relative to (in my case .).

There are 2 ways of achieving the following require() statement from anywhere in your application:

const Article = require('app/models/article');

5.1. Up-front

Before running your node app, first run:

Linux: export NODE_PATH=.
Windows: set NODE_PATH=.

Setting a variable like this with export or set will remain in your environment as long as your current shell is open. To have it globally available in any shell, set it in your userprofile and reload your environment.

5.2. Only while executing node

This solution will not affect your environment other than what node preceives. It does change your application start command.

Start your application like this from now on:
Linux: NODE_PATH=. node app
Windows: cmd.exe /C "set NODE_PATH=.&& node app"

(On Windows this command will not work if you put a space in between the path and the &&. Crazy shit.)

6. The Start-up Script

Effectively, this solution also uses the environment (as in 5.2), it just abstracts it away.

With one of these solutions (6.1 & 6.2) you can start your application like this from now on:
Linux: ./app (also for Windows PowerShell)
Windows: app

An advantage of this solution is that if you want to force your node app to always be started with v8 parameters like --harmony or --use_strict, you can easily add them in the start-up script as well.

6.1. Node.js

Example implementation: https://gist.github.com/branneman/8775568

6.2. OS-specific start-up scripts

Linux, create app.sh in your project root:

#!/bin/sh
NODE_PATH=. node app.js

Windows, create app.bat in your project root:

@echo off
cmd.exe /C "set NODE_PATH=.&& node app.js"

7. The Hack

Courtesy of @joelabair. Effectively also the same as 5.2, but without the need to specify the NODE_PATH outside your application, making it more fool proof. However, since this relies on a private Node.js core method, this is also a hack that might stop working on the previous or next version of node.

In your app.js, before any require() calls:

process.env.NODE_PATH = __dirname;
require('module').Module._initPaths();

8. The Wrapper

Courtesy of @a-ignatov-parc. Another simple solution which increases obviousness, simply wrap the require() function with one relative to the path of the application's entry point file.

Place this code in your app.js, again before any require() calls:

global.rootRequire = name => require(`${__dirname}/${name}`);

You can then require modules like this:

const Article = rootRequire('app/models/article');

Another option is to always use the initial require() function, basically the same trick without a wrapper. Node.js creates a new scoped require() function for every new module, but there's always a reference to the initial global one. Unlike most other solutions this is actually a documented feature. It can be used like this:

const Article = require.main.require('app/models/article');

Since Node.js v10.12.0 there's a module.createRequireFromPath() function available in the stdard library:

const { createRequireFromPath } = require('module')
const requireUtil = createRequireFromPath('../src/utils')

requireUtil('./some-tool')

Conclusion

0. The Alias
Great solution, and a well maintained and popular package on npm. The @-syntax also looks like something special is going on, which will tip off the next developer whats going on. You might need extra steps for this solution to work with linting and unit testing though.

1. The Container
If you're building a slightly bigger application, using a IoC Container is a great way to apply DI. I would only advise this for the apps relying heavily on Object-oriented design principals and design patterns.

2. The Symlink
If you're using CVS or SVN (but not Git!), this solution is a great one which works, otherwise I don't recommend this to anyone. You're going to have OS differences one way or another.

3. The Global
You're effectively swapping ../../../ for __base + which is only slightly better if you ask me. However it's very obvious for the next developer what's exactly happening. That's a big plus compared to the other magical solutions around here.

4. The Module
Great and simple solution. Does not touch other require calls to node_modules.

5. The Environment
Setting application-specific settings as environment variables globally or in your current shell is an anti-pattern if you ask me. E.g. it's not very handy for development machines which need to run multiple applications.

If you're adding it only for the currently executing program, you're going to have to specify it each time you run your app. Your start-app command is not easy anymore, which also sucks.

6. The Start-up Script
You're simplifying the command to start your app (always simply node app), and it gives you a nice spot to put your mandatory v8 parameters! A small disadvantage might be that you need to create a separate start-up script for your unit tests as well.

7. The Hack
Most simple solution of all. Use at your own risk.

8. The Wrapper
Great and non-hacky solution. Very obvious what it does, especially if you pick the require.main.require() one.

WARNING! I haven't investigated this thoroughly, but it seems that the symlink approach conflicts with how newer (6.11?) npm works. Now, if I have a symlink node_modules/@app => my_app_src, running npm i finds that @app is not in package.json and follows the symlink, thus removing everything in my_app_src!

I tried to add this to package.json:

"scripts": {
  "preinstall": "rm -f node_modules/@app"
}

This might fix the case of running npm i, but my_app_src still gets deleted in some cases, possibly when I run npm i some-package. Adding node_modules/.hooks/preinstall doesn't seem to help either.

@tuomasslo -- I just ran into this same issue. It seems like the behavior has changed recently and causes npm to unleash rimraf on your project root. The behavior is very destructive and dangerous so I've brought it up on the npm bug community: https://npm.community/t/npm-recursively-deletes-every-file-in-project-root-under-some-conditions/10426

I've switched to the "root": "file://./" line in package.json dependencies for now. This seems to create a symlink which is exactly what I wanted anyway.

@tuomassalo I ran into this issue too. And indeed the preinstall and postinstall hooks work partially. The problem remains when we install additional dependencies (e.g., npm install third-party-lib). The only way I could think of so far, and it seems to work, is creating a script to wrap npm install. Honestly, I don't like this solution very much, but I didn't find a way to run something before/after npm install third-party-lib. So here it goes

// File: install-wrapper.js
const util = require('util');
const exec = util.promisify(require('child_process').exec);
const { spawn } = require('child_process');

async function installWrapper() {
  try {
    console.log('Running preinstall hook');
    await exec('npm run preinstall');
    console.log(`Running npm ${process.argv[2]}`);
    const npmArgs = process.argv.slice(2);
    await new Promise((resolve, reject) => {
      const install = spawn('npm', npmArgs, { stdio: 'inherit' });
      install.on('error', reject);
      install.on('close', code => {
        if (code === 0) {
          resolve();
        } else {
          reject();
        }
      });
    });
    console.log('Running postinstall hook');
    await exec('node ./create-symlink.js');
  } catch (error) {
    console.error(error);
  }
}

installWrapper();

In my package.json

 "scripts": {
    "preinstall": "rm -f node_modules/@app",
    "postinstall": "node ./create-symlink.js",
    "i": "node ./install-wrapper.js install",
    "u": "node ./install-wrapper.js uninstall"
  }

Then to install the dependencies on a fresh clone, I do:

npm install

But to install third-party libs, I should do:

npm run i -- --save node-fetch

And to uninstall

npm run u -- node-fetch

For completeness, I include create-symlink.js

// script taken from this post https://gist.github.com/branneman/8048520#gistcomment-1412502
// This script run as a postinstall hook, it does not confuse IDEs (e.g., intellisense works)
const fs = require('fs');

// the src path relative to node_module
const srcpath = '../src';
const dstpath = 'node_modules/@app';

fs.exists(dstpath, function(exists) {
  // create the link only if the dest does not exist!
  if (!exists) {
    fs.symlinkSync(srcpath, dstpath, 'dir');
  }
});

Hope this helps someone :)

This is the solution I came up with: https://gist.github.com/aprilmintacpineda/66f6095ea7a9228db692a6dc794b19cb

UPDATE: No, the new npm version did not help. Good news! It seems that the newest npm fixes the deletion problem I mentioned above. Most likely related to: https://blog.npmjs.org/post/189613288975/release-6134

Maybe new import-map spec will help on this?

I'd love to see this feature be built in on NodeJS but so far my approach works and it works so well, I've been using it on my apps.

Thanks for an awesome writeup!

NIT: in https://gist.github.com/branneman/8048520#7-the-hack:
fool proof => full proof

This is the solution I came up with: https://gist.github.com/aprilmintacpineda/66f6095ea7a9228db692a6dc794b19cb

This is a different flavor of:
https://gist.github.com/branneman/8048520#3-the-global

This is the solution I came up with: https://gist.github.com/aprilmintacpineda/66f6095ea7a9228db692a6dc794b19cb

This is a different flavor of:
https://gist.github.com/branneman/8048520#3-the-global

Except it's much better because you don't have to manually concat every time you need to require a local module. Plus, it is sure to work on future version of nodes and probably even backward compatible since it doesn't change anything that's built-in to node unlike those hacks others suggest (unless there's a version of node where _require is actually a built-in function). It's also global and can be used on all your modules. And since it's using path module, it should (AFAIK) work on other OS.

I have been using the node_modules symlink option with a postinstall script and it works just fine

The Native Self-Dev-Dependency solution

I've found another cool way to do it with an "npm link" trick:

1. You need to choose a unique name for your package and fill the "name" field of package.json. For example:

"name": "@my-unique/package"

2. Add your package as a "self-dev-dependency" pointing to the root of your project:

"devDependencies": {
   "@my-unique/package" : "."

4. run npm install to let npm creating the link inside node_modules for you

5. Now you can use @my-unique/package inside your code to access to all files starting from the root of your project folder.

const myClass = require("@my-unique/package/src/myClass"); or import myClass from "@my-unique/package/src/myClass"

EXTRA: if you want full support of visual studio code intellisense you should set the "paths" option inside your tsconfig.json or jsconfig.json:

    "paths": {
      "@my-unique/package/*" : [
        "*"
      ]
    }

I think that this method is even more powerful than module-alias since it works also when your package is included as dependency (while module-alias has a limitation for it).

Let me know if it works for you

Another solution based on symlink: Rush/link-module-alias#25

The Native Self-Dev-Dependency solution

I've found another cool way to do it with an "npm link" trick:

1. You need to choose a unique name for your package and fill the "name" field of package.json. For example:

"name": "@my-unique/package"

1. Add your package as a "self-dev-dependency" pointing to the root of your project:

"devDependencies": {
   "@my-unique/package" : "."

1. run npm install to let npm creating the link inside node_modules for you
2. Now you can use @my-unique/package inside your code to access to all files starting from the root of your project folder.

const myClass = require("@my-unique/package/src/myClass"); or import myClass from "@my-unique/package/src/myClass"

EXTRA: if you want full support of visual studio code intellisense you should set the "paths" option inside your tsconfig.json or jsconfig.json:

    "paths": {
      "@my-unique/package/*" : [
        "*"
      ]
    }

I think that this method is even more powerful than module-alias since it works also when your package is included as dependency (while module-alias has a limitation for it).

Let me know if it works for you

Hey Yehonal, I really liked your approach and this was exactly what I was looking for. Following the line of thought that the browser key is offered out-of-the-box and do something very similar, I was thinking that it was very odd to leverage this to an external plugin.

I had some issues with my folder being deleted (even using ~ instead of @) using the link-module-alias package and your proposed solution works like a charm here as well. Thanks!

Simplest, Best Solution

npm install --save-dev basetag
npx basetag link --hook

And then

var Foo = require('$/lib/foo.js');

No need for special requires or code changes, no worries about npm install or npm ci blowing away links.

Just Works™ on Mac, Windows, and Linux.

https://github.com/janniks/basetag

(the way that it works is by running a postinstall script that creates a symlink on Mac and Linux, or a junction on Windows)

Because it's all real files, it also won't break your editor's ability to look up files and such.

Also, you can go back and forth between styles

# require('../../baseball') => require('$/baseball')
npx basetag rebase

# require('$/baseball') => require('../../baseball')
npx basetag rebase

There's another solution: yarn workspaces

Would be great if npm would support workspaces in the same way, so we don't have to install lerna. Not sure if lerna creates symlinks.

@fancydev18 npm@7 now supports workspaces.

https://docs.npmjs.com/cli/v7/using-npm/workspaces
https://github.blog/2020-10-13-presenting-v7-0-0-of-the-npm-cli/

so many solutions i wonder what's the best approach that would be great for my boilerplate?

https://github.com/W2AlharbiMe/express-boilerplate

i was going to add Awilix and avoid using require but after seeing this thread i think i might change this to npm workspaces

module-alias is great with commonJS, but it doesn't work well with ES Module, any suggestions?

Ok, I found a nice alternative for ES Module project

https://nodejs.org/api/packages.html#packages_subpath_patterns

By adding the following to package.json

{
  "imports":{
    "#root/*": "./*",
  }
}

Now you can use in your /sub/folder/test.js

import config from '#root/config.js'

It works for me, hope this helps :)

@branneman On solution 8 "The wrapper", the createRequireFromPath() has been deprecated. See here for deprecation. It suggests using createRequire() instead, which was introduced in v12.2.0. See here for documentation.

i would suggest taking a look into npm v7 workspaces feature
https://docs.npmjs.com/cli/v7/using-npm/workspaces/

it works like a charm for me

thanks @SeanDunford

@W2AlharbiMe i'm not really fond of having to make a package.json inside every folder that i want to turn into a workspace. Do you find it annoying at all?

@W2AlharbiMe i'm not really fond of having to make a package.json inside every folder that i want to turn into a workspace. Do you find it annoying at all?

no in fact it has more benefits for monorepos take a look here
https://dev.to/limal/simplify-your-monorepo-with-npm-7-workspaces-5gmj

i'm currently using it here
https://github.com/W2AlharbiMe/fastify-boilerplate

(for Yarn) How about: https://yarnpkg.com/features/protocols#why-is-the-link-protocol-recommended-over-aliases-for-path-mapping ?
Just define it in dependencies as such

"dependencies": {
  "src": "link:./src"
}

My IDE was easilly able to parse my linked src folder with zero configuration

(for Yarn) How about: https://yarnpkg.com/features/protocols#why-is-the-link-protocol-recommended-over-aliases-for-path-mapping ?
Just define it in dependencies as such

"dependencies": {
  "src": "link:./src"
}

My IDE was easilly able to parse my linked src folder with zero configuration

NICE, thanks for sharing

Another ugly solution (only for ESM modules) came with v16.12.0

Documentation

Here's the simplest example I personally use:

// my_project/loader.js
import path from 'path';
import fs from 'fs';
import { fileURLToPath } from 'url';

export async function resolve(specifier, context, defaultResolve) {
  if (specifier.startsWith('/')) {
    // try resolve absolute path imports relative to
    // project root (closest "package.json" location)
    let root = context.parentURL ? path.dirname(fileURLToPath(context.parentURL)) : process.cwd();
    while (root !== '/' && root !== '.') {
      try {
        await fs.promises.access(path.resolve(root, 'package.json'), fs.constants.R_OK);
        specifier = path.resolve(root, specifier.replace(/^\/+/, ''));
        break;
      } catch {
        root = path.dirname(root);
      }
    }
  }
  return defaultResolve(specifier, context, defaultResolve);
};

To make use of the loader you have to run node with flag:

cd my_project
node --experimental-loader ./loader.js start.js

And now

// my_project/deep/nested/script.js
import util from '/lib/util.js'; // -> resolves to my_project/lib/util.js

Or you can implement any other kind of resolution logic.

PROS: Native, no deps, full control, completely customizable
CONS: --experimental-loader flag, API is subject to change, only for ESM modules, possible side-effects??

Great post. Thanks.

If the project support .env file, I use NODE_PATH solution (5) , like this:

.env:

NODE_PATH=.
...

Thanks for your sharing