Dependency Resolution in CLJS+JS Projects

Part 1. Problem Definition

Dependency resolution and installation (provision) of mixed CLJS (maven) and JavaScript (npm) projects.

A Tangled Web

Lets consider a relatively simplified example in the following diagram:

+--------------+
|[email protected] +<--------------------------+
|from npm      |                           |
+-------+------+                           |
        ^                                  |
        |                                  |depends on via package.json peerDependencies
        |depends on via deps.cljs          |
        |                                  |
        |                                  |
+-------+-------+                  +-------+------------+
|[email protected] |                  |[email protected] |
|from maven     |                  |from npm            |
|               |                  |                    |
+-------+-------+                  +--------+-----------+
        ^                                   ^
        |                                   |
        |depends on via project.clj         |depends on via deps.cljs
        |                                   |
        |                                   |
        |        +--------------+           |
        |        |My Application|           |
        +--------+              +-----------+
                 |              |
                 +--------------+

'My Application' declares two dependencies:
1. [email protected] via its own my-app/project.clj/:dependencies
2. [email protected] via its own my-app/src/deps.cljs/:npm-deps which is equivalent to my-app/package.json/:dependencies
[email protected] declares many dependencies, but for our example only one we will mention:
1. [email protected] via its own reagent/src/deps.cljs which is found on the class path by shadow-cljs at build time (not lein-shadow, or other tools) and injected via npm install --save [email protected] into my-app/package.json
[email protected] declares many dependencies, but for our example only one we will mention:
1. react@^16.8 via its own framer-motion/package.json/:peerDependencies

Some important things to take away from this diagram include:

To successfully build 'My Application' there are TWO independent, yet merged into one graph, dependency systems. Maven and NPM.
A dependency can be required via multiple paths that need to be satisfied via both dependency systems; e.g. [email protected] vs react@^16.8
Transitive NPM dependencies can be via NPM package.json files (NPM dependencies of NPM dependencies), or via shadow-cljs deps.cljs files (NPM dependencies of CLJS dependencies).

Possible Build Stacks

'Vanilla shadow-cljs'. Just shadow-cljs and npm (as npm, or yarn which we don't consider in this discussion, is a required dependency of shadow-cljs).
shadow-cljs, npm and lein BUT NOT lein-shadow; probably via a lot of lein alaises and lein-shell out to the shadow-cljs CLI.
shadow-cljs, npm, lein and lein-shadow; uses shadow-cljs as a JAR library, not the shadow-cljs CLI.
Option 2. or 3. with the addition of a package-lock.json file

Dependency Resolution Use Cases

What dependency resolution use cases are actually different between these stacks ? Turns out, not a lot. Only Use Case 2. has any variation.

Use Case 1. Direct Dependency on CLJS

Only one option. Maven dependency in project.clj/:dependencies (or deps.edn, or shadow-cljs.edn).

Use Case 2. Direct Dependency on JS

There are two options, which are very similar:

NPM dependency in package.json/:dependencies, managed directly with npm and shadow-cljs ('Vanilla shadow-cljs')
NPM dependency in src/deps.cljs/:npm-deps, managed via lein-shadow which will write it out to package.json/:dependencies.

Technically, if we include tooling other than shadow-cljs there is the option of repackaged JavaScript in Maven dependencies via CLJSJS but we rule that out of the discussion as it has too many disadvantages.

Use Case 3. Transitive Dependency on JS via CLJS

Only one option. NPM dependency in deps.cljs in the CLJS dependency project, handled by shadow-cljs and npm. 'Vanilla shadow-cljs' searches for deps.cljs files on the classpath, does conflict resolution with the current project's package.json then if it is required executes npm install dep@ver --save....

Use Case 4. Transitive Dependency on CLJS via CLJS

Only one option. Maven dependency in project.clj in the CLJS dependency project, handled by lein.

Use Case 5. Transitive Dependency on JS via JS

Only one option. NPM dependency in package.json in the JS dependency project, handled by npm.

Part 2. Comparison

Feature	1. `shadow-cljs` & `npm`	2. `lein-shadow` & `npm`	3. `lein-shadow` & `npm` & `package-lock.json`
A. NPM interactions	❌ Manual	✔️ Automated	✔️ Automated
B. NPM integration	✔️ Simple	❌ Complex	❌ Complex
C. NPM peerDependencies	❌ Manual 1 2	✔️ Semi-Automated	✔️ Semi-Automated
D. NPM locked deps	✔️	❌	✔️
E. `package-lock.json` in Git	❌ Manual Commits	✔️ Nothing to Commit	❌ Manual Commits
F. Git tag-based versioning	❓ TBA	✔️ `lein-git-inject`	✔️ `lein-git-inject`
G. Dynamic shadow-cljs.edn config	⚠️ Limited	✔️	✔️

A. NPM interactions

NPM interactions include:

npm install or npm ci; i.e. creating a node_modules folder from package.json and package-lock.json
npm install name@ver --save{,-dev}; i.e. adding a dependency

B. NPM integration

NPM integration points include:

package.json and package-lock.json files
npm cli tool
shadow-cljs which, in any of the options, reads package.json to compare it with deps.cljs files on the classpath (transitive npm dependencies, e.g. react from reagent) then runs npm install react@ver --save.
lein-shadow which, if used, reads package.json and runs npm install and/or npm install dep@ver --save{,-dev}.

C. NPM `peerDependencies`

NPM 5 removed support for automatic installation of 'peerDependencies'. This includes react in some scenarios.

D. NPM locked dependencies

i.e. 'package-lock.json'

Solves the problem that we can specify exact versions for our own dependencies, but dependencies of our dependencies (transitive dependencies) may be pulled in via version ranges that auto-upgrade without our knowledge or intent.

E. `package-lock.json` in Git

If we track package-lock.json in Git then we need to regularly commit the changes manually at times that it should be changed such as adding or deliberately upgrading a dependency. This requires developers to:

recognise correctly when package-lock.json should be changed, and when it should not.
commit and push those changes.
avoid merge conflicts of package-lock.json such as by not maintain multiple branches of package-lock.json changes.

F. Git tag-based versioning

See lein-git-inject README.

Accurate versioning is critical to operational and technical support use cases; e.g. simply being able to answer the question, what code built this misbehaving app in production?

G. Dynamic `shadow-cljs.edn` Config

To reduce errors since the Figwheel days we have programatically generated parts of the compiler configuration in project.clj; e.g. dir/file paths.

Part 3. Recommendations (Draft)

Use `package-lock.json`

XXX

Use `npm ci`

Where we use npm install to mean 'install all the things in package.json', we would probably be better served by npm ci which has some key differences.

Fix `peerDependencies` in `package-lock.json` + `lein-shadow` Case

Currently peer dependencies work because shadow-cljs.edn and/or lein-shadow run multiple npm install dep@ver --save{,-dev} commands when a package.json file does NOT exist. In this case, the dependency (e.g. react) is always installed.

If a package.json file already exists, such as the case needs to be when using package-lock.json then in some cases peer dependencies are not installed with npm install because of it being removed in NPM 5.

superstructor/README.md