Using JavaScript modules in ClojureScript

js-in-cljs.md

Using JavaScript Libraries from ClojureScript

Using JavaScript libraries from ClojureScript involves two distinct concerns:

1. Packaging the code and delivering it to the browser
2. Making ClojureScript code that accesses JavaScript libraries safe for advanced optimization

Right now, the only single tool that solves these probems reliably, optimally, and with minimal configuration is shadow-cljs, and so that is what I favor. In paricular, shadow-cljs lets you install npm modules using npm or yarn and uses the resulting package.json to bundle external dependencies. Below I describe why, what alternatives there are, and what solutions I disfavor at this time.

Packaging and Delivering Code

My favored approaches in decreasing order of developer convenience:

1. Include the library by placing a <script> tag in your index.html. The library will load itself into a global object, which you can then access using js/LibraryObject. (This is the easiest but requires a round trip to the server for each library.)
2. Use shadow-cljs for dependency management. Install your libraries using npm or yarn. Import them directly into your source code. (This is my recommended approach: this provides the highest quality output and is easy to use once set up.)
3. Use the :foreign-libs feature of the normal cljs compiler. Find a UMD build from the npm distribution of your library, or create one using webpack if one does not exist. Import it using :foreign-libs.
4. The "double bundle" approach: same as (1) or (3), but use webpack to bundle all of your javascript libraries in a single bundle. Now, when you serve, you'll have one js bundle for your cljs compiled code and one bundle for your libraries. See this post for directions. It should be possible to get as good output here as with shadow-cljs, but you'll have to be good at configuring webpack.

I do not reccommend either of the following approaches for now:

1. Use the normal cljs compiler. Hope that a cljsjs version of your library exists and use it like a normal cljs library. Hope the author wrapped it properly. Hope they do an update when a new version comes out. If not, create your own externs file either manually or with the automatic externs tool. Be a nice person and submit a pull request to the cljsjs project.
2. Use the :npm-deps feature of the cljs compiler. I used it once. It broke with a perplexing error message. There was no way to debug it and nobody on slack ever seems to have answers about it. Although the documentation is clear that this feature is not expected to work all the time, it doesn't say when that might happen or what you are supposed to do about it. This feature will probably get better but it isn't very good as of ClojureScript 1.9.

Advanced Optimization

As described below, ClojureScript code compiled with advanced optimizations will break when accessing JavaScript code unless you take steps to protect the code from symbol munging. For the same reasons as described above, I do not recommend using cljsjs / externs approach because it is brittle and kind of a pain.

Here are the two approaches I like:

1. Use shadow-cljs automated externs inference. Shadow-cljs will give you warnings when it can't figure things out and you can just provide a type hint. Because shadow processes all of your JavaScript libraries, Shadow can infer externs from them, which is a real advantage over some other approaches relying on the built-in compiler's extern inference.
2. Use the cljs-oops library whenever you call a JavaScript function or access a JavaScript property. This library will prevent symbol munging.

Note: I don't know if the normal cljs automated externs inference feature is reliable. I can tell you that it is not as convenient to use as shadow's method.

Discussion and notes

Packaging and Delivering JavaScript: UMD Builds and Global Library Objects

The simplest mechanism to access a JavaScript library is to access it via the library's global object. Before es6 module syntax, libraries worked by setting a bunch of properties on a global object. When you included the library using a normal <script> tag, the library would set that global object once it was loaded. Most libraries come with at least one UMD distribution already (usually in the /dist directory of their node_modules installation--you can also pull one off of a site like unpkg). The top of it looks something like this:

$ head ReactDnD.js
(function webpackUniversalModuleDefinition(root, factory) {
	if(typeof exports === 'object' && typeof module === 'object')
		module.exports = factory(require("react"));
	else if(typeof define === 'function' && define.amd)
		define(["react"], factory);
	else if(typeof exports === 'object')
		exports["ReactDnD"] = factory(require("react"));
	else
		root["ReactDnD"] = factory(root["React"]);
})(this, function(__WEBPACK_EXTERNAL_MODULE_2__) {

This hideous code is here to ascertain what environment it is in (node, AMD, or browser) and to install a global object appropriately. When called inside a <script> tag, it will take the last else clause and root will be passed in as window. So you can access this library using an object called ReactDnD.

Delivering with <script> Tag / Accessing with the Global Object

Taking the above example, you can deliver the ReactDnD library by simply adding a <script> tag to your main html index file. You can either point the source to a CDN or to your own webserver. You then just access the object using js/ReactDnD.

Cons of this approach:

1. To translate from es6-style import syntax, you really need to understand what is going on under the covers
2. You will ship a lot of code to the browser and each library will make a round trip to the server
3. Not every library ships ready to be used in this mechanism, so you might have to run it through webpack first

Pros:

1. You can use a CDN and potentially save on bandwidth and speed up page loads if the library is likely already cached

The "Double Bundle": Use Webpack to Bundle All JavaScript Together

Another commonly used strategy is the "double bundle" approach, which uses webpack to bundle up all javascript libraries. You then load the compiled cljs as one js bundle and the javascript libraries as another bundle using two <script> tags (or one <script> tag for cljs and a :foreign-libs reference to the JavaScript bundle).

• https://github.com/pesterhazy/double-bundle
• http://blob.tomerweller.com/reagent-import-react-components-from-npm

Use shadow-cljs to Bundle (or Use a CDN)

Shadow-cljs augments the main cljs-compiler with some dependency management, much improved npm module integration, simpler configuration, better defaults, and other conveniences. With shadow-cljs, you don't specify npm modules with the rest of your cljs configuration. Instead, you install modules using npm or yarn and then shadow-cljs will read you package.json and will bundle the sources from node_modules. It is primarily a build tool, so aside from some extensions to the ns form and simpler type hints, it doesn't impact your code.

Shadow-cljs will replace:

1. The cljs compiler (although it still uses the cljs-compiler under the hood)
2. The dependency management portion of lein/boot (though you can still use lein/boot if you prefer)
3. Figwheel (it has its own version of hot reloading and heads up display)

You can still use lein and boot if your build is more complicated (like maybe if you have a css preprocessor step), but for simple projects you don't need them.

Use npm or yarn and install the package normally. Shadow-cljs will look at the node_modules directory and examine the information in package.json for each module. The module can usually be used directly in your cljs code like so:

  (:require ["react-dnd" :as react-dnd :refer (DropTarget)]))

A few notes:

• The "string" syntax above is standard ClojureScript and is not an extension.

• Npm modules can be imported using the same string name that you would use if you were doing an import FlipMove from "react-flip-move" in javascript. (You can also just use the symbol name react-flip-move which I prefer because my linter doesn't know about the string syntax.)

• Shadow-cljs supports every conceivable type of es6 import syntax, including default imports. This makes translation from javascript examples and documentation easy. See more here. (It allows a :default keyword that is technically an extension to the allowed syntax.)

• Shadow-cljs actually lets you load any JavaScript file on the classpath. So, (ns demo.bar (:require ["./foo" :as foo])) would load demo/foo.js from the classpath. This is experimental.. This technique allows interop in the other direction, by embedding cljs in an existing JavaScript app.

With shadow-cljs you import libraries using a real namespace with a syntax that is much more one-to-one with es6 syntax. This is described more below, but the syntax works something like this:

  (:require ["react-dnd" :as react-dnd :refer (DropTarget)]))

Shadow-cljs kind of operates like webpack for you in that it bundles all of your javascript for you, minifies it safely, and builds source maps for you. On slack, I saw one project that ported to shadow-cljs and saw big improvements in bundle size without doing anything.

Note: Shadow-cljs can also seamlessly resolve a require statement to a CDN. See this section of the manual.

One advantage of supporting the full es6 import syntax is that some libraries are built with components that can be imported individually (e.g. material-ui). You can potentially see big improvements in code size just because of this feature.

Shadow-cljs's import syntax makes it easy to limit code use with library that have been designed to be imported as components. For example, with material-ui, each component can be imported individually, and this is easy to do with shadow-cljs, since you just require the module using npm syntax.

Making Code Safe for Advanced Optimization

In advanced optimization mode, the Google Closure Compiler alters symbols in javascript code in order to get maximum minification. This is apparently important because the generated cljs code creates incredibly long function names.

Here's the problem. If you access a javascript library like this: (.method LibraryObject), Closure Compiler will come around an minify the method symbol and then you'll get a runtime error message like "Cannot read property 'q29' of null". Why? Because Closure Compiler doesn't optimize the external library but it does optimize the cljs code.

Notes:

1. Theoretically some javascript libraries are safe to be run through closure advanced optimizations, but in practice, it seems like virtually nothing you'll want to use is.
2. Automated externs inference in the compiler can be turned on and will solve some of this problem. But it can't do it for everything. You can set the :externs-inference true to turn it on and set *warn-on-infer* on a per-file basis to have it tell you where you need to provide type hints.
3. Externs inference is turned on in shadow-cljs with :compiler-options {:infer-externs :auto} and will provide you with any inference warnings for your source code files (i.e. you don't have to remember to turn it on in each file). Type hinting when inference fails is a bit easier in shadow because it accepts simpler type hints.

The problem with the "externs" file and cljsjs

The traditional solution is to create an "externs" file that contain symbols the Closure Compiler should not alter. Note, this affects compilation not of the external library, but of the cljs code. (This was initially confusing to me because the externs feel "attached" to the foreign library.) So when you write js/LibraryObject.method, the LibaryObject.method symbol in the compiled cljs code will remain untouched by the advanced optimizer, provided that it is properly declared in the extern files. Note, the cljs compiler has a feature called automated externs inference now, which, when turned on, will automatically avoid changing simple examples like this, but there are other instances where it cannot make the inference, so we have the same problem.

The cljsjs project has a bunch of libraries that other people have created externs for. These packages also contain deps.cljs files, which are little pieces of metadata that instruct the compiler how to load the javascript code and enable you to use a normal namespace when accessing the library (e.g. react/createElement).

Problem: what if your library isn't there? What if it is there but it's the wrong version? What if it is there but it was packaged incorrectly? Making these libraries involved learning how to use the extern inference tool, learning how to use the boot tool, and learning how to write the deps.cljs file, and then (if you are nice) submitting a pull request. This is a lot of friction as compared to npm install react-flip-move and just using it. It also is really difficult to make this work with more than just a few libraries because of the effort in resolving dependencies.

Use Shadow-cljs Automated Externs Inference

Shadow-cljs makes npm module integration work better by actually examining the source code of the modules and generating externs in addition to turning the cljs extern inference feature on.

Your shadow-cljs.edn will look something like this:

{...
  :builds
  {:app
    {:target :browser
     ...
     :compiler-options {:infer-externs :auto}
     }}}}

Use the :foreign-libs feature of the Standard Compiler and cljs-oops

The other technique that avoids the externs and cljsjs friction is to stick with the mainstream compiler and use the cljs-oops package to access the libraries. The advantage here is that you don't stray from the herd and your existing tooling will work. The disadvantage is that it's a little bit clunky in your source code and you have to be diligent never to access javascript objects using the js/ accessor.

The first step is to find a UMD build of your package. Typically, you can just npm install the package, go to node_modules,look inside the dist directory, and grab the build from there. The top of the file should look like the UMD example shown above.

Using that example, all you need to do now is inform the compiler that you want this included:

:foreign-libs [{:file "resources/lib/ReactDnD.js"
                :file-min "lib/ReactDnD.min.js"
                :provides ["react-dnd"]}]

Now, if we just do something like (.dropTarget monitor) the optimizer might alter the dropTargetsymbol and you'll get a runtime error in production. Instead, we use the cljs-oops library: (ocall dnd-connect "dropTarget") or like this (ocall dnd-connect :dropTarget) Because we are using strings or keywords, the optimizer will not alter this code. You can use oget and oset as well for properties. The library has a bunch of convenience macros and some error checking built in to to make this less error prone.

That's it. As long as you consistently use cljs-oops when you refer to foreign lib symbols, everything will work.

Note: Again, instead of using cljs-oops, you might give externs inference in the standard compiler a go. Shadow-cljs make this process smoother because it processes all of your npm modules and infers externs from them. It also automatically warns on anything it can infer in your files, whereas you have to turn that on file-by-file with the normal compiler. It might work for you, but I haven't tried in a serious way.

In short, this technique works, but it requires you to do some investigation to figure out how your library is packaged and how it exports its global object. If the examples are using sophisticated es6 import syntax, that may require some translation on your part. You don't get npm module resolution. You have to keep to a convention when you call javascript code. But it is still better than messing around with externs, and you may be more comfortable using the standard tool set.

Summary of issues and various approaches

Issue	<script>	:foreign-libs	Double Bundle	shadow-cljs
Bundled?	No	No	Yes	Yes
Configuration	Point <script> tag at correct location	Manually download and serve	Use npm + webpack	Use npm
Can use standard CDN distros?	Yes	No	No	Yes
Automatic externs inference	Either cljs-oops or built-in	Either cljs-oops or built-in	Either cljs-oops or built-in	Smooth Automatic Externs Inference Experience

Very informative. Thanks for putting this together

This is a great summary, thanks!

Thank you for doing this!

I added another topic in ClojureVerse showing how shadow-cljs imports npm modules, you may check: Guide on how to use/import npm modules/packages in ClojureScript?

Fantastic! Thank you so much for this, it's a big help 👍

This is brilliant — thank you!

What a mess! One decision to rely on Google's Closure compiler created this whole mess. Yeah, it probably made sense in 2011 or earlier but these days it seems more like a hindrance to ClosureScript adoption and good progress.
The development experience is so nice in JS these days (just run Create React App and your're set), things work well out of the box and there's even great progress with tools Snowpack while ClojureScript seems to be kinda doomed IMHO.
I'm wondering if ClojureScript is to provide great developer expericene then how could I achieve what Snowpack is doing for JavaScript?

The future for ClojureScript should be just to transpile to JavaScript ES6 modules on a file by file basis and then embrace the rich tooling there is for JavaScript already as well as NPM.

The future for ClojureScript should be just to transpile to JavaScript ES6 modules on a file by file basis and then embrace the rich tooling there is for JavaScript already as well as NPM.

👍