CLJS-2702: Accomodate new Closure Library dependency loading strategy

CLJS-2702.md

This pages logs work being done on CLJS-2702 under Clojurists Together Q2 2018 Funding. Overall funding round work is being logged here.

This work in particular covers the first bullet:

• Come up with a way to accommodate a recent change in the Closure Library that currently prevents ClojureScript from upgrading to newer versions of this library.

Work is being done in this branch, and when work is complete, the squashed results will be submitted as a patch.

01-May-2018

Evidently, internal changes have been made in Closure Library to accomodate a more sophisticated dependency tracking mechanism. ClosureScript relies on the internal details, and this is why things broke. Fortunately, it appears that—while still relying on private API—we can fairly easily rearrange our code so that things still continue to work. Commit 5070871 makes it so that the failure mode described in CLJS-2702 is addressed, while also preserving compatibility with the previous version of Closure Library.

As an aside, in the #cljs-dev Clojurians Slack (where ClojureScript compiler dev topics are discussed), we have discussed that it would be nice if changes made to support newer versions of the Closure Compiler can be made in a way that is compatible with previous versions, where reasonable. This lets us back out, if need be (to assess regressions, etc.), and it also affords downstream ClojureScript users the ability to use older versions of things if needed. It seems like the general consensus is that, if we can manage to straddle two versions of Closure, that flexibility would be useful. (We are actually dealing with the Closure Compiler and the Closure Library as two separate upstream codebases that ClosureScript depends on, but the concept is the same for both.)

Even though this gets things working again, there are several rough areas, defects, that will need to be sorted:

• Things work if you script/bootstrap --closure-library-head, but if you instead use the script in script/closure-library-release to build a relase, and then change project.clj and other items to depend on that release, script/noderepljs fails (evidently failing to load goog/deps.js). But in that mode, if you instead do script/uberjar, the resulting cljs.jar works. Here is the top part of the stack when it fails:

Exception in thread "main" java.lang.IllegalArgumentException: Cannot open <nil> as a Reader.
	at clojure.java.io$fn__10992.invokeStatic(io.clj:288)
	at clojure.java.io$fn__10992.invoke(io.clj:288)
	at clojure.java.io$fn__10894$G__10870__10901.invoke(io.clj:69)
	at clojure.java.io$reader.invokeStatic(io.clj:102)
	at clojure.java.io$reader.doInvoke(io.clj:86)
	at clojure.lang.RestFn.invoke(RestFn.java:410)
	at cljs.js_deps$goog_dependencies_STAR_.invokeStatic(js_deps.cljc:320)

• The browser REPL works but it complains whenever you require, with a message like this.

Error: Cannot write "http://localhost:9000/cljs/pprint.js" after document load
	 (goog/base.js:3266:20)
	 (goog/base.js:2881:9)
	 (goog/base.js:2779:23)
	 (goog/base.js:830:32)
	 require (clojure/browser/repl.cljs:221:33)

• There was a goog.dependencies_.visited internal collection that was manipulated, that is evidently no longer in the updated codebase. We'll have to assess what this means. (Perhaps it is related to the previous bullet.)
• This block of code that involves a new way of accessing loadFlags will need to be tested.

04-May-2018

The first bullet above, related to script/closure-library-release was being caused because the script attempts to download the deps, so if they haven't yet been deployed you need to do something like this:

cp ~/.m2/repository/org/clojure/google-closure-library/0.0-20180503-da9add34/google-closure-library-0.0-20180503-da9add34.jar lib
cp ~/.m2/repository/org/clojure/google-closure-library-third-party/0.0-20180503-da9add34/google-closure-library-third-party-0.0-20180503-da9add34.jar lib

With this, the Nashorn REPL works properly.

There is a Rhino REPL defect which manifests itself as

$ script/repljs
Exception in thread "main" java.lang.IllegalArgumentException: No matching method found: getProperty
   at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:80)
   at clojure.lang.Reflector.invokeStaticMethod(Reflector.java:207)
   at cljs.repl.rhino$rhino_eval.invokeStatic(rhino.clj:84)
   at cljs.repl.rhino$rhino_eval.invoke(rhino.clj:74)
   at cljs.repl.rhino.RhinoEnv._evaluate(rhino.clj:211)
   at cljs.repl$evaluate_form.invokeStatic(repl.cljc:554)
   at cljs.repl$evaluate_form.invoke(repl.cljc:485)
   at cljs.repl$evaluate_form.invokeStatic(repl.cljc:492)
   at cljs.repl$evaluate_form.invoke(repl.cljc:485)
   at cljs.repl$evaluate_form.invokeStatic(repl.cljc:490)
   at cljs.repl$evaluate_form.invoke(repl.cljc:485)
   at cljs.repl.rhino$rhino_setup.invokeStatic(rhino.clj:139)
   ...

This is really the result of the .require of cljs.core not causing a load-file here.

05-May-2018

The problem with the Rhino REPL above was that it wasn't returning true from CLOSURE_IMPORT_SCRIPT. This was always a requirement but now becomes critical otherwise the loading system will go into a paused state. This is fixed with commit 6cae334.

The problem with the browser REPL is that the goog/writeScriptTag_ behavior being monkey patches has been moved into the Dependency object itself, and thus the existing monkey-patch is doing nothing. The desired monkey-patched behavior can be conditionalled hooked in the new implementation of Closure Library by defining CLOSURE_IMPORT_SCRIPT. This is done with commit 5e1b152.

At this point, all shipping REPLs work properly when tested to see if they can (require 'clojure.zip) when tested in three modes:

• After script/bootstrap --closure-library-head
• After revising project.clj to depend on 0.0-20180503-da9add34 and copying in the JARs (as described above)
• After doing the same and building a cljs.jar uberjar

Unit tests pass when in the first two modes, except for cljs.build-api-tests/cljs-2077-test-loader: Evidently, ES6 code is used in some of Closure Library, with this being dealt with in commit f1a5c96.

Additionally no regressions are seen when doing the following the current Closure Library dep:

• After script/bootstrap
• After doing the same and building a cljs.jar uberjar

Self-host testing was patched up with 0e3fd4d.

Attached candidate patch with all of the above to CLJS-2702.