Experience report for creating a submodule within an existing module

report.md

Background

GopherJS is a compiler from Go to JavaScript that targets browser VMs.

The package github.com/gopherjs/gopherjs is the compiler program, responsible for transpiling Go code to JavaScript.

The package github.com/gopherjs/gopherjs/js provides an API for interacting with native JavaScript.

Because of the limited runtime environment afforded by a JavaScript VM (especially in the browser), GopherJS provides special implementations of certain core/standard library pacakages. As such it is closely linked to a given Go release (ignoring point releases). Therefore, a new version of the compiler (github.com/gopherjs/gopherjs) is released for every Go release. The implementation of github.com/gopherjs/gopherjs (and dependent "internal" packages) is in effect tied to a given release series, i.e. 1.10, 1.10.1.... then a new release for the 1.11 series. Point releases of GopherJS therefore correspond to bug fixes in GopherJS itself. The current policy is for the GopherJS version to "closely" follow the Go version, although these releases are not tagged in the GopherJS repository.

The JavaScript interop package github.com/gopherjs/gopherjs/js is, by contrast, not a function of Go release. Instead it is a function of the JavaScript language itself (GopherJS targets ECMAScript 5). As such, it does not change at all frequently, certainly not with each Go version.

Users of GopherJS broadly fall into two categories: people writing applications for browsers, and library authors writing packages to make writing such applications easier. Library authors invariably need to interact with JavaScript APIs, for things like DOM manipulation, wrapping of existing JavaScript libraries (referred to as GopherJS bindings). Those library authors often, therefore, import the github.com/gopherjs/gopherjs/js package, e.g. the canonical DOM package honnef.co/go/js/dom.

Both groups of users typically want to ensure their code works with the last two Go releases. Given the aforementioned implementation constraint on GopherJS itself, this requires them to depend on two versions of GopherJS. In the world of Go modules, this would translate to them relying on two major versions of github.com/gopherjs/gopherjs.

But such a major version policy within Go modules world would mean that any importers of github.com/gopherjs/gopherjs/js (which is a subpackage of github.com/gopherjs/gopherjs) would be forced to also follow the same policy; a new major version for each Go release.

Hence it seems to make sense to separate github.com/gopherjs/gopherjs/js into its own module; principally so that it can be versioned independently of github.com/gopherjs/gopherjs.

Given github.com/gopherjs/gopherjs/js is currently a subpackage of github.com/gopherjs/gopherjs, this involves creating a submodule.

One point of note is that github.com/gopherjs/gopherjs depends on github.com/gopherjs/gopherjs/js for:

• implementation reasons
• testing

But github.com/gopherjs/gopherjs/js, unsurprisingly, depends on github.com/gopherjs/gopherjs for testing.

Hence we have a cyclic module dependency:

github.com/gopherjs/gopherjs/js
     ^                +
     |                |
     +                v
 github.com/gopherjs/gopherjs

Another critical point here is that all of this work is happening in a fork of https://github.com/gopherjs/gopherjs, specifically https://github.com/myitcv/gopherjs. Go 1.11 support is being added in the https://github.com/myitcv/gopherjs/tree/go1.11 branch.

The goal

Given that background, our goal was therefore to create github.com/gopherjs/gopherjs/js as a submodule of github.com/gopherjs/gopherjs. Furthermore, we want to version github.com/gopherjs/gopherjs as github.com/gopherjs/gopherjs/v11 for the upcoming Go 1.11 release. All of this happening with the https://github.com/myitcv/gopherjs fork of the github.com/gopherjs/gopherjs project, with changes ultimately being "merged" into the https://github.com/myitcv/gopherjs/tree/go1.11 branch.

The process

github.com/gopherjs/gopherjs is, as the import path suggests, hosted on Github. So the description of the process below necessarily uses terminology like Pull Request (PR) to represent the equivalent of a Gerrit CL.

PR myitcv/gopherjs#21 captures all of the commits created in this process. Each commit is also part of a separate PR.

The 5 commits are roughly summarised as follows:

1. create github.com/gopherjs/gopherjs/js submodule; this breaks the existing CI build
2. use the github.com/gopherjs/gopherjs/js submodule from github.com/gopherjs/gopherjs
3. use github.com/gopherjs/gopherjs to test github.com/gopherjs/gopherjs/js
4. move github.com/gopherjs/gopherjs to v11
5. use github.com/gopherjs/gopherjs/v11 to test github.com/gopherjs/gopherjs/js

Each commit was maintained on a separate local branch. Where inter-branch references were required, branch names were used in go.mod as version specifications in module definitions.

Conclusion

Overall, the process worked very well. Unfortunately, however, the last step failed.

In the final step, we want to require github.com/gopherjs/gopherjs/v11 from the github.com/gopherjs/gopherjs/js module. But VCS https://github.com/gopherjs/gopherjs does not know anything about v11, only https://github.com/myitcv/gopherjs does. Despite there being an appropriate replace directive:

require github.com/gopherjs/gopherjs/v11 v11.0.0-20180628210949-0892b62f0d9f

replace (
	github.com/gopherjs/gopherjs => github.com/myitcv/gopherjs introduce_v11

	github.com/gopherjs/gopherjs/js => github.com/myitcv/gopherjs/js introduce_v11
)

per golang/go#26241 the go tool tried to resolve github.com/gopherjs/gopherjs/v11 before examining the replace directive, and hence failed:

go: github.com/gopherjs/gopherjs/[email protected]: missing github.com/gopherjs/gopherjs/go.mod and .../v11/go.mod at revision 0892b62f0d9f

Per golang/go#26241, one option here would be to obviate the requirement for a require directive in the case a corresponding (versionless) replace directive exists. This would also obviate the need for any sort of "fake" version in the require directive.

There was one major pain point. In the process of creating these commits, code often needed to move between commits as fixes/changes got pushed "up" the chain to ensure that changes in a given commit were logically related to the stage in the migration. This required a number of rebases against the previous step throughout the chain. As discussed above, inter-branch references were used in go.mod definitions in both require and replace directives. However, the go command erases these branch references, replacing them with a pseudo version corresponding to the HEAD commit on the named branch at that point in time. I therefore added comments with the original branch name above the corresponding directive to act as an aide memoire. But after each rebase I was still required to replace the pseudo-version with the branch name (from the comment) in order that the go command picked up the new HEAD commit for a new pseudo-version.

Minor pain points:

• Despite setting GO111MODULE=on I needed to move development outside of my GOPATH because of golang/go#26046 (comment)

There were a number of positive points from the process:

• Separate import paths for major versions; allows module-aware Go code to depend on different GopherJS major versions in order to test against different Go releases
• The ability to set GOFLAGS="-mod=readonly" to ensure CI did not add any unexpected dependencies; a good failsafe
• The updates to go/build to support module-aware code. Whilst things generally feel a bit slower, it made the migration trivial (GopherJS uses the go/build package all over the shop)

Open questions

• Whether it is necessary to have separate commits for each step described above
• Assuming separate commits are required, whether it makes sense to have separate PRs for each commit

Thanks for writing this up, Paul!