Remote dependencies proposal

This document describes a proposal to add support for remote dependencies to the juvix build tool.

Current dependencies support

Juvix projects can currently depend on other Juvix projects that are stored on the local file system using the dependencies field in the project's juvix.yaml file:

See for example the juvix.yaml in the juvix-containers repository:

name: containers
version: 0.4.1
dependencies:
  - deps/stdlib
  - deps/test

Use-case for remote dependencies

It's common for a programming language build tool to support remotely stored dependencies so that projects are able to share common code.

In the containers example above the test and stdlib dependencies are fetched from remote locations using a Makefile target:

deps/stdlib:
	@mkdir -p deps/
	@git clone https://github.com/anoma/juvix-stdlib.git deps/stdlib
	@git -C deps/stdlib checkout caffc3b9bfde589e5ef3fca3f835200ce78ab312

deps/test:
	@mkdir -p deps/
	@git clone --branch v0.4.0 --depth 1 https://github.com/anoma/juvix-test.git deps/test
	$(MAKE) -C deps/test deps

deps: deps/stdlib deps/test

This approach works but has some issues:

To support transitive dependencies, all dependencies must use the same Makefile target name and call this target for each dependency.
The juvix.yaml and the Makefile synchronised manually.
Users must be familiar with Makefile syntax and tooling.

Syntax proposal for `juvix.yaml`

To support the remote dependencies use-case we would extend the dependencies field in juvix.yaml to support a git object.

The git object is required to have both url and hash field that specifies the location and repository hash for remote git repository that contains a Juvix project with a juvix.yaml in its root.

name: containers
version: 0.4.1
dependencies:
  - git:
      url: https://github.com/anoma/juvix-test.git
      hash: 7e82aea8becba3badf9adab4dc68df11005e3c62
  - deps/anotherDep

Support for the following will be considered in a future iteration of the design:

Fields to specify a git tag, branch or other type of git reference
Remote repositories that contain multiple Juvix projects, e.g by specifying a path field.

Behaviour proposal

The following describes how the juvix compile / juvix typecheck / juvix format etc. would behave in different scenarios related to remote dependencies.

Case 0. No remote dependencies specified in `juvix.yaml`

The Juvix tools would behave in the same way as before.

Case 1. A remote dependency is specified in `juvix.yaml` its clone is not present

During the Juvix compiler pipeline setup step the remote dependency is cloned to a location within the internal build dir (i.e .juvix-build) and the clone's working copy is checked out to the specified hash.

If the repository cannot be cloned or the hash reference does not exist in the clone then an error is reported to the user and the pipeline terminates.

Case 2. A remote dependency is specified in `juvix.yaml` its clone is present

During the Juvix compiler pipeline setup step the clone's HEAD reference is compared to the corresponding hash field in the juvix.yaml.

If the hash matches then the pipeline proceeds.

If the hash does not match then a git fetch followed by a git checkout at the hash is attempted. If there's an error at this step it is reported to the user and the pipeline terminates.

Case 3. A remote dependency is specified in `juvix.yaml` that has remote dependencies itself

The dependencies of remote dependencies are fetched in the same way as the root project. All dependencies are fetched in the setup phase of the compiler pipeline.

Case 4. A remote dependency that was previously specified in `juvix.yaml` is removed.

The removed dependency is deleted from the internal build dir and is not used during the build.

Future enhancements

The following are enhancements that could be implemented after the intial proposal.

Specification of other types of git ref (branches, tags)

Support for arbitrary git references including branches and tags should be added at at later stage as this would be useful to depend on particular versions of a dependency.

This would require the implementation of a freeze file mechanism that would store and persist the specific git hashes of branches/tags that existed at the time that the build was run.

Considering this, priority should be given to first getting remote dependencies working with fixed commit hashes.

Caching of remote dependencies

Multiple projects may use the same dependency. Builds could be made more efficient if we stored caches of the git dependencies.

Supporting repositories that contain multiple Juvix projects.

As discussed above we could add a path field to the remote dependency spec to support a single repository that contains multiple projects.

name: containers
version: 0.4.1
dependencies:
  - git:
      url: https://github.com/anoma/juvix-test.git
      hash: 7e82aea8becba3badf9adab4dc68df11005e3c62
      path: depA
  - git:
      url: https://github.com/anoma/juvix-test.git
      hash: 7e82aea8becba3badf9adab4dc68df11005e3c62
      path: depB

paulcadman/remote-reps.md