The purpose of this Julep[1] is to propose a revised mechanism for dealing with external dependencies in Julia packages in the new revision of the package manager to be included in the Julia 0.2 release.
Please post any comments and suggestions directly on the julia-dev mailing list thread[2] for discussion.
In order to allow for the installation of external dependencies (i.e. those dependencies which are not part of the package repository), the package manger in Julia 0.1 simply evaluates a script located at PKG/deps/build.jl when the package is installed by the package manger.
The reason for this design is to minimize the degree of coupling between the package manager and the method by which binary binary dependencies are installed. This separation is akin to a separation of policy and mechanism often times encountered in systems design.
Since the package manager does not provide a lot of support to assist the package author with the common tasks involved in providing external dependencies(e.g. downloading binaries, installing them in the appropriate path, etc.), it is desirable to provide such common functionalities as a Julia package that can be easily included by any packages needing the functionality. The BinDeps.jl [3]currently provides such functionality, though it needs to be explicitly said that the package manager in no way depends on the BinDeps package and it is conceivable, even encouraged, that other packages will be developed to complement or even entirely replace BinDeps functionality.
This proposal covers solely changes to the Julia package manger itself. The process by which external dependencies are actually installed (e.g. binary downloads, source installation, etc. ) is outside the scope of the current proposal.
A common problem encountered by many users of the current package manger is that failing installations of external dependencies may easily corrupt package manger state.
The proposed process maintains the core idea of the current build.jl process but attempts to address some of the problems and challenges seen in production. Currently we may describe the state of a package as one of the following states (for a more complete description of the package manager see the METADATA.jl README [4]):
- Not Installed
We know about the package (i.e. we have its information in METADATA),
but it is not present in our .julia directory
- Installed
Some version of the package is installed in our .julia directory
- Required:
The package was explicitly added using Pkg.add(). In particular this
implies that the package may not be removed by the package manager
during Pkg.update().
In this scheme, a package being required automatically implies that it is installed.
The core of this proposal is to add "Working"/"Not working" as an additional state. In particular a package is supposed to be marked "Not working" until all external dependencies are resolved at which point it is to be marked as "Working". Setting and checking these states however is largely left to the packages themselves and no kind of enforcement is provided within the package manger.
In order to still allow external dependencies to be automatically installed in the common case, I propose to implement a two-stage process to the installation of external dependencies:
The first stage roughly corresponds to how the package manager worked before the introduction of the build.jl mechanism. In particular during this stage the repositories of the package being installed as well as well as the repositories of all declared dependencies are to be downloaded to the .julia directory. Should the download of one repository fail all other repositories are to be downloaded as planned. After the first stage, all packages are automatically in the "Installed" state (as well as the "Required" state if appropriate), but also in the "Not Working" state by default.
The second stage triggers the deps/build.jl script for every package that was installed in the first stage. It is important that the order of execution here respect the dependencies between packages[5]. Ideally the order would the same as the order in which the packages were downloaded during the first stage, though this is not strictly necessary. It is the responsibility of the build script to adjust the package state appropriately upon success of the installation of the external dependency. The following three scenarios may be commonly encountered:
- The build script sets the package status to "Working", everything
proceeds as expected.
- The build script does not set the package state to "Working", the build
script for other packages will still be run as usual. This will allow for
the implementation of systems that list the system packages to be installed
all at once without requiring an endless back and forth for every
single package.
- The build script throws an error. In this case the installation of
binary dependencies is aborted completely and the user is expected to
resolve any problems. It is recommended that this option is avoided save
for fatal errors that may jeopardize the installation process of
subsequent packages.
Should the deps/build.jl script not exist for a given package, it will automatically be marked working provided all of its dependencies are marked as working as well.
In all of this it is important to note that the "Working"/"Not working" distinction does not have any impact on package loading. In particular this implies that if a package does not check its own state there is no distinction between "Working" and "Not working" packages.
Furthermore most of the bookkeeping work should be managed by support packages such as BinDeps.jl (or packages like it) and packages that do not depend on external dependencies at all do not need to worry about this system either, since they will be automatically marked as working.
Names for these methods are preliminary. Please suggest alternatives, if you come up with a better name.
Pkg.isworking(pkg)
Check whether the package is marked as working.
Pkg.hasworkingdeps(pkg) = all(map(Pkg.isworking,Pkg.dependencies(pkg)))
Check whether all dependencies of the package are marked as working.
Pkg.markworking(working::Bool)
Mark the package as working (or not)
Pkg.fixup()
For all packages that are marked as not working, rerun the build script
(again paying attention to dependency ordering)
Pkg.fixup(pkg; force=false, forceall=false)
Re-run the build script for the package `pkg` and all of its dependencies
that are marked as "Not working". If `force` is set to true, the build
script for the package will be run regardless of its status, if `forceall`
is set to true the build script of all dependencies will be run, regardless
of status.
The following provides examples of how a user might interact with this system.
julia> OS_NAME
:Linux
julia> Pkg.add("Winston")
>> Downloading Package BinDeps
>> Downloading Package Cairo
>> Downloading Package Color
>> Downloading Package IniFile
>> Downloading Package Tk
>> Downloading Package Winston
>> Running build script for package Tk
>> Running build script for package Cairo
>> Done
NOTICE: Please run the following command:
- sudo apt-get install libtk libcairo
julia> using Tk
ERROR: Missing external dependencies. Please run Pkg.fixup("Tk")
julia> Pkg.status()
NOTICE: Some packages have incomplete external dependencies (indicated by *)
Winston 0.2.0
Cairo* 0.2.2
Color 0.2.2
IniFile 0.2.0
Tk* 0.2.2
BinDeps 0.0.1
julia>
$ sudo apt-get install libtk libcairo > /dev/null
$ ./julia
NOTICE: Some packages have incomplete external dependencies. Run Pkg.fixup()
julia> Pkg.fixup()
>> Running build script for package Tk
>> Running build script for package Cairo
>> Done
julia> Pkg.status()
Winston 0.2.0
Cairo 0.2.2
Color 0.2.2
IniFile 0.2.0
Tk 0.2.2
BinDeps 0.0.1
julia> using Tk
julia>
On windows the same may look like this:
julia> OS_NAME
:Windows
julia> Pkg.add("Winston")
>> Downloading Package BinDeps
>> Downloading Package Cairo
>> Downloading Package Color
>> Downloading Package IniFile
>> Downloading Package Tk
>> Downloading Package Winston
>> Running build script for package Tk
>> Downloading binaries: 100%
>> Running build script for package Cairo
>> Downloading binaries: 100%
>> Done
julia> using Tk
julia> Pkg.status()
Winston 0.2.0
Cairo 0.2.2
Color 0.2.2
IniFile 0.2.0
Tk 0.2.2
BinDeps 0.0.1
[1] https://groups.google.com/forum/?fromgroups#!searchin/julia-dev/julep/julia-dev/QIZ-9DyndIw/PdhJSZfqaV8J [2] [3] https://github.com/loladiro/BinDeps.jl [4] https://github.com/JuliaLang/METADATA.jl#package-manager-overview
[5] To see why this important, consider the following hypothetical situation: Say Clang.jl needs to compile LLVM from source using the cmake build system. To do so, it might require CMake.jl which itself needs to compile cmake from source (or at least download the appropriate binary for the current operating system). Thus is is necessary that the CMake.jl build script be executed before that for Clang.jl
It would be nice to have something similar to the existing
build.jl, or the ability to run a developer provided Makefile. This is only for people who are building packages for distribution, or need a highly customized build.Even in build.jl, we need two separate things:
Thus: