Julep 2: Package Requirements and Quality Standards

julep2.md

Julep 2: Package Requirements and Quality Standards

This is based on the discussion started here:

• https://github.com/johnmyleswhite/PackageTesting.jl
• https://groups.google.com/forum/#!searchin/julia-dev/package/julia-dev/I_Ul4TgpcZw/z0PVPNurn3cJ

as well as the manual.

This Julep sets the rules for packages to improve the quality of the packaging ecosystem. All packages submitted to METADATA.jl must meet these requirements.

Requirements for the Package Itself

REQUIRE file

• Packages must have a REQUIRE file that, as well as listing all packages it depends on, explicitly states the version of Julia it depends on.

License information

• Packages should have license information, either in a dedicated LICENSE file or stated in a README

Package Testing

• Packages should have a test/ directory, that contains a file runtests.jl
• The runtests.jl file will be called by an automatic package ecosystem testing system. As such, it should not call long-running performance tests, and should not require any configuration above-and-beyond that is automatically carried out during package installation. If a test fails it should throw an error that causes Julia to exit with an error code.
• Packages may contain other testing code in this folder of any nature, it will not be called.
• Packages are encouraged to have a .travis.yml file for TravisCI integration.

Requirements of the Package's METADATA.jl entry

Each package has a folder in METADATA.jl, with the following contents

url

• Every package must have a url file in the root of their folder in METADATA.jl
• The file should contain a single line that states the address of the package's git repository, e.g. git://github.com/JuliaStats/DataFrames.jl.git

DESCRIPTION.md

• Every package must have a DESCRIPTION.md file in the root of their folder in METADATA.jl
• The format of the file should be as follows:

# Description
A short description of the package
# Keywords
A comma separated list of keywords, e.g. distributions, probability, random normals, monte carlo
# Maintainers
A comma separated list of maintainers
# Install notes
A short description that will be displayed when the package is installed. Should be used only if manual operations must be performed to complete installation.

versions

• Every package must have a versions folder, see the README in https://github.com/JuliaLang/METADATA.jl/tree/devel

@staticfloat, Only reason to separate INSTALL and TEST is that some package's full test suite might take too long to run - slowing down package installation. I imagine running the DataFrames.jl test suite takes a non-trivial amount of time (does it?). If this is a non-factor, we should combine them, because it definitely adds complexity.

Automatic issue submission is awesome.

Given we're looking at rolling this out post 0.2, or at least not blocking 0.2 on it, I'm going to take some time prototyping some code in https://github.com/IainNZ/PackageEvaluator.jl

Example output from script so far:

---

Package Analysis Results

REQUIRE file

• Requirement: packages must have a REQUIRE file
  • ✓ Passed (+1.0)
• Requirement: REQUIRE file specifies a Julia version
  • ✗ Failed!

Licensing

• Recommendation: Packages should have a license
  • ✓ Passed (+1.0)
  • Detected license in JuMP.jl/LICENSE.md: 0.0

---

Summary

• Total score: 2.0
• One or more requirements failed - please fix and try again.

What tests would one realistically want to do for INSTALL besides checking that julia and external dependencies are properly installed? I agree with @staticfloat that this is very similar to what's already guaranteed by Pkg2 and BinDeps.

JuliaLang/julia#3921

@mlubin so no INSTALL test option then? Just TEST, PERF, and ALL options? If BinDeps is good enough, I don't really have any good ideas for reasons for INSTALL, anything I could think of related to that.

You know, the more I thought about it - I think we are very unlikely to run centralized performance testing, so why should we even bother creating a requirement to provide that functionality. Instead, we just say there is a file, you should provide it, tests will run - don't put performance tests in there. Minimal set of requirements = best. Thoughts?

Minimal set of requirements = best

Absolutely agree

@IainNZ Is it your intention to add PackageEvaluator to the METADATA.jl testsuite? Then all PR's for new packages will get automatically tested on Travis.

Right now its just experimental, but yes, if we can get everyone to agree on the above requirements, it will.
My timeline is vaguely:
a) People seem fairly OK with the Julep, so
b) I wrote PackageEvaluator to give people a feel for how they are doing.
c) Announce: packages must meet these clear rules, and should probably do these other things too. See how you are doing using PackageEvaluator
d) After x weeks, put it into METADATA.jl test suite. It'll almost surely fail on some unmaintained package. We try and address all the troublesome packages before doing this though.
e) Do we kick out bad packages?
f) In parallel, we have the package ecosystem testing stuff. I think it'd be cool to have a score for each package based on its pass rate and PackageEvaluator score. I stole this idea from Perl CPAN, and I know that Node.js/NPM are planning on doing this too.

More attention needed on the DESCRIPTION.md file, I don't know I like the idea of doing it as Markdown + reinventing the wheel...
@johnmyleswhite was suggesting we take the R package format. I'd feel comfortable with JSON personally, but I think this is all very personal-taste so maybe we should just semi-arbitrarily pick something.

Another thought: the description should probably go in the package repository as well as in METADATA. I dislike the duplication...

The R DESCRIPTION file format is actually a Debian format. With trivial modifications, it can be treated as YAML.

An idea for the testing infrastructure: http://jenkins-ci.org/. While Travis-CI is great for what it does, I think we need something a bit more flexible that we can run nightly with any combination of julia versions/package versions (metadata vs. master) that we like. This means we also need our own hardware.