Guidelines on evaluating health & quality of third-party crates at Embark

crate-health.md

What to evaluate and consider before adding usage of new third-party crates.

These are not exact requirements but questions to investigate and discuss to help reason around the health, safety, maintainability, and more around crates.

This can also be read as an opinionated guide for crate authors of what our (Embark's) guidelines and recommendations are, though should not be taken too literally.

Legend: 🔒 Must have, ⭐️ Should have, 👍 Nice to have, ℹ️ Info

1. Licensing

1. 🔒 Is the project using a license we support?
   • Motivation: For libraries we only use OSI permissive licenses. For binary apps/tools copyleft licenses are also ok.
   • Preferred license is the Rust-standard dual-license: MIT OR Apache-2.0.
   • Individual licenses we support generally for libraries: Apache-2.0, Apache-2.0 WITH LLVM-exception, BSD-2-Clause, BSD-3-Clause, MIT, Zlib, ISC
   • Exceptions can be made on a case by case basis depending on the license. See deny.toml in the Embark project you plan to use the crate in for full list of allowed licenses and rationales.
   • ☑️ We validate and enforce this on our projects with cargo-deny
2. 🔒 Are all transitive dependencies using compatible license with the main project?
   • Motivation: This makes it easier to reason about the project and is generally good practice.
   • Example: One would want to avoid having a MIT-licensed library accidentally depend on a GPLv3 licensed library.
   • ☑️ We validate and enforce this on our projects with cargo-deny

2. Maintainers

1. ⭐️ Is the project & repository actively maintained?
   • Motivation: If not then it likely will be a bottleneck later when dependencies change or bugs are found.
   • Example: when were the last commits, merges, and releases for the crate?
2. ⭐️ Are PRs from external contributors welcomed and actively merged?
   • Motivation: Hard to help with dependency upgrades or bug fixes if not, and we may then have to fork and maintain it fully ourselves
3. 👍 Is the project maintained by a company?
   • Motivation: It can be a benefit if the project is professionally maintained and proven to be used in production by another company.
4. ℹ️ How many active maintainers are there on the project?
   • Motivation: Multiple maintainers is generally good and reduces the bus factor.
5. ℹ️ What other open source projects are the maintainers supporting?
   • Motivation: Helps to tell if it is an experienced maintainer or a first time maintainer.
6. 👍 Does any of the maintainers have any active public presence where they can be reached out on?
   • Motivation: Can be a nice bonus to communicate casually or for collaborations.
   • Example: Twitter, Mastodon, Zulip, email, etc
7. 👍 Have we (Embark) previously contributed to the project and have had changes merged & published?
   • Motivation: If we have, then it is higher chance that we both are familiar and know the project or maintainer, as well as chance of having issues or PRs be resolved again.

3. Unsafe

1. ⭐️ Is the unsafe code sound? 😅
   • Motivation: Should strive towards having well reviewed and well understood unsafe code without unsoundness and UB, to avoid it breaking or causing unexpected behavior and issues. Though it can be hard in some projects so depends on the use cases of how much effort to put into it.
2. 👍 Is [forbid(unsafe_code)] used in the root of lib.rs?
   • Motivation: For crates that do not need unsafe, this enforces it and makes auditing significantly easier as one knows there aren't any unsafe at all.
3. ⭐️ Are # Safety comments used on public unsafe functions?
   • Motivation: This is best practice and important to know what the requirements and invariants are for the unsafe functions and how to call it in a safe way.
4. ⭐️ Are /// SAFETY: comments used on unsafe blocks?
   • Motivation: This is best practice and easier to see how a project reasons around the invariants and soundness of their unsafe code if it is written down next to it in this standard way.
5. ⭐️ Is the clippy::undocumented_unsafe_blocks lint used?
   • Motivation: This enforces that /// SAFETY: comments are used on unsafe blocks blocks to describe why the code is sound.
   • This is part of Embark's standard lints.
6. ⭐️ Is the unsafe_op_in_unsafe_fn lint used?
   • Motivation: This is best practice and is going to be more common in Rust (see lint description & RFCs) to clearly separate the safety description of the invariants needed to be upheld by the function, from the documentation of why the individual unsafe blocks are sound. So a project using this likely are more likely to follow and enforce best practice.
   • This is part of Embark's standard lints.
7. 👍 Is the amount of unsafe functions and unsafe blocks sane for the type of project and benefit?
   • Motivation: While the specific number is not important, the higher it is the more work it is to review and verify it is and continues to be sound.
8. 👍 Is Miri used for unsafe testing in CI?
   • Motivation: For particularly tricky unsafe code this is a great additional layer to validate that the the unsafe code is sound.

4. Auditing

1. 👍 Is it audited by us or our partners with cargo-vet?
   • Motivation: If it is that is a great sign of quality and that the project does what it says it does. Ultimately we would like to have all of our dependencies manually audited. But if a crate is not already audited we will have to do it, at least as a sanity check around appropriate ambient authority usage a project has before adding it.
   • Audits: Ours, Mozilla, Bytecode Alliance
2. ⭐️ Are the Rust API Guidelines generally followed?
   • Motivation: These are best practices and general expectations of how to build libraries and APIs so following them overall is a good sign of quality.
   • Clippy does help to validate some of these best practices.

5. Testing & CI

1. ⭐️ Are CI tests passing?
   • Motivation: Hard to validate current state or PRs if CI isn't even working, a warning sign.
2. ⭐️ Is CI building & running tests on all target platforms and architectures we support?
   • Motivation: We support all 3 main desktop platforms (Windows, Linux, Mac) and 2 main architectures (x86_64, aarch64, for some: Wasm) so best to have coverage of of all of them. Esp. important for crates that interact with desktop and system features.
   • Exception if crate is only used for services (Linux) or Wasm
3. ⭐️ Can one easily run tests locally on target platforms after cloning the repo?
   • Motivation: If it is complicated to run tests locally then that makes it harder to investigate issues and contribute changes.
4. 👍 Does CI test latest stable Rust?
   • Motivation: We use latest stable Rust compiler so prefer that to be tested and work well.
5. 👍 Is minimum supported Rust version specified in Cargo.toml
   • Motivation: It is a good best practice, though as we use latest stable it is rarely an issue for us.

6. Clippy

1. ⭐️ Does it use Clippy?
   • Motivation: We really like Clippy and have found it catches and fixes many bugs as well as helps with style and consistency, so it is a plus when other projects use it also.
2. 👍 Is Clippy run and validated to not fail in CI?
   • Motivation: Makes it easier to iterate and contribute to the project
3. 👍 Does it use most of the additional lints from our Embark's standard lints set?
   • Motivation: We've over time put together this set of lints we get value for and prefer for our projects, if other projects use it or similar list then they likely reason around code quality and best practices similar to how we do which can make it easier to use, contribute to, and collaborate with. Just a bonus though, not a requirement and different types of projects have different needs.

7. Rustfmt

1. ⭐️ Does it use rustfmt?
   • Motivation: It is easier to contribute to projects that use rustfmt and it is standard practice to use it in the Rust community
2. 👍 Is rustfmt run and validated in CI?
   • Motivation: Makes it easier to iterate and contribute to the project

8. Dependencies

1. 🔒 Is it described what system dependencies are used or required?
   • Motivation: System dependencies have to be installed manually by the developer or user and something we generally prefer libraries and tools not to use or require, but if they do it should be well documented and described for the target platforms.
2. 🔒 Are C/C++ dependencies built with standard cc?
   • Motivation: We do not allow CMake or other build systems, as it is much easier to set up and use Rust crates with just plain cc compiler building.
   • ☑️ We validate and enforce this on our projects with cargo-deny
3. ⭐️ Is Cargo.lock committed to the repository?
   • Motivation: To be able to reproduce and use the same dependencies locally and on CI. Esp. important for applications and projects with many dependencies.
4. ⭐️ Does the full set of of transient dependencies look reasonable and sane?
   • Motivation: Too many small dependencies increase build times and supply chain exposure, so important to keep projects clearly focused on just the dependencies they need. Which also includes transient dependencies (the dependencies of the dependencies) and having Cargo feature flags and optional dependencies where appropriate.
5. 👍 Is cargo-deny used?
   • Motivation: It is a quality sign that the project actively manages their dependencies and validates licenses used.
6. 👍 Are duplicate dependencies (multiple versions of same crate) detected in CI and opt-in?
   • Motivation: Duplicate dependencies increase build times and if used in libraries propagate to our projects where we have to fix or opt-in to it which takes extra time. A project detecting this in CI helps to avoid and keep the amount of multiple versions of same crates low.
   • ☑️ We validate and enforce this on our projects with cargo-deny
7. 👍 Are RustSec security advisories detected and verified in CI?
   • Motivation: This is great because then the maintainers of the project likely will find and fix advisories earlier before they reach our projects.
   • Typically done with cargo-deny, cargo-audit, or Github Depandabot.
   • ☑️ We validate and enforce this on our projects with cargo-deny

9. Change management

1. ⭐️ Are git tags used for published releases?
   • Motivation: This is standard practice and makes it easy to diff and compare what has changed between versions
2. ⭐️ Is semantic versioning correctly practiced for public libraries?
   • Motivation: This is best practice and it is hard to use libraries that accidentally or intentionally break semantic versioning.
3. ⭐️ Does it have a changelog or publish release notes?
   • Motivation: This is best practice and makes it easier to use, upgrade, and figure out what has changed in the project.
4. 👍 Is the source on GitHub?
   • Motivation: It is easier to contribute to and cross-reference issues and PRs on GitHub as our projects use GitHub

10. Sponsorship & support

1. ℹ️ Is the project actively looking for sponsorship / funding?
   • Motivation: Good to know as we may able to help support the maintainers and the projects development
2. ℹ️ How many others; individuals, organizations, or companies, are sponsoring the project or contributing to its funding and support in other ways?
   • Motivation: Good to know if we are alone in supporting the development or if there are a few or many other supporters, esp. of interest would be other companies/organizations.
3. 👍 Are we (Embark) sponsoring or supporting the project in any form?
   • Motivation: Maintainers are likely more interested in listening to us and taking our changes if we are doing our part to help support them.
   • See list of projects and individual developers we are sponsoring: OpenCollective, GitHub Sponsors and Patreon.

Very nice guide! 👍

Small nitpick:
The correct attribute to deny the use of unsafe is #![forbid(unsafe_code)].

Thanks fixed the typo!

Gotta say: Excellent guide 🙇

Is it intentional that the gist is currently marked "Secret"?

thanks! and good point, made it public!