Lock file proposal

Lockfile.md

Dependency Locking with the Swift Package Manager

Introduction

In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and feature improvements. When working against a collection of dependencies, especially in a team, it is vital to lock the graph and update it so that all team members receive the exact same graph at the same time.

The typical flow for this in dependency management is to create and use a dependency “Lockfile”.

Motivation

Our Lockfile system would be used for:

• Ensuring that at any specific version-control revision the exact state of the dependency graph is recorded
• Allowing users to lock, alter or “override” the URLs and or versions of packages in the graph. This is often required because:
  • They need to use a fork of a dependency that has a fix, or a private additional feature
• Allowing users to commit and thus lock source alterations to their dependencies

Proposed Solution

A file: Packages/Lockfile exists in the source tree alongside the cloned packages.

• The URL and versions of cloned packages
• An inline diff of any modifications made to those packages relative to their pristine cloned states

This file should be checked-in with projects.

This file may be checked-in with packages designed for consumption in projects, if so it would allow the user to attempt to resolve a graph that exactly matches that of package author (though this will not always be possible, in some situations it is useful to try and replicate an exact dependency graph for reliability and bug fix purposes)

The Lockfile is generated by SwiftPM.

Any modifications made to the clones in Packages are recorded in the Lockfile as part of the flow described in the next section. Modifications here means: changes to git remotes and any local changes to the sources.

Detailed Design

In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate Packages/Lockfile.

The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:

error: modified sources and unlocked sources
Execute `swift build --lock` or `swift build --ignore-lockfile`

It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.

When swift build --lock is specified the package manager regenerates the lockfile detailing the active git remote and the SHA that is checked-out. For local modifications it generates a diff against the check out and stores that in the Lockfile too.

Packages/Lockfile

There was concern with this feature using a file with lock in its name since this implies UNIX lockfiles. However, the emerging world of language dependency-managers has more or less settled on this term, so in order to be consistent our compromise is to call the file Lockfile and place it in the generated Packages directory. Thus it is clear that it is the lock for the cloned Packages and be not being named foo.lock it does not appear to be a UNIX lock.

The exact design of the contents of the Lockfile will be explored during iterative development, but here is a possible example (using TOML):

[package]
clone: Packages/PromiseKit-3.0.3
origin: https://github.com/mxcl/PromiseKit
version: 3.0.3

[package]
clone: Packages/Alamofire-1.2.3
origin: https://github.com/a-fork-somewhere/Alamofire
branch: crucial-fix

[package]
clone: Packages/Quick-1.2.3
origin: https://github.com/Quick/Quick
tag: 1.2.3
diff: INLINE-DIFF

Rationale for Local Diffs

Storing local diffs encourages users to edit, fix and improve their packages while preventing them from making changes that will not be stored as part of the dependency information revision history.

Workflow — Regular Build

The user is expected to commit the lock file into the git repo for others to reproduce the exact versions of dependencies on their system. This is optional, but strongly encouraged. Not checking it in is essentially saying: my project has uncontrolled dependencies: good luck!

1. User runs swift build
2. If Packages/ contains clones and a Lockfile SwiftPM skips to 7.
3. If Packages/ contains clones and no Lockfile the lockfile is generated from the clones
4. If Packages/ contains checked out sources without git information and no Lockfile SwiftPM fetches the git information and provided there is no diff, genereates the Lockfile, if there is variation it is an error *
5. If Packages/Lockfile is present its dependency graph is used
6. If Packages doesn't exist or is empty the depedency graph is resolved, packages are cloned and the Lockfile is generated
7. Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.

• This scenario is for users who choose to check in their complete dependency sources instead of a Lockfile, this is in fact a superior way to lock dependencies (because otherwise there is no gauarentee your dependencies will be unchanged since when you locked—it's the Internet dude) but to many it is distateful to check-in “derived data”.

Workflow — Making Modifications

1. User makes local modification to a dependency’s sources
2. User runs swift build
3. swift build errors out.
4. User must either lock the graph or run with --ignore-lock

Runing swift build --lock regenerates the lockfile, but doesn not build.

Workflow — Overriding Packages

1. User steps into a Package directory eg. Packages/Foo-1.2.3
2. User changes the origin of Foo to their own fork
3. User alters HEAD to point to a fix in their fork
4. swift build errors out.
5. User must either lock the graph or run with --ignore-lock

Running swift build --lock regenerates the lockfile, the new origin and tag is stored so if this project is freshly cloned it will use the overrides.

Workflow — Consuming a Package that wants you to override a dependency

A package, foo, depends on a package bar, bar has a bug, the author of foo fixes the bug in their own fork. The author of foo wants consumers of foo to use the bug fixed fork. What should happen here?

This is potentially a source of dependency hell, so we must be careful here to provide tooling that makes this situation tennable.

Solution: Dependent packages cannot cause overrides in root packages.

This is certainly unpleasant for package authors, but we have a responsibility to ensure a solid packaging ecosystem.

A package author would then specify the override that an end-user should configure for their own Lockfile.

As an alternative, the package author could change the dependency in their Package.swift to point to their own fork, and this is in fact fine. Especially if we follow through on our promise to lint the APIs of packages as part of a future publish step. As long as the API of a package is precise (and module collisions can be avoided via a namespacing system that is being proposed.) then we have avoided dependency hell.

Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.

Miscellaneous Details

• The Lockfiles of dependencies are ignored and only the root Packages/Lockfile is used when resolving the graph (though we can add an optional feature to take into account other Lockfiles in the future)
• The user is not expected to interact with this file as it'll always be generated by SwiftPM.

##Alternatives Considered

One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this thread it might not be the best idea.

Using Git submodules for this feature was considered. It still could be implemented this way. We won't do it this way however because git-submodules are not widely understood and a Lockfile is clear. Ultimately it was firmly rejected because the local-diff feature could not combined.

Oh I'm so glad this is coming. I really like the details of this proposal as well.

Just one thing - having the Lockfile in Packages/Lockfile seems like a bad idea to me. Couple of hurdles right away

• for people who don't want to check in their dependencies, we can't just add Packages to our .gitignore, because that would make it impossible to check in the Lockfile
• swift build --clean=dist deletes the whole Packages directory, removing the Lockfile, while I can imagine just wanting to delete and re-pull my dependencies with swift build --clean=dist; swift build. This will potentially generate a new Lockfile, even though there was a completely valid one which I didn't explicitly say I want to overwrite.

The two points above are just the low-hanging fruit that came into my mind, but I prefer the approach CocoaPods takes with Podfile.lock being next to the repo manifest, instead of in the Pods folder (which instead contains a Manifest.lock AFAIK).

For me, a regular troubleshooting step is removing the Packages and .build folders and rebuilding. In my opinion, this step should not include the risk of altering the locked dependency graph, because both folders are just "Derived Data", however the Lockfile is a source of truth. And mixing truth with generated files feels wrong to me.

I'm happy to be proven wrong, but I'd suggest to move the location of the Lockfile next to Package.swift.

for people who don't want to check in their dependencies, we can't just add Packages to our .gitignore, because that would make it impossible to check in the Lockfile

You can:

/Packges
!/Packages/Lockfile

And we can make this the default gitignore swift init generates.

swift build --clean=dist deletes the whole Packages directory, removing the Lockfile, while I can imagine just wanting to delete and re-pull my dependencies with swift build --clean=dist; swift build. This will potentially generate a new Lockfile, even though there was a completely valid one which I didn't explicitly say I want to overwrite.

We have control over that, so we could just not delete the Lockfile.

For me, a regular troubleshooting step is removing the Packages and .build folders and rebuilding. In my opinion, this step should not include the risk of altering the locked dependency graph, because both folders are just "Derived Data", however the Lockfile is a source of truth. And mixing truth with generated files feels wrong to me.

True.

I'm happy to be proven wrong, but I'd suggest to move the location of the Lockfile next to Package.swift.

This makes people who think of UNIX lockfiles unhappy.

I'll incorporate this feedback.

👍

@mxcl I realized one more thing.

You can:

/Packges
!/Packages/Lockfile

Adding this specific entry into .gitignore will only work if the user doesn't first create a repo on GitHub (or other service), which already offers to create a .gitignore for you (e.g. Swift specific entries). I actually always use that path, meaning that in our case, even if the user runs swift build --init, they won't end up with this entry in their .gitignore (we only add it if no existing .gitignore is found). Which might lead to them never committing their Lockfile into git and thus possibly ending up with an unstable dependency tree.

I guess this again strengthens my view of keeping the Lockfile out of Packages. And from what you mentioned, it seems like people have a larger issue with the name containing the word "lock" rather than with the location of it. So maybe we could solve both problems by moving the Lockfile next to Package.swift and also offering to rename it to something that doesn't scare people away.