[swift-build-dev] [swift-evolution] [Proposal] Lock file for Swift Package Manager

Rick Ballard rballard at apple.com
Tue Dec 22 11:57:18 CST 2015 

Thanks for pushing on this, Ankit, Thomas, and Paul. As Doug mentioned in another thread, Apple is heading into our holiday shutdown until the new year, so I think we should schedule the actual evolution review for the first week of January. Let's get this hashed out in the meantime and ready for that review now, though.

Here are my comments:

– I know Cargo uses these terms already, but --lock and "lockfile" are very generic terms. A "package manager lockfile" could easily refer to a file multiple package manager processes use to avoid corrupting a shared database. I think calling this a "deplock" file (for "dependency lock") is much more specific without being much more verbose. And I'd suggest calling the option something like --lock-deps.

– Likewise, --bootstrap isn't very clear. I'd suggest --use-locked-deps instead. (If we have this flag at all... see below).

– I'm concerned about the lockfile being a 2nd source of truth (vs Package.swift) that could easily get out of sync. For example, if you update Package.swift to require a new minimum version, but forget to update the lockfile (or forget to commit the updated lockfile), users will wind up silently using a different version than that allowed by the Package.swift's specified version.

– I'm also uneasy with the lockfile being toml while Package.swift is swift. That seems inconsistent and requires users to work in two different configuration file syntaxes (even if toml is very simple).

– To address both of the above points, maybe the dep-lock info should be stored in Package.swift itself. In this case, any tool which automatically modifies your Package.swift for you would autoclear the dependency lock when it updates a package version, and since that data is in the same file, those two changes would get checked in together. When you're hand-editing the file, it'd be a lot easier to remember to clear (or update) the locked dependency. And you'd ensure that the Package.swift data is always in sync with your dependency lock across revisions and branches, since both data is in the same file (at least for direct dependencies).

Since dependency locks apply to non-direct dependencies as well, we would need to add a new package property for modeling the dependency lock for an otherwise-unspecified dependency. And we might require that the locks fully specify the properties of the dependency they apply to, so if the required version of an indirect dependency changes, we can tell that the dependency lock is out of date. For example, say your "FooApp" package depends on "Lib1" and "Lib2", which both depend on "LibBar". "Lib1" might specify that it depends on LibBar versions: Version(1,0,1)..<Version(2,0,0)), while "Lib2" might specify LibBar versions: Version(1,0,0)..<Version(1,5,0)). FooApp's indirect dependency on LibBar is thus constrained to versions: Version(1,0,1)..<Version(1,5,0)), and that's what we'd record for the dependencyLock:

let package =
 Package(
    name: "FooApp",
    dependencies: [
        .Package(url: "ssh://git@example.com/Lib1.git"),
	.Package(url: "ssh://git@example.com/Lib2.git"),
    ],
    dependencyLocks: [
	.DependencyLock(lockRevision: c611ad62500182cae041abe83db908c2ea8e4485, .Package(url: "ssh://git@example.com/Lib1.git"),
	.DependencyLock(lockRevision: 1fb095a46ff55161876380067344ff641b8e95e2, .Package(url: "ssh://git@example.com/Lib2.git"),
        .DependencyLock(lockRevision: db2e873d530c72023af00ce7fe9a37211b8d2fbc, .Package(url: "ssh://git@example.com/LibBar.git", versions: Version(1,0,1)..<Version(1,5,0)),
    ],
)

If Lib2 then updated its version specification to  Version(1,0,0)..<Version(1,6,0). we could tell that our dependency lock was out of date and prompt you to create a new lock.

Note that we wouldn't expect you to have to hand-author the .DependencyLock (and manually repeat the version range for packages you depend on); normally this would be autogenerated by the package manager.

– It would be nice if we could warn, when building with dependency locks, if your dependencyLock revision does not match a dependency's version specifier. For example, if your dependency specifies (2,0,1)..<Version(2,1,0), and your dependencyLock is db2e873d530c72023cf00ce7fe9a37211b8d2fbc, we would check and make sure that some revision 2.0.1 or greater contains db2e873d530c72023cf00ce7fe9a37211b8d2fbc, but that it's not reachable from 2.1.0 or greater. That said, `git tag --contains` is probably not fast enough for nontrivial repositories to run on each build for the purposes of issuing this warning, so this might be a non-starter for performance reasons.

– Should we always automatically use the deplock info? Ankit's proposal said no, while Thomas and Paul said yes. I think that it makes sense for most use cases to use a stable version of your dependencies and only update to a newer version explicitly, instead of having that happen implicitly when you build if there happens to be a new version. That favors always using the deplock info. Thus I think I like Paul's proposal to "always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version." So I'd suggest:

	– `swift build --update-deps` updates all dependencies to the latest allowed version and sets/updates dependency locks for each dependency.
	– `swift build --update-dep=<package name>` updates the named dependency to the latest allowed version and sets/updates dependency lock for that dependency.
	– `swift build --lock-deps` looks at the HEAD of all cloned dependencies and sets the dependency lock in the top-level Package.swift to that HEAD commit. This is useful when you want to explicitly lock to something other than the latest allowed version of a package.
	– `swift build --lock-dep=<package name> looks at HEAD of the cloned named dependency and sets the dependency lock in the the top-level Package.swift for that one dependency.
	– `swift build` clones any dependencies that aren't already cloned, checks out the locked commit for all dependencies (whether they were already cloned before or not), and adds dependency locks to Package.swift for any packages that don't have a lock already. This also warns if the dependency lock's package specifier doesn't match the actual package specifier where that dependency is defined, which indicates that your locks are out of date with respect to your dependency specifications.

One downside to this behavior is it makes it easy to mess up when modifying your dependencies locally. If you make an edit to a cloned dependency and commit it, and then `swift build` the top-level package, swiftpm will automatically revert HEAD of that dependency to the locked commit, so it won't actually build your change. In order to avoid this, you need to run `swift build --lock-deps` after committing a local change to a cloned dependency, and it's easy to forget to do so. That said, the alternative behavior – where `swift build` preserves the state of your dependencies by default – means that if you've built a package in the past, and you pull and get a new Package.swift with new dependency locks, you won't automatically get those dependencies updated when you build, since you already have cloned dependencies whose HEADs are stale and don't match the new dependency locks. The latter problem seems worse than the former. I'm open to ideas to how to solve both problems nicely; the ways I've thought of so far make this proposal even more complex.

– This proposal still doesn't fully address how you can use a branch for your dependency instead of a version tag, which was one of the reasons this topic came up in the first place. You could do so by checking out the branch in your cloned dependency and then using `swift build --lock-deps` to lock to that commit, but if you do a `swift build --update-deps` we'll blow that away. To solve this, perhaps a dependency lock could have an additional optional "overridingRef" property which, if specified, overrides the version specifier for the package. That means that `swift build --update-deps` will now update the package to what that ref points to. The --lock-dep option could allow a follow-on option --lock-overriding-ref which takes the overriding ref to set.

– Likewise, this mechanism could be used to allow you to override the source of a dependency for your indirect dependencies. For example, if you depend on "Lib1", which depends on git at github.com:Somewhere/LibFoo.git, but you actually want to use your own fork of LibFoo – git at github.com:YourName/LibFoo.git – the dependency lock would allow that override. This would be done with an "overridingURL" property on the dependency lock.

– I am concerned about the complexity and additional learning curve this behavior brings to the package manager. That said, this seems like important functionality.

Thoughts?

	- Rick

> On Dec 20, 2015, at 1:22 PM, Paul Cantrell via swift-evolution <swift-evolution at swift.org> wrote:
> 
> +1 for Ankit’s general idea. Details of the proposal aside, I’ll say from experience with bundler that it’s immensely useful — a lifesaver! — to know the exact version of the dependencies another author was using. This has saved my neck more than once.
> 
> IMO it’s useful to have a lock file checked in even for libraries — just not pushed forward to client projects. You still want to know what versions the library’s tests last passed against, both for CI and for diagnosing downstream breakage.
> 
> -1 to this:
> 
>> [The] lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
>> $ swift build always ignores the lock file and uses local state of Packages dir / Package.swift
>> …
>> To lock the current state of Packages user can run  $ swift build --lock
> 
> 
> A couple of problems with that:
> 
> (1) Package.swift can specify a version range. You may want to update to the latest patch release without actually modifying Package.swift. I agree with Thomas: there should be a command to update dependencies to the latest matching version. This command should also be able update a single dependency:
> 
> 	swift build --update SomePackage
> 
> (2) I don’t like the idea of the build system running in two separate modes, where sometimes the lock file is ignored and sometimes takes precedence. (If there’s a desire to run in an “unlocked” mode, how about it just doesn't generate the .lock if not already present, and always uses it if it is present?) In practice, though, I’ve found the bundler model works quite well: always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version.
> 
> Cheers,
> 
> Paul
> 
>> On Dec 20, 2015, at 9:51 AM, Thomas Guthrie via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>> 
>> Personally, I’d be more in favour of having something similar to Cargo (Rust’s package/crate manager):
>> 
>> 1. `swift build`
>> 
>> Almost the same as it is now, expect if there’s no Package.lock it creates one and subsequent builds use the same dependencies.
>> 
>> 2. `swift build --update` or maybe eventually `swift update`
>> 
>> Updates the dependencies in Package.lock to their newest versions and writes them to disk. It should probably build the project as well but possibly makes less sense if its `swift update`.
>> 
>> Similar to Bundler and Cargo you’d check in your Package.lock for app projects and ignore it for library projects.
>> 
>> I’m not really sure what their motivation was for having a lock file always created, it definitely favours “app” projects heavily, but I’ve been messing around with Rust recently and it works pretty well honestly. Maybe there’s a way of making the experience better when the package is solely a library? Personally, if you’re developing a library and `swift build` updates a dependency that breaks everything it’s probably better to know then, whereas with an app you probably want to be working to a lock file and checking what happens when you update dependencies individually.
>> 
>> As for the format of Package.lock, I think it might have to be more complicated than shown to be able to handle the possibility of multiple versions of a dependency etc? Haven’t had a chance to mess around with swiftpm enough yet to say though.
>> 
>> (/end ramble of first thoughts)
>> 
>> — Thomas
>> 
>>> On 20 Dec 2015, at 09:01, Ankit Agarwal via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>>> 
>>> Lock File for Swift Package Manager
>>> Introduction
>>> A Package.lock file containing list of resolved dependencies generated by swiftpm.
>>> 
>>> Motivation
>>> Package.lock file can be helpful in situations like : 
>>> 
>>> Reproduce exact versions of dependencies on different machine
>>> 
>>> * Multiple developers working on a package would want to use the exact versions (including minor and patch) of the dependencies declared in the manifest file
>>> * Also helpful when a build is being performed on a remote machine eg CI 
>>> Pointing a dependency to an untagged commit
>>> 
>>> Sometimes it might be helpful to lock a dependency to a particular commit ref for which a tagged version is unavailable in cases such as :
>>> 
>>> * Forking a 3rd party library and making it swiftpm compatible for temporary use until officially supported by the author
>>> * Package is in active development and not ready for a release tag
>>> Proposed Solution
>>> swiftpm generates a simple Package.lock file after resolving the dependency graph for that package in some simple format.
>>> 
>>> Detailed Design
>>> 1. Initial$ swift build resolves the dependency graph and generates a Package.lock file similar to :
>>> ssh://github.com/foo/bar <http://github.com/foo/bar> "v1.2.3"
>>> http://github.com/foo/baz <http://github.com/foo/baz> "v1.0.0"
>>> ../local/git/repo "v3.4.4"
>>> lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
>>> $ swift build always ignores the lock file and uses local state of Packages dir / Package.swift
>>> 
>>> 2. User modifies the cloned packages in Packages dir and when satisfied with the current code of the dependency, commits and pushes it.
>>> To lock the current state of Packages user can run  $ swift build --lock which might result something similar to 
>>> ssh://github.com/foo/bar <http://github.com/foo/bar> "248441ff375a19c4365d00d6b0706e11173074f6"
>>> http://github.com/foo/baz <http://github.com/foo/baz> "v1.0.0"
>>> ../local/git/repo "v3.4.4"
>>> the lock file is committed into the package repo for others to use.
>>> 
>>> 3. A command like $ swift build --bootstrap will always use the lock file to fetch and checkout the dependencies.
>>> This is useful because running  $ swift build might give a higher patch or minor version of the dependency.
>>> 
>>> 4. If some dependency depends on commit hash (ie non-tagged commit) the author mentions that in their readme and the end user and maybe other parallel dependencies will have to use only that commit hash in order to avoid dependency hell.
>>> 
>>> 5. Allow declaring a dependency without versions in the manifest file for user wanting to use a untagged dependency. This should probably only be allowed in debug configuration.
>>> Impact on existing code
>>> None as this would be additional functionality to swift package manager
>>> Alternatives Considered
>>> One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html> thread it might not be the best idea.
>>> 
>>> -- 
>>> Ankit
>>>  _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
>> 
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
> 
>  _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20151222/73c6fc7b/attachment.html>

More information about the swift-build-dev mailing list