[swift-build-dev] [swift-evolution] [Proposal] Lock file for Swift Package Manager

[Max Howell]

> – Likewise, --bootstrap isn't very clear. I'd suggest --use-locked-deps instead. (If we have this flag at all... see below).

I’d prefer a workflow that omits —bootstrap, to repeat my previous email:

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

This makes understanding what happens simpler: `swift build` always uses the lock file if it is present.

This makes reliably building apps possible since you will always be building what everyone else built when the sources where committed.

> – I'm also uneasy with the lockfile being toml while Package.swift is swift. That seems inconsistent and requires users to work in two different configuration file syntaxes (even if toml is very simple).

The proposal is not TOML, just some trivial unspecified markup. If we are to have two files then I think it should either be Swift or just a basic list like the proposal says.

> – To address both of the above points, maybe the dep-lock info should be stored in Package.swift itself. In this case, any tool which automatically modifies your Package.swift for you would autoclear the dependency lock when it updates a package version, and since that data is in the same file, those two changes would get checked in together. When you're hand-editing the file, it'd be a lot easier to remember to clear (or update) the locked dependency. And you'd ensure that the Package.swift data is always in sync with your dependency lock across revisions and branches, since both data is in the same file (at least for direct dependencies).

This certainly needs more discussion. I’ll wait for Daniel to chime in.

> Since dependency locks apply to non-direct dependencies as well, we would need to add a new package property for modeling the dependency lock for an otherwise-unspecified dependency. And we might require that the locks fully specify the properties of the dependency they apply to, so if the required version of an indirect dependency changes, we can tell that the dependency lock is out of date. For example, say your "FooApp" package depends on "Lib1" and "Lib2", which both depend on "LibBar". "Lib1" might specify that it depends on LibBar versions: Version(1,0,1)..<Version(2,0,0)), while "Lib2" might specify LibBar versions: Version(1,0,0)..<Version(1,5,0)). FooApp's indirect dependency on LibBar is thus constrained to versions: Version(1,0,1)..<Version(1,5,0)), and that's what we'd record for the dependencyLock:
> 
> let package =
>  Package(
>     name: "FooApp",
>     dependencies: [
>         .Package(url: "ssh://git@example.com/Lib1.git <ssh://git@example.com/Lib1.git>"),
> 	.Package(url: "ssh://git@example.com/Lib2.git <ssh://git@example.com/Lib2.git>"),
>     ],
>     dependencyLocks: [
> 	.DependencyLock(lockRevision: c611ad62500182cae041abe83db908c2ea8e4485, .Package(url: "ssh://git@example.com/Lib1.git <ssh://git@example.com/Lib1.git>"),
> 	.DependencyLock(lockRevision: 1fb095a46ff55161876380067344ff641b8e95e2, .Package(url: "ssh://git@example.com/Lib2.git <ssh://git@example.com/Lib2.git>"),
>         .DependencyLock(lockRevision: db2e873d530c72023af00ce7fe9a37211b8d2fbc, .Package(url: "ssh://git@example.com/LibBar.git <ssh://git@example.com/LibBar.git>", versions: Version(1,0,1)..<Version(1,5,0)),
>     ],
> )

This makes sense, if we go one-file.

> If Lib2 then updated its version specification to  Version(1,0,0)..<Version(1,6,0). we could tell that our dependency lock was out of date and prompt you to create a new lock.

Indeed, this is what I envisage: respect the lock file as the primary truth and if the manifest and lock file conflict, error (for major version changes) or warn.

> Note that we wouldn't expect you to have to hand-author the .DependencyLock (and manually repeat the version range for packages you depend on); normally this would be autogenerated by the package manager.

Indeed.

> – Should we always automatically use the deplock info? Ankit's proposal said no, while Thomas and Paul said yes. I think that it makes sense for most use cases to use a stable version of your dependencies and only update to a newer version explicitly, instead of having that happen implicitly when you build if there happens to be a new version. That favors always using the deplock info. Thus I think I like Paul's proposal to "always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version." So I'd suggest:

> 	– `swift build --update-deps` updates all dependencies to the latest allowed version and sets/updates dependency locks for each dependency.
> 	– `swift build --update-dep=<package name>` updates the named dependency to the latest allowed version and sets/updates dependency lock for that dependency.
> 	– `swift build --lock-deps` looks at the HEAD of all cloned dependencies and sets the dependency lock in the top-level Package.swift to that HEAD commit. This is useful when you want to explicitly lock to something other than the latest allowed version of a package.
> 	– `swift build --lock-dep=<package name> looks at HEAD of the cloned named dependency and sets the dependency lock in the the top-level Package.swift for that one dependency.
> 	– `swift build` clones any dependencies that aren't already cloned, checks out the locked commit for all dependencies (whether they were already cloned before or not), and adds dependency locks to Package.swift for any packages that don't have a lock already. This also warns if the dependency lock's package specifier doesn't match the actual package specifier where that dependency is defined, which indicates that your locks are out of date with respect to your dependency specifications.

Sounds good, though I’d drop -deps from both.

> One downside to this behavior is it makes it easy to mess up when modifying your dependencies locally. If you make an edit to a cloned dependency and commit it, and then `swift build` the top-level package, swiftpm will automatically revert HEAD of that dependency to the locked commit, so it won't actually build your change. In order to avoid this, you need to run `swift build --lock-deps` after committing a local change to a cloned dependency, and it's easy to forget to do so. That said, the alternative behavior – where `swift build` preserves the state of your dependencies by default – means that if you've built a package in the past, and you pull and get a new Package.swift with new dependency locks, you won't automatically get those dependencies updated when you build, since you already have cloned dependencies whose HEADs are stale and don't match the new dependency locks. The latter problem seems worse than the former. I'm open to ideas to how to solve both problems nicely; the ways I've thought of so far make this proposal even more complex.

IMO if the Packages/ directory has unstaged or uncommitted/unpushed changes should be a build error.

However it is important to allow editing of Packages/. In other language packaging systems, fixing or modifying your dependencies is tedious, and this discourages people from improving the ecosystem. 

So in my opinion it is important that this is possible and thus I’d like to propose a flag, —ignore-lock perhaps that allows the above error to be ignored, albeit with a very visible warning with a link to documentation about this.

> – This proposal still doesn't fully address how you can use a branch for your dependency instead of a version tag, which was one of the reasons this topic came up in the first place. You could do so by checking out the branch in your cloned dependency and then using `swift build --lock-deps` to lock to that commit, but if you do a `swift build --update-deps` we'll blow that away. To solve this, perhaps a dependency lock could have an additional optional "overridingRef" property which, if specified, overrides the version specifier for the package. That means that `swift build --update-deps` will now update the package to what that ref points to. The --lock-dep option could allow a follow-on option --lock-overriding-ref which takes the overriding ref to set.

The proposal should mention this.

> – Likewise, this mechanism could be used to allow you to override the source of a dependency for your indirect dependencies. For example, if you depend on "Lib1", which depends on git at github.com:Somewhere/LibFoo.git, but you actually want to use your own fork of LibFoo – git at github.com:YourName/LibFoo.git – the dependency lock would allow that override. This would be done with an "overridingURL" property on the dependency lock.
> 
> – I am concerned about the complexity and additional learning curve this behavior brings to the package manager. That said, this seems like important functionality.

I think we can minimize the complexity further.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20151222/cbeb9ba7/attachment.html>