[swift-build-dev] [swift-evolution] Proposal: Package Manager Version Pinning

Fri Oct 14 12:51:25 CDT 2016

> On Oct 14, 2016, at 10:03 AM, Max Desiatov via swift-evolution <swift-evolution at swift.org> wrote:
> 
> The other point is that when working in a multi-language environment, having conventions such as this broken causes additional mental burden. That is, after working with JavaScript/Rust/iOS with CocoaPods and then switching to Swift, would require a lot of unneeded context switching, as in: "Where's that lockfile? I can't find it. Oh great, turns out it's not a lockfile, it's something else".

In Swift, we have pretty consistently tried to choose the "right" answer to make the resulting language consistent and beautiful.

I'm perfectly happy to have a discussion about the naming, but I would like it to be driven by what we believe the "right" answer is, not simply by deference to existing solutions. I do sympathize with wanting to be consistent where there is little value in diverging, but most people I have discussed this with agree that "lock" is actually the *wrong* word to use for this operation.

Also, please be mindful that your perspective is biased by the tools you are familiar with. Python's pip tool, for example, uses "freeze" for this, and Heroku expects "requirements.txt", so someone coming from that ecosystem could make the same argument in another direction. I find arguments about "A does X so we should do X" most compelling when it comes with a relatively complete survey of existing tools. not just a particular slice of possible tools.

Whether "pinning" is the right word is a different debate, but when we view pinning as a workflow-focused feature, versus the specification in the manifest (which is the "requirement"), then I think the connotation actually works fairly well (e.g., a pinboard is something you pin to while working, or pinning a dress while you stitch it). I also wasn't a huge fan of pin initially, but as it bounced around in my head for a while I really started to like it, for exactly this connotation reason.

 - Daniel

> 
> With best regards, Max.
> 
>> On 14 Oct 2016, at 17:59, Max Desiatov <max.desiatov at gmail.com <mailto:max.desiatov at gmail.com>> wrote:
>> 
>> I also agree with the point that .lock extension better suits here and adheres to a convention already established in other languages. I personally would prefer the file to be named Package.lock, not Package.pins and also think that it would help newcomers who already used other package managers, especially CocoaPods.
>> 
>> With best regards, Max.
>> 
>>> On 14 Oct 2016, at 17:43, Alexis via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>>> 
>>> 
>>> 
>>>> On Oct 14, 2016, at 2:01 AM, Ankit Aggarwal via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:
>>>> 
>>>> Feedback welcomed!
>>>> 
>>>> Thanks,
>>>> Ankit
>>>> 
>>>> --------
>>>> 
>>>> Package Manager Version Pinning
>>>> Proposal: SE-XXXX
>>>> Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
>>>> Review Manager: TBD
>>>> Status: Discussion
>>>> Introduction
>>>> This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.
>>>> 
>>>> Motivation
>>>> As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.
>>>> 
>>>> Terminology
>>>> 
>>>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
>>>> 
>>> I’ve never seen this cause any actual confusion, nor has anyone I know who teaches/develops these sorts of tools. As far as I can tell, the broader programming community is rapidly converging on this as standard terminology:
>>> 
>>> * Gemfile.lock (Ruby)
>>> * Cargo.lock (Rust)
>>> * Composer.lock (PHP)
>>> * yarn.lock (JS)
>>> * pubspec.lock (Dart)
>>> * Podfile.lock (Swift/Objc!)
>>> 
>>> Diverging from this seems counter-productive.
>>>> Philosophy
>>>> 
>>>> Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.
>>>> 
>>>> Use Cases
>>>> 
>>>> Our proposal is designed to satisfy several different use cases for such a behavior:
>>>> 
>>>> Standardizing team workflows
>>>> 
>>>> When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.
>>>> 
>>>> This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.
>>>> 
>>>> Difficult to test packages or dependencies
>>>> 
>>>> Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.
>>>> 
>>>> Dependency locking w.r.t. deployment
>>>> 
>>>> When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.
>>>> 
>>>> Proposed solution
>>>> We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.
>>>> 
>>>> This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).
>>>> 
>>>> The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.
>>>> 
>>>> This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.
>>>> 
>>>> In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).
>>>> 
>>>> The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.
>>>> 
>>>> Detailed Design
>>>> We will add a new command pin to swift package tool with following semantics:
>>>> 
>>>> $ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
>>>> The package-name refers to the name of the package as specified in its manifest.
>>>> 
>>>> This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples: 
>>>> 
>>>> $ swift package pin --all - pins all the dependencies.
>>>> $ swift package pin Foo - pins Foo at current resolved version.
>>>> $ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
>>>> The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example: 
>>>> 
>>>> $ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
>>>> Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.
>>>> 
>>>> We will add a new command unpin:
>>>> 
>>>> $ swift package unpin ( [--all] | [<package-name>] )
>>>> This is the counterpart to the pin command, and unpins one or all packages.
>>>> 
>>>> We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.
>>>> 
>>>> We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:
>>>> 
>>>> $ swift package update [--repin]
>>>> Update command errors if there are no unpinned packages which can be updated.
>>>> 
>>>> Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.
>>>> 
>>>> The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.
>>>> 
>>>> The update and checkout will both emit logs, notifying the user that pinning is in effect.
>>>> 
>>>> The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.
>>>> 
>>>> As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.
>>>> 
>>>> Impact on existing code
>>>> There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.
>>>> 
>>>> Alternative considered
>>>> We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.
>>>> 
>>> 
>>> I agree with the others that this is the better solution.
>>> 
>>> With regards to the constraining problem, the key insight adopted by Cargo/Yarn/Bundler is to distinguish libraries from applications. A library shouldn’t pin its dependencies, while an application should. This ensures that the ecosystem itself is maximally unconstrained, while ensuring actual applications continue to reliably build, regardless of ecosystem changes and the computer that it was built on. If a version of a library has trouble building with different versions, it should ideally specify that with its dependency constraints, not a lockfile.
>>> 
>>> This also ensures that there’s diverse testing of versions: CI for applications will verify particular configurations, while CI for libraries will verify the latest-and-greatest works.
>>> 
>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> swift-evolution mailing list
>>>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>>>> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
>>> 
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
> 
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20161014/3713212e/attachment.html>