[swift-build-dev] [swift-evolution] Proposal: Package Manager Version Pinning

Daniel Dunbar daniel_dunbar at apple.com
Fri Oct 14 13:06:02 CDT 2016 

> On Oct 14, 2016, at 10:37 AM, Eloy Duran <eloy.de.enige at gmail.com> wrote:
> 
> Hey Daniel,
> 
> Totally đź‘Ť

I'm not sure what this is to. :)

> I think you should pin by default, I wouldn't even provide some option to disable it.

Pin by default in the sense you are using it just means we automatically would create "Package.pins", that is the only behavior change (i.e., you don't inherit it from your dependencies).

Can you explain why simply needing to run one more command when you want to do this is a big deal? If you don't check it in, it doesn't change anything when you run `swift build`, that always kept a consistent set of versions. It changes `swift update`, but you were already running it at that point, so I'm not sure why that is desired.

Is it possible that we have a different expectation about what the tools do when you don't create this file? Our expectation is that you only ever get updated packages in a local build when you explicitly run `swift package update`. We expect to add features to notice when new dependencies are available, and let you choose to update, but for any one local build we are never talking about changing your dependencies without your interaction.

> As others have touched on, which I forgot to include, is that a library can choose to not include the lock file in SCM. Especially if the lib uses a CI for testing, that should bring up any issues with the latest versions of dependencies while keeping a paper trail (local lock file) of dependency versions that did work.
> 
> The key here is that it remains an explicit choice. The default should imo always be consistent builds.

Consistent across what axis? If you don't inherit it, then you still don't get consistency between yourself and your clients. If you don't check it in, you don't get much of a behavior change. However, as is touched on in other places, choosing to inherit has big downsides.

> ----
> 
> With regards to being in a different position, I doubt it. Swift developers are, like in other languages, just humans, we all make mistakes. I think we've all accepted that we'll always introduce bugs any time we have to make decisions, the same applies to determining semantic versions.
> 
> You could automate some of that (e.g. check interfaces), but you cannot guarantee behavioral intent and thus there's room left for human mistakes.

You can, however, in theory, apply a lot of automation for testing downstream dependencies (see earlier response).

> Please note, though, that I'm kinda pessimistic (realistic) about this and it's solely based on experience with established ecosystems, where people generally don't like being told how to do things. It may well be that your situation will turn out to be different.

This is the trickiest part of this design question. We don't really know how the ecosystem will behave, we don't know how resilient Swift packages will be w.r.t. bugs and semantic versioning, etc. At the end of the day, we all just are sticking our fingers to the wind and hoping to do the best thing for everyone.

One argument I haven't heard anyone refute is that we can always back down from this position, but the converse isn't true (because its impact will have permeated the ecosystem). I consider that an important point.

> As the Queen would say, Godspeed!

:)

 - Daniel

> 
> On 14 Oct 2016, at 18:42, Daniel Dunbar <daniel_dunbar at apple.com <mailto:daniel_dunbar at apple.com>> wrote:
> 
>> Hey Eloy,
>> 
>> Same question as I sent to Orta, can you detail exactly what you would prefer to change?
>> 
>>> On Oct 14, 2016, at 4:06 AM, Eloy Durán via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>>> 
>>> I cannot agree more with Orta.
>>> 
>>> > It drives the user away from taking full advantage of semantic versioning.
>>> 
>>> While I ideally subscribe to this thought, the truth of the matter is that this has proven unreliable on multiple occasions. Promoting the idea that it works is going to inevitably lead users into problems when they did not take explicit user action to update dependencies.
>> 
>> This makes a lot of sense to me. The open question in my mind is, given our other goals, are we in a position to make this model work, or to make it "the right model for us". Some of the other things I am considering (and maybe should go into the proposal):
>> 
>> 1. Swift is still evolving in a much greater way than other languages. That means we continue to need to have a very strong push towards forward versions. This is also true for the package manager features, we need the ecosystem to move forward aggressively so that we can move our own features forward aggressively.
>> 
>> 2. We have some lofty goals around semantic versioning, like trying to have a much deeper integration with the compiler, API, and ABI, and testing in order to help manage this more effectively.
>> 
>>> Semantic versioning is a great tool when you decide to sit down and explicitly update dependencies.
>> 
>> The problem is they go hand in hand. The more people pin versus following the semantic versioning, the more and more likely it is that those specifications are wrong. That leads to more pinning, which leads to more wrong specifications.
>> 
>> My preference is that we very aggressively commit to using semantic versioning, and then follow up with the entire cross functional design (an index, if we ever do one, the compiler features, any IDE integration) to make this work. I think we are in a better position than other tools which didn't have the ability to make as many cross functional changes (e.g., integration with the compiler to assist in semantic versioning).
>> 
>> On the other hand, I am also very pragmatic, and I respect your experience here... if this model simply isn't going to work, then we shouldn't try to go that way.
>> 
>> The big struggle I have is that if we go the other direction, and as a result people's semantic versions become poorly specified, we will never be able to recover. The converse is not true, if we start with this direction and realize it doesn't work, we can relax our behavior.
>> 
>>> > We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.
>>> 
>>> Again, ideally I would subscribe to this, but the truth is that when people are working on their projects they *really really really* do *not* like their builds breaking, especially not when it’s seemingly caused outside of their own fault.
>> 
>> One discussion we had a lot was that there are very different workflows between when you are largely the consumer of packages, versus when you are working on a package that is shared.
>> 
>> We generally agree that when you are simply the consumer of packages, pinning makes sense.
>> 
>> However, when you are primarily a distributor of packages (and this is expected to *most* of the momentum behind SwiftPM in the near term), I believe that it is very important to the ecosystem that the semver specs be correct, and so even if the team *wants* to pin, doing so would be actively harmful.
>> 
>>> In the end, it comes down to whether you prioritise surfacing bugs (long term happiness) over user-experience (short term happiness). As a dependency manager author, I can tell you that I agree with your ideals and would want to choose the former, but as a user of dependency managers I would choose UX and Getting (The) Things (I Really Wanted To Do) Done over surfacing bugs *any* day.
>> 
>> I really appreciate the feedback. Do my arguments that we might be in a position to do better here have any weight with you?
>> 
>>  - Daniel
>> 
>>> 
>>> [A.]		Eloy Durán
>>> 		Artsy <http://artsy.net/>
>>> 		
>>> 		eloy at artsy.net <mailto:eloy at artsy.net>		Twitter <https://twitter.com/alloy>
>>> 		GitHub <https://github.com/alloy>
>>>> On 14 Oct 2016, at 09:29, orta therox via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>>> 
>>>> Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.
>>>> 
>>>> NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the  feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.
>>>> 
>>>> -- 
>>>> 
>>>> [A.]	    Orta Therox
>>>> 
>>>>> w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
>>>>> @orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
>>>>> Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM
>>>> 
>>>>> On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:
>>>>> 
>>>>> Feedback welcomed!
>>>>> 
>>>>> Thanks,
>>>>> Ankit
>>>>> 
>>>>> --------
>>>>> 
>>>>> Package Manager Version Pinning
>>>>> Proposal: SE-XXXX
>>>>> Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
>>>>> Review Manager: TBD
>>>>> Status: Discussion
>>>>> Introduction
>>>>> This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.
>>>>> 
>>>>> Motivation
>>>>> As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.
>>>>> 
>>>>> Terminology
>>>>> 
>>>>> We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
>>>>> 
>>>>> Philosophy
>>>>> 
>>>>> Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.
>>>>> 
>>>>> Use Cases
>>>>> 
>>>>> Our proposal is designed to satisfy several different use cases for such a behavior:
>>>>> 
>>>>> Standardizing team workflows
>>>>> 
>>>>> When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.
>>>>> 
>>>>> This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.
>>>>> 
>>>>> Difficult to test packages or dependencies
>>>>> 
>>>>> Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.
>>>>> 
>>>>> Dependency locking w.r.t. deployment
>>>>> 
>>>>> When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.
>>>>> 
>>>>> Proposed solution
>>>>> We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.
>>>>> 
>>>>> This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).
>>>>> 
>>>>> The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.
>>>>> 
>>>>> This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.
>>>>> 
>>>>> In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).
>>>>> 
>>>>> The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.
>>>>> 
>>>>> Detailed Design
>>>>> We will add a new command pin to swift package tool with following semantics:
>>>>> 
>>>>> $ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
>>>>> The package-name refers to the name of the package as specified in its manifest.
>>>>> 
>>>>> This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples: 
>>>>> 
>>>>> $ swift package pin --all - pins all the dependencies.
>>>>> $ swift package pin Foo - pins Foo at current resolved version.
>>>>> $ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
>>>>> The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example: 
>>>>> 
>>>>> $ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
>>>>> Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.
>>>>> 
>>>>> We will add a new command unpin:
>>>>> 
>>>>> $ swift package unpin ( [--all] | [<package-name>] )
>>>>> This is the counterpart to the pin command, and unpins one or all packages.
>>>>> 
>>>>> We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.
>>>>> 
>>>>> We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:
>>>>> 
>>>>> $ swift package update [--repin]
>>>>> Update command errors if there are no unpinned packages which can be updated.
>>>>> 
>>>>> Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.
>>>>> 
>>>>> The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.
>>>>> 
>>>>> The update and checkout will both emit logs, notifying the user that pinning is in effect.
>>>>> 
>>>>> The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.
>>>>> 
>>>>> As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.
>>>>> 
>>>>> Impact on existing code
>>>>> There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.
>>>>> 
>>>>> Alternative considered
>>>>> We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> swift-build-dev mailing list
>>>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
>>>> 
>>>> _______________________________________________
>>>> swift-build-dev mailing list
>>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
>>> 
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20161014/80910fbf/attachment.html>

More information about the swift-build-dev mailing list