[swift-build-dev] Proposal: Git Branch and ref support for dependencies in Swift Package Manager

Tue Dec 15 14:00:41 CST 2015

Hi Ankit,

Thanks for starting work on this proposal, I agree this is a feature we sorely need. I think there are two problems this proposal is tackling, and it may be worth to call them out separately:

1. It may be necessary to point at an alternate "namespace" for pulling a dependency from (branch, etc.), because swiftpm support is not in the mainline.

2. It is useful when developing packages and hierarchies of packages to be able to point to non-tagged commits.

The reason I think it makes sense to separate these is that #1 is a problem which applies to all of the packages in a dependency tree. For example, some popular Swift projects may want to add support for swiftpm + Swift 2.2, but they can't put that into their mainline, or their mainline tags. One natural way to support that use case would be to have "namespace tags", like "swiftpm-1.2". That is something which deserves its own proposal, if anyone wants to tackle it.

For #2, I think we should encourage a usage model where the "Package.swift is the thing you should check in". We want to always encourage people to write manifests appropriate for sharing. The need to *override* the behavior strikes me as a mostly local issue, which should be addressed by a mechanism outside of the Package.swift.

Max and I just discussed this briefly, and came up with what I think is a pretty good strategy here:

1. Instead of tackling this problem in the manifest, start by coming up with a proposal for supporting "lock files". Something like:
 - Support a Packages.lock file in the root folder.
 - Define a simple syntax for associating package URLs with locked versions.
 - Implement package manager support for using that file.

Once that is in place, you will already be able to use that file to solve this particular problem (you can just edit the lock file to point at a branch or at a commit). We already know we need "lock file" support, so this is also handy in that it is solving this problem with a generally useful feature.

2. Max had a great suggestion for how "locking" should work:
 - "swift build --lock" (or however it is spelled) would simply create the .lock file based on the current state of your Packages tree.

This simple idea gives a great workflow for developing packages: you start by declaring your dependencies in your Package.swift, you use `swift build` to pull them down for you, then you start hacking on them (in your local checkout) to get the changes you need, iterating using `swift build --lock && swift build`, and then once you have something that works you can move to pushing those changes back and you already have the right .lock file

That is another feature to be proposed and implemented, but I think can be decoupled from the basic initial Packages.lock support.

What do you think?

 - Daniel

> On Dec 15, 2015, at 6:09 AM, Ankit Agarwal via swift-build-dev <swift-build-dev at swift.org> wrote:
> 
> I think even the source URL will also work on tags 
> 
> On Tue, Dec 15, 2015 at 1:33 PM, Kostiantyn Koval <konstantin.koval1 at gmail.com <mailto:konstantin.koval1 at gmail.com>> wrote:
> There is a plan to add ability to support fetching packages by url for development purpose described here
> Importing Dependencies by Source URL form "PackageManagerCommunityProposal" 
> https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#importing-dependencies-by-source-url <https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#importing-dependencies-by-source-url>
> 
> Kostiantyn
> 
>> On 14 Dec 2015, at 22:36, Kostiantyn Koval <konstantin.koval1 at gmail.com <mailto:konstantin.koval1 at gmail.com>> wrote:
>> 
>> +1 for adding this.
>> 
>> Ability to point to commit instead of a tag is crucial. 
>> Often happens that a bug was fixed in the depended library but it wasn't released/tagged.
>> As well it helps during development.
>> 
>> Kostiantyn
>> 
>>> On 14 Dec 2015, at 20:14, Marc Knaup via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>> 
>>> +1 since I had to do exactly that a couple of times with CocoaPods.
>>> There were issue in third party Pods which were fixed in a specific commit or branch but the official release would still take a while. Delaying our app release because of that was not an option and referring to a fixed yet unreleased version the best solution. 
>>> 
>>> On Dec 14, 2015 8:04 PM, "Ankit Agarwal via swift-build-dev" <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>> Correcting one of the sentences in the Detailed design section
>>> 
>>> * If pointed to a branch, there might be two broad use cases
>>> 1. User wants to point a branch due to active development of that dep and wants latest ref available in that branch
>>> 2. User is actively developing a dep in that branch and want to test it out in the current package
>>> 
>>> 
>>> On Tue, Dec 15, 2015 at 12:13 AM, Ankit Agarwal <ankit at ankit.im <mailto:ankit at ankit.im>> wrote:
>>> 
>>> Hi,
>>> 
>>> Here is a proposal of the adding git branch support feature in SPM
>>> 
>>> Introduction
>>> 
>>> Pointing to branch or a commit ref for dependencies in Package.swift as opposed to only a tagged release.
>>> 
>>> Motivation
>>> 
>>> * Try a package which is almost stable or useable but not yet ready for a release/pre-release so not tagged (eg: new feature being introduced by a library)
>>> * While developing packages, one would want to point a package that uses the package to a develop branch (eg: Developing Foo package, Bar uses Foo and wants to point Foo dep to develop branch)
>>> * One would want to point to his own fork but not create a release while developing/testing (eg: Fork a library not compatible with SPM to make it compatible)
>>> * One wants to point to some commit but doesn't have a branch/tag created for that
>>> 
>>> Proposed solution
>>> 
>>> Allow refs and branch in Package.swift
>>> 
>>> let package = Package(
>>>     name: "Hello",
>>>     dependencies: [
>>>         .Package(url: "ssh://git@example.com/Greeter.git <http://git@example.com/Greeter.git>", branch: "develop", shouldFastForward: true),
>>>         .Package(url: "ssh://git@example.com/FooBar.git <http://git@example.com/FooBar.git>", commit: "d8ec7ca398a3ac3990477028117384d05ca7734e"),
>>>     ]
>>> )
>>> 
>>> Detailed design
>>> 
>>> * Only the root Package.swift would be able to use branch/ref feature to avoid dependency hell, any other dependency fetched in current Package should not compile if that dependency contains another dependency pointing to a branch/ref
>>> * This feature should strictly be used for testing/developing purpose and should not be deployed to production environments
>>> 
>>> SPM could have the following behavior when running `swift build` :
>>> 
>>> * If pointed to a branch, there might be two use cases 
>>> Since there is a high probability that user wants to point a branch due to active development of that dep and wants latest ref available in that branch
>>> If a dependency is not cloned, clone it and checkout that branch
>>> If shouldFastForward is on -> Always try to be on the latest ref, disregard any local changes made to the checked out package
>>> If shouldFastForward is false -> Always try to be on the latest ref unless any local changes made to the checked out package
>>> 
>>> * If pointed to a ref : 
>>> If that dependency is not cloned, clone it and checkout that ref.
>>> Consecutive `swift build` will not affect the cloned package
>>> If changes are made in the cloned repo, rebuild that package with those changes
>>> 
>>> Impact on existing code
>>> 
>>> None as this will be a new functionality
>>> 
>>> Alternatives considered
>>> 
>>> One option is to only allow a commit ref and not a branch so SPM will not have to worry about fast forwarding but this is a desired feature.
>>> 
>>> On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard at apple.com <mailto:rballard at apple.com>> wrote:
>>> > On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit at ankit.im <mailto:ankit at ankit.im>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Is pointing to a branch instead of version for a package in scope of SPM?
>>> > if it is, I'd love to try to implement it
>>> 
>>> Hi Ankit,
>>> 
>>> This is in scope, though not yet designed. Prior to anyone working on an implementation, we should agree on a design for how you'd do this. While this isn't at the top of our priority list at the moment, we'd welcome both design contributions and eventual implementation.
>>> 
>>> If you'd like to put a proposal together for this, please see the Swift evolution process at https://github.com/apple/swift-evolution/blob/master/process.md <https://github.com/apple/swift-evolution/blob/master/process.md>. We'd be happy to discuss this here as part of your process for putting a proposal together. Some things to think about in this area are:
>>> 
>>> – How should refs (branches or tags) that aren't simple version numbers be specified?
>>> 
>>> – Right now we require you to tag something as a versioned "release". Should we require that you tag a branch before someone can make a package depend on it? It could be convenient to be able to just depend on a branch, but the meaning of depending on a branch changes over time as more commits come in. Is it harmful to allow packages to depend on something that's not an identified commit?
>>> 
>>>         – Note that we have yet to design our security story (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing <https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing>); what we settle on there might require dependencies to be specified as a specific tagged commit, so that it can be signed.
>>> 
>>> – Should it be possible to override a package's dependency to use a different branch, without having to modify and commit a change to that package's Package.swift?
>>> 
>>> – We may want to design a way for packages to support different versions of the Swift language, as the language continues to change – e.g. a branch of the package for the last released swift vs the current under development swift snapshot. Is supporting dependencies on package branches a part of how we'll do that?
>>> 
>>> Thanks,
>>> 
>>>         - Rick
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Ankit
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Ankit
>>> 
>>>  
>>> _______________________________________________
>>> swift-build-dev mailing list
>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
>>> 
>>>  _______________________________________________
>>> swift-build-dev mailing list
>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
> 
> 
> 
> 
> -- 
> Ankit
> 
>  _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20151215/2fa6801a/attachment.html>