<html><body><p><font size="4">I think that a post-clone script or similar functionality could be very useful. There are other systems language communities also recognize the need for such support to enhance the build process in some fashion. For example, Rust/Cargo has its build.rs file (</font><a href="http://doc.crates.io/build-script.html"><font size="4">http://doc.crates.io/build-script.html</font></a><font size="4">) that serves the purpose of running these custom tasks before the Cargo package manager builds the project. Platforms where Swift projects with C-interop without easy access to a system library Package manager includes Buildpacks (like to Heroku and Bluemix) through Cloud Foundry.</font><br><br><font size="4">I think that the current C-proposal that builds C code if it exists in a properly structured project with a file structure hits a great niche that can serve many use-cases that the C code is basically owned by the user writing the Swift code. It is very similar functionally to the node-gyp package. I have tested it in the latest SNAPSHOT 3-21 and it works very well!</font><br><br><font size="4">However, there are many libraries out there as you pointed out like libSSL and libCurl that have more complex build steps. Given the current C-proposal, these repositories such as libCurl would be needed to have a full overhaul to their structure in order for SPM to build them. Even with this overhaul, there would be no way to pass in arguments to the configure script in libCurl in order to add in additional support for HTTP2 or SSL. </font><br><br><font size="4">I have found a "hack" that can work in order to bring in a prebuilt library for x86-64 Linux and Mac OS before the Swift build is run. It works by checking the platform in the Package.swift file and copying the prebuilt dynamic library to the .build/debug directory before kicking off a build. This is not an elegant solution. It would not handle platforms that we did not anticipate (like ARM, zOS, PowerPC, etc.) but since SPM will read the Package.swift files through the dependency chain before actually calling swiftc, this solution works. Here is the modulemap for libCurl and the "post-clone" step:</font><br><br><a href="https://github.com/IBM-Swift/CCurl-bin/"><font size="4">https://github.com/IBM-Swift/CCurl-bin/</font></a><br><br><font size="4">I think we still need to be careful, however not to try to recreate what Ninja, CMake, Automake does. Also, not recreate what apt-get, yum, brew, etc. is trying to achieve. Maybe if the Swift Package Manager had more "hooks" such as post-clone, post-compile, etc. throughout the lifecycle of the application build?</font><br><br>
<table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="259" valign="middle"><font size="4">Robert F. Dickerson, PhD</font><br>Software Engineer, Swift@IBM Engineering</td><td width="227" valign="middle"><img width="1" height="1" src="cid:2__=09BBF5EDDFE91AFA8f9e8a93df938690918c09B@" border="0" alt=""></td></tr></table><br><img width="16" height="16" src="cid:3__=09BBF5EDDFE91AFA8f9e8a93df938690918c09B@" border="0" alt="Inactive hide details for swift-build-dev-request---03/22/2016 11:58:02 AM---Send swift-build-dev mailing list submissions to "><font color="#424282">swift-build-dev-request---03/22/2016 11:58:02 AM---Send swift-build-dev mailing list submissions to swift-build-dev@swift.org</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">swift-build-dev-request@swift.org</font><br><font size="2" color="#5F5F5F">To: </font><font size="2">swift-build-dev@swift.org</font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">03/22/2016 11:58 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">swift-build-dev Digest, Vol 4, Issue 13</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">swift-build-dev-bounces@swift.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>Send swift-build-dev mailing list submissions to<br> swift-build-dev@swift.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> </tt><tt><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></tt><tt><br>or, via email, send a message with subject or body 'help' to<br> swift-build-dev-request@swift.org<br><br>You can reach the person managing the list at<br> swift-build-dev-owner@swift.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of swift-build-dev digest..."<br><br><br>Today's Topics:<br><br> 1. Re: How to depend on a specified Pre-release version of<br> package with the Swift Package Manager (Max Howell)<br> 2. Re: [Discussion Starter] Package post-clone script (Max Howell)<br> 3. Re: [Discussion Starter] Package post-clone script (Honza Dvorsky)<br> 4. Re: [swift-evolution] Proposal Discussion Thread: SwiftPM:<br> Locking and Overriding Dependencies (Kostiantyn Koval)<br> 5. Re: [swift-evolution] Proposal Discussion Thread: SwiftPM:<br> Locking and Overriding Dependencies (Max Howell)<br> 6. Re: [swift-evolution] Proposal Discussion Thread: SwiftPM:<br> Locking and Overriding Dependencies (Max Howell)<br> 7. Re: Proposal Discussion Thread: SwiftPM: Locking and<br> Overriding Dependencies (Max Howell)<br> 8. Re: [Discussion Starter] Package post-clone script (Chris Bailey)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Mon, 21 Mar 2016 11:10:02 -0700<br>From: Max Howell <max.howell@apple.com><br>To: Kostiantyn Koval <konstantin.koval1@gmail.com><br>Cc: swift-build-dev@swift.org, Martha JP <jpmartha.jp@gmail.com><br>Subject: Re: [swift-build-dev] How to depend on a specified<br> Pre-release version of package with the Swift Package Manager<br>Message-ID: <179AFB3A-AF8D-4D1D-8774-F694DF1C5C85@apple.com><br>Content-Type: text/plain; charset="utf-8"<br><br>Strictly, we don’t really want people specifying such exact version requirements in Package.swift, since it would be easy to forget and then be trapped on a single version forever despite updates being available.<br><br>The lockfile proposal is more for this kind of restriction, but this is not yet implemented (or approved).<br><br>However Kostiantyn’s suggestion should work, though you’ll need to specify it as an inclusive range of the provided Version() twice.<br><br>> On Mar 21, 2016, at 2:36 AM, Kostiantyn Koval via swift-build-dev <swift-build-dev@swift.org> wrote:<br>> <br>> Hi JPMartha,<br>> <br>> I think it should be possible to specify a dependency package with a specific pre-release identifier in Manifest and swiftpm should fetch that exact version. <br>> Example:<br>> Package(url: ""</tt><tt><a href="https://github.com/ikesyo/Himotoki.git">https://github.com/ikesyo/Himotoki.git</a></tt><tt> <</tt><tt><a href="http://github.com/ikesyo/Himotoki.git">http://github.com/ikesyo/Himotoki.git</a></tt><tt>>",<br>> versions: Version(1, 0, 0, prereleaseIdentifiers: “beta.1”)<br>> <br>> The swiftPM should fetch a package with v-1.0.0-beta-1 tag <br>> <br>>> On 21 Mar 2016, at 06:43, Martha JP via swift-build-dev <swift-build-dev@swift.org <</tt><tt><a href="mailto:swift-build-dev@swift.org">mailto:swift-build-dev@swift.org</a></tt><tt>>> wrote:<br>>> <br>>> Hi, I'm JPMartha.<br>>> <br>>> I had filed an issue of the Swift Package Manager first.<br>>> But, I should have heard your feedback on my opinion beforehand.<br>>> <br>>> Please see the following:<br>>> </tt><tt><a href="https://bugs.swift.org/browse/SR-965">https://bugs.swift.org/browse/SR-965</a></tt><tt> <</tt><tt><a href="https://bugs.swift.org/browse/SR-965">https://bugs.swift.org/browse/SR-965</a></tt><tt>><br>>> <br>>> I think it's unexpected behavior.<br>>> I'm curious to hear your feedback!<br>>> <br>>> Best regards,<br>>> JPMartha<br>>> _______________________________________________<br>>> swift-build-dev mailing list<br>>> swift-build-dev@swift.org <</tt><tt><a href="mailto:swift-build-dev@swift.org">mailto:swift-build-dev@swift.org</a></tt><tt>><br>>> </tt><tt><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></tt><tt><br>> <br>> _______________________________________________<br>> swift-build-dev mailing list<br>> swift-build-dev@swift.org<br>> </tt><tt><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></tt><tt><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</tt><tt><a href="https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/3571291e/attachment-0001.html">https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/3571291e/attachment-0001.html</a></tt><tt>><br><br>------------------------------<br><br>Message: 2<br>Date: Mon, 21 Mar 2016 11:19:40 -0700<br>From: Max Howell <max.howell@apple.com><br>To: Honza Dvorsky <jan.dvorsky@me.com><br>Cc: "swift-build-dev@swift.org" <swift-build-dev@swift.org><br>Subject: Re: [swift-build-dev] [Discussion Starter] Package post-clone<br> script<br>Message-ID: <C52F9EDA-6859-44A5-95FC-F8A9CCB734F9@apple.com><br>Content-Type: text/plain; charset="utf-8"<br><br><br>> On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev <swift-build-dev@swift.org> wrote:<br>> <br>> Hi All,<br>> <br>> today I was trying to investigate how to build a proper cross-platform server-side Swift project and only use SwiftPM to do it. In particular, I was interested in getting OpenSSL/libcurl working as a dependency (built from source). The main reason is that currently the big Swift server frameworks all require manual steps during deployment (running apt-get/brew to install libcurl/openssl/others) and I am trying to get rid of these manual steps, eventually only requiring a `swift build` and run.<br>> <br>> With the great initial work <</tt><tt><a href="https://github.com/apple/swift-package-manager/pull/183">https://github.com/apple/swift-package-manager/pull/183</a></tt><tt>> to support building C-family packages in SwiftPM done by Ankit, I was able to have a simple C package as a dependency, as advertised. However, the reality of big cross-platform open source projects like OpenSSL and libcurl is that in order to actually build it from source, a configure (or similar) script has to be run to get the source code ready for compilation. And this is where I got stuck.<br>> <br>> Thus I wanted to kick off a discussion of what approach of solving this problem with SwiftPM should be. The simple solution, which I'm tentatively proposing, is to have an optional "post-clone" script in the package's repository. Then, the Package.swift would optionally contain a field for the path to this script - and if present, it'd get run after SwiftPM clones this package. Note that this would be a "post-clone" script, not a "prebuild" script - I imagine it'd only run once when cloning and then only after each clean. <br>> <br>> In order to get users to stop having to run manual script to install all dependencies, I believe that we need to allow packages to declare what work needs to be done on its source before compilation can begin.<br><br>I have a proposal I’d like to push today that adds knowledge to SwiftPM about how to install system dependencies for System Module Packages.<br><br>We are not keen to add arbitrary script execution to SwiftPM, because: 1) Arbitrary scripts cannot be controlled and this leads to dependency hell and 2) arbitrary scripts mean your package graph may do anything, which nobody wants.<br><br>Now I’m fine with the “root package” ie. the package the user has control over doing more, but this would be a different discussion.<br><br>So the question becomes, is this sufficient? To have `brew install openssl` run, or do you really need to install from source?<br><br>Max<br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</tt><tt><a href="https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/e84bfa9f/attachment-0001.html">https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/e84bfa9f/attachment-0001.html</a></tt><tt>><br><br>------------------------------<br><br>Message: 3<br>Date: Mon, 21 Mar 2016 18:38:14 +0000<br>From: Honza Dvorsky <jan.dvorsky@me.com><br>To: Max Howell <max.howell@apple.com><br>Cc: "swift-build-dev@swift.org" <swift-build-dev@swift.org><br>Subject: Re: [swift-build-dev] [Discussion Starter] Package post-clone<br> script<br>Message-ID:<br> <CAJi4s=TSPC0tncHGOmZL6QbhirO0uG57BvtEWLdMnJ82T4zGCQ@mail.gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><br>I personally don't like the idea of an arbitrary script either, it was just<br>the first obvious solution to a problem I believe needs solving. The<br>advantage of building from source even of these base dependencies is that<br>the exact version of system dependencies<br>- can be ensured to be the same version across platforms<br>- their version is part of the explicit dependency graph<br><br>Currently, AFAIR, installing Redis with homebrew and apt-get each produce a<br>different version, which IMO defeats the point of having a dependency<br>manager in the first place. But I guess if your proposal could ensure that<br>the same version is installed on all platforms, this particular problem<br>would be solved.<br><br>> So the question becomes, is this sufficient? To have `brew install<br>openssl` run, or do you really need to install from source?<br><br>I'm still not sure. Now I think your proposal will probably help with most<br>cases (which is a very good way to approach things), but definitely not<br>all. Imagine you yourself having a cross-platform C project which needs<br>platform configuring before compilation. Even when you're completely in<br>control, the fact that you still can't get this working is troubling. But<br>maybe I'm just inflating extremely uncommon usecases.<br><br>One more thing - consider two Swift packages each needing a different<br>version of OpenSSL. Does your proposal handle that case?<br><br>Anyway, I'd love to read your proposal and then re-evaluate whether I still<br>feel like there's a large enough benefit in introducing such customization<br>to SwiftPM :)<br><br>Honza<br><br>On Mon, Mar 21, 2016 at 6:19 PM Max Howell <max.howell@apple.com> wrote:<br><br>><br>> On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev <<br>> swift-build-dev@swift.org> wrote:<br>><br>> Hi All,<br>><br>> today I was trying to investigate how to build a proper cross-platform<br>> server-side Swift project and only use SwiftPM to do it. In particular, I<br>> was interested in getting OpenSSL/libcurl working as a dependency (built<br>> from source). The main reason is that currently the big Swift server<br>> frameworks all require manual steps during deployment (running apt-get/brew<br>> to install libcurl/openssl/others) and I am trying to get rid of these<br>> manual steps, eventually only requiring a `swift build` and run.<br>><br>> With the great initial work<br>> <</tt><tt><a href="https://github.com/apple/swift-package-manager/pull/183">https://github.com/apple/swift-package-manager/pull/183</a></tt><tt>> to support<br>> building C-family packages in SwiftPM done by Ankit, I was able to have a<br>> simple C package as a dependency, as advertised. However, the reality of<br>> big cross-platform open source projects like OpenSSL and libcurl is that in<br>> order to actually build it from source, a configure (or similar) script has<br>> to be run to get the source code ready for compilation. And this is where I<br>> got stuck.<br>><br>> Thus I wanted to kick off a discussion of what approach of solving this<br>> problem with SwiftPM should be. The simple solution, which I'm tentatively<br>> proposing, is to have an optional "post-clone" script in the package's<br>> repository. Then, the Package.swift would optionally contain a field for<br>> the path to this script - and if present, it'd get run after SwiftPM clones<br>> this package. Note that this would be a "post-clone" script, not a<br>> "prebuild" script - I imagine it'd only run once when cloning and then only<br>> after each clean.<br>><br>> In order to get users to stop having to run manual script to install all<br>> dependencies, I believe that we need to allow packages to declare what work<br>> needs to be done on its source before compilation can begin.<br>><br>><br>> I have a proposal I’d like to push today that adds knowledge to SwiftPM<br>> about how to install system dependencies for System Module Packages.<br>><br>> We are not keen to add arbitrary script execution to SwiftPM, because: 1)<br>> Arbitrary scripts cannot be controlled and this leads to dependency hell<br>> and 2) arbitrary scripts mean your package graph may do anything, which<br>> nobody wants.<br>><br>> Now I’m fine with the “root package” ie. the package the user has control<br>> over doing more, but this would be a different discussion.<br>><br>> So the question becomes, is this sufficient? To have `brew install<br>> openssl` run, or do you really need to install from source?<br>><br>> Max<br>><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</tt><tt><a href="https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/5717a2ac/attachment-0001.html">https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/5717a2ac/attachment-0001.html</a></tt><tt>><br><br>------------------------------<br><br>Message: 4<br>Date: Mon, 21 Mar 2016 21:12:47 +0100<br>From: Kostiantyn Koval <konstantin.koval1@gmail.com><br>To: Max Howell <max.howell@apple.com><br>Cc: "swift-evolution@swift.org" <swift-evolution@swift.org>,<br> "swift-build-dev@swift.org" <swift-build-dev@swift.org><br>Subject: Re: [swift-build-dev] [swift-evolution] Proposal Discussion<br> Thread: SwiftPM: Locking and Overriding Dependencies<br>Message-ID: <EF0A84B5-9C45-4979-A59F-94E8F52D533A@gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><br>I like the proposal.<br><br>I have 1 concern. <br>As a package author, like PromiseKit <</tt><tt><a href="https://github.com/mxcl/PromiseKit">https://github.com/mxcl/PromiseKit</a></tt><tt>> I often have a need to fix some of my dependencies and publish it to my package users.<br><br>With the current proposal I can do it:<br> - Advise it in README, which is not really a solution done by swift package manager.<br> - update Manifest.swift file. There is no convent way to do it now.<br><br>I would love to have some functionality to make it simpler for me to specify (modify Manifest.swift or smt else) that Dependency X should: <br> - use specific tag X<br> - use commit X<br><br>But maybe that should be part of another proposal. The lock file would perfectly solve the problems for developing projects that uses packages.<br><br>- Kostiantyn<br><br>> On 20 Mar 2016, at 06:07, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:<br>> <br>> My thoughts on this proposal:<br>> <br>> 1. I agree with some of the other comments that I would prefer the version file be adjacent to "Package.swift". When the Packages directory isn't being checked in, I really would like to think of it as an "implementation detail" and not embed functionality inside it that would make it hard to remove or change in the future.<br>> <br>> 2. I like VersionLocks.json well enough, but would like to see a discussion about possible alternatives. My personal proposal (in line with #1) is to use "PackageVersions.json" which has a nice agreement with Package.swift and would mean two common metadata files show up adjacent. I don't really want to bike shed on the name, but I suspect whatever we pick first will last for a while so I would at least like to review the various alternatives. I also will throw out that my personal opinion is we don't need to pick a name that bears much resemblance with existing terminology, whatever we pick will eventually become "the standard" for the SwiftPM ecosystem so I would prefer to pick the most-descriptive-possible name up front, not one that alludes to the same concept in other systems.<br>> <br>> 3. I like the terminology section here, I almost feel like we should adopt that as official terminology in our documentation (which I don't think we have yet, correct me if I am wrong).<br>> <br>> 4. I would like it if the lock file recorded the exact SHA it received, and validate that when retrieving. This helps protect users against MITM attacks or unexpected changes if an upstream modifies a tag. It also can be used as part of safety checks when migrating to an alternate repository host which is expected to have the same content.<br>> <br>> 5. The "workflow - build" sections #2,3,4 are rather complicated. Is this because the proposal is trying to work with existing Packages layouts, or because the proposal is trying to handle the various variations of what the user may have checked in inside the Packages subdirectory?<br>> <br>> 6. I wonder if we should be defining, as Eloy alludes to, two different things:<br>> - The version lock file, which defines the expected versions for the package manager to use when it is doing package resolution.<br>> - The package state file (in Packages.swift), which is used by the package manager to track information on the Packages/ subdir state in order to provide useful features primarily focused at the scenarios when the user is modifying those files.<br>> Currently it seems like a lot of the behaviors in the proposal are focused at the latter case, but they feel like they should be decoupled problems to me.<br>> <br>> - Daniel<br>> <br>>> On Mar 17, 2016, at 11:23 AM, Max Howell via swift-evolution <swift-evolution@swift.org <</tt><tt><a href="mailto:swift-evolution@swift.org">mailto:swift-evolution@swift.org</a></tt><tt>>> wrote:<br>>> <br>>> The following is a draft proposal, feedback welcome.<br>>> <br>>> ____________<br>>> SwiftPM Dependency Version Locking<br>>> Proposal: SE-NNNN <</tt><tt><a href="https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md">https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md</a></tt><tt>><br>>> Author(s): Ankit Agarwal <</tt><tt><a href="https://github.com/aciidb0mb3r">https://github.com/aciidb0mb3r</a></tt><tt>>, Max Howell <</tt><tt><a href="https://github.com/mxcl">https://github.com/mxcl</a></tt><tt>><br>>> Status: Discussion<br>>> Review manager: Rick Ballard<br>>> Introduction<br>>> This proposal seeks to declare a new, generated file Packages/VersionLocks.json that describes the exact state of a package’s dependency graph and then by default will be respected when executing most package manager commands. Thus it is considered a “version lock” for a package’s dependency sources.<br>>> <br>>> Swift-evolution thread <</tt><tt><a href="https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html">https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html</a></tt><tt>><br>>> Terminology<br>>> A package refers to a published, versioned git repository designed to be consumed as a dependency by SwiftPM.<br>>> A project refers to an end-user workspace that uses SwiftPM (via a Package.swift and swift build) fetching and building packages as part of its build<br>>> Describing this distinction is required because both the above have the same form, but are used differently by an end-user. An end-user may publish packages, but will eventually consume those packages in a project.<br>>> <br>>> As justification for this confusion, it is considered a feature that projects can easily and trivially become packages when using SwiftPM. Encouraging a vibrant packaging ecosystem is one of our goals.<br>>> <br>>> Motivation<br>>> In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and new features. A development team needs:<br>>> <br>>> To ensure they are all using the same versions of their dependencies for any given version-control commit.<br>>> Ensure they are all using the same versions of the underlying Swift toolchain<br>>> Be able to override or modify dependency specifications for the whole team for specific commits.<br>>> Currently with SwiftPM it is possible to fulfill 1. by committing the sources of a package’s dependencies with the package itself, but this is not always desirable. There is no way to achieve 2. and 3. with SwiftPM alone.<br>>> <br>>> Additionally, there is not currently a way to know which version of Swift a package requires to build. At this time this situation is particularly precarious because Swift itself is not backwards compatible. As a Swift developer at the very least recording which Swift version a package was built with by the package developer is essential information in order to assess a package's suitability. Practically the package manager could in the future use this information to aid an end-user or even fix the problem when packages fail to compile.<br>>> <br>>> Proposed Solution<br>>> A file: Packages/VersionLocks.json will be created alongside the Package.swift file. Its contents will describe:<br>>> <br>>> The URL and versions of cloned dependencies<br>>> An inline diff of any local modifications made to those packages relative to their pristine cloned states<br>>> The exact version of the Swift toolchain used as part of the last successful build of the package<br>>> This file is generated by SwiftPM.<br>>> <br>>> This file should be checked-in with projects.<br>>> <br>>> This file is generated and should not be edited by users. If the file is edited by users the behavior is undefined.<br>>> <br>>> This file should be checked-in with packages designed for consumption in projects, however SwiftPM will not use the checkout files of dependencies when determining a project’s dependency graph (this would make dependency graphs much less likely to resolve due to overly strict versioning requirements). In the future we may choose to make it possible for end-users to attempt to build a package using all checkout files since in certain deployment scenarios where an exact graph has already been tested, this is a solid reliabiity feature.<br>>> <br>>> Any local, modifications made to the clones in Packages are recorded in Packages/VersionLocks.json as part of the flow described in the next section. Modifications here means: changes to git remotes and the git-ref of the checked-out HEAD.<br>>> <br>>> Detailed Design<br>>> In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate a Packages/VersionLocks.json file.<br>>> <br>>> The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:<br>>> <br>>> error: dependency sources have been modified<br>>> execute `swift build --lock` or `swift build --ignore-lock`<br>>> It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.<br>>> <br>>> When swift build --lock is specified the package manager regenerates the lockfile detailing the active git remote and the SHA that is checked-out.<br>>> <br>>> Every time swift build completes a build the lockfile is updated (if necessary) recording the current version of the Swift toolchain that achieved the build.<br>>> <br>>> Packages/VersionLocks.json<br>>> <br>>> The exact design of the contents of this file will be explored during iterative development, but here is a possible example:<br>>> <br>>> json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "</tt><tt><a href="https://github.com/mxcl/PromiseKit">https://github.com/mxcl/PromiseKit</a></tt><tt> <</tt><tt><a href="https://github.com/mxcl/PromiseKit">https://github.com/mxcl/PromiseKit</a></tt><tt>>" "ref": "3.0.3" }, { "clone": "Packages/Alamofire-1.2.3", "origin": "</tt><tt><a href="https://github.com/a-fork-somewhere/Alamofire">https://github.com/a-fork-somewhere/Alamofire</a></tt><tt> <</tt><tt><a href="https://github.com/a-fork-somewhere/Alamofire">https://github.com/a-fork-somewhere/Alamofire</a></tt><tt>>" "ref": "crucial-fix" }, { "clone": "Packages/Quick-1.2.3", "origin": "</tt><tt><a href="https://github.com/Quick/Quick">https://github.com/Quick/Quick</a></tt><tt> <</tt><tt><a href="https://github.com/Quick/Quick">https://github.com/Quick/Quick</a></tt><tt>>" "ref": "1.2.3" } ] }<br>>> <br>>> Workflow — Regular Build<br>>> <br>>> User runs swift build<br>>> If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips to 7.<br>>> If Packages/ contains clones and no VersionLocks.json the lockfile is generated from the clones<br>>> If Packages/ contains checked out sources without git information and no VersionLocks.json SwiftPM fetches the git information and provided there is no diff, generates the Lockfile, if there is variation it is an error *<br>>> If Packages/VersionLocks.json is present its dependency graph is used<br>>> If Packages doesn't exist or is empty the dependency graph is resolved, packages are cloned and the Lockfile is generated<br>>> Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.<br>>> <br>>> This scenario is so users can check in their complete dependency sources to their tree instead of / as well as the VersionLocks.json file: a situation which sometimes is necessary if your dependencies are removed from their third party online location, etc.<br>>> <br>>> Workflow — Making Modifications<br>>> <br>>> User makes local modification to a dependency’s sources<br>>> User runs swift build<br>>> swift build errors out.<br>>> User must either lock the graph or run with --ignore-lock<br>>> The error-out is likely to be considered tedious by users, however we consider it important that users are made aware and forced to act when they modify their dependencies and thus are exposing their team/users to so-called “dependency hell”.<br>>> <br>>> Runing swift build --lock regenerates the lockfile, but does not build.<br>>> <br>>> Modifications must be committed. This means that if the modifications are not uploaded to a location accessible to the rest of the team they will fail to build when they update their checkouts.<br>>> <br>>> The package manager could check for this by asking git if the specified origin has the current locked ref and error out as appropriate.<br>>> <br>>> Workflow — Overriding Packages<br>>> <br>>> User steps into a Package directory eg. Packages/Foo-1.2.3<br>>> User changes the origin of Foo to their own fork<br>>> User alters HEAD to point to a fix in their own fork<br>>> swift build errors out.<br>>> User must either lock the graph or run with --ignore-lock<br>>> Running swift build --lock regenerates the lockfile, the new origin and tag is stored. Thus a fresh clone of this project would use these overrides.<br>>> <br>>> It is important to note that this workflow will not be respected for dependencies, only for projects.<br>>> <br>>> If a package author requires an override they have a few options:<br>>> <br>>> Change the Package.swift dependency specification. This should only be done as a last resort, for example, a critical bug must be fixed in a dependency and that dependency author is not being responsive. It is up to the Package author to ensure this scenario goes well. SwiftPM itself wants to guard against these conditions with our proposed “publish & lint” step that validates such decisions before signing a published package tag. But we are not there yet and thus package authors should be responsible.<br>>> Advise end-users in a package README that they should override the dependency themselves.<br>>> 2 is preferred, but 1 will happen. We consider it our responsibility to develop tooling that makes 1. safe or unnecessary, but we are not there yet.<br>>> <br>>> Workflow — Updating Packages<br>>> <br>>> SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.<br>>> <br>>> Impact on existing code<br>>> This proposal will have no impact on existing code.<br>>> <br>>> Alternatives Considered<br>>> One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <</tt><tt><a href="http://markdownlivepreview.com/%22https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/">http://markdownlivepreview.com/%22https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/</a></tt><tt>> thread it might not be the best idea.<br>>> <br>>> Using Git submodules for this feature was considered. However something additionally would be required to specify swift version and record local diffs. Also this would lock us into git, and despite the fact that currently we only use git, we have not yet ruled out supporting other version control systems.<br>>> <br>>> _______________________________________________<br>>> swift-evolution mailing list<br>>> swift-evolution@swift.org <</tt><tt><a href="mailto:swift-evolution@swift.org">mailto:swift-evolution@swift.org</a></tt><tt>><br>>> </tt><tt><a href="https://lists.swift.org/mailman/listinfo/swift-evolution">https://lists.swift.org/mailman/listinfo/swift-evolution</a></tt><tt><br>> <br>> _______________________________________________<br>> swift-build-dev mailing list<br>> swift-build-dev@swift.org<br>> </tt><tt><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></tt><tt><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</tt><tt><a href="https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/3f3452fe/attachment-0001.html">https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/3f3452fe/attachment-0001.html</a></tt><tt>><br><br>------------------------------<br><br>Message: 5<br>Date: Mon, 21 Mar 2016 15:57:49 -0700<br>From: Max Howell <max.howell@apple.com><br>To: Kostiantyn Koval <konstantin.koval1@gmail.com><br>Cc: "swift-evolution@swift.org" <swift-evolution@swift.org>,<br> "swift-build-dev@swift.org" <swift-build-dev@swift.org><br>Subject: Re: [swift-build-dev] [swift-evolution] Proposal Discussion<br> Thread: SwiftPM: Locking and Overriding Dependencies<br>Message-ID: <28D54CDA-9388-4938-B0AF-C895B54D9134@apple.com><br>Content-Type: text/plain; charset=us-ascii<br><br>> - update Manifest.swift file. There is no convent way to do it now.<br><br>It is not convenient enough to edit it with a text editor and commit? Seems pretty convenient to me.<br><br><br><br>------------------------------<br><br>Message: 6<br>Date: Mon, 21 Mar 2016 16:05:46 -0700<br>From: Max Howell <max.howell@apple.com><br>To: Daniel Dunbar <daniel_dunbar@apple.com><br>Cc: "swift-evolution@swift.org" <swift-evolution@swift.org>,<br> "swift-build-dev@swift.org" <swift-build-dev@swift.org><br>Subject: Re: [swift-build-dev] [swift-evolution] Proposal Discussion<br> Thread: SwiftPM: Locking and Overriding Dependencies<br>Message-ID: <D2A832AB-4B07-418E-9D6F-32A76EF1EB58@apple.com><br>Content-Type: text/plain; charset="utf-8"<br><br>> 2. I like VersionLocks.json well enough, but would like to see a discussion about possible alternatives. My personal proposal (in line with #1) is to use "PackageVersions.json" which has a nice agreement with Package.swift and would mean two common metadata files show up adjacent. I don't really want to bike shed on the name, but I suspect whatever we pick first will last for a while so I would at least like to review the various alternatives. I also will throw out that my personal opinion is we don't need to pick a name that bears much resemblance with existing terminology, whatever we pick will eventually become "the standard" for the SwiftPM ecosystem so I would prefer to pick the most-descriptive-possible name up front, not one that alludes to the same concept in other systems.<br><br>I like PackageVersions.json<br>> <br>> 3. I like the terminology section here, I almost feel like we should adopt that as official terminology in our documentation (which I don't think we have yet, correct me if I am wrong).<br><br>We don’t, but I agree, we should aim to pick some names and use them consistently.<br><br>> 4. I would like it if the lock file recorded the exact SHA it received, and validate that when retrieving. This helps protect users against MITM attacks or unexpected changes if an upstream modifies a tag. It also can be used as part of safety checks when migrating to an alternate repository host which is expected to have the same content.<br><br>Good point, this should be there.<br><br>> 5. The "workflow - build" sections #2,3,4 are rather complicated. Is this because the proposal is trying to work with existing Packages layouts, or because the proposal is trying to handle the various variations of what the user may have checked in inside the Packages subdirectory?<br><br>The latter, if we are to support checking in the `Packages` directory, we should handle it when it is so. Is there a simpler way you can see?<br><br>> 6. I wonder if we should be defining, as Eloy alludes to, two different things:<br>> - The version lock file, which defines the expected versions for the package manager to use when it is doing package resolution.<br>> - The package state file (in Packages.swift), which is used by the package manager to track information on the Packages/ subdir state in order to provide useful features primarily focused at the scenarios when the user is modifying those files.<br>> Currently it seems like a lot of the behaviors in the proposal are focused at the latter case, but they feel like they should be decoupled problems to me.<br><br>I’m not sure we need a second file, since the versions of the “installed dependencies” are recorded in the directory names as well as that we also do full clones, so that information is part of the clone & checkout.<br><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</tt><tt><a href="https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/83fb74bd/attachment-0001.html">https://lists.swift.org/pipermail/swift-build-dev/attachments/20160321/83fb74bd/attachment-0001.html</a></tt><tt>><br><br>------------------------------<br><br>Message: 7<br>Date: Mon, 21 Mar 2016 16:15:06 -0700<br>From: Max Howell <max.howell@apple.com><br>To: Max Howell <max.howell@apple.com><br>Cc: swift-evolution <swift-evolution@swift.org>,<br> swift-build-dev@swift.org<br>Subject: Re: [swift-build-dev] Proposal Discussion Thread: SwiftPM:<br> Locking and Overriding Dependencies<br>Message-ID: <7F5E17C6-BCAD-4AC7-AE14-3E5E35FE68D1@apple.com><br>Content-Type: text/plain; charset=us-ascii<br><br>I have revised the proposal based on feedback:<br><br></tt><tt><a href="https://github.com/mxcl/swift-evolution/blob/lockfiles/proposals/NNNN-swiftpm-dependency-lockfiles.md">https://github.com/mxcl/swift-evolution/blob/lockfiles/proposals/NNNN-swiftpm-dependency-lockfiles.md</a></tt><tt><br><br>Changed location and name, included SHAs, revised build-workflow steps.<br><br>------------------------------<br><br>Message: 8<br>Date: Tue, 22 Mar 2016 09:43:42 +0000<br>From: Chris Bailey <BAILEYC@uk.ibm.com><br>To: Honza Dvorsky <jan.dvorsky@me.com><br>Cc: "swift-build-dev@swift.org" <swift-build-dev@swift.org><br>Subject: Re: [swift-build-dev] [Discussion Starter] Package post-clone<br> script<br>Message-ID:<br> <201603220944.u2M9i1k5017204@d06av07.portsmouth.uk.ibm.com><br>Content-Type: text/plain; charset="utf-8"<br><br>To add one more question on the installation of system dependencies - does <br>that then drive a need to install as root/sudo? If so, I can see that <br>being a challenge for any cloud based application deployments, where your <br>both unlikely to be able to run as root/sudo, or indeed be able to install <br>into /usr/lib<br><br><br>Chris<br><br><br><br>From: Honza Dvorsky via swift-build-dev <swift-build-dev@swift.org><br>To: Max Howell <max.howell@apple.com><br>Cc: "swift-build-dev@swift.org" <swift-build-dev@swift.org><br>Date: 21/03/2016 18:38<br>Subject: Re: [swift-build-dev] [Discussion Starter] Package <br>post-clone script<br>Sent by: swift-build-dev-bounces@swift.org<br><br><br><br>I personally don't like the idea of an arbitrary script either, it was <br>just the first obvious solution to a problem I believe needs solving. The <br>advantage of building from source even of these base dependencies is that <br>the exact version of system dependencies<br>- can be ensured to be the same version across platforms<br>- their version is part of the explicit dependency graph<br><br>Currently, AFAIR, installing Redis with homebrew and apt-get each produce <br>a different version, which IMO defeats the point of having a dependency <br>manager in the first place. But I guess if your proposal could ensure that <br>the same version is installed on all platforms, this particular problem <br>would be solved.<br><br>> So the question becomes, is this sufficient? To have `brew install <br>openssl` run, or do you really need to install from source?<br><br>I'm still not sure. Now I think your proposal will probably help with most <br>cases (which is a very good way to approach things), but definitely not <br>all. Imagine you yourself having a cross-platform C project which needs <br>platform configuring before compilation. Even when you're completely in <br>control, the fact that you still can't get this working is troubling. But <br>maybe I'm just inflating extremely uncommon usecases.<br><br>One more thing - consider two Swift packages each needing a different <br>version of OpenSSL. Does your proposal handle that case?<br><br>Anyway, I'd love to read your proposal and then re-evaluate whether I <br>still feel like there's a large enough benefit in introducing such <br>customization to SwiftPM :)<br><br>Honza<br><br>On Mon, Mar 21, 2016 at 6:19 PM Max Howell <max.howell@apple.com> wrote:<br><br>On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev <<br>swift-build-dev@swift.org> wrote:<br><br>Hi All,<br><br>today I was trying to investigate how to build a proper cross-platform <br>server-side Swift project and only use SwiftPM to do it. In particular, I <br>was interested in getting OpenSSL/libcurl working as a dependency (built <br>from source). The main reason is that currently the big Swift server <br>frameworks all require manual steps during deployment (running <br>apt-get/brew to install libcurl/openssl/others) and I am trying to get rid <br>of these manual steps, eventually only requiring a `swift build` and run.<br><br>With the great initial work to support building C-family packages in <br>SwiftPM done by Ankit, I was able to have a simple C package as a <br>dependency, as advertised. However, the reality of big cross-platform open <br>source projects like OpenSSL and libcurl is that in order to actually <br>build it from source, a configure (or similar) script has to be run to get <br>the source code ready for compilation. And this is where I got stuck.<br><br>Thus I wanted to kick off a discussion of what approach of solving this <br>problem with SwiftPM should be. The simple solution, which I'm tentatively <br>proposing, is to have an optional "post-clone" script in the package's <br>repository. Then, the Package.swift would optionally contain a field for <br>the path to this script - and if present, it'd get run after SwiftPM <br>clones this package. Note that this would be a "post-clone" script, not a <br>"prebuild" script - I imagine it'd only run once when cloning and then <br>only after each clean. <br><br>In order to get users to stop having to run manual script to install all <br>dependencies, I believe that we need to allow packages to declare what <br>work needs to be done on its source before compilation can begin.<br></tt><br><tt>I have a proposal I’d like to push today that adds knowledge to SwiftPM <br>about how to install system dependencies for System Module Packages.<br><br>We are not keen to add arbitrary script execution to SwiftPM, because: 1) <br>Arbitrary scripts cannot be controlled and this leads to dependency hell <br>and 2) arbitrary scripts mean your package graph may do anything, which <br>nobody wants.<br><br>Now I’m fine with the “root package” ie. the package the user has control <br>over doing more, but this would be a different discussion.<br><br>So the question becomes, is this sufficient? To have `brew install <br>openssl` run, or do you really need to install from source?<br><br>Max<br>_______________________________________________<br>swift-build-dev mailing list<br>swift-build-dev@swift.org<br></tt><tt><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></tt><tt><br><br><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</tt><tt><a href="https://lists.swift.org/pipermail/swift-build-dev/attachments/20160322/97f0ffac/attachment-0001.html">https://lists.swift.org/pipermail/swift-build-dev/attachments/20160322/97f0ffac/attachment-0001.html</a></tt><tt>><br><br>------------------------------<br><br>_______________________________________________<br>swift-build-dev mailing list<br>swift-build-dev@swift.org<br></tt><tt><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></tt><tt><br><br><br>End of swift-build-dev Digest, Vol 4, Issue 13<br>**********************************************<br></tt><br><br><BR>
</body></html>