<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Geordie,<div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 1, 2017, at 5:16 AM, Geordie J via swift-build-dev <<a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hi Ankit, thanks for your quick reply.</span><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class=""></div><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :) The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.</div></div></blockquote><div><br class=""></div><div>We honestly appreciate the feedback about pain points, as long as the tone is respectful (which I think yours is) we are more than happy to hear constructive criticism!</div><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just<span class="Apple-converted-space"> </span><i class="">builds</i><span class="Apple-converted-space"> </span>with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.</div></div></blockquote><div><br class=""></div>This is certainly our goal, getting it right requires tuning and time.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><blockquote type="cite" class=""><div class="">Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <<a href="mailto:ankit_aggarwal@apple.com" class="">ankit_aggarwal@apple.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:swift-build-dev@swift.org" target="_blank" class="">swift-build-dev@swift.org</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;">Hello Swift Build Devs!<div class=""><br class=""></div><div class="">I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)</div><div class=""><br class=""></div><div class="">We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.</div><div class=""><br class=""></div><div class="">The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.</div></div></blockquote><div class=""><br class=""></div><div class="">In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:</div><div class=""><br class=""></div><div class="">1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:</div><div class=""><br class=""></div><div class="">2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to <a href="https://bugs.swift.org/browse/SR-6492" class="">https://bugs.swift.org/browse/SR-6492</a> and a seemingly endless cycle of frustration, particularly when combined with the next point:</div><div class=""><br class=""></div><div class="">3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).</div><div class=""><br class=""></div><div class="">Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)</div><div class=""><br class=""></div><div class="">4. The more recent bug (<a href="https://bugs.swift.org/browse/SR-6493" class="">https://bugs.swift.org/browse/SR-6493</a>) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.</div></div></div></div></blockquote><div><br class=""></div>I agree, this one is really frustrating me too and I think we need to get it resolved.</div><div><br class=""><blockquote type="cite" class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><div class="">5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.</div><div class=""><br class=""></div><div class="">6. Why are we dealing with all this complexity we don’t need?</div><div class=""><br class=""></div><div class="">The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.</div></div></div></blockquote><div><br class=""></div>One thing to be clear, your way of using SwiftPM is not the “golden path” that we believe most users are using. Deviating that is probably causing you to bump into more pain points and workflows that need tuning.</div><div><br class=""></div><div>Thank you for writing up the concrete list of issues, that is very helpful. Here is my perspective on this:</div><div>1. As I said, I think your way of using SwiftPM is slightly unorthodox. One thing that means is we have to carefully balance the features you want versus their impact on all the other workflows.</div><div>2. I agree there is a problem here. I myself encounter a similar situation relatively frequently with the work I am doing, and don’t yet know how to solve it.</div><div>3. Ultimately, the most constructive form of conversation is to try and identify individual features which (a) help and (b) integrate well with all the other features, and then write (or coauthor, if you want help) a proposal for it.</div><div><br class=""></div><div>As one example, a feature that I have long believed makes sense for SwiftPM is a way for developers to have a global “set of packages I contribute to”, and then SwiftPM features to always use those versions when resolving dependencies, instead of manually managing them. I think such a feature might fit with your use case, and solve many of your problems. What do you think?</div><div><br class=""><blockquote type="cite" class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class="">Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: <a href="https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md" target="_blank" class="">https://github.com/<wbr class="">ddunbar/swift-evolution/blob/<wbr class="">multi-package-repos/proposals/<wbr class="">NNNN-swiftpm-multi-package-<wbr class="">repos.md</a>. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.</div><div class=""><br class=""></div><div class="">I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?</div></div></blockquote><div class=""><br class=""></div><div class="">We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!</div></div></div></blockquote><div class=""><br class=""></div><div class="">Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?</div></div></div></blockquote><div><br class=""></div>Rick (CC’d) is working on revising the next version of this proposal and will post it once ready.</div><div><br class=""><blockquote type="cite" class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class=""><br class=""></div><div class="">I did find this commit by Ankit Aggarwal: <a href="https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e" target="_blank" class="">https://github.com/<wbr class="">apple/swift-package-manager/<wbr class="">commit/<wbr class="">1ff987c64c00e9dc60e040f6e96060<wbr class="">2c84eede5e</a> which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.</div></div></blockquote><div class=""><br class=""></div><div class="">The isLocal property is an implementation detail and is not expected to be modified by the user.</div></div></div></blockquote><div class=""><br class=""></div><div class="">I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class=""><br class=""></div><div class="">So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do? </div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class=""></div><div class="">At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.</div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).</div></div></div></blockquote><div class=""><br class=""></div><div class="">For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.</div><div class=""><br class=""></div><div class="">I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.</div><div class=""><br class=""></div><div class="">Kind Regards,</div><div class="">Geordie</div><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><br class=""></div><div class="">We also have a<span class="Apple-converted-space"> </span><a href="https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html" class="">slack channel</a><span class="Apple-converted-space"> </span>if you want to chat about some more workarounds.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?</div></div></div></blockquote><div><br class=""></div>The mailing list and the swift-evolution repository are the “authoritative” sources for discussion, Slack is great for working out details but ultimately any decisions should be communicated through one of the those.</div><div><br class=""></div><div> - Daniel</div><div><br class=""><blockquote type="cite" class=""><div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word; line-break: after-white-space;"><div class=""></div><div class="">Cheers and thank you in advance,</div><div class="">Geordie</div><div class=""><br class=""></div></div><br class="">______________________________<wbr class="">_________________<br class="">swift-build-dev mailing list<br class=""><a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a><br class=""><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev" rel="noreferrer" target="_blank" class="">https://lists.swift.org/<wbr class="">mailman/listinfo/swift-build-<wbr class="">dev</a></blockquote></div></div></blockquote></div><br class=""></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">swift-build-dev mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev" class="">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></span></blockquote></div><br class=""></div></body></html>