<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Ankit, thanks for your quick reply.<div class=""><br class=""></div><div class="">I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :) The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.</div><div class=""><br class=""></div><div class="">I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just <i class="">builds</i> with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <<a href="mailto:ankit_aggarwal@apple.com" class="">ankit_aggarwal@apple.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:swift-build-dev@swift.org" target="_blank" class="">swift-build-dev@swift.org</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class="">Hello Swift Build Devs!<div class=""><br class=""></div><div class="">I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)</div><div class=""><br class=""></div><div class="">We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.</div><div class=""><br class=""></div><div class="">The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.</div></div></blockquote><div class=""><br class=""></div><div class="">In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.</div></div></div></blockquote><div><br class=""></div><div>Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:</div><div><br class=""></div><div>1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:</div><div><br class=""></div><div>2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to <a href="https://bugs.swift.org/browse/SR-6492" class="">https://bugs.swift.org/browse/SR-6492</a> and a seemingly endless cycle of frustration, particularly when combined with the next point:</div><div><br class=""></div><div>3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).</div><div><br class=""></div><div>Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)</div><div><br class=""></div><div>4. The more recent bug (<a href="https://bugs.swift.org/browse/SR-6493" class="">https://bugs.swift.org/browse/SR-6493</a>) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.</div><div><br class=""></div><div>5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.</div><div><br class=""></div><div>6. Why are we dealing with all this complexity we don’t need?</div><div><br class=""></div><div>The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.</div><div><br class=""></div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: <a href="https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md" target="_blank" class="">https://github.com/<wbr class="">ddunbar/swift-evolution/blob/<wbr class="">multi-package-repos/proposals/<wbr class="">NNNN-swiftpm-multi-package-<wbr class="">repos.md</a>. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.</div><div class=""><br class=""></div><div class="">I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?</div></div></blockquote><div class=""><br class=""></div><div class="">We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!</div></div></div></blockquote><div><br class=""></div><div>Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?</div><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">I did find this commit by Ankit Aggarwal: <a href="https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e" target="_blank" class="">https://github.com/<wbr class="">apple/swift-package-manager/<wbr class="">commit/<wbr class="">1ff987c64c00e9dc60e040f6e96060<wbr class="">2c84eede5e</a> which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.</div></div></blockquote><div class=""><br class=""></div><div class="">The isLocal property is an implementation detail and is not expected to be modified by the user.</div></div></div></blockquote><div><br class=""></div><div>I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do? </div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class=""><div class=""></div><div class="">At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.</div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).</div></div></div></blockquote><div><br class=""></div><div>For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.</div><div><br class=""></div><div>I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.</div><div><br class=""></div><div>Kind Regards,</div><div>Geordie</div><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""><br class=""></div><div class="">We also have a<span class="Apple-converted-space"> </span><a href="https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html" class="">slack channel</a><span class="Apple-converted-space"> </span>if you want to chat about some more workarounds.</div></div></div></blockquote><div><br class=""></div><div>Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?</div><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word; line-break: after-white-space;" class=""><div class=""></div><div class="">Cheers and thank you in advance,</div><div class="">Geordie</div><div class=""><br class=""></div></div><br class="">______________________________<wbr class="">_________________<br class="">swift-build-dev mailing list<br class=""><a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a><br class=""><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev" rel="noreferrer" target="_blank" class="">https://lists.swift.org/<wbr class="">mailman/listinfo/swift-build-<wbr class="">dev</a></blockquote></div></div></blockquote></div><br class=""></div></body></html>