[swift-build-dev] Advice on using SwiftPM for developing large projects

Fri Dec 1 09:48:51 CST 2017

Hi Geordie,

> On Dec 1, 2017, at 5:16 AM, Geordie J via swift-build-dev <swift-build-dev at swift.org> wrote:
> 
> Hi Ankit, thanks for your quick reply.
> 
> I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :) The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.

We honestly appreciate the feedback about pain points, as long as the tone is respectful (which I think yours is) we are more than happy to hear constructive criticism!

> I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just builds with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.

This is certainly our goal, getting it right requires tuning and time.

>> Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <ankit_aggarwal at apple.com <mailto:ankit_aggarwal at apple.com>>:
>> 
>> 
>> 
>> On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>> Hello Swift Build Devs!
>> 
>> I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)
>> 
>> We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.
>> 
>> The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.
>> 
>> In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.
> 
> Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:
> 
> 1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:
> 
> 2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to https://bugs.swift.org/browse/SR-6492 <https://bugs.swift.org/browse/SR-6492> and a seemingly endless cycle of frustration, particularly when combined with the next point:
> 
> 3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).
> 
> Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)
> 
> 4. The more recent bug (https://bugs.swift.org/browse/SR-6493 <https://bugs.swift.org/browse/SR-6493>) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.

I agree, this one is really frustrating me too and I think we need to get it resolved.

> 5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.
> 
> 6. Why are we dealing with all this complexity we don’t need?
> 
> The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.

One thing to be clear, your way of using SwiftPM is not the “golden path” that we believe most users are using. Deviating that is probably causing you to bump into more pain points and workflows that need tuning.

Thank you for writing up the concrete list of issues, that is very helpful. Here is my perspective on this:
1. As I said, I think your way of using SwiftPM is slightly unorthodox. One thing that means is we have to carefully balance the features you want versus their impact on all the other workflows.
2. I agree there is a problem here. I myself encounter a similar situation relatively frequently with the work I am doing, and don’t yet know how to solve it.
3. Ultimately, the most constructive form of conversation is to try and identify individual features which (a) help and (b) integrate well with all the other features, and then write (or coauthor, if you want help) a proposal for it.

As one example, a feature that I have long believed makes sense for SwiftPM is a way for developers to have a global “set of packages I contribute to”, and then SwiftPM features to always use those versions when resolving dependencies, instead of manually managing them. I think such a feature might fit with your use case, and solve many of your problems. What do you think?

>> Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md <https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md>. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.
>> 
>> I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?
>> 
>> We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!
> 
> Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?

Rick (CC’d) is working on revising the next version of this proposal and will post it once ready.

>> 
>> I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e <https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e> which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.
>> 
>> The isLocal property is an implementation detail and is not expected to be modified by the user.
> 
> I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.
> 
>>  
>> 
>> So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do? 
>> 
>> 
>> Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.
>>  
>> At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.
>> 
>> 
>> One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).
> 
> For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.
> 
> I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.
> 
> Kind Regards,
> Geordie
> 
>> 
>> We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.
> 
> Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?

The mailing list and the swift-evolution repository are the “authoritative” sources for discussion, Slack is great for working out details but ultimately any decisions should be communicated through one of the those.

 - Daniel

> 
>>  
>> Cheers and thank you in advance,
>> Geordie
>> 
>> 
>> _______________________________________________
>> swift-build-dev mailing list
>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20171201/878d3a3a/attachment.html>