[swift-build-dev] Advice on using SwiftPM for developing large projects

Daniel Dunbar daniel_dunbar at apple.com
Fri Dec 1 10:32:26 CST 2017 


> On Dec 1, 2017, at 8:24 AM, Geordie J <geojay at gmail.com> wrote:
> 
> Hi Daniel,
> 
> Thanks for your reply!
> 
>> Am 01.12.2017 um 16:48 schrieb Daniel Dunbar <daniel_dunbar at apple.com <mailto:daniel_dunbar at apple.com>>:
>> 
>> Hi Geordie,
>> 
>>> On Dec 1, 2017, at 5:16 AM, Geordie J via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>> 
>>> Hi Ankit, thanks for your quick reply.
>>> 
>>> I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :) The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.
>> 
>> We honestly appreciate the feedback about pain points, as long as the tone is respectful (which I think yours is) we are more than happy to hear constructive criticism!
>> 
>>> I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just builds with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.
>> 
>> This is certainly our goal, getting it right requires tuning and time.
>> 
>>>> Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <ankit_aggarwal at apple.com <mailto:ankit_aggarwal at apple.com>>:
>>>> 
>>>> 
>>>> 
>>>> On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>>> Hello Swift Build Devs!
>>>> 
>>>> I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)
>>>> 
>>>> We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.
>>>> 
>>>> The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.
>>>> 
>>>> In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.
>>> 
>>> Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:
>>> 
>>> 1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:
>>> 
>>> 2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to https://bugs.swift.org/browse/SR-6492 <https://bugs.swift.org/browse/SR-6492> and a seemingly endless cycle of frustration, particularly when combined with the next point:
>>> 
>>> 3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).
>>> 
>>> Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)
>>> 
>>> 4. The more recent bug (https://bugs.swift.org/browse/SR-6493 <https://bugs.swift.org/browse/SR-6493>) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.
>> 
>> I agree, this one is really frustrating me too and I think we need to get it resolved.
>> 
>>> 5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.
>>> 
>>> 6. Why are we dealing with all this complexity we don’t need?
>>> 
>>> The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.
>> 
>> One thing to be clear, your way of using SwiftPM is not the “golden path” that we believe most users are using. Deviating that is probably causing you to bump into more pain points and workflows that need tuning.
> 
> I agree with this point wholeheartedly. I just have the impression that, if SwiftPM were structured around the case of just “building” rather than on managing dependencies, our case would be trivial and every other case would be no more complex. I have been very surprised from the outset that even the most basic of SPM demos require initting a git repo. I have the impression I’m not alone here.
> 
> I don’t know enough about the behind the scenes nor the current SPM codebase to comment on how realistic this wish is, but as I alluded to in previous emails, I kind of wish there were two projects, “swift build” (which builds) and “swift package” (which manages deps). It seems the two are mixed together in a way that is currently very difficult (at least as a user) to disentangle. I’m not suggesting they’d have to be two codebases either. Those two commands already exist under one project as is, just their functionality doesn’t always match what is on the tin, in my opinion.
> 
>> 
>> Thank you for writing up the concrete list of issues, that is very helpful. Here is my perspective on this:
>> 1. As I said, I think your way of using SwiftPM is slightly unorthodox. One thing that means is we have to carefully balance the features you want versus their impact on all the other workflows.
> 
> The docker case building for two platforms is definitely unorthodox, and I realise that optimising towards that is (very) unrealistic. But I don’t see anything unorthodox at all about the vision of just being able to:
> 
> - create a directory on an offline machine without git installed
> - make a Package.swift and a Sources directory with some code in it
> - type "swift build” to build it
> - add this newly created package as a dependency of another one (its parent, say) via the filesystem, without any behind the scenes copying or magic required.
> 
> The git requirement (etc.) seems antithetical to Swift’s progressive disclosure principle. This is what I’m referring to filesystem-based packages being the simple “base case”. Structuring in this way would allow for the docker case and many others (running "swift build" from a more minimal ubuntu image, etc).

Just to make sure I am on the same page — the first three of these already work, it is the last bullet point that you find objectionable? I think we are all in agreement that this is a pain point we would like to solve (the question is how).

> 
>> 2. I agree there is a problem here. I myself encounter a similar situation relatively frequently with the work I am doing, and don’t yet know how to solve it.
>> 3. Ultimately, the most constructive form of conversation is to try and identify individual features which (a) help and (b) integrate well with all the other features, and then write (or coauthor, if you want help) a proposal for it.
> 
> Can you think of a case where filesystem-based packages would break existing features or make the SPM codebase more complex? Because the more I write about this, the more I think they’d be the basis of any proposal I would like to make.

The big concern is not wanting users to unintentionally end up in a situation where their package cannot be effectively shared or built reproducibly by other developers.

>> As one example, a feature that I have long believed makes sense for SwiftPM is a way for developers to have a global “set of packages I contribute to”, and then SwiftPM features to always use those versions when resolving dependencies, instead of manually managing them. I think such a feature might fit with your use case, and solve many of your problems. What do you think?
> 
> Hmm this may work, but it again seems to be straying into the land of an unpredictable level of complexity in practice. It’s unlikely to work with the docker environment, for example. And a many of the packages this would apply to would only used in a single project, so it’d be centralising things unnecessarily (== complexity). I could see great use for in this concept for smaller “toolset” library packages though, say with some helpers for scripting or async.
> 
> All this aside, it’d be hard to argue that this would be simpler than filesystem-based packages for the SPM codebase or for the user’s mental model.

How would you imagine filesystem based packages working — that you would just give SwiftPM the path to the package, and it would accept it if it finds a Package.swift there but no git repository?

 - Daniel

> 
>> 
>>>> Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md <https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md>. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.
>>>> 
>>>> I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?
>>>> 
>>>> We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!
>>> 
>>> Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?
>> 
>> Rick (CC’d) is working on revising the next version of this proposal and will post it once ready.
> 
> Great, thanks.
> 
>> 
>>>> 
>>>> I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e <https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e> which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.
>>>> 
>>>> The isLocal property is an implementation detail and is not expected to be modified by the user.
>>> 
>>> I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.
>>> 
>>>>  
>>>> 
>>>> So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do? 
>>>> 
>>>> 
>>>> Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.
>>>>  
>>>> At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.
>>>> 
>>>> 
>>>> One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).
>>> 
>>> For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.
>>> 
>>> I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.
>>> 
>>> Kind Regards,
>>> Geordie
>>> 
>>>> 
>>>> We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.
>>> 
>>> Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?
>> 
>> The mailing list and the swift-evolution repository are the “authoritative” sources for discussion, Slack is great for working out details but ultimately any decisions should be communicated through one of the those.
> 
> Ok thanks for the info!
> 
> - Geordie
> 
>> 
>>  - Daniel
>> 
>>> 
>>>>  
>>>> Cheers and thank you in advance,
>>>> Geordie
>>>> 
>>>> 
>>>> _______________________________________________
>>>> swift-build-dev mailing list
>>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
>>> _______________________________________________
>>> swift-build-dev mailing list
>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20171201/af216de6/attachment.html>

More information about the swift-build-dev mailing list