[swift-build-dev] [swift-users] SwiftPM manual dependency management

[Geordie Jay]

Hi Ankit, thanks for your explanations.

Ankit Aggarwal <ankit_aggarwal at apple.com> schrieb am Sa. 22. Juli 2017 um
13:44:

> On 22-Jul-2017, at 3:37 PM, Geordie Jay <geojay at gmail.com> wrote:
>
>
> Geordie J <geojay at gmail.com> schrieb am Fr. 21. Juli 2017 um 14:39:
>
>> Hi Ankit, thanks for your reply.
>>
>> Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users <
>> swift-users at swift.org>:
>>
>>
>>
>> On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users <
>> swift-users at swift.org> wrote:
>>
>>> Hi all,
>>>
>>> My team and I are trying to use SwiftPM to develop a relatively complex
>>> app with multiple dependencies, all of which are being developed locally
>>> and in parallel. The reason for this is compatibility with an existing
>>> module/import structure used by our iOS app. Maybe I’m doing something very
>>> wrong but my experience so far (2 months in) is that this is extremely
>>> difficult with SwiftPM.
>>>
>>> What I’d love to be able to do is to just run `git add submodule
>>> http://blah.com/mysubmodule.git` in the Packages subdirectory and
>>> SwiftPM would just let me manage dependencies from there myself.
>>>
>>> I was excited to see that SwiftPM 4 has a "Top of Tree" development
>>> option for this purpose. So far my experience with this has not been good.
>>> Firstly because SwiftPM *still* unnecessarily tries to clone my repos
>>> itself (some of which are huge), and secondly because this creates an
>>> absolute path dependency in `.build/dependencies-state.json`, meaning this
>>> setup isn’t sharable within our dev team.
>>>
>>> Attempting this with "local" git urls adds an almost absurd level of
>>> complexity, having to tag each commit for SwiftPM to build. The fact that
>>> we'd need to make a commit to test whether the project even builds is
>>> insane enough as is, let alone the tagging and trying to tell the base
>>> project to use a newer minor version etc etc.
>>>
>>> Adding multiple subtargets is also not an option because the
>>> dependencies (as dynamic libraries) really are shared between multiple
>>> targets/sub-dependencies, which SwiftPM seems to deal with quite well.
>>>
>>> *tldr;* *Please* let us manage dependencies ourselves. It’d be so easy
>>> if Package.swift had an option along the lines of *.Package.local(named:
>>> "XYZ")* that it then looked for in ./Packages/XYZ. Again, maybe I’m
>>> overlooking something but this seems like an obvious and vital option to
>>> have. It’d also simplify the introductory SwiftPM docs significantly.
>>>
>>> Is anyone else having this issue? Would this change really be as simple
>>> and painless as it sounds? I would be prepared to make a pull request along
>>> these lines.
>>>
>>
>> I think you're not really using the Top of Tree feature. You need to add
>> each dependency using its canonical URL, hosted at some server like github.
>> After adding the dependencies, you can use edit feature to put a dependency
>> in Top of Tree mode. To do so, run:
>>
>> $ swift package edit <PackageName> --path
>> ../path/to/self/managed/checkout/of/the/package
>>
>>
>> Yes, this is what I tried this week. I’m pretty sure this is not a case
>> of misunderstanding the feature or the docs.
>>
>>
>> The package manager will then stop using the cloned repository and use
>> the checkout present at that path (regardless of the state it is in).
>>
>>
>> Yes, but then I have – per dependency – two checkouts of a potentially
>> huge repository. Why force everyone on the dev team to clone a huge repo
>> twice, only to *never* use one of the clones. Also, SwiftPM breaks when
>> —path points at Packages/PackageName, which is exactly where I’d expect the
>> package to be, not in some arbitrary external path (+ some kind of internal
>> checkout cache that will never be used) as well.
>>
>> I haven’t tried to test this recently because it’s a slow process but I
>> have the impression the deps could be even be cloned more than twice,
>> depending on how cleverly SwiftPM realises that multiple Packages have the
>> same dependency.
>>
>> Also, this makes managing interdependent state of development amongst
>> dependencies more difficult than needed. How do we guarantee that devs are
>> on the same commit when using top of tree development? Tagging and managing
>> version numbers etc for day-to-day development is emphatically not an
>> option for us. Since SwiftPM packages only work from a git context anyway,
>> why not allow use of git’s established pattern of dealing with this, namely
>> submodules?
>>
>> Sharing this setup is not automatic, but simple. Each user just needs to
>> run the above command once per dependency.
>>
>>
>> We have about 10 dependencies, *all *of which will* always* be in this
>> state. This seems like a lot of overhead and room for user error, plus it’s
>> a huge workaround for something that could be very simple.
>>
>> Also, you only need to do this if you're actively working on a dependency.
>>
>>
>> The point is that we will *always* be working on the dependencies. This
>> is the core of what we’re doing, not a short aside. This is what makes me
>> think we are either doing something wrong, or there is a big feature gap
>> (as it appears from here).
>>
>> The new manifest also supports using branch instead of version range,
>> which is very helpful during the development period.
>>
>>
>> This has much the same result as top-of-tree development, but it is how
>> we were able to "hack" SwiftPM 3 into leaving us alone.
>>
>> Let me know if something is unclear or if you have more questions!
>>
>>
>> Maybe an overview of our structure would be helpful to make our use case
>> clearer:
>>
>> Main Project (git repo, not a Swift Package, contains no swift code
>> directly)
>> –– Dependencies (external)
>> –– Subproject (*internal* git submodule, is a Swift Package, has
>> multiple Swift Targets)
>> –––– Dependency A (*internal*, git submodule)
>> –––––––– Huge external C-language dependencies (managed via git
>> submodules)
>> –––– Dependency B (*internal*, git submodule)
>> –––––––– Depends on internal dependency D
>> –––– Dependency C (*internal*, git submodule)
>> –––––––– Depends on *internal* dependency A
>> –––––––– Depends on *internal* dependency B
>> –––––––– etc.
>> –––– Dependency D (*internal*, git submodule)
>>
>
> Reading over this entire thread, I've come across this again, which I
> think sums up the pain point better than anything else:
>
>
>> *I think the friction is coming from the fact that we’d like to use
>> SwiftPM just to build, rather than to manage our dependencies.*
>>
>
> When I say "swift build", I expect swift to build! Not to check for
> commits and tags and dependencies, not to clone anything, just build what
> is there. For all the other stuff we have the "swift package ..." commands.
>
> Imagine if "swift package update" did the step that currently happens
> before "swift build" builds the project (dep management, cloning etc). And
> to retain old behaviour we could have "swift build --update-deps" with a
> note on failing "swift build" builds suggesting users try the update flag
> to get the old behaviour.
>
>
> One of the core functionality of the package manager is managing
> dependencies. The "swift build" command doesn't do any dependency related
> operation unless it needs to. If there are no dependencies cloned yet, it
> doesn't make sense for "swift build" to error out and say run "swift
> package resolve" because it can already run that command if needed.
>
> I think this would be an equally welcome addition for users not currently
> on a strong internet connection.
>
>
> If you already have all the dependencies cloned, the "swift build" command
> will never try to connect to internet.
>

Assuming multipackage repos are accepted this will be true for all
important cases, I agree.

>

We're also considering an offline mode for the "package resolve" command
> but that is for different reasons.
>
>
> The question would be how Swift knows which dependency is which. The
> answer should be pretty simple: look in each Packages/package/Package.swift
> for the available package names and build them if there's a dependency on
> them somewhere in the graph.
>
>
> That sounds like a very fragile implementation. The package authors
> shouldn't need to care how their dependencies are being handled by the
> package manager, unless of course it is being edited.
>
> There's something about that idea that seems at odds with the current
> git-centric model, though I'm still skeptical that the git-centric model is
> a reasonable base case.  For example, it's confusing because
> Sources/TargetName uses a filesystem convention while package dependencies
> currently do not, but kind of actually do after "swift package edit" has
> been run, and would have to with any of the current proposals. Is the
> complexity of using git as the base case becoming clear? No matter what
> we're doing, the reality is that we end up with files in our filesystem. To
> me having the dep as a remote git repo is actually the edge case, which is
> the opposite of the current model.
>
>
> Note that the convention system for targets is greatly simplified in Swift
> 4 and fully customizable.
>

I did notice yesterday that tools version 4.0 allows many different target
structures. I guess this is nice but having a "convention" that could be
one of many things (different locations for target sources) also makes
SwiftPM more complex than simple in my opinion.

I don't think target and package dependencies are similar enough to relate
> them to the convention system. The package manager is fully responsible for
> managing the dependencies. The dependencies need to come from somewhere,
> which is the git url currently. The multipackage repository will allow
> having the dependencies in the same repository, I think this is what really
> fits your usecase. Unfortunately, we couldn't finish design for this
> feature in enough timeframe for Swift 4.
>

How can I get involved in the evolution of this? The evolutions are always
uneditable uncommentable markdown files on a repo somewhere and the mailing
lists are in my practical experience inpentrible especially for entries you
weren't subscribed to the list for, or were "only" subscribed to the digest
for. I wish they were on google docs or a hackpad equivalent. Is there
"officially recognised" discussion on the SwiftPM dev slack channel
regarding evolution topics?


> To do anything with a moderate level of complexity took me weeks to get
> figure out the ins and outs of and get to a point where it was repeatable
> within the team. Again, I think this can be done better. And I think
> assuming that dependencies are local unless specified otherwise would make
> this a lot easier. Why not for example just have "swift package update"
> check out the git repos into its internal build cache and symlink them into
> Packages/PackageName, always building packages from ./Packages? I'd find
> that a lot more consistent and transparent.
>
>
> Again, the package authors shouldn't need to worry about how the
> dependencies are being managed. The editable packages proposal
> <https://github.com/apple/swift-evolution/blob/master/proposals/0082-swiftpm-package-edit.md> briefly
> explains the motivation behind making the dependencies an implementation
> detail.
>
> We have discussed whether or not hiding the sources for non-editable
> packages is the right default. The motivation for hiding the sources is
> that in a large, mature, stable ecosystem there are likely to be a large
> number of packages involved in any particular project build, and many of
> those are likely to be uninteresting to the package developer. In
> particular, while a project developer might be interested in the source of
> their direct dependencies, the sources of that packages own dependencies is
> an "implementation detail" from the perspective of the project developer.
>
>
Being interested in the source of my direct dependencies is exactly the
current case. I can see the point about having many indirect package
dependencies, but this is what I'm used to from other ecosystems and to be
honest it has never bothered me there. More often I have found it
convenient to track down bugs in 3rd party libraries etc. but I agree this
would still be possible with the checkouts in .build

Have a great weekend,
Geordie


> The package update command does actually need to fetch the latest tags
> from internet to do the dependency resolution. It may be possible to
> resolve the dependencies with the tags we have in cache but that may not be
> ideal. However, this is still useful in case there is no internet
> connection, so we plan to introduce an offline mode to handle those cases.
>
>
> In the meantime the multi-package repo proposal seems like a step in the
> right direction. In a way though it seems to me like a crutch for an
> underlying inconsistency in both tooling namespaces ("swift build" vs
> "swift package") and in convention vs configuration (specific directory
> structure vs automated dep management via git).
>
> I'm not sure what the implications of this are, but I'm starting to wonder
> whether "swift build" and "swift package" are conceptually two different
> but related projects, and whether it'd be a good idea in the medium-long
> term to more clearly separate them.
>
>
> I think I tried to explain the reason why "swift build" can run git
> commands above.
>
> Cheers,
> Geordie
>
>
> PS. The top-of-tree workaround does work after all, but is complicated
> because we're also running "swift build" from within a docker image to
> build for other platforms, so the absolute paths are (inescapably)
> different between the environments.
>
> Again, the idea of having absolute paths there at all seems unnecessary,
> but until multi-package repos are available it seems the best option will
> be scripting a find-and-replace in ".build/dependencies-state.json" before
> running "swift build"...
>
>
> One solution could be using a different build folder when using docker.
>


Possible, true, I didn't think of that. It'd mean even more git checkout
deadweight of huge dependencies though. And the script to put the Packages
in editable mode would then have to be maintained for multiple
environments. The easier option is just altering the absolute paths for now.



> ---
>
> Lets see what others think about reconsidering local dependencies feature
> as that can probably help many such usecases until multi-package repository
> feature arrives.
>
>
>
>> Again, this could be solved with a simple API addition in the manifest:
>>
>> Package(
>>   …
>>   dependencies: [
>>     .package.local(named: "Dependency A")
>>     .package.local(named: "Dependency B")
>>     ...
>>   ]
>> )
>>
>>
>> At the end of the day it seems we can work around this by cloning the
>> submodules at *Project/Submodule*instead of *Project/Package/Submodule* and
>> then running *swift package edit Submodule —path ./Submodule*, just that
>> this process would have to be manual for each new dev cloning the repo. And
>> then we’d still have two checkouts of the same thing. Yes, this works, it
>> just seems very inefficient and still hacky. And it’s very possible it'll
>> break again with future SwiftPM versions.
>>
>
>> I’m just surprised the idea of a "local dependency" is not seen as a
>> first class citizen in SwiftPM, still trying to understand the logic behind
>> that. Maybe you can give me an idea of the reasoning behind this?
>>
>> Best Regards,
>> Geordie
>>
>
>>
>>
>>> Best Regards,
>>> Geordie
>>>
>>>
>>> PS. In SwiftPM 3 we had been using a hack that worked great: by filling
>>> in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just
>>> left us alone.. We were able to commit `workspace-state.json` into our base
>>> project’s git repo and the rest Just Worked™. Now with the absolute paths
>>> being checked for this doesn’t seem to be an option.
>>>
>>>
>> Please do not rely on internals of the package manager as they're not
>> stable and will change without notice.
>>
>>
>> This was not our preferred way of going about it of course. But it was
>> (unfortunately) the best solution to the problem.
>>
>>
>>
>>
>>> _______________________________________________
>>> swift-users mailing list
>>> swift-users at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>
>>>
>> _______________________________________________
>> swift-users mailing list
>> swift-users at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-users
>>
>>
>
> --
> Ankit
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20170722/16a09d63/attachment.html>