<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">+swift-build-dev<div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 21-Jul-2017, at 6:09 PM, Geordie J <<a href="mailto:geojay@gmail.com" class="">geojay@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Ankit, thanks for your reply.<div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users <<a href="mailto:swift-users@swift.org" class="">swift-users@swift.org</a>>:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users <span dir="ltr" class=""><<a href="mailto:swift-users@swift.org" target="_blank" class="">swift-users@swift.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">Hi all,<div class=""><br class=""></div><div class="">My team and I are trying to use SwiftPM to develop a relatively complex app with multiple dependencies, all of which are being developed locally and in parallel. The reason for this is compatibility with an existing module/import structure used by our iOS app. Maybe I’m doing something very wrong but my experience so far (2 months in) is that this is extremely difficult with SwiftPM.</div><div class=""><br class=""></div><div class="">What I’d love to be able to do is to just run `git add submodule <a href="http://blah.com/mysubmodule.git%60" target="_blank" class="">http://blah.com/mysubmodule.<wbr class="">git`</a> in the Packages subdirectory and SwiftPM would just let me manage dependencies from there myself.</div><div class=""><br class=""></div><div class="">I was excited to see that SwiftPM 4 has a "Top of Tree" development option for this purpose. So far my experience with this has not been good. Firstly because SwiftPM <i class="">still</i> unnecessarily tries to clone my repos itself (some of which are huge), and secondly because this creates an absolute path dependency in `.build/dependencies-state.<wbr class="">json`, meaning this setup isn’t sharable within our dev team.</div><div class=""><br class=""></div><div class="">Attempting this with "local" git urls adds an almost absurd level of complexity, having to tag each commit for SwiftPM to build. The fact that we'd need to make a commit to test whether the project even builds is insane enough as is, let alone the tagging and trying to tell the base project to use a newer minor version etc etc.</div><div class=""><br class=""></div><div class="">Adding multiple subtargets is also not an option because the dependencies (as dynamic libraries) really are shared between multiple targets/sub-dependencies, which SwiftPM seems to deal with quite well.</div><div class=""><br class=""></div><div class=""><b class="">tldr;</b> *Please* let us manage dependencies ourselves. It’d be so easy if Package.swift had an option along the lines of <b class="">.Package.local(named: "XYZ")</b> that it then looked for in ./Packages/XYZ. Again, maybe I’m overlooking something but this seems like an obvious and vital option to have. It’d also simplify the introductory SwiftPM docs significantly.</div><div class=""><br class=""></div><div class="">Is anyone else having this issue? Would this change really be as simple and painless as it sounds? I would be prepared to make a pull request along these lines.</div></div></blockquote><div class=""><br class=""></div><div class="">I think you're not really using the Top of Tree feature. You need to add each dependency using its canonical URL, hosted at some server like github. After adding the dependencies, you can use edit feature to put a dependency in Top of Tree mode. To do so, run:</div><div class=""><br class=""></div><div class="">$ swift package edit <PackageName> --path ../path/to/self/managed/checkout/of/the/package</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Yes, this is what I tried this week. I’m pretty sure this is not a case of misunderstanding the feature or the docs.</div><br class=""></div></div></div></div></blockquote><div><br class=""></div><div>Ah, cool!</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><div class="">The package manager will then stop using the cloned repository and use the checkout present at that path (regardless of the state it is in).</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Yes, but then I have – per dependency – two checkouts of a potentially huge repository. Why force everyone on the dev team to clone a huge repo twice, only to *never* use one of the clones.</div></div></div></div></div></blockquote><div><br class=""></div><div>Sure, you could end up with clones but that is only an implementation detail. SwiftPM can become smart enough and remove a clone that is not being used.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><div class=""> Also, SwiftPM breaks when —path points at Packages/PackageName, which is exactly where I’d expect the package to be, not in some arbitrary external path (+ some kind of internal checkout cache that will never be used) as well.</div></div></div></div></div></blockquote><div><br class=""></div><div>If --path Packages/PackageName doesn't work, that is a bug! Do you mind filing a JIRA at <a href="http://bugs.swift.org" class="">bugs.swift.org</a>? </div><div><br class=""></div><div>I understand internal caches could be an inconvenience if repositories are large but we can definitely improve how caching works.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><div class=""><br class=""></div><div class="">I haven’t tried to test this recently because it’s a slow process but I have the impression the deps could be even be cloned more than twice, depending on how cleverly SwiftPM realises that multiple Packages have the same dependency.</div></div></div></div></div></blockquote><div><br class=""></div><div>Right now, SwiftPM will never clone a dependency twice.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><div class="">Also, this makes managing interdependent state of development amongst dependencies more difficult than needed. How do we guarantee that devs are on the same commit when using top of tree development? Tagging and managing version numbers etc for day-to-day development is emphatically not an option for us.</div></div></div></div></div></blockquote><div><br class=""></div><div>When you use top of the tree mode, the checkout is expected to be managed by the user. That is exactly what top of the tree mode means. If we get local package support in the manifest, you will still need to manage the checkout state of those dependencies.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><div class=""> Since SwiftPM packages only work from a git context anyway, why not allow use of git’s established pattern of dealing with this, namely submodules?</div></div></div></div></div></blockquote><div><br class=""></div><div>I am not what do you mean here exactly.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Sharing this setup is not automatic, but simple. Each user just needs to run the above command once per dependency.</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">We have about 10 dependencies, <i class="">all </i>of which will<i class=""> always</i> be in this state. This seems like a lot of overhead and room for user error, plus it’s a huge workaround for something that could be very simple.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Also, you only need to do this if you're actively working on a dependency. </div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">The point is that we will <i class="">always</i> be working on the dependencies. This is the core of what we’re doing, not a short aside. This is what makes me think we are either doing something wrong, or there is a big feature gap (as it appears from here).</div></div></div></div></div></blockquote><div><br class=""></div><div>I agree that a local dependency feature will be very helpful here but also note that you only need to run the edit command once (unless you delete the build folder). You can write a simple script that just put every dependency you have in edit mode.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">The new manifest also supports using branch instead of version range, which is very helpful during the development period.</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">This has much the same result as top-of-tree development, but it is how we were able to "hack" SwiftPM 3 into leaving us alone.</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Let me know if something is unclear or if you have more questions!</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Maybe an overview of our structure would be helpful to make our use case clearer:</div><div class=""><br class=""></div><div class="">Main Project (git repo, not a Swift Package, contains no swift code directly)</div><div class="">–– Dependencies (external)</div><div class="">–– Subproject (<b class="">internal</b> git submodule, is a Swift Package, has multiple Swift Targets)</div><div class="">–––– Dependency A (<b class="">internal</b>, git submodule)</div><div class="">–––––––– Huge external C-language dependencies (managed via git submodules)</div><div class="">–––– Dependency B (<b class="">internal</b>, git submodule)</div><div class=""><div class="">–––––––– Depends on internal dependency D</div></div><div class="">–––– Dependency C (<b class="">internal</b>, git submodule)</div><div class=""><div class="">–––––––– Depends on <b class="">internal</b> dependency A</div><div class="">–––––––– Depends on <b class="">internal</b> dependency B</div><div class="">–––––––– etc.</div></div><div class="">–––– Dependency D (<b class="">internal</b>, git submodule)</div><div class=""><br class=""></div><div class=""><b class="">I think the friction is coming from the fact that we’d like to use SwiftPM <i class="">just to build</i>, rather than to manage our dependencies.</b></div><div class=""><br class=""></div><div class="">Again, this could be solved with a simple API addition in the manifest:</div><div class=""><br class=""></div><div class="">Package(</div><div class=""> …</div><div class=""> dependencies: [</div><div class=""> .package.local(named: "Dependency A")</div><div class=""> .package.local(named: "Dependency B")</div><div class=""> ...</div><div class=""> ]</div><div class="">)</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">At the end of the day it seems we can work around this by cloning the submodules at <b class="">Project/Submodule</b> instead of <b class="">Project/Package/Submodule</b> and then running <b class="">swift package edit Submodule —path ./Submodule</b>, just that this process would have to be manual for each new dev cloning the repo. And then we’d still have two checkouts of the same thing. Yes, this works, it just seems very inefficient and still hacky. And it’s very possible it'll break again with future SwiftPM versions.</div></div></div></div></div></blockquote><div><br class=""></div><div>I think we do allow editing a package that is inside another package as a submodule. If that doesn't work, do you mind filing a JIRA with a sample project setup?</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><div class="">I’m just surprised the idea of a "local dependency" is not seen as a first class citizen in SwiftPM, still trying to understand the logic behind that. Maybe you can give me an idea of the reasoning behind this?</div></div></div></div></div></blockquote><div><br class=""></div><div>I understand your usecase will benefit a lot from the local dependency feature. We did consider adding this feature, I think the main concern was package authors ending up publishing package which only works for them incase they forget updating their manifest before a release. Also, note that SwiftPM is still under heavy development and we can definitely re-consider adding this!</div><div><br class=""></div><div>Also, I think this is a good usecase of a multipackage repository feature. We don't have that yet but you can read some of the discussion <a href="https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20161107/000722.html" class="">here</a>.</div><div><br class=""></div><div>PS: We also have a slack channel if you want to hang out - <a href="https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html" class="">https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html</a></div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><div class="">Best Regards,</div><div class="">Geordie</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""></div><div class="">Best Regards,</div><div class="">Geordie</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">PS. In SwiftPM 3 we had been using a hack that worked great: by filling in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just left us alone.. We were able to commit `workspace-state.json` into our base project’s git repo and the rest Just Worked™. Now with the absolute paths being checked for this doesn’t seem to be an option.</div></div><br class=""></blockquote><div class=""><br class=""></div><div class="">Please do not rely on internals of the package manager as they're not stable and will change without notice.</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">This was not our preferred way of going about it of course. But it was (unfortunately) the best solution to the problem.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">______________________________<wbr class="">_________________<br class="">
swift-users mailing list<br class="">
<a href="mailto:swift-users@swift.org" class="">swift-users@swift.org</a><br class="">
<a href="https://lists.swift.org/mailman/listinfo/swift-users" rel="noreferrer" target="_blank" class="">https://lists.swift.org/<wbr class="">mailman/listinfo/swift-users</a><br class="">
<br class=""></blockquote></div><br class=""></div></div>
_______________________________________________<br class="">swift-users mailing list<br class=""><a href="mailto:swift-users@swift.org" class="">swift-users@swift.org</a><br class=""><a href="https://lists.swift.org/mailman/listinfo/swift-users" class="">https://lists.swift.org/mailman/listinfo/swift-users</a><br class=""></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""><div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="">--<br class="">Ankit</div></div>
</div>
<br class=""></div></body></html>