[swift-build-dev] Draft SwiftPM proposal: Multi-package repositories

[Russ Bishop]

> On Nov 12, 2016, at 9:54 PM, Daniel Dunbar <daniel_dunbar at apple.com> wrote:
>> 
>> 
>> It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one. 
>> 
>> Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.
> 
> Mostly because I am concerned this doesn't scale well to *very* large repositories, in which commits to that file would be "contentious" (in the lock contention sense, not subject to debate sense). Of course, this argument is a little bogus as the current proposal doesn't scale that great either since we have to discover the packages (although I believe we can probably do a good job of caching this information).

If you’re big enough that you’re creating sub packages frequently then you’re big enough to use automated tooling and it won’t matter so much what we do as long as the file format doesn’t promote impossible merges. Ideally the sub package specifiers can be one-liners so most 3-way merge tools won’t have trouble.

Let’s be absurd and imagine a monorepo with 100,000 packages created over a 5-year period. That’s one new package every ~6 minutes only during 8-hr workdays in a single timezone. It would not be possible to manually commit anything to the monorepo in that scenario let alone a change to a package file, you would have to use tooling that rolled changes to various levels of upstream. It would be trivial for a package-specific script to simply form the union of all unique packages across the downstream sources it is looking at to create the proper top-level manifest.

More realistically if you took 10,000 as the upper-bound that’s one commit per hour to the top-level manifest which is easily doable with manual merging.

> 
> It certainly would simplify the implementation & proposal to have this.
> 
> The other reason is it is yet another thing for people to maintain (and remember the syntax for). Most repos are small enough that I think the current proposal would perform fine and have a tendency to do what people might naively expect (even if they didn't really think about why). On the other hand, this file is likely to be quite static, so I'm not sure that is a very important issue.
> 
> I was already on the fence on this, but I hadn't considered the benefits you mention of allowing import of the package w/ no sub package specifier to mean import of all sub-packages. That tips me a little more towards thinking maybe a better proposal is to KISS and require this in some root file (whether or not that root file is itself a package manifest or a different kind of file is another question, you assume it would be the regular package manifest but I don't think it *need* be, and there is some value in not having any nesting relationship amongst packages).
> 
> - Daniel


I can see pros and cons to a new file format or using the Package manifest so I would go along with either.


Russ