[swift-build-dev] Allowing further module organization

[Daniel Dunbar]

> On Apr 19, 2016, at 1:47 PM, Max Howell <max.howell at apple.com> wrote:
> 
>> Just to make sure I understand, the idea here is that
>> 
>> - Sources
>> - Sources/A
>> - Sources/A/B
>> - Sources/A/C
>> 
>> produces two modules, B & C, because A has no sources in it directly (following the convention for Sources/ itself)?
> 
> Yes.
> 
>> Following the SwiftPM example, why do you want BuildNode and BuildDescription to be separate modules?
> 
> I have enjoyed splitting code out into more encapsulated units. This is one example of that.
> 
>> The biggest problem I see here is that modules may mean different things to different people, and they have significant semantic meaning (since they change visibility). If I compare to LLVM, for example, the individual equivalent of a "module" is typically quite large. That leads to occasionally wanting to subdivide the sources of the module into subfolders without changing which module they contribute to, which I think this proposal would require?
> 
> Indeed, my proposal is basically saying: modules should be small. But maybe this is too much.

I don't think I am comfortable with that assumption, currently.

In particular, module subdivision and visibility have substantial impact on performance (e.g., whole-module optimization), and performance sensitive clients are likely to want to put critical functionality in the same module even if it may be conceptually encapsulated. Modules are also very important for the Swift ABI resiliency model, which will encourage the same behavior.

Right now, for better or worse, modules are very similar to traditional libraries or frameworks, and not just because that is how we typically build them. It makes sense for a large project to subdivide code into modules along the same boundaries that would be done for those, which is typically fairly large.

> There seems to be three options:
> 
> 1. Leave it as is, subfolders of Sources are modules, subfolders of module folders are organizational.

One benefit of this is that it is very clear exactly what modules a package produces just from the Sources subdirectory structure. I tend to think this is an actively good thing. I agree it doesn't scale up.

> 2. What I propose, where empty folders in Sources indicate organizational folders, you can’t have subfolders to make our rules less “surprising”
> 3. What I propose, where empty folders indicate organizational folders, but you can have subfolders with source files in modules, this could be “surprising” where adding sources to empty folders completely changes the outputs of a build
> 4. Some setting in Package.swift that can mark folders as organizational
> 
> 4 seems more like what you think should be done and that would be fine with me if people don’t think saying: modules should be small is acceptable.

I'm not sure about overrides for this specifically. I think we need a general purpose override mechanism, and think maybe the capability should fall out of that?

Here is an alternative proposal along a different line:

We have outstanding requests for better support for having multiple packages within a single repository. This is useful both for people who are building "monorepo" style projects, as well as for large projects which want to impose more organization. I have a design in my head sketched out, but not written up yet. If we had this support, one way to accomplish what you want without needing to change the existing conventions would be to use that support and subdivide the project into sub packages.

What do you think of the idea of using that feature to solve this problem?

 - Daniel

> 
> Max
>