[swift-evolution] [Pitch] Let's talk about submodules

[Matthew Johnson]

Sent from my iPad

On Feb 21, 2017, at 4:01 AM, Brent Royal-Gordon <brent at architechies.com> wrote:

>> On Feb 20, 2017, at 7:59 PM, Matthew Johnson <matthew at anandabits.com> wrote:
>> 
>>> Any group of Swift files can be grouped together to form a submodule.
>> 
>> It isn't clear to me how you express that a particular group of files forms a submodule.  Can you elaborate?
> 
> I discuss this later, but maybe don't describe it that clearly. Basically, submodule membership is not directly expressed in the source code; rather, it's a function of how the build system invokes the compiler.
> 
> To understand what I'm suggesting, you must first understand how the Swift compiler is invoked to build code today. (The following is very simplified—there are some steps I've left out—but it's basically accurate.) Suppose your app, `PersonTalk`, has four files:
> 
>    AppDelegate.swift
>    RootViewController.swift
>    Person.swift
>    Message.swift
> 
> When Xcode or SwiftPM or whatever goes to compile your app, the build system will invoke the compiler once for each file, passing *all* of the files but specifying only one as primary:
> 
>    swiftc -emit-object -module-name PersonTalk -primary-file AppDelegate.swift RootViewController.swift Person.swift Message.swift ...
>    swiftc -emit-object -module-name PersonTalk AppDelegate.swift -primary-file RootViewController.swift Person.swift Message.swift ...
>    swiftc -emit-object -module-name PersonTalk AppDelegate.swift RootViewController.swift -primary-file Person.swift Message.swift ...
>    swiftc -emit-object -module-name PersonTalk AppDelegate.swift RootViewController.swift Person.swift -primary-file Message.swift ...
> 
> Each of these compiler invocations compiles its primary file into a .o file. The build system then asks Clang to link them all together, and you get a fully built module of some sort:
> 
>    clang -o PersonTalk AppDelegate.o RootViewController.o Person.o Message.o …
> 
> In my proposal, if you wanted Person.swift and Message.swift to be in a `PersonTalk.Model` submodule, you would do that by invoking the build tools in a slightly different way. First you would compile the two files in the submodule with one another:
> 
>    swiftc -emit-object -module-name PersonTalk.Model -primary-file Person.swift Message.swift ...
>    swiftc -emit-object -module-name PersonTalk.Model Person.swift -primary-file Message.swift ...
> 
> Then you would emit a single PersonTalk.Model.o file encompassing both of them:
> 
>    swiftc -emit-submodule -module-name PersonTalk.Model Person.o Message.o ...
> 
> Then you would compile the other two files, providing the PersonTalk.Model.o file as context:
> 
>    swiftc -emit-object -module-name PersonTalk -primary-file AppDelegate.swift RootViewController.swift -with-submodule PersonTalk.Model.o ...
>    swiftc -emit-object -module-name PersonTalk AppDelegate.swift -primary-file RootViewController.swift -with-submodule PersonTalk.Model.o ...
> 
> And finally, you would link everything together:
> 
>    clang -o PersonTalk AppDelegate.o RootViewController.o PersonTalk.Model.o …
> 
> So whether files belong to a submodule or the main module is not a property of the files themselves, but rather of how the build system chooses to combine them during compilation. That leaves the build system free to specify submodules in a way appropriate to its design.

Hmm interesting.  I'll have to give this further thought but I can see how this approach could have advantages.  Would WMO still work across the whole module or would it be restricted to individual submodules?  I think we would definitely want to see it work across all submodules.  I think this would work because it runs during linking no, right?

> 
> I assume that Xcode would group files into submodules with per-file or per-target metadata. For instance, there might be a "Submodule" field in the File inspector of each Swift source file, or an additional tab in the Project editor. Upon building, it would examine all its metadata, create a list of submodules and files in each one, examine each submodule to determine its dependencies, order the submodules so that each one's dependencies will be available, and invoke the compiler dance properly to build everything.
> 
> SwiftPM, which relies mainly on the directory structure to organize your code, would probably have each submodule be a subdirectory of the Sources directory, using the presence of dots in the filename to match submodules:
> 
>    Package.swift
>    Sources/
>        PersonTalk/
>            AppDelegate.swift
>            RootViewController.swift
>        PersonTalk.Model/
>            Person.swift
>            Message.swift
>        TalkKit/
>            Talker.swift
>            TalkUser.swift
>            TalkMessage.swift
> 
> Or perhaps it would use nested subdirectories to represent the submodules:
> 
>    Package.swift
>    Sources/
>        PersonTalk/
>            AppDelegate.swift
>            RootViewController.swift
>            Model/
>                Person.swift
>                Message.swift
>        TalkKit/
>            Talker.swift
>            TalkUser.swift
>            TalkMessage.swift
> 
> In either case, I imagine that SwiftPM would try to build `PersonTalk`, detect the error indicating that the `PersonTalk.Model` submodule was not available, recurse in order to build `PersonTalk.Model`, and then try to build `PersonTalk` again.
> 
> Other build systems would do whatever made sense for them. If you were using `make`, for instance, you'd probably have either a target for each submodule, or a directory and `Makefile` for each submodule. If you used Ant, you'd write some sort of XML horror. The point is, specifying which files belong to which submodules is a build system responsibility, not a language responsibility. This keeps source code portable, avoids boilerplate, and keeps submodule information from being duplicated (and thus potentially becoming inconsistent) in file system structure or build system metadata.

Like I said earlier, I'll have to give this more thought.  In some ways I think it fits pretty well with what I want out of submodules.  

The downside is that it's potentially harder to move code between build systems or have it work with multiple build systems simultaneously (i.e. Xcode and SPM).  Have you given any thought to how to address this?  It would be unfortunate to have to duplicate submodule structure information for each build system.

> 
> Hope this helps,

Very much, thanks!

> -- 
> Brent Royal-Gordon
> Architechies
>