[swift-build-dev] Allowing further module organization

[Drew Crawford]

I hate to be "that guy", but I believe the problem under study is a special case of a much more general paradox.

Swift presently lacks any concept of "module encapsulation".  A module's members are either available everywhere, or available nowhere.  I refer to the following diagram:


> - Sources
>     - Country
>          Venezuela.swift
>        - United_States
>            Rhode_island.swift      
>           - Texas
>              Austin.swift

Consider the symbol Austin.SXSW.  Either Austin.SXSW is public, in which case it is accessible to Texas, Rhode Island, United_States, etc, or it is private/internal, in which case it is only accessible to Austin.  In no case it is accessible to Texas and not further, in no case is it "encapsulated" somehow by the Texas module.  It is either invisible or global, Swift does not have another option.  The namespace is flat. [0]

Meanwhile, Swift does have a concept of dependencies.  We must compile *some* module first, and that module cannot depend on another module which is not yet compiled.  Here, Austin has no dependencies and so it is compiled first.  As a consequence it cannot depend on United States, and therefore it cannot access Rhode Island.  However that has nothing to do with encapsulation, it has to do with a limitation of the Swift compiler model.  A limitation which by the way, does not exist in other SwiftPM languages such as C.  But in Swift, modules form a dependency tree [1].

The paradox is that, traditionally, a filesystem connotes both [0] and [1], while from a language point of view, we mean only one or the other.  For example, in Unix, /usr/bin/git is an encapsulated path; if you try to run /usr/bin/git we first check that you are allowed to traverse /usr/, then /usr/bin/, and only finally /usr/bin/git.  Whereas in Swift, to access Sources/Country/United_States/Texas/Austin.SXSW, we only check Austin; because Swift modules are not encapsulated.  So a filesystem that promised meaning [0] did not deliver.

On the other hand, a folder depends on its contents; if you ask Unix to copy a folder you expect not just an empty folder, but a folder of the same contents, e.g. an identical folder.  But if you copy a C module, you do not get all its dependencies, because there exists such a keyword as `extern`.  So a filesystem that promised meaning [1] did not deliver it either.

> If we then extend the same rules to modules then there will be no surprising behavior.

I suppose "surprise" is subjective, but I don't believe there is a solution to the paradox that is not surprising.  We could resolve the problem in the first diagram by reorganizing to solve [0], requiring

> - Sources
>     - Country
>        Venezuela.swift
>     - United_States
>        Rhode_island.swift      
>     - Texas
>        Austin.swift


which I think is one of your proposals.  But now we are wrong on [1].  That is, as a matter of compile-order, Rhode_island.Providence is not available in Austin while Austin.SXSW is available in Rhode_island, and this result is surprising given the flat filesystem layout shown.

Perhaps this sheds some light on my opposition to filesystem-defined build systems.  The paradox above is not accidentally complex, that we could resolve it by "tightening the rules" to mandate some particular filesystem layout.  It is essentially complex, because no filesystem layout can describe Swift's actual behavior. 

We could of course redefine Swift to mean both [0] and [1], while that would break... pretty much all existing code, I would support it.  However that does nothing to resolve the problem for C, so unless we plan to ban the extern keyword (which would no longer be C) that is no way out of the paradox.


> On Apr 19, 2016, at 2:42 PM, Max Howell via swift-build-dev <swift-build-dev at swift.org> wrote:
> 
> SwiftPM is a pioneer for SwiftPM, and thus we have many modules. For some of these modules I’d like to split them out even more eg:
> 
> - Sources
>     - Build
> 
> Build is a module, but really I’d like:
> 
> - Sources
>     - Build
>         - BuildNode
>         - BuildDescription
> 
> If I do this currently I still will end up with one module: `Build`, this is how the module layout rules that SwiftPM uses work. But instead I’d like Build to be purely an organizational folder and to get two modules *BuildNode* and *BuildDescription*.
> 
> I’d like to propose some method to allow further organization, but without ruining our existing rules.
> 
> I’d also like to propose tightening up our rules a little.
> 
> —————————
> 
> One clear way to accomplish the first goal is to extend the rules and say empty folders are not considered modules.
> 
> I am unhappy with this however, already there are easy ways to completely transform the build of a package, and the rules we have are somewhat confusing, one can see this in newcomer questions on StackOverflow.
> 
> For example:
> 
> - Foo
>     - Sources
>         - Bar
>             - main.swift
> 
> Will build a single executable called `Bar`. However if one accidentally creates a new swift file:
> 
> - Foo
>     - Sources
>         - baz.swift
>         - Bar
>             - main.swift
> 
> Then you will get a single executable called baz, or more likely a compile error. Diagnosing this is currently an exercise in frustration (though we could improve this with better diagnostics when compiles fail, eg. we could output the modules structure we inferred).
> 
> To prevent these transformations we could just tighten up our rules.
> 
> If one browses swift packages online there is a clear preference towards a root `Sources` directory.
> 
> Thus I propose we only allow
> 
> A Sources directory with sub folders for modules
> A Sources directory with sources directly in the Sources directory and NO sub folders
> 
> Any other options become errors.
> 
> If we then extend the same rules to modules then there will be no surprising behavior. At first I resisted this idea as folders are for organization, however in the new world of SwiftPM *modules* are for organization.
> 
> Modules have internal accessibility modifiers that allow powerful architectural designs. Modules have a defined public interface that maps onto folder organizational practices better than 
> 
> However this prohibits eg:
> 
> - NetEngine/
>     - CommonCode.swift
>     - HTTP/
>         - foo.swift
>     - HTTPS/
>         - bar.swift
> 
> And perhaps thus is overly restrictive. Though my argument is that HTTP and HTTPS would be better off being their own modules: it would encourage true encapsulation and a better code architecture.
> 
> I am unsure and thus am opening a discussion here. Certainly there are other ways to accomplish the goals here so please let us know what you think a better solution (or not) would be. Thanks for your time.
> 
> Max
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160420/3bc4f1fc/attachment.html>