<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 1, 2017, at 1:56 PM, David Hart <<a href="mailto:david@hartbit.com" class="">david@hartbit.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="content-type" content="text/html; charset=utf-8" class=""><div dir="auto" class=""><div class="">Wonderful comments. I really enjoy your take on submodules which keeps most of the power while keeping the simplicity. Comments below:</div><div class=""><br class=""><br class="">Sent from my iPhone</div><div class=""><br class="">On 1 Mar 2017, at 07:55, Brent Royal-Gordon via swift-evolution <<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class=""><blockquote type="cite" class=""><span class="">On Feb 24, 2017, at 11:34 AM, Matthew Johnson via swift-evolution <<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a>> wrote:</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">Scope-based submodules</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Proposal: SE-NNNN</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Authors: Matthew Johnson</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Review Manager: TBD</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Status: Awaiting review</span><br class=""></blockquote><span class=""></span><br class=""><span class="">Well, this is certainly comprehensive! Sorry about the delay in answering; I've been hosting a house guest and haven't had a lot of free time.</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it. </span><br class=""></blockquote><span class=""></span><br class=""><span class="">I agree with this as the primary goal of a submodule system.</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">Some other goals of this proposal are:</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Submodules should help us to manage and understand the internal dependencies of a large, complex system.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • A module should not be required to expose its internal submodule structure to users when symbols are exported.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">One goal I don't see mentioned here is "segment the API surface exposed to importing code". The `UIGestureRecognizerSubclass` use case has been thoroughly discussed, but I think there are probably a lot of cases where there are two "sides" to an API and it'd often be helpful to hide one unless it's needed. `URLProtocol` and `URLProtocolClient` come to mind; the many weird little classes and symbols related to `NSAtomicStore` and `NSIncrementalStore` might be another.</span><br class=""><span class=""></span><br class=""><span class="">(I'm not necessarily suggesting that the Foundation and Core Data overlays should move these into submodules—I'm suggesting that, if they were implemented in a Swift with a submodule feature, they would be candidates for submodule encapsulation.)</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">Submodule names form a hierarchical path:</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • In this example, InnerSubmodule is a child of Submodule.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • A submodule may not have the same name as any of its ancestors. This follows the rule used by types.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">Does being in a nested submodule have any semantic effect, or is it just a naming trick?</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">Submodules may not be extended. They form strictly nested scopes.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • The only way to place code in a submodule is with a submodule declaration at the top of a file.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • All code in a file exists in a single submodule.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">I'm a big supporter of the 1-to-N submodule-to-file approach.</span><br class=""></div></blockquote><div class=""><br class=""></div><div class="">Agreed.</div><br class=""><blockquote type="cite" class=""><div class=""><blockquote type="cite" class=""><span class="">There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file. </span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">The alternatives are:</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Use a manifest file. This would be painful to maintain.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">I'm going to push back on this a little. I don't like the top-of-file `submodule` declaration for several reasons:</span><br class=""><span class=""></span><br class=""><span class=""> 1. Declarations in a Swift file are almost always order-independent. There certainly aren't any that must be the first thing in the file to be valid.</span><br class=""><span class=""></span><br class=""><span class=""> 2. Swift usually keeps configuration stuff out of source files so you can copy and paste snippets of code or whole files around with minimum fuss. Putting `submodule` declarations in files means that developers would need to open and modify those files if they wanted to copy them to a different project. (It's worth noting that your own goal of making it easy to extract submodules into separate modules is undermined by submodule declarations inside files.)</span><br class=""><span class=""></span><br class=""><span class=""> 3. However you're organizing your source code—whether in the file system, an IDE project, or whatever else—it's very likely that you will end up organizing files by submodule. That means either information about submodules will have to be specified twice—once in a canonical declaration and again in source file organization—and kept in sync, or IDEs and tooling will have to interpret the `submodule` declarations in source files and reflect that information in their UIs.</span><br class=""><span class=""></span><br class=""><span class=""> 4. Your cited reason for rejecting build system-based approaches is that "This makes it more difficult for a module to support multiple build systems", but Swift has this same problem in *many* other parts of its design. For instance, module names and dependencies are build system concerns, despite the fact that this makes it harder to support multiple build systems. I can only conclude that supporting multiple build systems with a single code base is, in the long term, a non-goal, presumably by improving the Xcode/SwiftPM story in some way.</span><br class=""><span class=""></span><br class=""><span class="">I'm still a fan of build-system-based approaches because I think they're better about these issues. The only way that they're worse is that—as you note—it may not be clear which submodule a particular file is in. But I think this is basically a UI problem for editors.</span><br class=""></div></blockquote><div class=""><br class=""></div><div class="">I'm also a big fan of the build-system or file-system approach, for exactly the same reasons. The copy/pasting argument is important IMHO. For example, the suggestion which has been discussed a lot on here to make <b class="">private</b> change to submodule scope when inside a submodule would have the same negative effect on the ability to copy paste code between a submodule and top-module.</div><br class=""><blockquote type="cite" class=""><div class=""><blockquote type="cite" class=""><span class="">Top-level export</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">All export statements consist of an access modifier, the export keyword, and a submodule name:</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">open export ChildSubmodule</span><br class=""></blockquote><span class=""></span><br class=""><span class="">Is the access control keyword mandatory?</span><br class=""><span class=""></span><br class=""><span class="">If we do indeed use `submodule` statements, could we attach the attributes to them, rather than having a separate statement in a different file?</span><br class=""><span class=""></span><br class=""><span class="">What does it mean if a `public` or `open` symbol is in a submodule which is not `export`ed?</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class=""> • A submodule may be published under a different external name using the export as NewName syntax*.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">What's the use case for this feature?</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class=""> • @implicit causes symbols from the submodule to be implicitly imported when the module is imported.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • @inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">So if you write `@implicit public export Bar` in module `Foo`, then writing `import Foo` also imports `Foo.Bar.Baz` *as* `Foo.Bar.Baz`, whereas `@inline public export Bar` copies `Foo.Bar.Baz` into `Foo`, so it imports as `Foo.Baz`?</span><br class=""><span class=""></span><br class=""><span class="">What's the use case for supporting both of these behaviors?</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">Exports within the module</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:</span><br class=""></blockquote><span class=""></span><br class=""><span class="">I'm not sure how valuable this feature is in this kind of submodule design.</span><br class=""><span class=""></span><br class=""><span class="">***</span><br class=""><span class=""></span><br class=""><span class="">To avoid being coy, here's the export control model that *I* think would make the most sense for this general class of submodule system designs:</span><br class=""><span class=""></span><br class=""><span class="">1. A submodule with `public` or `open` symbols is importable from outside the module. There is no need to separately mark the submodule as importable.</span><br class=""></div></blockquote><div class=""><br class=""></div><div class="">I had not thought of that but it's very elegant.</div><br class=""><blockquote type="cite" class=""><div class=""><span class="">2. Normally, references within the module to submodule symbols need to be prefixed with the submodule name. (That is, in top-level `Foo` code, you need to write `Bar.Baz` to access `Foo.Bar.Baz`). As a convenience, you can import a submodule, which makes the submodule's symbols available to that file as though they were top-level module symbols.</span><br class=""><span class=""></span><br class=""><span class="">3. When you import a submodule, you can mark it with `@exported`; this indicates that the symbols in that submodule should be aliased and, if `public` or `open`, re-exported to other modules.</span><br class=""></div></blockquote><div class=""><br class=""></div><div class="">Could you explain this in more detail?</div><br class=""><blockquote type="cite" class=""><div class=""><span class="">4. There are no special facilities for renaming submodules or implicitly importing submodules.</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">Importing submodules</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">Submodules are imported in exactly the same way as an external module by using an import statement.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">Okay, but what exactly does importing *do*? Set up un-prefixed private aliases for the submodule's internal-and-up APIs?</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">There are a few additional details that are not applicable for external modules:</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class=""> • Circular imports are not allowed.</span><br class=""></blockquote><span class=""></span><br class=""><span class="">Why not? In this design, all submodules are evaluated at once, so I'm not sure why circular imports would be a problem.</span><br class=""><span class=""></span><br class=""><blockquote type="cite" class=""><span class="">// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)</span><br class=""></blockquote><blockquote type="cite" class=""><span class="">// This reads: `Child1` is scoped to `Grandparent`.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">scoped(Grandparent) export Child1</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">// `Child2` is visible throughout the module but may not be exported for use by clients.</span><br class=""></blockquote><blockquote type="cite" class=""><span class="">// This reads: `Child2` is scoped to the module.</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">scoped(module) export Child2</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">With parameterization, scoped has the power to specify all access levels that Swift has today:</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">`scoped` == `private` (Swift 3)</span><br class=""></blockquote><blockquote type="cite" class=""><span class="">`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)</span><br class=""></blockquote><blockquote type="cite" class=""><span class="">`scoped(submodule)` == `internal`</span><br class=""></blockquote><blockquote type="cite" class=""><span class="">`scoped(public) scoped(internal, inherit)`* == `public`</span><br class=""></blockquote><blockquote type="cite" class=""><span class="">`scoped(public)` == `open`</span><br class=""></blockquote><blockquote type="cite" class=""><span class=""></span><br class=""></blockquote><blockquote type="cite" class=""><span class="">The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).</span><br class=""></blockquote><span class=""></span><br class=""><span class="">What is the purpose of creating more verbose aliases for existing access levels? I can't think of one, which means that these are redundant.</span><br class=""><span class=""></span><br class=""><span class="">And if we remove them as redundant, the remaining access control levels look like:</span><br class=""><span class=""></span><br class=""><span class=""> scoped</span><br class=""><span class=""> private</span><br class=""><span class=""> scoped(TypeName)</span><br class=""><span class=""> internal</span><br class=""><span class=""> scoped(SomeModule)</span><br class=""><span class=""> scoped(module)</span><br class=""><span class=""> scoped(extension)</span><br class=""><span class=""> public</span><br class=""><span class=""> open</span><br class=""><span class=""></span><br class=""><span class="">There's just no logic to the use of the `scoped` keyword here—it doesn't really mean anything other than "we didn't want to assign a keyword to this access level".</span><br class=""><span class=""></span><br class=""><span class="">***</span><br class=""><span class=""></span><br class=""><span class="">I think we need to go back to first principles here. The reason to introduce a new access level is that we believe that a submodule is a large enough unit of code that it will simultaneously need to encapsulate some of its implementation details from other submodules, *and* have some of its own implementation details encapsulated from the rest of the submodule.</span></div></blockquote><div class=""><br class=""></div><div class="">That's where I disagree. Why would Swift need that level of differentiation and not C# or Java, which have had packages and namespaces without it.</div></div></div></blockquote><div><br class=""></div><div>Need is a relative thing. There are many languages that succeed without many of the features Swift has and many languages that succeed partly because of features Swift does not have. </div><div><br class=""></div><div>Learning from other languages is a good thing, but in the end the goal is to make Swift the best language it can be. In many cases that happens when we carefully consider our options and find a solution that in some cases is unique. </div><div><br class=""></div><div>I am happy that Swift doesn’t follow the example of C# and Java too closely. There are things we can learn from them but also many things we *don’t* want to learn from them. Introducing namespaces without scope boundaries would be a mistake IMO.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="auto" class=""><br class=""><blockquote type="cite" class=""><div class=""><span class="">Thus, we need at least three access levels within a submodule: one that exposes an API to other submodules, one that exposes an API throughout a submodule, and one that exposes it to only part of a submodule.</span><br class=""><span class=""></span><br class=""><span class="">What we do *not* need is a way to allow access only from certain other named submodules. The goal is to separate external and internal interfaces, not to micromanage who can access what.</span><br class=""><span class=""></span><br class=""><span class="">Basically, that means we need one of two things. Keeping all existing keywords the same—i.e., not removing either `private` or `fileprivate`— and using `semi-` as a placeholder, we want to either have:</span><br class=""><span class=""></span><br class=""><span class=""> private: surrounding scope</span><br class=""><span class=""> fileprivate: surrounding file</span><br class=""><span class=""> semi-internal: surrounding submodule</span><br class=""><span class=""> internal: surrounding module</span><br class=""><span class=""> public: all modules (no subclassing)</span><br class=""><span class=""> open: all modules (with subclassing)</span><br class=""><span class=""></span><br class=""><span class="">Or:</span><br class=""><span class=""></span><br class=""><span class=""> private: surrounding scope</span><br class=""><span class=""> fileprivate: surrounding file</span><br class=""><span class=""> internal: surrounding submodule</span><br class=""><span class=""> semi-public: surrounding module</span><br class=""><span class=""> public: all modules (no subclassing)</span><br class=""><span class=""> open: all modules (with subclassing)</span><br class=""><span class=""></span><br class=""><span class="">The difference between the two is that, with `semi-internal` below `internal`, submodule APIs are exposed by default to other submodules; with `semi-public` above `internal`, submodule APIs are encapsulated by default from other submodules.</span><br class=""><span class=""></span><br class=""><span class="">I think encapsulating by default is the right decision, so we want the `semi-public` design. But there's also a second reason to use that design: We can anticipate another use case for it. The library resilience design document discusses the idea of "resilience domains"—groups of libraries whose versions are always matched, and which therefore don't need to use resilient representations of each others' data structures—and the idea of having "SPIs", basically APIs that are only public to certain clients. I think these ideas could be conflated, so that a semi-public API would be available both to other submodules in the module and to other libraries in your resilience domain, and that this feature could be used to expose SPIs.</span><br class=""><span class=""></span><br class=""><span class="">So, that leaves an important question: what the hell do you call this thing? My best suggestions are `confidential` and `privileged`; in the context of information, these are both used to describe information which *is* shared, but only within a select group. (Think, for instance, of attorney-client privilege: You can share this information with your lawyer, but not with anyone else.)</span><br class=""><span class=""></span><br class=""><span class="">So in short, I suggest adding a single access level to the existing system:</span><br class=""><span class=""></span><br class=""><span class=""> private</span><br class=""><span class=""> fileprivate</span><br class=""><span class=""> internal</span><br class=""><span class=""> confidential/privileged</span><br class=""><span class=""> public</span><br class=""><span class=""> open</span><br class=""><span class=""></span><br class=""><span class="">This is orthogonal to any other simplification of the access control system, like removing `private` or `fileprivate`.</span><br class=""></div></blockquote><div class=""><br class=""></div><div class="">Yes, I would still like to see scope private go away.</div><br class=""><blockquote type="cite" class=""><div class=""><blockquote type="cite" class=""><span class="">Appendix A: file system independence</span><br class=""></blockquote><span class=""></span><br class=""><span class="">I think we need to decide: Is a translation unit of some sort—whether it's a physical on-disk file or some simulacrum like a database record or just a separate string—something intrinsic to Swift? I think it should be; it simplifies a lot of parts of the language that would otherwise require nesting and explicit scoping.</span><br class=""><span class=""></span><br class=""><span class="">If translation units are an implicit part of Swift, then this section is not really necessary. If translation units aren't, then we need to rethink a lot of things that are already built in.</span><br class=""><span class=""></span><br class=""><span class="">-- </span><br class=""><span class="">Brent Royal-Gordon</span><br class=""><span class="">Architechies</span><br class=""><span class=""></span><br class=""><span class="">_______________________________________________</span><br class=""><span class="">swift-evolution mailing list</span><br class=""><span class=""><a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a></span><br class=""><span class=""><a href="https://lists.swift.org/mailman/listinfo/swift-evolution" class="">https://lists.swift.org/mailman/listinfo/swift-evolution</a></span><br class=""></div></blockquote></div></div></blockquote></div><br class=""></body></html>