[swift-build-dev] POSIX, Foundation, and layering

[Daniel Dunbar]

Hi all,

We have started taking some commits to swift-package-manager to incorporate Foundation. The original idea was that we would try and move a lot of our support facilities onto Foundation (https://bugs.swift.org/browse/SR-1005).

However, we have hit a couple issues that have made me want to take a different route:
1. In some cases, our needs (solved by existing code in the POSIX or Utilities modules) don't cleanly match the API Foundation provides. For example, we used the ability to pass arbitrary file descriptors to a subprocess in one place (see https://github.com/apple/swift-package-manager/pull/303) which is trivial with `posix_spawn` but difficult with `NSTask`.
2. In another case, we had to go to substantial effort to workaround the cross-platform bridging of Foundation. See https://github.com/apple/swift-package-manager/pull/333.

Based on this, I am proposing the following direction:

1. We keep our POSIX module. The intent of this module should be to provide "Swifty" APIs to access portable, POSIX facilities. "Swifty" here does not mean completely changing the API to work more conveniently, it just means translating the raw POSIX API into something which is easy to call from Swift and handles errors in a "native" fashion. Almost all APIs in this module should match the underlying POSIX name, and `man <name>` should describe the semantics even if the API is called differently. This module should only depend on `libc` and hopefully one day something like this module can be part of the standard library.

2. We always use `Foundation` when its APIs are the right tool for the job, but if not (because our behavior is in some places fairly low-level, with a neat mapping to POSIX) then we continue to build on the POSIX module. We shouldn't re-implement any significant system provided behavior, we should only be choosing between using a `POSIX` or a `Foundation` API for a task. If the `Foundation` API exists, but isn't implemented yet, then we should work to implement that API in `swift-corelibs-foundation`.

3. We introduce/rename a new `Basic` module out of `Utility`. This matches a convention used in Clang, Swift, and llbuild for the "base support module". This module will define the "convenient" APIs for system services, shared ADTs, etc. and will sit on top of `libc`, `POSIX`, and `Foundation`. We should try to limit cross-platform adaption code to this layer -- if we need a `Foundation` API, but it is not yet implemented in such a way that use of that API is *syntactically* cross-platform (e.g., it needs different bridging behavior) then we should provide an abstraction in `Basic` that hides that divergence. The goal here is that any code above `Basic` can be written without fear of cross-platform incompatibility (which is a serious burden on developer productivity).

4. We should incrementally audit our existing API usage to fit into this conceptual model.

Thoughts?

 - Daniel