[swift-evolution] Looking back on Swift 3 and ahead to Swift 4

[Chris Lattner]

Hello Everyone,

The Swift 3 release is nearing completion, so it is time to look back on the release, learn from what happened, and use it to shape what we (the Swift community) do in the year ahead.  Overall, Swift 3 is going to be an absolutely *amazing* release, and it is impressive how much got done.  Thank you to everyone who contributed to making it happen.  Instead of diving into a flurry of new proposals immediately, it is important to take stock of where we are, and look at the bigger picture.

Metapoint: this email is ridiculously long and covers multiple topics.  Instead of replying to it directly, it is best to start new threads on individual topics that you’d like to discuss.  Just tag them with “[Swift 4]” in the subject line.


### Swift 3 Retrospective ###

Every year of Swift’s development has been completely different from the previous one, and I expect Swift 4 to continue this trend.  With a goal of learning and improving year over year, here are some observations & retrospective about Swift 3:

 - Open source is pretty great.  It has been incredible to see such a vibrant community working so well together, and to see you come together practically overnight.  It is really fantastic to work with such a talented and enthusiastic group of people!

 - Open source also brings challenges.  I think it is fair to say that "open design" is slower and less predictable than "closed design”.  However, the end result is significantly better, and therefore the tradeoff is worth it.  A big “Thank you!” goes out to everyone who helps improve Swift through the evolution process.

 - Software scheduling (particularly with open source) continues to be difficult-to-impossible to predict.  We set off on Swift 3 with a lofty set of goals that had to be trimmed back later.  It is *good* to have high goals, but we need to do a better job of communicating that “goals” are not “promises” so that people don’t feel misled.

 - The community benefits from keeping focus on a limited number of topics, because if there is too much going on, no one can follow and keep track of it all.  It is important for the core team to be involved in key discussions up front.  In the Swift 3 cycle, it was problematic that many folks had no time to follow the threads until after the review period completed.

 - Having clear goals is liberating.  In the December/January timeframe in particular, we loosely scoped what sorts of ideas would fit into Swift 3, and embarked on several projects that ended up being far beyond what we could handle.  Later in the release, we had very specific goals (for example, “no additive proposals”) which made it much easier for everyone to focus on what was important.

 - It is impossible to make everyone happy, particularly when discussing which features to pick up and prioritize - because that implicitly de-prioritizes something else. This is inevitable though, because there is no way that all the interesting work will fit into a one-year release cycle.  Fortunately, there will “always be another release,” and each new release will have its share of big improvements.

With that as context, let’s look ahead!


### Swift Release Plan ###

Over the next year, the core team expects to ship two major releases of Swift: Swift 3.x in Spring 2017 and Swift 4 in Fall 2017.   In addition to the major releases, we will surely ship some minor releases (e.g. Swift 3.0.1) to fix bugs or service the needs of the corelibs or other swift.org <http://swift.org/> projects.


### Swift 4 Release Cycle Planning Approach ###

From our experience with Swift 3, we know we need to pick and choose what we’re going to tackle.  For Swift 4, the primary goals are to deliver on the promise of source stability from 3.0 on, and to provide ABI stability for the standard library.  As such, the core team has decided to take a two-stage approach to the next year:

Stage #1: Focus on the essentials required for Source and ABI stability, and keep reasonably strict focus on only that work.  This means that any features that don’t fundamentally change the ABI of existing language features or imply an ABI-breaking change to the standard library will not be considered in this stage. For example, generics features like conditional conformance <https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#conditional-conformances-> is an additive feature, but since it is expected to reshape much of the standard library it would be part of Stage 1.  On the other hand, language support for regular expressions would not affect existing ABI nor cause major changes to existing standard library features, so it would not fit in Stage 1.

The work entailed by Stage 1 is non-trivial (explored in somewhat more detail below), and will probably keep us busy until the Spring.

Stage #2: As the design and implementation work on the Stage 1 features is cresting, we’ll scope and plan a few other large features based on how much time we have left.  I’m optimistic that we’ll have time to pick up some of the long list of features explored below, but we won’t know which ones those are until we know how much time we have to develop them with.


In addition to new features, we also need to reevaluate accepted source breaking proposals that didn’t make it into Swift 3.  These proposals will not necessarily be grandfathered in - we will need to evaluate them against the goals of Swift 4 and decide what to do with each of them on a case by case basis.

Finally, while it isn’t specifically related to swift-evolution, I want to call out quality & performance work.  The core team wants to continue to improve quality, including fixing compiler bugs and improving error and warning diagnostics.  Performance is also an ongoing area of critical development, including improving the performance of the generated code, improving the standard library implementation, speeding up compile time, etc.  All of this work can happen in either stage.


### Swift 4 Stage 1 Goals ###

With a focus on source and ABI stability, the core team had an initial discussion about what makes up the Stage 1 effort.  Here are the features that we are prioritizing for Stage 1:

- Source stability features: These should be relatively small, but important.  For example, we need a “-std=swift3” sort of compiler flag.  We may also add a way to conditionally enable larger efforts that are under development but not yet stable - in order to make it easier to experiment with them.

- Resilience: This provides a way for public APIs to evolve over time, even in the face of ABI stability.  For example, we don’t want the C++ “fragile base class" problem to ever exist in Swift.  Much of the design and implementation work was done in the Swift 3 timeframe, but there are still major missing pieces, including the user-visible part of the model (e.g. new attributes).

 - ABI Details: There are a ton of small details that should be audited and improved in the code generation model.  This is mostly relevant to swift-dev, and not specifically a swift-evolution topic.

 - Generics improvements needed by the standard library: I expect conditional conformances <https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#conditional-conformances-> to be the very top of this list, and for recursive protocol requirements <https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#recursive-protocol-constraints-> and more powerful associated type constraints to be close followers.  However, the standard library gurus need to break down what is absolutely essential to finally eliminate the rest of the “_” protocols and manifest the public API of the standard library in the right way for the long term.

 - String re-evaluation: String is one of the most important fundamental types in the language.  The standard library leads have numerous ideas of how to improve the programming model for it, without jeopardizing the goals of providing a unicode-correct-by-default model.  Our goal is to be better at string processing than Perl!

 - Memory ownership model: Adding an (opt-in) Cyclone/Rust inspired memory ownership model to Swift is highly desired by systems programmers and folks who want predictable and deterministic performance (for example, in real time audio processing code).  More pertinent to the goals of Swift 4, this feature is important because it fundamentally shapes the ABI.  It informs code generation for “inout", how low-level “addressors” work in the ABI, impacts the Swift runtime, and will have a significant impact on the type system and name mangling.

Each of these areas has had some thought put into them already, but are still a long way from being formal proposals.  I expect and hope that they will turn into major discussions early in the Swift 4 cycle.  Further, since we haven’t fully scoped what impacts ABI stability, there may be other specific additions as we learn more.  Finally, it is also possible that we may choose to scope in specific small features that are high value to unblock SwiftPM or other swift.org <http://swift.org/> projects.


### Possible Swift 4 Stage 2 Efforts ###

As I mentioned above, at this point it is impossible to know what can be achieved in the Stage 2 timeframe, because we don’t know how long that timeframe is.  The core team would also like to converge Swift 4 development earlier in its cycle than Swift 3 did, in order to fix more bugs late in the release and provide a longer bake time.

That said, I’m optimistic that we’ll be able to pick up and tackle some of the commonly requested new features.  To give you an idea of some of them, here is a list.  Please note that this is not a plan or commitment, it is just a laundry list of commonly requested features:

- Reflection: The core team is committed to adding powerful dynamic features to Swift.  For example, Swift 3 already added nearly all the infrastructure for data reflection (which is already used by the Xcode memory debugger).  We should use this infrastructure to build out a powerful user-facing API.  Similarly, we would like to design and build out the implementation for dynamic method reflection runtime + API support.

- First class concurrency: Actors, async/await, atomicity, memory model, and related topics.  This area is highly desired by everyone, as it will open the door for all sorts of new things on the client, server and more.  We plan to start formal *discussions* about this in Phase 2, but it is unfortunately crystal clear that a new concurrency model won’t be done in time for the Swift 4 release.  This is simply because it will take more than a 12 months to design and build, and we want to make sure to take time to do it right.  It also makes sense for the memory ownership model to be better understood before taking this on.

- Generics improvements: The generics manifesto <https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md> includes many exciting enhancements to the generics system, many of which will not be specifically required for ABI stability of the standard library, but would make Swift generics more powerful and expressive.

- .swiftmodule stability: At some point we need to stabilize the “.swiftmodule” binary file format (or replace it with a different mechanism) to allow 3rd party binary frameworks.  This is a very large amount of work over and above what is required for ABI stability of the standard library.

- New scripting features: Regular expressions, multi-line string literals, etc.  Having these would make Swift much more appealing to the crowd doing scripting and those building web technologies, among others.  They’d also help round out the String model.

- Property behaviors: This feature promises to provide powerful abstractions over the existing property model.  The deferred SE-0030 <https://github.com/apple/swift-evolution/blob/master/proposals/0030-property-behavior-decls.md> proposal describes this opportunity well.

- So many others: Submodules, implicit promotions between numeric types, importing C++ APIs, hygenic macro system, guaranteed tail calls, making enums enumerable, typed ‘throws’, user defined attributes, abstract methods/classes, better SIMD support, ‘dynamic’ for non- at objc, data parallelism support, higher kinded types, …

- Syntactic sugar: I won’t list them all, but there are a ton of other trivial to small proposals that frequently come up, typically things seen in other languages that solve specific problems.  These are the lowest priority to tackle for Swift 4.



So there you go, an overly long email laying out some thoughts and ideas of what to do over the next year.  One specific thing to keep in mind is that Swift 3 isn’t done yet.  While source breaking changes are (nearly) done, there is still time for bug fixing and other quality work, and that is very important to the release.

I think it would be helpful for us to spend the immediate future discussing the general parameters of how we tackle the releases over the next year, then iron out the Stage 1 features conceptually.  We should only start writing proposals when the specific design of them are well understood.  The core team does *not* want to be in a situation where there are so many proposals floating around that we can’t keep track of them, or that they prevent tackling the large and important high priority projects in front of us.

Thank you.  Again, please start new threads if you’d like to dive in more deeply into specific topics!

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160729/3b4efd36/attachment.html>