[swift-evolution] [discussion] compile-time execution and metaprogramming (was: "constexpr" for Swift 5)

[Gor Gyolchanyan]

As with many other threads, the "constexpr" one changed into a fixed-size array thread 😆, so instead of fighting it, let's branch off into another topic that also almost took over.

The bottom line of this discussion is to weight pros and cons of a compile-time execution feature and a metaprogramming feature built on top of it.

The way I imagine it to work is to have a new keyword `compiletime` (or something similar) that can be applied to any executable block (property accessor, property observer, function, closure), which would restrict the use of runtime-only features of swift (the exact list of which is yet to be defined) and make that block of code eligible for guaranteed compiletime execution. Being eligible for compile-time execution essentially means having the return value (and values of mutated state) pass as a literal in the eyes of the compiler. Obviously, restrictions of the subset of Swift (which would most definitely include access to source code) in the compile-time context are transitive: the entire call hierarchy has to adhere to the restrictions with no exceptions. Also, obviously, in order to eligible code blocks to execute, all their captured context and parameters have to also be compiletime. Compiletime variables are mutable state during the compilation, which are demoted to compile-time constants after the compilation.

According to John McCall, the beginnings of this already exist in the form of a simple expression folding that could be extended to encompass more types of code to be executed at compile-time.

The metaprogramming part is all about generating language constructs programmatically. Currently, that niche is occupied by text-based boilerplate generators like Gyb, Sourcery and the like.
With the compile-time execution available, boilerplate could be remove from the language entirely due to an unprecedented ability to modify the fully parsed, structurized binary representation of the code form inside the code.
A set of built-in types and functions could be used to generate instances of TypeAliasDeclaration, StructureDefinition, VariableDeclaration and such. These would then be fed into a compile-time in-language representation of the compiler to cause it to insert them into the current module for use by other code. This would work very similarly to LLVM's instruction builder class.

According to David Sweeris, getting rid of compiler magic is one of Swift's goals and you can see how this will be single most powerful way of reducing the compiler to its bare minimum and moving most of the runtime into the standard library.

My main point of why I'd like to start an in-depth discussion about this now is because this is a huge endeavor with tremendous consequences, which need time to think through and plan accordingly. Swift 4 is about to release in a few months and Swift 5's stages and priorities will be on the table. I'd like to see where this can go *before* we miss the window of opportunity on such a proposal and are stuck waiting for another release cycle.

I'd like to hear as many opinions as possible (especially from the core team, which is ultimately going to have to decide whether or not this is ever going to happen).