[swift-evolution] [Proposal] Refining Identifier and Operator Symbology

[Chris Lattner]

> On Oct 21, 2016, at 9:38 PM, Jonathan S. Shapiro via swift-evolution <swift-evolution at swift.org> wrote:
> 
> Well, it seems that I jumped the gun and sent my document link to swift-evolution by mistake. Since I can't take it back, it might be good to say what it's about. Because of my mistake, this version has not had any review by the rest of the author group, which probably would have improved it a lot. I had intended one more round of edits to deal with a few things that still say "FIX" and a few minor cleanups, but I think the substance of the proposal is sound.
> 
> This revised proposal <https://github.com/jsshapiro/swift-evolution/blob/unicode-id-op/proposals/NNNN-refining-identifiers-and-operators.md> tries to take into account most of the feedback we have received here. Lots of nitty-gritty changes, but here's the big picture of the changes:

This is looking like great progress over the original proposal and (to restate the obvious) I’m a huge fan of your (and Jacob, Erica, and Xiaodi’s) work on this proposal.  Thank you ALL for driving this forward, it is a very messy but critical task to get right for Swift 4.

My biggest concern is the Operator Definition section (https://github.com/jsshapiro/swift-evolution/blob/unicode-id-op/proposals/NNNN-refining-identifiers-and-operators.md#operator-definition)

The issue is that this breaks the ability to separately parse files.  The C family of languages generally requires the parser to reason about a “translation unit”, and used a technique colloquially known as the “lexer hack” to determine whether an identifier token refers to a type or a value definition.  Once the lexer determines this, the parser behaves differently.  Your proposal introduces an analogous “lexer hack”, for the purposes of determining whether an token is an identifier or an operator.

This is deeply concerning to me, because this would break the ability to parse a file without parsing all of its dependencies. Our current support for this has some well known problems (e.g. you can’t resolve expressions to a particularly useful representation: you get SequenceExprs unless you can perform name lookup) but we do have the ability parse *declarations* with high fidelity.  

This ability is extremely important for us to be able to get incremental compilation of an individual source file (which typically depends on declarations from other source files, but does not depend on the contents of their bodies) and is important for various latency-critical IDE features.  For example, if you perform a code completion within a function, you generally only need to fully type check the function body that you’re inside, everything else in the file can be ignored, and only referenced declarations need to be (recursively) checked.


I think you have a good set of goals and motivations, and I agree that aligning with UAX31’s ultimate resolution is important.  That said, I hope that we can tackle this in Swift 4 by taking a small but reasonable subset of what UAX31 is “certainly” going to support, and then wait for UAX31 to be finalized before expanding the rest out.

I’m not a unicode expert by any stretch of the imagination, but would it be possible to carve off some obvious blocks of emoji support as identifiers (e.g. not symbols, flags, or anything else complicated), and carve off the most obvious blocks of the math operators as operators?  For the operator set, maybe we could start with some small subset of 100 (totally random number here) operators that are commonly requested and seem obvious, then expand it out to a principled set once UAX31 is resolved?  This would avoid regressing too much in Swift 4 from Swift 3, but also dramatically limit the risk of painting ourselves into a corner that turns out to be incompatible with UAX31.

-Chris


> Emojis are admitted, subject to reasonable sanity conditions.
> A (significantly) broader, but still conservative set of code points are admitted for symbol identifiers. Hopefully this addresses current need, but I remain open to adopting full-on [:Sm:] and [:So:] if there is a strong push for that.
> Operator definition is orthogonal to identifier specification, which deals with the noun/verb confusion and also addresses the widely-expressed feeling that some symbols aren't operators and their conventional meaning should be usable. The term "operator" no longer has anything to do with identifiers.
> A laundry list of potential parsing gotchas are addressed. The previous proposal would have broken the generics syntax and also the binding syntax. This isn't a substantive conceptual change, but it's important if the proposal is going to, you know, actually work. :-)
> Dollar is admitted in identifiers.
> Explicitly addresses anonymous closure parameters in a way that reflects how the compiler actually needs to deal with such things. Might be I've written a compiler or two in my career. :-)
> Consistent with the current direction of UAX31 on these issues.
> Susan Kare's legacy is preserved. :-) If you don't know who Susan is, look her up and learn why Chris loves the dogcow emoji pair.
> The new proposal remains entirely compatible with Swift 3, except where existing source runs up against the narrower symbol identifier space. It's a specific goal to avoid breaking reasonable current practice where possible, though we're surely going to break something with this one.
> 
> I was trained to write specifications in a school that favored rigorous writing. In order to make sure I didn't lose track of something I rewrote the proposal in a form that I know how to use effectively. Any loss of "fun" in the text is my fault alone.
> 
> Interested to see how this will be received.
> 
> 
> Jonathan
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20161024/5e9c1bfc/attachment.html>