<div dir="ltr">After reading the original proposal and the Unicode Annex #31 document that underlies it (https://<span id="m_-7165443368052766486:5p4.1">unicode</span>.org/reports/<wbr>tr31/) I think that the existing work as an underlying layer could help frame the discussion and push it forward. <div><br></div><div>Although I do see the concerns about defining things too strictly in Unicode terms, the proposal brings in some really helpful work without really tethering Swift to Unicode categories now or in the future. If nothing else, throwing out the Unicode framework to start from scratch was always likely to send the discussion into a massive detailed, unstructured discussion of how to handle individual characters, which seems to be what happened.<div><br></div><div><div>Particularly regarding identifiers, taking advantage of the Unicode work on Unicode characters seems to be more promising to make progress than starting from scratch.</div><div><div><br></div><div>My suggestion is to revive the proposal in part, keeping identifier work and leaving operator work at the structural level: </div><div><br></div><div>1) Identifiers: Swift valid identifier characters will be aligned with <span id="m_-7165443368052766486:5p4.2">UAX</span>#31, with exceptions. </div><div>2) Operators: Swift valid operator characters will be defined as an arbitrary list, subject to community discussion within the Unicode structure.</div><div><br></div><div>Identifiers: </div><div><br></div><div>The existing Unicode work in document <span id="m_-7165443368052766486:5p4.3">UAX</span>#31 is entirely about identifiers and like the proposal authors, I think Swift can use this, with exceptions, as the proposal specifies. In particular, the work in <span id="m_-7165443368052766486:5p4.4">UAX</span>#31 for identifiers seems well-thought out and worthy of inclusion, and specifies that individual programming languages can define their syntax *relative* to these defaults, which is probably what Swift wants to do. 1) Swift definitely will have "_" as a valid identifier-head. 2) Swift may need additional rules for using <span id="m_-7165443368052766486:5p4.5">Emoji</span> as identifiers. </div><div><br></div><div>This is described more clearly later in the proposal, under details-identifiers https://<span id="m_-7165443368052766486:5p4.6">github</span>.com/<span id="m_-7165443368052766486:5p4.7">xwu</span>/swift-<wbr>evolution/blob/master/<wbr>proposals/<span id="m_-7165443368052766486:5p4.8">NNNN</span>-refining-<wbr>identifier-and-operator-<span id="m_-7165443368052766486:5p4.9">symbol<wbr>ogy</span>.<span id="m_-7165443368052766486:5p4.10">md</span>#identifiers It also has some guidelines on identifier equivalence that would be worth pulling in. </div><div><br></div><div>Not sure if there is objection to the identifier part of the proposal. It may have been phrased too prescriptively in terms of Unicode, but that doesn't seem to be the intent. </div><div><br></div><div>Operators: </div><div><br></div><div>Unicode doesn't provide any guidance on operators, so what Swift includes is arbitrary regardless. The rule in the proposal was solid but, first, too complex: Default operator characters would be all <span id="m_-7165443368052766486:5p4.11">unicode</span> characters 1) tagged by Unicode as 1a) "Pattern Syntax" characters and 1b) "Mathematical" but 2) excluding characters in the blocks for 2a) "Geometric Shapes", 2b) "Miscellaneous Symbols" and 2c) "Miscellaneous Technical". I can see why this was considered too complex, but as an attempt for a non-arbitrary definition it was a great start. </div><div><br></div><div>So the <span id="m_-7165443368052766486:5p4.12">Swit</span> list of valid <span id="m_-7165443368052766486:5p4.13">operatior</span> characters is arbitrary: embrace it. </div><div><br></div><div>Right now Swift's arbitrary list of Unicode ranges that are valid operator-head or operator-character is unreadable and therefore mostly useless for looking up which characters are valid (<a href="https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/" target="_blank">https://developer.apple.com/<wbr>library/content/documentation/<wbr>Swift/Conceptual/Swift_<wbr>Programming_Language/</a><span id="m_-7165443368052766486:5p4.14">LexicalSt<wbr>ructure</span>.html#//apple_ref/doc/<span id="m_-7165443368052766486:5p4.15">u<wbr>id</span>/TP40014097-CH30-ID418). It's introduced in text as: </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Custom operators can begin with one of the ASCII characters /, =, -, +, !, *, %, <, >, &, |, ^, ?, or ~, or one of the Unicode characters defined in the grammar below (which include characters from the Mathematical Operators, Miscellaneous Symbols, and Dingbats Unicode blocks, among others). After the first character, combining Unicode characters are also allowed.</blockquote><div><br></div><div>This description is more helpful. Discussions of characters should make reference to which code blocks they belong to, and to attempt to align with Unicode groupings or categories when it makes sense to do so. </div><div><br></div><div>The online "<span id="m_-7165443368052766486:5p4.16">UnicodeSet</span>" functionality is very useful for seeing what characters are included (https://<span id="m_-7165443368052766486:5p4.17">unicode</span>.org/<span id="m_-7165443368052766486:5p4.18">cldr</span>/<wbr>utility/list-<span id="m_-7165443368052766486:5p4.19">unicodeset</span>.<span id="m_-7165443368052766486:5p4.20">jsp</span>). For the current Swift 4 Language Reference, I made a list to see what was included, which for this tool required manually expanding some ranges and making a few other inferences, i.e., Swift allows the "dingbats" code block with the exception of "Dingbat circled digits."</div><div><br></div><div>You can see the <span id="m_-7165443368052766486:5p4.21">unicode</span>-set list of existing Swift <a href="http://bit" target="_blank">http://bit</a>.<span id="m_-7165443368052766486:5p4.22">ly</span>/2yYzduM (warning: may not be perfect). </div><div><br></div><div>The <span id="m_-7165443368052766486:5p4.23">UnicodeSet</span> tool doesn't scale very well for discussions. Regardless, it would be very helpful if the final lists were accordingly grouped by Unicode block or subhead, together with at least some descriptions / examples:</div><div><br></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><u><span id="m_-7165443368052766486:5p4.24">Ascii</span></u></div></div><div><div>/ = - + ! * % < > & | ^ ~ ?</div></div><div><br></div><div><div><u>Latin 1 Supplement</u></div></div><div><div>Latin-1 punctuation and symbols: (INVERTED EXCLAMATION MARK) U+00A1–U+00A7, U+00A9, U+00AB, U+00AC, U+00AE, U+00B0–U+00B1, U+00B6, U+00BB (RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK)</div></div><div><div>Punctuation: U+00BF (INVERTED QUESTION MARK)</div></div><div><div>Mathematical operator: U+00D7 (<wbr>MULTIPLICATION SIGN), U+00F7 (DIVISION SIGN)</div></div><div><div><br></div></div><div><div><u>General Punctuation</u></div></div><div><div>General punctuation: U+2016 (DOUBLE VERTICAL LINE)-U+2057 (QUADRUPLE PRIME)</div></div><div><div>Quotation marks: U+2039, U+203A</div></div><div><div>Double punctuation for vertical text: U+203C, U+2047-U+2049</div></div><div><div>Archaic punctuation: U+2056 (THREE DOT PUNCTUATION) - U+205E (VERTICAL FOUR DOTS)</div></div></blockquote><div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div> ...</div></div></blockquote><div><div><br></div><div>The included names of characters are just to help illustrate what kind of marks are in each block, which with the titles helps to stay oriented. If some kind of organization like this made it into Swift documentation, it would be hugely helpful. </div><div><br></div><div>Of course it does raise the question of why certain characters made it in and others didn't--did someone affirmatively decide Ancient Greek Textual Symbols should be valid operators? It appears operators can start General Punctuation, Arrows, <wbr>Miscellaneous Technical, Box Drawing, Dingbats (except Dingbat Circled Digits), Supplemental Punctuation and some <span id="m_-7165443368052766486:5p4.25">CJK</span> Symbols and Punctuation. </div></div><div><br></div><div>Some criteria on how to decide what to include would be helpful -- I would suggest that criteria go into this proposal, and the final list be assembled in the future, perhaps subject to regular (annual?) review. </div><div><br></div><div>And for the mailing list (as long as we're still on it) can we have a tag for discussions of specific operators, something like [operator-[X]-discussion]? Probably many people would like to follow the overall discussion but not most discussions of specific operators. I think the forum-based discussion will help, and my point is mainly that discussion of the criteria or broad scope is different from the discussion of any particular character's inclusion or exclusion. </div></div><div><br></div><div>Does this direction seem like it could preserve flexibility while taking advantage of Unicode standards for identifiers as a starting point for Unicode characters as identifiers, while creating a criteria and structure for future discussion of operators? </div><div><br></div><div>Mike <span id="m_-7165443368052766486:5p4.28">Sand</span></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 30, 2017 at 6:59 PM, Chris Lattner via swift-evolution <span dir="ltr"><<a href="mailto:swift-evolution@swift.org" target="_blank">swift-evolution@swift.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>The core team recently met to discuss PR609 - Refining identifier and operator symbology:</div><div><a href="https://github.com/xwu/swift-evolution/blob/7c2c4df63b1d92a1677461f41bc638f31926c9c3/proposals/NNNN-refining-identifier-and-operator-symbology.md" target="_blank">https://github.com/xwu/swift-<wbr>evolution/blob/<wbr>7c2c4df63b1d92a1677461f41bc638<wbr>f31926c9c3/proposals/NNNN-<wbr>refining-identifier-and-<wbr>operator-symbology.md</a></div><div><br></div><div>The proposal correctly observes that the partitioning of unicode codepoints into identifiers and operators is a mess in some cases. It really is an outright bug for 🙂 to be an identifier, but ☹️ to be an operator. That said, the proposal itself is complicated and is defined in terms of a bunch of unicode classes that may evolve in the “wrong way for Swift” in the future.</div><div><br></div><div>The core team would really like to get this sorted out for Swift 5, and sooner is better than later :-). Because it seems that this is a really hard problem and that <a href="https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good" target="_blank">perfection is becoming the enemy of good</a>, the core team requests the creation of a new proposal with a different approach. The general observation is that there are three kinds of characters: things that are obviously identifiers, things that are obviously math operators, and things that are non-obvious. Things that are non-obvious can be made into invalid code points, and legislated later in follow-up proposals if/when someone cares to argue for them.</div><div><br></div><div><br></div><div>To make progress on this, we suggest a few separable steps:</div><div><br></div><div><div>First, please split out the changes to the ASCII characters (e.g. . and \ operator parsing rules) to its own (small) proposal, since it is unrelated to the unicode changes, and can make progress on that proposal independently.</div></div><div><br></div><div><br></div><div>Second, someone should take a look at the concrete set of unicode identifiers that are accepted by Swift 4 and write a new proposal that splits them into the three groups: those that are clearly identifiers (which become identifiers), those that are clearly operators (which become operators), and those that are unclear or don’t matter (these become invalid code points).</div><div><br></div><div>I suggest that the criteria be based on <b>utility for Swift code</b>, not on the underlying unicode classification. For example, the discussion thread for PR609 mentions that the T character in “ xᵀ ” is defined in unicode as a latin “letter”. Despite that, its use is Swift would clearly be as a postfix operator, so we should classify it as an operator.</div><div><br></div><div>Other suggestions:</div><div> - Math symbols are operators excepting those primarily used as identifiers like “alpha”. If there are any characters that are used for both, this proposal should make them invalid.</div><div> - While there may be useful ranges for some identifiers (e.g. to handle european accented characters), the Emoji range should probably have each codepoint independently judged, and currently unassigned codepoints should not get a meaning defined for them.</div><div> - Unicode “faces”, “people”, “animals” etc are all identifiers.</div><div> - In order to reduce the scope of the proposal, it is a safe default to exclude characters that are unlikely to be used by Swift code today, including Braille, weird currency symbols, or any set of characters that are so broken and useless in Swift 4 that it isn’t worth worrying about.</div><div> - The proposal is likely to turn a large number of code points into rejected characters. In the discussions, some people will be tempted to argue endlessly about individual rejections. To control that, we can require that people point out an example where the character is already in use, or where it has a clear application to a domain that is known today: the discussion needs to be grounded and practical, not theoretical.</div><div><br></div><div><br></div><div>Third, if there is interest sometime in the future, we can have subsequent proposals that expand the range of accepted code points, motivated by the specific application domain that cares about them. These proposals will not be source breaking, so they can happen at any time.</div><div><br></div><div><br></div><div>Is anyone interested in helping to push this effort forward?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-Chris</div><div><br></div></font></span></div><br>______________________________<wbr>_________________<br>
swift-evolution mailing list<br>
<a href="mailto:swift-evolution@swift.org">swift-evolution@swift.org</a><br>
<a href="https://lists.swift.org/mailman/listinfo/swift-evolution" rel="noreferrer" target="_blank">https://lists.swift.org/<wbr>mailman/listinfo/swift-<wbr>evolution</a><br>
<br></blockquote></div><br></div>