<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div><blockquote type="cite" class=""><div class="">On Jan 13, 2018, at 10:30 AM, Michael Ilseman via swift-dev <<a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a>> wrote:</div><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><div class=""><div class=""> I wouldn’t overly rely on it for guidance on these issues give that it it stuck so squarely in the realm of UTF16.</div><div class=""><br class=""></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Wading a little into the weeds here, CharacterSet’s builtins model older Unicode semantics than what we probably want to provide. E.g. CharacterSet.lowercaseLetters means general category “Ll”, while modern Unicode defines property Lowercase which includes more scalars. The Lowercase property vs CharacterSet.lowercaseLetters is equivalent to ICU’s u_isULowercase [1] vs u_islower [2], where the former is the modern recommended API. Contrarily, Perl 6’s <lower> built-in rule[3] seems to also just be “Ll” and I’m not familiar with the rationale there (compatibility?). Certainly needs investigation.</div></div></div></div></blockquote><div><br class=""></div><div>Makes sense. It would be great to standardize towards the grapheme cluster as the canonical concept of character (this is why Character is defined that way) and by pushing regex’s to use that definition of character, we can simplify and reduce aggregate complexity. This is a definition that “just works” most often, and the introduction of regex’s will eliminate all of the awkardness of having to use them - by eliminating some of the most common reasons that people want to use integer indexes into strings.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class="">I supposed by “guidance” I meant “here are some of the things people commonly ask about characters”. Built-in character classes in various regex flavors also provide guidance.</div><div class=""><br class=""></div><div class="">[1] u_isULowercase: <a href="http://icu-project.org/apiref/icu4c/uchar_8h.html#a8321c9ba617ed00787f20c4b23a254bc" class="">http://icu-project.org/apiref/icu4c/uchar_8h.html#a8321c9ba617ed00787f20c4b23a254bc</a></div><div class="">[2] u_islower: <a href="http://icu-project.org/apiref/icu4c/uchar_8h.html#aa26a51b768147fbf34029f8141132815" class="">http://icu-project.org/apiref/icu4c/uchar_8h.html#aa26a51b768147fbf34029f8141132815</a></div><div class="">[3] <a href="https://docs.perl6.org/language/regexes#Backslashed,_predefined_character_classes" class="">https://docs.perl6.org/language/regexes#Backslashed,_predefined_character_classes</a></div></div></div></div></blockquote><div><br class=""></div><div>One of the frustrating things about the regex state of the art is that most engines were built pre-modern-unicode-complexity. As you say, it is surprising that Perl6 didn’t fix some of this, but it could either be because of compatibility or (more likely) that they made many of these decisions 15 years ago when Perl 6 was just getting going. Perl 6 had a long gestation period.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Note: In no way would these properties obviate the need for CharacterSet, as CharacterSet API is independently useful and the right model for many whole-string operations.<br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">No it isn’t. A Grapheme-cluster based analog of CharacterSet would be a reasonable model to consider though. It could conceptually support predicates like isEmoji()</div><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">In the regex section I talk about character classes as being modeled by `(Character) -> Bool` rather than something that conforms to SetAlgebra. One big reason (beyond being awesomely convenient) is because I don’t think graphemes are something to be enumerated.</div></div></div></div></blockquote><div><br class=""></div><div>+1</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class="">Theoretically speaking, I’m not sure if the set of graphemes is even recursively enumerable (e.g. zalgo-text or zalgo-emoji). Practically speaking, it darn-well might as well be uncountable. Best case, if even possible, any enumeration would be emergent behavior of encoding details and change significantly with each Unicode version. If we cannot enumerate graphemes, we cannot answer equality, isDisjoint, isSubset, etc.</div></div></div></div></blockquote><div><br class=""></div>+2</div><div><div><br class=""></div></div><div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class="">I don’t know if those operations are important for a grapheme analogue. They just demonstrate that a grapheme set follows a different model than a scalar set, and different than what we have traditionally used the word “Set” for in Swift, though you would know the history here better.</div></div></div></div></blockquote><div><br class=""></div><div>+3. I completely agree that a “set” in the computer sciency usage is no longer a useful notion any longer for characters. It is still technically correct with the mathematical definition, but any alignment of terminology towards this will just be confusing to users.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class="">As far as uses of such a grapheme set are concerned, they seem equivalent to application of a regex consisting of a character class. For example, rangeOfCharacter(from:options:) could be accomplished with the below. </div></div></div></div></blockquote><div><br class=""></div><div>Yes, that’s a great way to consider it. </div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class="">In an ever-futile attempt to avoid focusing on specific syntax, I’ll form an army of straw-people, using 「」 delimiters for a regex literal, « » to denote character classes, subscript on String to model application of a Regex, parenthesis to denote a capture that must always be a decl (even if the name is dropped by using `_`), and Regex literals defaulting to whole-string matching rather than the first partial-string match:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(186, 45, 162); background-color: rgb(255, 255, 255);" class="">extension<span style="" class=""> </span><span style="color: rgb(112, 61, 170);" class="">Character</span><span style="" class=""> {</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(186, 45, 162);" class="">var</span> isEmoji: <span style="color: rgb(112, 61, 170);" class="">Bool</span> { get }</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">}</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(186, 45, 162); background-color: rgb(255, 255, 255);" class="">extension<span style="" class=""> </span><span style="color: rgb(112, 61, 170);" class="">Regex</span><span style="" class=""> {</span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); background-color: rgb(255, 255, 255);" class=""><span style="" class=""> </span><span style="color: rgb(186, 45, 162);" class="">var</span><span style="" class=""> rangeOfMatch: </span><span style="color: rgb(112, 61, 170);" class="">Regex</span><span style="" class=""><(</span><span style="color: rgb(112, 61, 170);" class="">Range</span><span style="" class=""><</span><span style="color: rgb(112, 61, 170);" class="">String</span><span style="" class="">.</span><span style="color: rgb(112, 61, 170);" class="">Index</span><span style="" class="">>)> </span>// Drops the captured value, just wants the range</div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""> <span style="color: rgb(186, 45, 162);" class="">var</span> allMatches: <span style="color: rgb(112, 61, 170);" class="">Regex</span><<span style="color: rgb(112, 61, 170);" class="">LazySequence</span><<span style="color: rgb(112, 61, 170);" class="">T</span>>></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class="">}</div><div style="margin: 0px; font-stretch: normal; line-height: normal; background-color: rgb(255, 255, 255); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""><span style="color: rgb(186, 45, 162);" class="">let</span> emojiPattern =「(<span style="color: rgb(186, 45, 162);" class="">let</span> <span style="color: rgb(186, 45, 162);" class="">_</span> = «.isEmoji»)」 <span style="color: rgb(0, 132, 0);" class="">// Regex<(Character)></span></div></div></div></div></div></blockquote><div><br class=""></div><div>I assume that we will support an escape sequence of some sort and use the \ character for it. Given that, it makes sense to use \(.isEmoji) as the syntax for referring to user-defined predicates, given that we use it for string literal interpolation, and that regex’s are the dual of it.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; background-color: rgb(255, 255, 255); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""><span style="color: rgb(186, 45, 162);" class="">let</span> theFirst = myString[emojiPattern.rangeOfMatch.firstPartial] <span style="color: rgb(0, 132, 0);" class="">// Range<String.Index></span></div><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);" class=""><span style="color: rgb(186, 45, 162);" class="">let</span> allOfThem = myString[emojiPattern.rangeOfMatch.allMatches] <span style="color: rgb(0, 132, 0);" class="">// LazySequence<Range<String.Index>></span></div><div style="margin: 0px; font-stretch: normal; line-height: normal; background-color: rgb(255, 255, 255); min-height: 14px;" class=""><div style="margin: 0px; font-stretch: normal; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><br class=""></div></div></div><div class="">(Alternatively, Regex could have an init taking a (Character) -> Bool)</div><div class=""><br class=""></div><div class="">In such a world, what do you image the role of a grapheme set type is? It might have more uses as provided by the platform, alongside things such as localization, linguistic analysis, etc.</div></div></div></div></blockquote><div><br class=""></div><div>I think you’re arguing here that CharacterSet should go away. If that is possible, then that would be great. I’d check to see how CharacterSet is used across Cocoa, I’m not an expert on that. It would be fantastic if those could be replaced with versions that take predicate functions or regex’s.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">* Interpolated expressions are supplied inside the literal, meaning they cannot be passed around like format strings without extra boilerplate (e.g. a wrapping function).<br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">The original (2013 era) idea was to allow for interpolants to take multiple arguments (the contents of \(….) would be passed as the argument list, so you could specify formatting information as subsequent parameters, e.g.: </div><div class=""><br class=""></div><div class=""> “your file access is set to \(mode, .Octal)”.</div><div class=""><br class=""></div><div class="">or:</div><div class=""> “your file access is set to \(mode, format: .Octal)”.</div><div class=""> </div><div class="">or something like that. Of course each type would define its own formatting modes, and this would probably be reflected through a type-specific initializer.</div><div class=""><br class=""></div></div></div></blockquote><div class=""><br class=""></div><div class="">That looks very similar to one of Brent’s proposals (that I can no longer find a link to). Was there a reason it didn’t happen other than not getting around to it?</div></div></div></div></blockquote><div><br class=""></div><div>It simply wasn’t the priority at the time (e.g. we didn’t even have let vs var yet :-). There were bigger fish to fry and we didn’t come back to it.</div><div><br class=""></div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">### Regex: Tearing Strings Apart<br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">This is clearly necessary but also clearly not part of Swift 5. Regex’s should themselves be the subject of an intense design process. Swift 6 pretty please??? I’d suggest splitting this section out and starting a regex manifesto. :-)</div><div class=""><br class=""></div></div></div></blockquote><div class=""><br class=""></div><div class="">Since it is so big, I feel like if we’re going to make any progress we need to start soon. Of course, any focus will be preempted as needed by ABI stability. Even if Swift N doesn’t get the fully formed feature, we’ll pick up things along the way. E.g. as you mention there’s a parity between Character’s Bool properties and character classes.</div><div class=""><br class=""></div><div class="">I definitely want to circulate and discuss these ideas a little bit before writing a “manifesto”.</div></div></div></div></blockquote><div><br class=""></div>K, I’m just saying that this is meaty enough and should continue to develop, so I think that it being a manifesto is an inevitability. :-) I’m thrilled you’re pushing this forward btw.</div><div><br class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"></div></div></blockquote></div></div></blockquote></div></div></blockquote></div>-Chris<div class=""><br class=""></div></body></html>