<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Hi Michael,</div><div class="gmail_quote"><br></div><div class="gmail_quote">On Thu, Dec 17, 2015 at 9:16 PM, Michael Buckley via swift-dev <span dir="ltr"><<a href="mailto:swift-dev@swift.org" target="_blank">swift-dev@swift.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>I would like to fix rdar://20511834 , which is that the new skin tone and multi-person grouping emoji introduced with iOS 8.3 and OS X 10.10.3 are represented as multiple extended grapheme clusters by Swift.String, and I have a few questions.</div><div><br></div><div>1. Is this something we want to fix at this time, considering these emoji are part of UTR #51, but not part of an official Unicode standard, or do we want to wait for these emoji groupings to be published as part of the Unicode standard.</div></div></blockquote><div><br></div><div>The issue you are describing is indeed important. There are multiple considerations here. One of them is that we currently describe the segmentation that Swift performs to be "extended grapheme clusters", which has a precise definition in the spec. Changing that would mean introducing tailoring, which is allowed by the spec, but the algorithm would be custom.</div><div> <br></div><div>One thing to do would be to check the Apple's ICU implementation, which (I think) implements some extra handling for UTR #51 (<a href="http://opensource.apple.com/release/os-x-1011/">http://opensource.apple.com/release/os-x-1011/</a>) to see how it deals with this, whether it introduces tailoring, and if so, in what way.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>2. What is the best way to fix this?</div></div></blockquote><div><br></div><div>My primary concern with the fix in the PR is that it seems to change the segmentation behavior for other sequences. The grapheme cluster segmentation algorithm is local and stateless. It only looks at two adjacent Unicode scalars. This means that adding a rule like "ZWJ no_boundary Emoji" will affect all sequences, even those that are not a grouping as defined in UTR #51 (for example, "Latin letter, ZWJ, Emoji": the three scalars would be grouped).</div><div><br></div><div>This is the same issue as multiple flags pasted together (which are represented as regional indicator characters). The current algorithm just does not have enough information to split them apart, it needs to look at a wider part of the string.</div><div><br></div><div>I would be much happier with a solution only changed the segmentation for the cases covered by the TR, but I understand it might have performance implications. I think we should try to add such a tailoring, and benchmark it.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div><div>3. Is this a significant enough change to warrant going through the entire swift-evolution process, or a simple bug fix?</div></div></blockquote><div><br></div><div>The change that adds the first tailoring to the algorithm might be significant enough. But I think it would be a question of whether we want any tailoring at all, not about specific tailoring.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>As I see it, there are three disadvantages to this approach: The hardcoded character class, the reliance on a second emoji data file, and the fact that the trie bitmap had to be extended from 16 bits to 32 bits. This last change was probably inevitable in the future, and it only increases the trie size by 4096 bytes (from 18961 bytes to 23057 bytes).</div></div></blockquote><div><br></div><div>In my opinion, the biggest disadvantage is that it would change segmentation for other sequences.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>But probably the biggest reason for seeking an alternative implementation is that the existing behavior is not always incorrect. It's incorrect on the most recent versions of Apple operating systems when rendered in contexts that support these emoji, but it's correct everywhere else. It seems as though Swift perhaps needs to allow users to change grapheme clustering behavior based on a user setting, and perhaps even allow users to specify which unicode version they want to use, but that's a much larger change, which may not be worth the costs.</div></div></blockquote><div><br></div><div>I would prefer to avoid platform-specific differences here. Unicode is hard as it is, and adding context-sensitivity to algorithms in an unusual way (that is, not through existing established mechanisms like locales) just calls for interoperability issues.</div><div></div></div><div class="gmail_extra"><br></div>Dmitri<br clear="all"><div><br></div>-- <br><div class="gmail_signature">main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if<br>(j){printf("%d\n",i);}}} /*Dmitri Gribenko <<a href="mailto:gribozavr@gmail.com" target="_blank">gribozavr@gmail.com</a>>*/</div>
</div></div>