<div dir="ltr">It actually appears that Swift already links against ICU. I'll see if I can hook Swift up to ICU's grapheme separation code.</div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 20, 2015 at 10:41 PM, Michael Buckley <span dir="ltr"><<a href="mailto:michael@buckleyisms.com" target="_blank">michael@buckleyisms.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">After reading through the ICU sources, if I understand them correctly, ICU uses the Aho–Corasick algorithm to determine grapheme breaks, word breaks and line breaks, and then does some post-processing after matching using the algorithm.<div><br></div><div>This allows ICU to solve the regional indicator problem by including a pattern that matches 3 regional indicator characters in a row and inserts a grapheme break after the second. This does not actually modify the string by adding a zero-width space or something.</div><div><br></div><div>While this approach can solve the regional indicator problem efficiently, it cannot solve the problem with the zero-width joiner emoji sequences as easily. This is because Aho-Corasick is liner on the length of the text + the number of patterns + the number of matches, and the emoji problem would require a pattern for every emoji sequence we want to support.</div><div><br></div><div>However, after reading UTR #51 again, we may want to treat all emoji joined by a ZWJ as a single extended grapheme cluster, whether they form a known sequence or not. That's because UTR#51 leaves the exact sequences as implementation-defined. It includes a list of currently-known implemented sequences, but allows for implementers to add their own sequences.</div><div><br></div><div>Which means that Ubuntu could, for example, support a sequence of DOG FACE + ZWJ + PILE OF POO, and represent it with a glyph of a dog doing its business. We basically have two options here. We could treat Swift as an Apple-platform centric language and implement only the sequences that appear on Apple platforms, or we could implement a rule of any emoji + ZWJ + any emoji has no break. As Dmitri pointed out, this would mean Swift would mean Swift would report strings of invalid sequences as a single character, which could be confusing. But I posit that the situation we have now, reporting valid strings as multiple characters is also confusing, and much more likely. It's unlikely that anyone is going to stick a ZWJ between emoji unless they intend to make a sequence from it.</div><div><br></div><div>Incidentally, this is what ICU does. You can test this yourself in TextEdit by typing HEAVY BLACK HEART followed by ZWJ ad infinitum, then press the left arrow key once and watch TextEdit treat the sequence as a single character, causing the cursor to jump to the beginning of the string. ICU, however, does hard-code the emoji that are currently used by Apple emoji sequences, so you can't do the same thing with PILE OF POO. This makes sense in an ICU context, since it's only implementing the Apple sequences, but if we want Swift to be more platform-agnostic, we would want this behavior for any emoji.</div><div><br></div><div><br></div><div>ICU's implementation fixes the regional indicator problem, but the implementation is large and moderately complicated. Just throwing this out there, but would it be possible to add ICU as a dependency to Swift and just use its implementation? I'm sure this would be a nightmare to work out license and logistics-wise. (It would probably necessitate that ICU development be opened up to the same degree that other Swift dependencies are). I also understand that adding any dependencies at all is less than ideal. But this seems like a perfect situation for some code sharing. We have a moderately large and complicated library that is being updated with new Emoji support when new Emoji are added anyway. It's fast, it's already well-used, and we'd have to duplicate a lot of what it does to solve the same problems if we didn't use it.</div><div><br></div><div>As a bonus, we could link to the system-supplied libicu on OS X and iOS, so Swift apps would automatically get the latest emoji support when users update their OSs. We would still have to bundle it for other OSs.</div><div><br></div><div>I know that there are a lot of downsides to making it a dependency, but I wanted to throw the idea out there to see if it made sense.</div></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Fri, Dec 18, 2015 at 6:22 AM, Michael Buckley <span dir="ltr"><<a href="mailto:michael@buckleyisms.com" target="_blank">michael@buckleyisms.com</a>></span> wrote:<br></span><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the response, Dimitri. My comments inline below.<br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, Dec 18, 2015 at 3:29 AM, Dmitri Gribenko <span dir="ltr"><<a href="mailto:gribozavr@gmail.com" target="_blank">gribozavr@gmail.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> <br></div><div>One thing to do would be to check the Apple's ICU implementation, which (I think) implements some extra handling for UTR #51 (<a href="http://opensource.apple.com/release/os-x-1011/" target="_blank">http://opensource.apple.com/release/os-x-1011/</a>) to see how it deals with this, whether it introduces tailoring, and if so, in what way.</div></div></div></div></blockquote><div><br></div></span><div> I will look into that. I had always thought that would have been part of Core Text, and not open sourced. It is great to know that it is open-sourced.</div><span><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>My primary concern with the fix in the PR is that it seems to change the segmentation behavior for other sequences. The grapheme cluster segmentation algorithm is local and stateless. It only looks at two adjacent Unicode scalars. This means that adding a rule like "ZWJ no_boundary Emoji" will affect all sequences, even those that are not a grouping as defined in UTR #51 (for example, "Latin letter, ZWJ, Emoji": the three scalars would be grouped).</div></div></div></div></blockquote><div><br></div></span><div>Apologies, I forgot to mention that disadvantage. It does change the segmentation behavior for other sequences, which was one of the reasons I was on the fence about whether this should go through the swift-evolution process.</div><span><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>This is the same issue as multiple flags pasted together (which are represented as regional indicator characters). The current algorithm just does not have enough information to split them apart, it needs to look at a wider part of the string.</div></div></div></div></blockquote><div><br></div></span><div>I could be reading the Unicode standard incorrectly, but it appears that this might be the intended behavior for the flag characters. I definitely agree that it's not ideal.</div><span><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I would be much happier with a solution only changed the segmentation for the cases covered by the TR, but I understand it might have performance implications. I think we should try to add such a tailoring, and benchmark it.</div></div></div></div></blockquote><div><br></div></span><div>Just so that I understand what you mean by tailoring, you mean switching to a possibly stateful algorithm which can consider more than just two adjacent characters when grouping, right?</div><span><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>The change that adds the first tailoring to the algorithm might be significant enough. But I think it would be a question of whether we want any tailoring at all, not about specific tailoring.</div></div></div></div></blockquote><div><br></div></span><div>Thanks for the clarification. Just to be sure, if this change wasn't as problematic, but still changed the behavior of Swift.String, you're saying it would not be important enough for swift-evolution? As a concrete example, if I was just proposing to fix the skin tone emoji, but not the SWJ sequences, would it be considered just a bug fix?</div></div></div></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>