<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Dec 4, 2015, at 11:47 AM, Dmitri Gribenko <<a href="mailto:gribozavr@gmail.com" class="">gribozavr@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">On Fri, Dec 4, 2015 at 11:25 AM, Jens Alfke <<a href="mailto:jens@mooseyard.com" class="">jens@mooseyard.com</a>> wrote:<br class=""><blockquote type="cite" class="">In other words, during UTF-8 encoding, some Unicode codepoints need to be broken into surrogate pairs. Therefore a UTF-8 decoder needs to recognize surrogate pairs and reassemble them into a single codepoint. That’s what’s not happening in this case.<br class=""></blockquote><br class="">No, surrogate code points can not appear in a UTF-8 stream, they can<br class="">only appear in UTF-16.<br class=""><br class=""><a href="http://www.unicode.org/versions/Unicode8.0.0/ch02.pdf" class="">http://www.unicode.org/versions/Unicode8.0.0/ch02.pdf</a> page 30, table 2-3.<br class=""></div></div></blockquote></div><br class=""></div><div class=""><div class="">Thanks for referencing the spec — that was useful to me. However, this looks like an issue where spec conformance clashes with real-world desire for compatibility. After some more reading I found that it’s actually pretty common to find UTF-8 containing surrogate pairs, mostly due to software that was using 16-bit Unicode before it got formalized as UTF-16. <a href="https://en.wikipedia.org/wiki/CESU-8" class="">According to Wikipedia</a>*, Java encodes UTF-8 this way, as do Oracle and MySQL databases.</div><div class=""><br class=""></div><div class="">If there are enough text editors/processors that do this, it might be a good idea for Swift’s lexer to accept surrogate pairs even if they’re technically invalid.</div></div><div class=""><br class=""></div><div class="">—Jens</div><div class=""><br class=""></div><div class="">* <a href="https://en.wikipedia.org/wiki/CESU-8" class="">https://en.wikipedia.org/wiki/CESU-8</a></div></body></html>