<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">Er, typo in the first sentence! I meant to say:</div><div class=""><br class=""></div><div class="">I was quite surprised to learn that it’s possible to create Swift strings that contain things other than valid Unicode characters.</div></div></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 18, 2015, at 3:47 PM, Paul Cantrell via swift-evolution <<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I was quite surprised to learn that it’s possible to create Swift strings that do not contain things other than valid Unicode characters. Is it feasible to guarantee that this cannot happen?</div><div class=""><br class=""></div><div class="">String.init(bytes:encoding:) is failable, and does in fact validate that the given bytes are decodable with the given encoding in most circumstances:</div><div class=""><br class=""></div><div class=""><div style="color: rgb(102, 139, 73); font-family: Menlo; font-size: 10.5px; margin: 0px; line-height: normal;" class=""><span style="" class=""> </span>// Returns nil</div><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo; color: rgb(88, 126, 168);" class=""><span style="" class=""> </span>String<span style="" class="">(</span></div><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo;" class=""> bytes: [<span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">0xD8</span>, <span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">0x00</span>] <span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">as</span> [<span style="font-variant-ligatures: no-common-ligatures; color: #587ea8" class="">UInt8</span>],</div><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo;" class=""> encoding: <span style="font-variant-ligatures: no-common-ligatures; color: #587ea8" class="">NSUTF8StringEncoding</span>)</div></div><div class=""><br class=""></div><div class="">However, that initializer does <i class="">not</i> reject invalid surrogate characters in UTF-16:</div><div class=""><br class=""></div><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo; color: rgb(102, 139, 73);" class=""><span style="" class=""> </span>// Succeeds (wat?!)</div><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">let</span> bogusStr = <span style="font-variant-ligatures: no-common-ligatures; color: #587ea8" class="">String</span>(</div><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo;" class=""> bytes: [<span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">0xD8</span>, <span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">0x00</span>] <span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">as</span> [<span style="font-variant-ligatures: no-common-ligatures; color: #587ea8" class="">UInt8</span>],</div><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo; color: rgb(88, 126, 168);" class=""><span style="" class=""> encoding: </span>NSUTF16BigEndianStringEncoding<span style="" class="">)!</span></div><div class=""><br class=""></div><div class="">Ever wonder why dataWithJSONObject(…) is declared “throws?” Now you know!</div><div class=""><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo; color: rgb(102, 139, 73);" class=""><div style="margin: 0px; font-size: 10.5px; line-height: normal; min-height: 12px;" class=""><div class=""><br class=""></div></div><div style="margin: 0px; font-size: 10.5px; line-height: normal;" class=""><span style="font-size: 10.5px;" class=""> </span><span style="font-size: 10.5px;" class="">// Throws an error</span></div><div style="margin: 0px; font-size: 10.5px; line-height: normal; color: rgb(88, 126, 168);" class=""><span style="" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #323e7d" class="">try</span><span style="" class="">! </span>NSJSONSerialization<span style="" class="">.</span>dataWithJSONObject<span style="" class="">(</span></div><div style="margin: 0px; font-size: 10.5px; line-height: normal;" class=""> [<span style="font-variant-ligatures: no-common-ligatures; color: #843e64" class="">"foo"</span>: <span style="font-variant-ligatures: no-common-ligatures; color: #587ea8" class="">bogusStr</span>], options: [])</div><div style="margin: 0px; font-size: 10.5px; line-height: normal;" class=""><br class=""></div><div style="margin: 0px; font-size: 10.5px; line-height: normal;" class=""><div style="font-family: 'Helvetica Neue'; font-size: 13px;" class="">And why does the URL escaping method in Foundation return an optional even though it escapes the string using UTF-8, which is a complete Unicode encoding? Same reason:</div><div style="font-family: 'Helvetica Neue'; font-size: 13px;" class=""><div style="margin: 0px; font-size: 10.5px; line-height: normal; font-family: Menlo; color: rgb(102, 139, 73);" class=""><div style="margin: 0px; font-size: 10.5px; line-height: normal; min-height: 12px;" class=""></div></div></div></div><div class=""><div style="font-size: 10.5px; margin: 0px; line-height: normal;" class=""><span style="" class=""> </span>// Returns nil</div><div style="font-size: 10.5px; margin: 0px; line-height: normal; color: rgb(88, 126, 168);" class=""><span style="" class=""> </span>bogusStr<span style="" class="">.</span>stringByAddingPercentEncodingWithAllowedCharacters<span style="" class="">(</span></div><div style="font-size: 10.5px; margin: 0px; line-height: normal; color: rgb(88, 126, 168);" class=""><span style="" class=""> </span>NSCharacterSet<span style="" class="">.</span>alphanumericCharacterSet<span style="" class="">())</span></div></div><div class=""><span style="" class=""><br class=""></span></div></div></div><div class="">AFAIK, the first method could lose its “throws” modifier and the second method would not need to return an optional if only String itself guaranteed that it would always contain valid Unicode. There are likely other APIs that would see similar benefits.</div><div class=""><br class=""></div><div class="">Are there downsides to making all String initializers guarantee that the Strings always contain valid Unicode? I can think of two possibilities:</div><div class=""><br class=""></div><div class=""><ul class="MailOutline"><li class="">Is there some circumstance where you actually want a String to contain unpaired UTF-16 surrogate characters? I can’t imagine what that would be, but perhaps someone else can.</li><li class="">Is it important to ensure that String.init(…) is O(1) when it uses UTF-16? This seems thin: I assume that the library has to copy the raw bytes regardless, and it’s O(n) for other character encodings, so…?</li></ul></div><div class=""><br class=""></div><div class="">Cheers,</div><div class=""><br class=""></div><div class="">Paul</div><div class=""><br class=""></div>
<img src="https://u2002410.ct.sendgrid.net/wf/open?upn=zCg-2FSGF9Wk188a6c55kLyEbrj7YhaXxFEHM-2F-2B0YAlzXBuiMxVKrFw6jmCVQmn-2FCchYCpqD8UxhCufUdbwXQre20UkHMe4-2FoIXjR7Y15AA-2FDbX2c-2BD-2FSZi5Bsx3hT-2FYR0sakbmlDz68pfZ-2BXBZ9t4Pt3JevLoCHDCumQXuf5aZDSlsO-2BY-2BuAkXnbRWVIYn9PWFi9YIuTpHnVugERtFqrsZSVdMmE9AX16g2jS5Khzufk-3D" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;" class="">
</div>
_______________________________________________<br class="">swift-evolution mailing list<br class=""><a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-evolution<br class=""></div></blockquote></div><br class=""></body></html>