<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 22, 2017, at 6:05 PM, Mohit Athwani via swift-users <<a href="mailto:swift-users@swift.org" class="">swift-users@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-family: Alegreya-Regular; font-size: 15px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">I don't understand why we need the usedEncoding parameter? I understand that it's a pointer but how do we decide what encoding to use? Do we default to NSUTF8StringEncoding?</div></div></blockquote></div><br class=""><div class="">The original implementation in Foundation uses some heuristics to try to guess the encoding, since there are unfortunately billions of plain text files out there that don’t explicitly state their encoding. It’s not open source, so we can’t know for sure [except for the people who work at Apple], but I’m sure it includes things like:</div><div class=""><br class=""></div><div class="">- Look for a Unicode BOM at the start, in which case it’s probably UTF-16 (or maybe UTF-32? I don’t know the details.)</div><div class="">- If not, see whether all bytes are 0x00-0x7F ⟶ in that case use ASCII</div><div class="">- If not, does it contain any byte sequences that are illegal in UTF-8? ⟶ If not, use UTF-8</div><div class="">- Otherwise, does it contain any bytes in the range 0x80-0xBF?</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>⟶ If not, ISO-8859-1 (aka ISO-Latin-1) is a good guess</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>⟶ If so, CP-1252 (aka WinLatin1) is a good guess; it’s a nonstandard but very common superset of ISO-8859-1 with extra characters in that byte range</div><div class=""><br class=""></div><div class="">There are likely other heuristics too. It used to be important to detect the old MacRoman encoding used in pre-OS X apps, but it’s been long enough that there shouldn’t be many docs like that in the wild anymore. There are multibyte non-Unicode encodings that used to be very common in non-Roman languages, like Shift-JIS, but I have no idea how to detect them or if they’re even still relevant.</div><div class=""><br class=""></div><div class="">It could also be useful to check whether the start of the file looks like XML or HTML, and if so, parse it enough to find where it specifies its encoding. (Are there other text formats that include encodings? I’ve seen special markings at the top of source files used for emacs or vi, specifying tab widths and such, but I don’t know if those can specify encodings too.)</div><div class=""><br class=""></div><div class="">I’m not involved in Swift development, but IMHO a basic implementation that just uses the rules I sketched above would be pretty useful, and then people with more domain knowledge could enhance that code to add more heuristics later on.</div><div class=""><br class=""></div><div class="">—Jens</div></body></html>