<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Looks pretty good in general from my quick glance–at least, it’s much better than the current situation. I do have a couple of comments and questions, which I’ve inlined below.<div class=""><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Saagar Jha</div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Jan 19, 2017, at 6:56 PM, Ben Cohen via swift-evolution <<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi all,<br class=""><br class="">Below is our take on a design manifesto for Strings in Swift 4 and beyond.<br class=""><br class="">Probably best read in rendered markdown on GitHub:<br class=""><a href="https://github.com/apple/swift/blob/master/docs/StringManifesto.md" class="">https://github.com/apple/swift/blob/master/docs/StringManifesto.md</a><br class=""><br class="">We’re eager to hear everyone’s thoughts.<br class=""><br class="">Regards,<br class="">Ben and Dave<br class=""><br class=""><br class=""># String Processing For Swift 4<br class=""><br class="">* Authors: [Dave Abrahams](https://github.com/dabrahams), [Ben Cohen](https://github.com/airspeedswift)<br class=""><br class="">The goal of re-evaluating Strings for Swift 4 has been fairly ill-defined thus<br class="">far, with just this short blurb in the<br class="">[list of goals](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160725/025676.html):<br class=""><br class=""><blockquote type="cite" class="">**String re-evaluation**: String is one of the most important fundamental<br class="">types in the language. The standard library leads have numerous ideas of how<br class="">to improve the programming model for it, without jeopardizing the goals of<br class="">providing a unicode-correct-by-default model. Our goal is to be better at<br class="">string processing than Perl!<br class=""></blockquote><br class="">For Swift 4 and beyond we want to improve three dimensions of text processing:<br class=""><br class=""> 1. Ergonomics<br class=""> 2. Correctness<br class=""> 3. Performance<br class=""><br class="">This document is meant to both provide a sense of the long-term vision <br class="">(including undecided issues and possible approaches), and to define the scope of<br class="">work that could be done in the Swift 4 timeframe.<br class=""><br class="">## General Principles<br class=""><br class="">### Ergonomics<br class=""><br class="">It's worth noting that ergonomics and correctness are mutually-reinforcing. An<br class="">API that is easy to use—but incorrectly—cannot be considered an ergonomic<br class="">success. Conversely, an API that's simply hard to use is also hard to use<br class="">correctly. Acheiving optimal performance without compromising ergonomics or<br class="">correctness is a greater challenge.<br class=""></div></div></blockquote><div><br class=""></div><div>Minor typo: acheiving->achieving</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">Consistency with the Swift language and idioms is also important for<br class="">ergonomics. There are several places both in the standard library and in the<br class="">foundation additions to `String` where patterns and practices found elsewhere<br class="">could be applied to improve usability and familiarity.<br class=""><br class="">### API Surface Area<br class=""><br class="">Primary data types such as `String` should have APIs that are easily understood<br class="">given a signature and a one-line summary. Today, `String` fails that test. As<br class="">you can see, the Standard Library and Foundation both contribute significantly to<br class="">its overall complexity.<br class=""><br class="">**Method Arity** | **Standard Library** | **Foundation**<br class="">---|:---:|:---:<br class="">0: `ƒ()` | 5 | 7<br class="">1: `ƒ(:)` | 19 | 48<br class="">2: `ƒ(::)` | 13 | 19<br class="">3: `ƒ(:::)` | 5 | 11<br class="">4: `ƒ(::::)` | 1 | 7<br class="">5: `ƒ(:::::)` | - | 2<br class="">6: `ƒ(::::::)` | - | 1<br class=""><br class="">**API Kind** | **Standard Library** | **Foundation**<br class="">---|:---:|:---:<br class="">`init` | 41 | 18<br class="">`func` | 42 | 55<br class="">`subscript` | 9 | 0<br class="">`var` | 26 | 14<br class=""><br class="">**Total: 205 APIs**<br class=""><br class="">By contrast, `Int` has 80 APIs, none with more than two parameters.[0] String processing is complex enough; users shouldn't have<br class="">to press through physical API sprawl just to get started.<br class=""><br class="">Many of the choices detailed below contribute to solving this problem,<br class="">including:<br class=""><br class=""> * Restoring `Collection` conformance and dropping the `.characters` view.<br class=""> * Providing a more general, composable slicing syntax.<br class=""> * Altering `Comparable` so that parameterized<br class=""> (e.g. case-insensitive) comparison fits smoothly into the basic syntax.<br class=""> * Clearly separating language-dependent operations on text produced <br class=""> by and for humans from language-independent<br class=""> operations on text produced by and for machine processing.<br class=""> * Relocating APIs that fall outside the domain of basic string processing and<br class=""> discouraging the proliferation of ad-hoc extensions.<br class=""><br class=""><br class="">### Batteries Included<br class=""><br class="">While `String` is available to all programs out-of-the-box, crucial APIs for<br class="">basic string processing tasks are still inaccessible until `Foundation` is<br class="">imported. While it makes sense that `Foundation` is needed for domain-specific<br class="">jobs such as<br class="">[linguistic tagging](<a href="https://developer.apple.com/reference/foundation/nslinguistictagger" class="">https://developer.apple.com/reference/foundation/nslinguistictagger</a>),<br class="">one should not need to import anything to, for example, do case-insensitive<br class="">comparison.<br class=""><br class="">### Unicode Compliance and Platform Support<br class=""><br class="">The Unicode standard provides a crucial objective reference point for what<br class="">constitutes correct behavior in an extremely complex domain, so<br class="">Unicode-correctness is, and will remain, a fundamental design principle behind<br class="">Swift's `String`. That said, the Unicode standard is an evolving document, so<br class="">this objective reference-point is not fixed.[1] While<br class="">many of the most important operations—e.g. string hashing, equality, and<br class="">non-localized comparison—will be stable, the semantics<br class="">of others, such as grapheme breaking and localized comparison and case<br class="">conversion, are expected to change as platforms are updated, so programs should<br class="">be written so their correctness does not depend on precise stability of these<br class="">semantics across OS versions or platforms. Although it may be possible to<br class="">imagine static and/or dynamic analysis tools that will help users find such<br class="">errors, the only sure way to deal with this fact of life is to educate users.<br class=""><br class="">## Design Points<br class=""><br class="">### Internationalization<br class=""><br class="">There is strong evidence that developers cannot determine how to use<br class="">internationalization APIs correctly. Although documentation could and should be<br class="">improved, the sheer size, complexity, and diversity of these APIs is a major<br class="">contributor to the problem, causing novices to tune out, and more experienced<br class="">programmers to make avoidable mistakes.<br class=""><br class="">The first step in improving this situation is to regularize all localized<br class="">operations as invocations of normal string operations with extra<br class="">parameters. Among other things, this means:<br class=""><br class="">1. Doing away with `localizedXXX` methods <br class="">2. Providing a terse way to name the current locale as a parameter<br class="">3. Automatically adjusting defaults for options such<br class=""> as case sensitivity based on whether the operation is localized.<br class="">4. Removing correctness traps like `localizedCaseInsensitiveCompare` (see<br class=""> guidance in the<br class=""> [Internationalization and Localization Guide](<a href="https://developer.apple.com/library/content/documentation/MacOSX/Conceptual/BPInternational/InternationalizingYourCode/InternationalizingYourCode.html" class="">https://developer.apple.com/library/content/documentation/MacOSX/Conceptual/BPInternational/InternationalizingYourCode/InternationalizingYourCode.html</a>).<br class=""><br class="">Along with appropriate documentation updates, these changes will make localized<br class="">operations more teachable, comprehensible, and approachable, thereby lowering a<br class="">barrier that currently leads some developers to ignore localization issues<br class="">altogether.<br class=""><br class="">#### The Default Behavior of `String`<br class=""><br class="">Although this isn't well-known, the most accessible form of many operations on<br class="">Swift `String` (and `NSString`) are really only appropriate for text that is<br class="">intended to be processed for, and consumed by, machines. The semantics of the<br class="">operations with the simplest spellings are always non-localized and<br class="">language-agnostic.<br class=""><br class="">Two major factors play into this design choice:<br class=""><br class="">1. Machine processing of text is important, so we should have first-class,<br class=""> accessible functions appropriate to that use case.<br class=""><br class="">2. The most general localized operations require a locale parameter not required<br class=""> by their un-localized counterparts. This naturally skews complexity towards<br class=""> localized operations.<br class=""><br class="">Reaffirming that `String`'s simplest APIs have<br class="">language-independent/machine-processed semantics has the benefit of clarifying<br class="">the proper default behavior of operations such as comparison, and allows us to<br class="">make [significant optimizations](#collation-semantics) that were previously<br class="">thought to conflict with Unicode.<br class=""><br class="">#### Future Directions<br class=""><br class="">One of the most common internationalization errors is the unintentional<br class="">presentation to users of text that has not been localized, but regularizing APIs<br class="">and improving documentation can go only so far in preventing this error.<br class="">Combined with the fact that `String` operations are non-localized by default,<br class="">the environment for processing human-readable text may still be somewhat<br class="">error-prone in Swift 4.<br class=""><br class="">For an audience of mostly non-experts, it is especially important that naïve<br class="">code is very likely to be correct if it compiles, and that more sophisticated<br class="">issues can be revealed progressively. For this reason, we intend to<br class="">specifically and separately target localization and internationalization<br class="">problems in the Swift 5 timeframe.<br class=""><br class="">### Operations With Options<br class=""><br class="">There are three categories of common string operation that commonly need to be<br class="">tuned in various dimensions:<br class=""><br class="">**Operation**|**Applicable Options**<br class="">---|---<br class="">sort ordering | locale, case/diacritic/width-insensitivity<br class="">case conversion | locale<br class="">pattern matching | locale, case/diacritic/width-insensitivity<br class=""><br class="">The defaults for case-, diacritic-, and width-insensitivity are different for<br class="">localized operations than for non-localized operations, so for example a<br class="">localized sort should be case-insensitive by default, and a non-localized sort<br class="">should be case-sensitive by default. We propose a standard “language” of<br class="">defaulted parameters to be used for these purposes, with usage roughly like this:<br class=""><br class="">```swift<br class=""> x.compared(to: y, case: .sensitive, in: swissGerman)<br class=""><br class=""> x.lowercased(in: .currentLocale)<br class=""><br class=""> x.allMatches(<br class=""> somePattern, case: .insensitive, diacritic: .insensitive)<br class="">```<br class=""><br class="">This usage might be supported by code like this:<br class=""><br class="">```swift<br class="">enum StringSensitivity {<br class="">case sensitive<br class="">case insensitive<br class="">}<br class=""><br class="">extension Locale {<br class=""> static var currentLocale: Locale { ... }<br class="">}<br class=""><br class="">extension Unicode {<br class=""> // An example of the option language in declaration context,<br class=""> // with nil defaults indicating unspecified, so defaults can be<br class=""> // driven by the presence/absence of a specific Locale<br class=""> func frobnicated(<br class=""> case caseSensitivity: StringSensitivity? = nil,<br class=""> diacritic diacriticSensitivity: StringSensitivity? = nil,<br class=""> width widthSensitivity: StringSensitivity? = nil,<br class=""> in locale: Locale? = nil<br class=""> ) -> Self { ... }<br class="">}<br class="">```<br class=""></div></div></blockquote><div><br class=""></div><div>Any reason why Locale is defaulted to nil, instead of currentLocale? It seems more useful to me.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">### Comparing and Hashing Strings<br class=""><br class="">#### Collation Semantics<br class=""><br class="">What Unicode says about collation—which is used in `<`, `==`, and hashing— turns<br class="">out to be quite interesting, once you pick it apart. The full Unicode Collation<br class="">Algorithm (UCA) works like this:<br class=""><br class="">1. Fully normalize both strings<br class="">2. Convert each string to a sequence of numeric triples to form a collation key<br class="">3. “Flatten” the key by concatenating the sequence of first elements to the<br class=""> sequence of second elements to the sequence of third elements<br class="">4. Lexicographically compare the flattened keys <br class=""><br class="">While step 1 can usually<br class="">be [done quickly](<a href="http://unicode.org/reports/tr15/#Description_Norm" class="">http://unicode.org/reports/tr15/#Description_Norm</a>) and<br class="">incrementally, step 2 uses a collation table that maps matching *sequences* of<br class="">unicode scalars in the normalized string to *sequences* of triples, which get<br class="">accumulated into a collation key. Predictably, this is where the real costs<br class="">lie.<br class=""><br class="">*However*, there are some bright spots to this story. First, as it turns out,<br class="">string sorting (localized or not) should be done down to what's called<br class="">the<br class="">[“identical” level](<a href="http://unicode.org/reports/tr10/#Multi_Level_Comparison" class="">http://unicode.org/reports/tr10/#Multi_Level_Comparison</a>),<br class="">which adds a step 3a: append the string's normalized form to the flattened<br class="">collation key. At first blush this just adds work, but consider what it does<br class="">for equality: two strings that normalize the same, naturally, will collate the<br class="">same. But also, *strings that normalize differently will always collate<br class="">differently*. In other words, for equality, it is sufficient to compare the<br class="">strings' normalized forms and see if they are the same. We can therefore<br class="">entirely skip the expensive part of collation for equality comparison.<br class=""><br class="">Next, naturally, anything that applies to equality also applies to hashing: it<br class="">is sufficient to hash the string's normalized form, bypassing collation keys.<br class="">This should provide significant speedups over the current implementation.<br class="">Perhaps more importantly, since comparison down to the “identical” level applies<br class="">even to localized strings, it means that hashing and equality can be implemented<br class="">exactly the same way for localized and non-localized text, and hash tables with<br class="">localized keys will remain valid across current-locale changes.<br class=""><br class="">Finally, once it is agreed that the *default* role for `String` is to handle<br class="">machine-generated and machine-readable text, the default ordering of `String`s<br class="">need no longer use the UCA at all. It is sufficient to order them in any way<br class="">that's consistent with equality, so `String` ordering can simply be a<br class="">lexicographical comparison of normalized forms,[4]<br class="">(which is equivalent to lexicographically comparing the sequences of grapheme<br class="">clusters), again bypassing step 2 and offering another speedup.<br class=""><br class="">This leaves us executing the full UCA *only* for localized sorting, and ICU's<br class="">implementation has apparently been very well optimized.<br class=""><br class="">Following this scheme everywhere would also allow us to make sorting behavior<br class="">consistent across platforms. Currently, we sort `String` according to the UCA,<br class="">except that—*only on Apple platforms*—pairs of ASCII characters are ordered by<br class="">unicode scalar value.<br class=""><br class="">#### Syntax<br class=""><br class="">Because the current `Comparable` protocol expresses all comparisons with binary<br class="">operators, string comparisons—which may require<br class="">additional [options](#operations-with-options)—do not fit smoothly into the<br class="">existing syntax. At the same time, we'd like to solve other problems with<br class="">comparison, as outlined<br class="">in<br class="">[this proposal](<a href="https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e" class="">https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e</a>)<br class="">(implemented by changes at the head<br class="">of<br class="">[this branch](<a href="https://github.com/CodaFi/swift/commits/space-the-final-frontier" class="">https://github.com/CodaFi/swift/commits/space-the-final-frontier</a>)).<br class="">We should adopt a modification of that proposal that uses a method rather than<br class="">an operator `<=>`:<br class=""></div></div></blockquote><div><br class=""></div><div>Why not both? Have the “UFO” operator, with the methods as support for more complicated use cases where the sugar doesn’t hold up.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class="">```swift<br class="">enum SortOrder { case before, same, after }<br class=""><br class="">protocol Comparable : Equatable {<br class=""> func compared(to: Self) -> SortOrder<br class=""> ...<br class="">}<br class="">```<br class=""><br class="">This change will give us a syntactic platform on which to implement methods with<br class="">additional, defaulted arguments, thereby unifying and regularizing comparison<br class="">across the library.<br class=""><br class="">```swift<br class="">extension String {<br class=""> func compared(to: Self) -> SortOrder<br class=""><br class="">}<br class="">```<br class=""><br class="">**Note:** `SortOrder` should bridge to `NSComparisonResult`. It's also possible<br class="">that the standard library simply adopts Foundation's `ComparisonResult` as is,<br class="">but we believe the community should at least consider alternate naming before<br class="">that happens. There will be an opportunity to discuss the choices in detail<br class="">when the modified<br class="">[Comparison Proposal](<a href="https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e" class="">https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e</a>) comes<br class="">up for review.<br class=""><br class="">### `String` should be a `Collection` of `Character`s Again<br class=""><br class="">In Swift 2.0, `String`'s `Collection` conformance was dropped, because we<br class="">convinced ourselves that its semantics differed from those of `Collection` too<br class="">significantly.<br class=""><br class="">It was always well understood that if strings were treated as sequences of<br class="">`UnicodeScalar`s, algorithms such as `lexicographicalCompare`, `elementsEqual`,<br class="">and `reversed` would produce nonsense results. Thus, in Swift 1.0, `String` was<br class="">a collection of `Character` (extended grapheme clusters). During 2.0<br class="">development, though, we realized that correct string concatenation could<br class="">occasionally merge distinct grapheme clusters at the start and end of combined<br class="">strings.<br class=""><br class="">This quirk aside, every aspect of strings-as-collections-of-graphemes appears to<br class="">comport perfectly with Unicode. We think the concatenation problem is tolerable,<br class="">because the cases where it occurs all represent partially-formed constructs. The<br class="">largest class—isolated combining characters such as ◌́ (U+0301 COMBINING ACUTE<br class="">ACCENT)—are explicitly called out in the Unicode standard as<br class="">“[degenerate](<a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries" class="">http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries</a>)” or<br class="">“[defective](<a href="http://www.unicode.org/versions/Unicode9.0.0/ch03.pdf" class="">http://www.unicode.org/versions/Unicode9.0.0/ch03.pdf</a>)”. The other<br class="">cases—such as a string ending in a zero-width joiner or half of a regional<br class="">indicator—appear to be equally transient and unlikely outside of a text editor.<br class=""><br class="">Admitting these cases encourages exploration of grapheme composition and is<br class="">consistent with what appears to be an overall Unicode philosophy that “no<br class="">special provisions are made to get marginally better behavior for… cases that<br class="">never occur in practice.”[2] Furthermore, it seems<br class="">unlikely to disturb the semantics of any plausible algorithms. We can handle<br class="">these cases by documenting them, explicitly stating that the elements of a<br class="">`String` are an emergent property based on Unicode rules.<br class=""><br class="">The benefits of restoring `Collection` conformance are substantial: <br class=""><br class=""> * Collection-like operations encourage experimentation with strings to<br class=""> investigate and understand their behavior. This is useful for teaching new<br class=""> programmers, but also good for experienced programmers who want to<br class=""> understand more about strings/unicode.<br class=""><br class=""> * Extended grapheme clusters form a natural element boundary for Unicode<br class=""> strings. For example, searching and matching operations will always produce<br class=""> results that line up on grapheme cluster boundaries.<br class=""><br class=""> * Character-by-character processing is a legitimate thing to do in many real<br class=""> use-cases, including parsing, pattern matching, and language-specific<br class=""> transformations such as transliteration.<br class=""><br class=""> * `Collection` conformance makes a wide variety of powerful operations<br class=""> available that are appropriate to `String`'s default role as the vehicle for<br class=""> machine processed text.<br class=""><br class=""> The methods `String` would inherit from `Collection`, where similar to<br class=""> higher-level string algorithms, have the right semantics. For example,<br class=""> grapheme-wise `lexicographicalCompare`, `elementsEqual`, and application of<br class=""> `flatMap` with case-conversion, produce the same results one would expect<br class=""> from whole-string ordering comparison, equality comparison, and<br class=""> case-conversion, respectively. `reverse` operates correctly on graphemes,<br class=""> keeping diacritics moored to their base characters and leaving emoji intact.<br class=""> Other methods such as `indexOf` and `contains` make obvious sense. A few<br class=""> `Collection` methods, like `min` and `max`, may not be particularly useful<br class=""> on `String`, but we don't consider that to be a problem worth solving, in<br class=""> the same way that we wouldn't try to suppress `min` and `max` on a<br class=""> `Set([UInt8])` that was used to store IP addresses.<br class=""><br class=""> * Many of the higher-level operations that we want to provide for `String`s,<br class=""> such as parsing and pattern matching, should apply to any `Collection`, and<br class=""> many of the benefits we want for `Collections`, such<br class=""> as unified slicing, should accrue<br class=""> equally to `String`. Making `String` part of the same protocol hierarchy<br class=""> allows us to write these operations once and not worry about keeping the<br class=""> benefits in sync.<br class=""><br class=""> * Slicing strings into substrings is a crucial part of the vocabulary of<br class=""> string processing, and all other sliceable things are `Collection`s.<br class=""> Because of its collection-like behavior, users naturally think of `String`<br class=""> in collection terms, but run into frustrating limitations where it fails to<br class=""> conform and are left to wonder where all the differences lie. Many simply<br class=""> “correct” this limitation by declaring a trivial conformance:<br class=""><br class=""> ```swift<br class=""> extension String : BidirectionalCollection {}<br class=""> ```<br class=""><br class=""> Even if we removed indexing-by-element from `String`, users could still do<br class=""> this:<br class=""><br class=""> ```swift<br class=""> extension String : BidirectionalCollection {<br class=""> subscript(i: Index) -> Character { return characters[i] }<br class=""> }<br class=""> ```<br class=""><br class=""> It would be much better to legitimize the conformance to `Collection` and<br class=""> simply document the oddity of any concatenation corner-cases, than to deny<br class=""> users the benefits on the grounds that a few cases are confusing.<br class=""><br class=""></div></div></blockquote><div><br class=""></div><div>Will String also conform to SequenceType? I’ve seen many users (coming from other languages) confused that they can’t “just” loop over a String’s characters.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">Note that the fact that `String` is a collection of graphemes does *not* mean<br class="">that string operations will necessarily have to do grapheme boundary<br class="">recognition. See the Unicode protocol section for details.<br class=""><br class="">### `Character` and `CharacterSet`<br class=""><br class="">`Character`, which represents a<br class="">Unicode<br class="">[extended grapheme cluster](<a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries" class="">http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries</a>),<br class="">is a bit of a black box, requiring conversion to `String` in order to<br class="">do any introspection, including interoperation with ASCII. To fix this, we should:<br class=""><br class=""> - Add a `unicodeScalars` view much like `String`'s, so that the sub-structure<br class=""> of grapheme clusters is discoverable.<br class=""> - Add a failable `init` from sequences of scalars (returning nil for sequences<br class=""> that contain 0 or 2+ graphemes).<br class=""> - (Lower priority) expose some operations, such as `func uppercase() -><br class=""> String`, `var isASCII: Bool`, and, to the extent they can be sensibly<br class=""> generalized, queries of unicode properties that should also be exposed on<br class=""> `UnicodeScalar` such as `isAlphabetic` and `isGraphemeBase` .<br class=""><br class="">Despite its name, `CharacterSet` currently operates on the Swift `UnicodeScalar`<br class="">type. This means it is usable on `String`, but only by going through the unicode<br class="">scalar view. To deal with this clash in the short term, `CharacterSet` should be<br class="">renamed to `UnicodeScalarSet`. In the longer term, it may be appropriate to<br class="">introduce a `CharacterSet` that provides similar functionality for extended<br class="">grapheme clusters.[5]<br class=""><br class="">### Unification of Slicing Operations<br class=""><br class="">Creating substrings is a basic part of String processing, but the slicing<br class="">operations that we have in Swift are inconsistent in both their spelling and<br class="">their naming: <br class=""><br class=""> * Slices with two explicit endpoints are done with subscript, and support<br class=""> in-place mutation:<br class=""><br class=""> ```swift<br class=""> s[i..<j].mutate()<br class=""> ```<br class=""><br class=""> * Slicing from an index to the end, or from the start to an index, is done<br class=""> with a method and does not support in-place mutation:<br class=""> ```swift<br class=""> s.prefix(upTo: i).readOnly()<br class=""> ```<br class=""><br class="">Prefix and suffix operations should be migrated to be subscripting operations<br class="">with one-sided ranges i.e. `s.prefix(upTo: i)` should become `s[..<i]`, as<br class="">in<br class="">[this proposal](<a href="https://github.com/apple/swift-evolution/blob/9cf2685293108ea3efcbebb7ee6a8618b83d4a90/proposals/0132-sequence-end-ops.md" class="">https://github.com/apple/swift-evolution/blob/9cf2685293108ea3efcbebb7ee6a8618b83d4a90/proposals/0132-sequence-end-ops.md</a>).<br class="">With generic subscripting in the language, that will allow us to collapse a wide<br class="">variety of methods and subscript overloads into a single implementation, and<br class="">give users an easy-to-use and composable way to describe subranges.<br class=""><br class="">Further extending this EDSL to integrate use-cases like `s.prefix(maxLength: 5)`<br class="">is an ongoing research project that can be considered part of the potential<br class="">long-term vision of text (and collection) processing.<br class=""><br class="">### Substrings<br class=""><br class="">When implementing substring slicing, languages are faced with three options:<br class=""><br class="">1. Make the substrings the same type as string, and share storage.<br class="">2. Make the substrings the same type as string, and copy storage when making the substring.<br class="">3. Make substrings a different type, with a storage copy on conversion to string.<br class=""><br class="">We think number 3 is the best choice. A walk-through of the tradeoffs follows.<br class=""><br class="">#### Same type, shared storage<br class=""><br class="">In Swift 3.0, slicing a `String` produces a new `String` that is a view into a<br class="">subrange of the original `String`'s storage. This is why `String` is 3 words in<br class="">size (the start, length and buffer owner), unlike the similar `Array` type<br class="">which is only one.<br class=""><br class="">This is a simple model with big efficiency gains when chopping up strings into<br class="">multiple smaller strings. But it does mean that a stored substring keeps the<br class="">entire original string buffer alive even after it would normally have been<br class="">released.<br class=""><br class="">This arrangement has proven to be problematic in other programming languages,<br class="">because applications sometimes extract small strings from large ones and keep<br class="">those small strings long-term. That is considered a memory leak and was enough<br class="">of a problem in Java that they changed from substrings sharing storage to<br class="">making a copy in 1.7.<br class=""><br class="">#### Same type, copied storage<br class=""><br class="">Copying of substrings is also the choice made in C#, and in the default<br class="">`NSString` implementation. This approach avoids the memory leak issue, but has<br class="">obvious performance overhead in performing the copies.<br class=""><br class="">This in turn encourages trafficking in string/range pairs instead of in<br class="">substrings, for performance reasons, leading to API challenges. For example:<br class=""><br class="">```swift<br class="">foo.compare(bar, range: start..<end)<br class="">```<br class=""><br class="">Here, it is not clear whether `range` applies to `foo` or `bar`. This<br class="">relationship is better expressed in Swift as a slicing operation:<br class=""><br class="">```swift<br class="">foo[start..<end].compare(bar)<br class="">```<br class=""><br class="">Not only does this clarify to which string the range applies, it also brings<br class="">this sub-range capability to any API that operates on `String` "for free". So<br class="">these other combinations also work equally well:<br class=""><br class="">```swift<br class="">// apply range on argument rather than target<br class="">foo.compare(bar[start..<end])<br class="">// apply range on both<br class="">foo[start..<end].compare(bar[start1..<end1])<br class="">// compare two strings ignoring first character<br class="">foo.dropFirst().compare(bar.dropFirst())<br class="">```<br class=""><br class="">In all three cases, an explicit range argument need not appear on the `compare`<br class="">method itself. The implementation of `compare` does not need to know anything<br class="">about ranges. Methods need only take range arguments when that was an<br class="">integral part of their purpose (for example, setting the start and end of a<br class="">user's current selection in a text box).<br class=""><br class="">#### Different type, shared storage<br class=""><br class="">The desire to share underlying storage while preventing accidental memory leaks<br class="">occurs with slices of `Array`. For this reason we have an `ArraySlice` type.<br class="">The inconvenience of a separate type is mitigated by most operations used on<br class="">`Array` from the standard library being generic over `Sequence` or `Collection`.<br class=""><br class="">We should apply the same approach for `String` by introducing a distinct<br class="">`SubSequence` type, `Substring`. Similar advice given for `ArraySlice` would apply to `Substring`:<br class=""><br class=""><blockquote type="cite" class="">Important: Long-term storage of `Substring` instances is discouraged. A<br class="">substring holds a reference to the entire storage of a larger string, not<br class="">just to the portion it presents, even after the original string's lifetime<br class="">ends. Long-term storage of a `Substring` may therefore prolong the lifetime<br class="">of large strings that are no longer otherwise accessible, which can appear<br class="">to be memory leakage.<br class=""></blockquote><br class="">When assigning a `Substring` to a longer-lived variable (usually a stored<br class="">property) explicitly of type `String`, a type conversion will be performed, and<br class="">at this point the substring buffer is copied and the original string's storage<br class="">can be released.<br class=""><br class="">A `String` that was not its own `Substring` could be one word—a single tagged<br class="">pointer—without requiring additional allocations. `Substring`s would be a view<br class="">onto a `String`, so are 3 words - pointer to owner, pointer to start, and a<br class="">length. The small string optimization for `Substring` would take advantage of<br class="">the larger size, probably with a less compressed encoding for speed.<br class=""><br class="">The downside of having two types is the inconvenience of sometimes having a<br class="">`Substring` when you need a `String`, and vice-versa. It is likely this would<br class="">be a significantly bigger problem than with `Array` and `ArraySlice`, as<br class="">slicing of `String` is such a common operation. It is especially relevant to<br class="">existing code that assumes `String` is the currency type. To ease the pain of<br class="">type mismatches, `Substring` should be a subtype of `String` in the same way<br class="">that `Int` is a subtype of `Optional<Int>`. This would give users an implicit<br class="">conversion from `Substring` to `String`, as well as the usual implicit<br class="">conversions such as `[Substring]` to `[String]` that other subtype<br class="">relationships receive.<br class=""><br class="">In most cases, type inference combined with the subtype relationship should<br class="">make the type difference a non-issue and users will not care which type they<br class="">are using. For flexibility and optimizability, most operations from the<br class="">standard library will traffic in generic models of<br class="">[`Unicode`](#the--code-unicode--code--protocol).<br class=""><br class="">##### Guidance for API Designers<br class=""><br class="">In this model, **if a user is unsure about which type to use, `String` is always<br class="">a reasonable default**. A `Substring` passed where `String` is expected will be<br class="">implicitly copied. When compared to the “same type, copied storage” model, we<br class="">have effectively deferred the cost of copying from the point where a substring<br class="">is created until it must be converted to `String` for use with an API.<br class=""><br class="">A user who needs to optimize away copies altogether should use this guideline:<br class="">if for performance reasons you are tempted to add a `Range` argument to your<br class="">method as well as a `String` to avoid unnecessary copies, you should instead<br class="">use `Substring`.<br class=""><br class="">##### The “Empty Subscript”<br class=""><br class="">To make it easy to call such an optimized API when you only have a `String` (or<br class="">to call any API that takes a `Collection`'s `SubSequence` when all you have is<br class="">the `Collection`), we propose the following “empty subscript” operation,<br class=""><br class="">```swift<br class="">extension Collection {<br class=""> subscript() -> SubSequence { <br class=""> return self[startIndex..<endIndex] <br class=""> }<br class="">}<br class="">```<br class=""><br class="">which allows the following usage:<br class=""><br class="">```swift<br class="">funcThatIsJustLooking(at: person.name[]) // pass person.name as Substring<br class="">```<br class=""><br class="">The `[]` syntax can be offered as a fixit when needed, similar to `&` for an<br class="">`inout` argument. While it doesn't help a user to convert `[String]` to<br class="">`[Substring]`, the need for such conversions is extremely rare, can be done with<br class="">a simple `map` (which could also be offered by a fixit):<br class=""><br class="">```swift<br class="">takesAnArrayOfSubstring(arrayOfString.map { $0[] })<br class="">```<br class=""><br class="">#### Other Options Considered<br class=""><br class="">As we have seen, all three options above have downsides, but it's possible<br class="">these downsides could be eliminated/mitigated by the compiler. We are proposing<br class="">one such mitigation—implicit conversion—as part of the the "different type,<br class="">shared storage" option, to help avoid the cognitive load on developers of<br class="">having to deal with a separate `Substring` type.<br class=""><br class="">To avoid the memory leak issues of a "same type, shared storage" substring<br class="">option, we considered whether the compiler could perform an implicit copy of<br class="">the underlying storage when it detects the string is being "stored" for long<br class="">term usage, say when it is assigned to a stored property. The trouble with this<br class="">approach is it is very difficult for the compiler to distinguish between<br class="">long-term storage versus short-term in the case of abstractions that rely on<br class="">stored properties. For example, should the storing of a substring inside an<br class="">`Optional` be considered long-term? Or the storing of multiple substrings<br class="">inside an array? The latter would not work well in the case of a<br class="">`components(separatedBy:)` implementation that intended to return an array of<br class="">substrings. It would also be difficult to distinguish intentional medium-term<br class="">storage of substrings, say by a lexer. There does not appear to be an effective<br class="">consistent rule that could be applied in the general case for detecting when a<br class="">substring is truly being stored long-term.<br class=""><br class="">To avoid the cost of copying substrings under "same type, copied storage", the<br class="">optimizer could be enhanced to to reduce the impact of some of those copies.<br class="">For example, this code could be optimized to pull the invariant substring out<br class="">of the loop:<br class=""><br class="">```swift<br class="">for _ in 0..<lots { <br class=""> someFunc(takingString: bigString[bigRange]) <br class="">}<br class="">```<br class=""><br class="">It's worth noting that a similar optimization is needed to avoid an equivalent<br class="">problem with implicit conversion in the "different type, shared storage" case:<br class=""><br class="">```swift<br class="">let substring = bigString[bigRange]<br class="">for _ in 0..<lots { someFunc(takingString: substring) }<br class="">```<br class=""><br class="">However, in the case of "same type, copied storage" there are many use cases<br class="">that cannot be optimized as easily. Consider the following simple definition of<br class="">a recursive `contains` algorithm, which when substring slicing is linear makes<br class="">the overall algorithm quadratic:<br class=""><br class="">```swift<br class="">extension String {<br class=""> func containsChar(_ x: Character) -> Bool {<br class=""> return !isEmpty && (first == x || dropFirst().containsChar(x))<br class=""> }<br class="">}<br class="">```<br class=""><br class="">For the optimizer to eliminate this problem is unrealistic, forcing the user to<br class="">remember to optimize the code to not use string slicing if they want it to be<br class="">efficient (assuming they remember):<br class=""><br class="">```swift<br class="">extension String {<br class=""> // add optional argument tracking progress through the string<br class=""> func containsCharacter(_ x: Character, atOrAfter idx: Index? = nil) -> Bool {<br class=""> let idx = idx ?? startIndex<br class=""> return idx != endIndex<br class=""> && (self[idx] == x || containsCharacter(x, atOrAfter: index(after: idx)))<br class=""> }<br class="">}<br class="">```<br class=""><br class="">#### Substrings, Ranges and Objective-C Interop<br class=""><br class="">The pattern of passing a string/range pair is common in several Objective-C<br class="">APIs, and is made especially awkward in Swift by the non-interchangeability of<br class="">`Range<String.Index>` and `NSRange`. <br class=""><br class="">```swift<br class="">s2.find(s2, sourceRange: NSRange(j..<s2.endIndex, in: s2))<br class="">```<br class=""><br class="">In general, however, the Swift idiom for operating on a sub-range of a<br class="">`Collection` is to *slice* the collection and operate on that:<br class=""><br class="">```swift<br class="">s2.find(s2[j..<s2.endIndex])<br class="">```<br class=""><br class="">Therefore, APIs that operate on an `NSString`/`NSRange` pair should be imported<br class="">without the `NSRange` argument. The Objective-C importer should be changed to<br class="">give these APIs special treatment so that when a `Substring` is passed, instead<br class="">of being converted to a `String`, the full `NSString` and range are passed to<br class="">the Objective-C method, thereby avoiding a copy.<br class=""><br class="">As a result, you would never need to pass an `NSRange` to these APIs, which<br class="">solves the impedance problem by eliminating the argument, resulting in more<br class="">idiomatic Swift code while retaining the performance benefit. To help users<br class="">manually handle any cases that remain, Foundation should be augmented to allow<br class="">the following syntax for converting to and from `NSRange`:<br class=""><br class="">```swift<br class="">let nsr = NSRange(i..<j, in: s) // An NSRange corresponding to s[i..<j]<br class="">let iToJ = Range(nsr, in: s) // Equivalent to i..<j<br class="">```<br class=""><br class="">### The `Unicode` protocol<br class=""><br class="">With `Substring` and `String` being distinct types and sharing almost all<br class="">interface and semantics, and with the highest-performance string processing<br class="">requiring knowledge of encoding and layout that the currency types can't<br class="">provide, it becomes important to capture the common “string API” in a protocol.<br class="">Since Unicode conformance is a key feature of string processing in swift, we<br class="">call that protocol `Unicode`:<br class=""></div></div></blockquote><div><br class=""></div><div>Another minor typo: capitalize “Swift"</div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class="">**Note:** The following assumes several features that are planned but not yet implemented in<br class=""> Swift, and should be considered a sketch rather than a final design.<br class=""><br class="">```swift<br class="">protocol Unicode <br class=""> : Comparable, BidirectionalCollection where Element == Character {<br class=""><br class=""> associatedtype Encoding : UnicodeEncoding<br class=""> var encoding: Encoding { get }<br class=""><br class=""> associatedtype CodeUnits <br class=""> : RandomAccessCollection where Element == Encoding.CodeUnit<br class=""> var codeUnits: CodeUnits { get }<br class=""><br class=""> associatedtype UnicodeScalars <br class=""> : BidirectionalCollection where Element == UnicodeScalar<br class=""> var unicodeScalars: UnicodeScalars { get }<br class=""><br class=""> associatedtype ExtendedASCII <br class=""> : BidirectionalCollection where Element == UInt32<br class=""> var extendedASCII: ExtendedASCII { get }<br class=""><br class=""> var unicodeScalars: UnicodeScalars { get }<br class="">}<br class=""><br class="">extension Unicode {<br class=""> // ... define high-level non-mutating string operations, e.g. search ...<br class=""><br class=""> func compared<Other: Unicode>(<br class=""> to rhs: Other,<br class=""> case caseSensitivity: StringSensitivity? = nil,<br class=""> diacritic diacriticSensitivity: StringSensitivity? = nil,<br class=""> width widthSensitivity: StringSensitivity? = nil,<br class=""> in locale: Locale? = nil<br class=""> ) -> SortOrder { ... }<br class="">}<br class=""><br class="">extension Unicode : RangeReplaceableCollection where CodeUnits :<br class=""> RangeReplaceableCollection {<br class=""> // Satisfy protocol requirement<br class=""> mutating func replaceSubrange<C : Collection>(_: Range<Index>, with: C) <br class=""> where C.Element == Element<br class=""><br class=""> // ... define high-level mutating string operations, e.g. replace ...<br class="">}<br class=""><br class="">```<br class=""><br class="">The goal is that `Unicode` exposes the underlying encoding and code units in<br class="">such a way that for types with a known representation (e.g. a high-performance<br class="">`UTF8String`) that information can be known at compile-time and can be used to<br class="">generate a single path, while still allowing types like `String` that admit<br class="">multiple representations to use runtime queries and branches to fast path<br class="">specializations.<br class=""><br class="">**Note:** `Unicode` would make a fantastic namespace for much of<br class="">what's in this proposal if we could get the ability to nest types and<br class="">protocols in protocols.<br class=""><br class=""><br class="">### Scanning, Matching, and Tokenization<br class=""><br class="">#### Low-Level Textual Analysis<br class=""><br class="">We should provide convenient APIs processing strings by character. For example,<br class="">it should be easy to cleanly express, “if this string starts with `"f"`, process<br class="">the rest of the string as follows…” Swift is well-suited to expressing this<br class="">common pattern beautifully, but we need to add the APIs. Here are two examples<br class="">of the sort of code that might be possible given such APIs:<br class=""><br class="">```swift<br class="">if let firstLetter = input.droppingPrefix(alphabeticCharacter) {<br class=""> somethingWith(input) // process the rest of input<br class="">}<br class=""><br class="">if let (number, restOfInput) = input.parsingPrefix(Int.self) {<br class=""> ...<br class="">}<br class="">```<br class=""><br class="">The specific spelling and functionality of APIs like this are TBD. The larger<br class="">point is to make sure matching-and-consuming jobs are well-supported.<br class=""><br class=""></div></div></blockquote><div><br class=""></div><div>+100, this kind of work is currently quite painful in Swift. Looking forward to seeing this implemented!</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">#### Unified Pattern Matcher Protocol<br class=""><br class="">Many of the current methods that do matching are overloaded to do the same<br class="">logical operations in different ways, with the following axes:<br class=""><br class="">- Logical Operation: `find`, `split`, `replace`, match at start<br class="">- Kind of pattern: `CharacterSet`, `String`, a regex, a closure<br class="">- Options, e.g. case/diacritic sensitivity, locale. Sometimes a part of<br class=""> the method name, and sometimes an argument<br class="">- Whole string or subrange.<br class=""><br class="">We should represent these aspects as orthogonal, composable components,<br class="">abstracting pattern matchers into a protocol like<br class="">[this one](<a href="https://github.com/apple/swift/blob/master/test/Prototypes/PatternMatching.swift#L33" class="">https://github.com/apple/swift/blob/master/test/Prototypes/PatternMatching.swift#L33</a>),<br class="">that can allow us to define logical operations once, without introducing<br class="">overloads, and massively reducing API surface area.<br class=""><br class="">For example, using the strawman prefix `%` syntax to turn string literals into<br class="">patterns, the following pairs would all invoke the same generic methods:<br class=""><br class="">```swift<br class="">if let found = s.firstMatch(%"searchString") { ... }<br class="">if let found = s.firstMatch(someRegex) { ... }<br class=""><br class="">for m in s.allMatches((%"searchString"), case: .insensitive) { ... }<br class="">for m in s.allMatches(someRegex) { ... }<br class=""><br class="">let items = s.split(separatedBy: ", ")<br class="">let tokens = s.split(separatedBy: CharacterSet.whitespace)<br class="">```<br class=""><br class="">Note that, because Swift requires the indices of a slice to match the indices of<br class="">the range from which it was sliced, operations like `firstMatch` can return a<br class="">`Substring?` in lieu of a `Range<String.Index>?`: the indices of the match in<br class="">the string being searched, if needed, can easily be recovered as the<br class="">`startIndex` and `endIndex` of the `Substring`.<br class=""><br class="">Note also that matching operations are useful for collections in general, and<br class="">would fall out of this proposal:<br class=""><br class="">```<br class="">// replace subsequences of contiguous NaNs with zero<br class="">forces.replace(oneOrMore([Float.nan]), [0.0])<br class="">```<br class=""><br class="">#### Regular Expressions<br class=""><br class="">Addressing regular expressions is out of scope for this proposal.<br class="">That said, it is important that to note the pattern matching protocol mentioned<br class="">above provides a suitable foundation for regular expressions, and types such as<br class="">`NSRegularExpression` can easily be retrofitted to conform to it. In the<br class="">future, support for regular expression literals in the compiler could allow for<br class="">compile-time syntax checking and optimization.<br class=""><br class="">### String Indices<br class=""><br class="">`String` currently has four views—`characters`, `unicodeScalars`, `utf8`, and<br class="">`utf16`—each with its own opaque index type. The APIs used to translate indices<br class="">between views add needless complexity, and the opacity of indices makes them<br class="">difficult to serialize.<br class=""><br class="">The index translation problem has two aspects:<br class=""><br class=""> 1. `String` views cannot consume one anothers' indices without a cumbersome<br class=""> conversion step. An index into a `String`'s `characters` must be translated<br class=""> before it can be used as a position in its `unicodeScalars`. Although these<br class=""> translations are rarely needed, they add conceptual and API complexity.<br class=""> 2. Many APIs in the core libraries and other frameworks still expose `String`<br class=""> positions as `Int`s and regions as `NSRange`s, which can only reference a<br class=""> `utf16` view and interoperate poorly with `String` itself.<br class=""><br class="">#### Index Interchange Among Views<br class=""><br class="">String's need for flexible backing storage and reasonably-efficient indexing<br class="">(i.e. without dynamically allocating and reference-counting the indices<br class="">themselves) means indices need an efficient underlying storage type. Although<br class="">we do not wish to expose `String`'s indices *as* integers, `Int` offsets into<br class="">underlying code unit storage makes a good underlying storage type, provided<br class="">`String`'s underlying storage supports random-access. We think random-access<br class="">*code-unit storage* is a reasonable requirement to impose on all `String`<br class="">instances.<br class=""><br class="">Making these `Int` code unit offsets conveniently accessible and constructible<br class="">solves the serialization problem:<br class=""><br class="">```swift<br class="">clipboard.write(s.endIndex.codeUnitOffset)<br class="">let offset = clipboard.read(Int.self)<br class="">let i = String.Index(codeUnitOffset: offset)<br class="">```<br class=""><br class="">Index interchange between `String` and its `unicodeScalars`, `codeUnits`,<br class="">and [`extendedASCII`](#parsing-ascii-structure) views can be made entirely<br class="">seamless by having them share an index type (semantics of indexing a `String`<br class="">between grapheme cluster boundaries are TBD—it can either trap or be forgiving).<br class="">Having a common index allows easy traversal into the interior of graphemes,<br class="">something that is often needed, without making it likely that someone will do it<br class="">by accident.<br class=""><br class=""> - `String.index(after:)` should advance to the next grapheme, even when the<br class=""> index points partway through a grapheme.<br class=""><br class=""> - `String.index(before:)` should move to the start of the grapheme before<br class=""> the current position.<br class=""><br class="">Seamless index interchange between `String` and its UTF-8 or UTF-16 views is not<br class="">crucial, as the specifics of encoding should not be a concern for most use<br class="">cases, and would impose needless costs on the indices of other views. That<br class="">said, we can make translation much more straightforward by exposing simple<br class="">bidirectional converting `init`s on both index types:<br class=""><br class="">```swift<br class="">let u8Position = String.UTF8.Index(someStringIndex)<br class="">let originalPosition = String.Index(u8Position)<br class="">```<br class=""><br class="">#### Index Interchange with Cocoa<br class=""><br class="">We intend to address `NSRange`s that denote substrings in Cocoa APIs as<br class="">described [later in this document](#substrings--ranges-and-objective-c-interop).<br class="">That leaves the interchange of bare indices with Cocoa APIs trafficking in<br class="">`Int`. Hopefully such APIs will be rare, but when needed, the following<br class="">extension, which would be useful for all `Collections`, can help:<br class=""><br class="">```swift<br class="">extension Collection {<br class=""> func index(offset: IndexDistance) -> Index {<br class=""> return index(startIndex, offsetBy: offset)<br class=""> }<br class=""> func offset(of i: Index) -> IndexDistance {<br class=""> return distance(from: startIndex, to: i)<br class=""> }<br class="">}<br class="">```<br class=""><br class="">Then integers can easily be translated into offsets into a `String`'s `utf16`<br class="">view for consumption by Cocoa:<br class=""><br class="">```swift<br class="">let cocoaIndex = s.utf16.offset(of: String.UTF16Index(i))<br class="">let swiftIndex = s.utf16.index(offset: cocoaIndex)<br class="">```<br class=""><br class="">### Formatting<br class=""><br class="">A full treatment of formatting is out of scope of this proposal, but<br class="">we believe it's crucial for completing the text processing picture. This<br class="">section details some of the existing issues and thinking that may guide future<br class="">development.<br class=""><br class="">#### Printf-Style Formatting<br class=""><br class="">`String.format` is designed on the `printf` model: it takes a format string with<br class="">textual placeholders for substitution, and an arbitrary list of other arguments.<br class="">The syntax and meaning of these placeholders has a long history in<br class="">C, but for anyone who doesn't use them regularly they are cryptic and complex,<br class="">as the `printf (3)` man page attests.<br class=""><br class="">Aside from complexity, this style of API has two major problems: First, the<br class="">spelling of these placeholders must match up to the types of the arguments, in<br class="">the right order, or the behavior is undefined. Some limited support for<br class="">compile-time checking of this correspondence could be implemented, but only for<br class="">the cases where the format string is a literal. Second, there's no reasonable<br class="">way to extend the formatting vocabulary to cover the needs of new types: you are<br class="">stuck with what's in the box.<br class=""><br class="">#### Foundation Formatters<br class=""><br class="">The formatters supplied by Foundation are highly capable and versatile, offering<br class="">both formatting and parsing services. When used for formatting, though, the<br class="">design pattern demands more from users than it should:<br class=""><br class=""> * Matching the type of data being formatted to a formatter type<br class=""> * Creating an instance of that type<br class=""> * Setting stateful options (`currency`, `dateStyle`) on the type. Note: the<br class=""> need for this step prevents the instance from being used and discarded in<br class=""> the same expression where it is created.<br class=""> * Overall, introduction of needless verbosity into source<br class=""><br class="">These may seem like small issues, but the experience of Apple localization<br class="">experts is that the total drag of these factors on programmers is such that they<br class="">tend to reach for `String.format` instead.<br class=""><br class="">#### String Interpolation<br class=""><br class="">Swift string interpolation provides a user-friendly alternative to printf's<br class="">domain-specific language (just write ordinary swift code!) and its type safety<br class="">problems (put the data right where it belongs!) but the following issues prevent<br class="">it from being useful for localized formatting (among other jobs):<br class=""><br class=""> * [SR-2303](<a href="https://bugs.swift.org/browse/SR-2303" class="">https://bugs.swift.org/browse/SR-2303</a>) We are unable to restrict<br class=""> types used in string interpolation.<br class=""> * [SR-1260](<a href="https://bugs.swift.org/browse/SR-1260" class="">https://bugs.swift.org/browse/SR-1260</a>) String interpolation can't<br class=""> distinguish (fragments of) the base string from the string substitutions.<br class=""><br class="">In the long run, we should improve Swift string interpolation to the point where<br class="">it can participate in most any formatting job. Mostly this centers around<br class="">fixing the interpolation protocols per the previous item, and supporting<br class="">localization.<br class=""><br class="">To be able to use formatting effectively inside interpolations, it needs to be<br class="">both lightweight (because it all happens in-situ) and discoverable. One <br class="">approach would be to standardize on `format` methods, e.g.:<br class=""><br class="">```swift<br class="">"Column 1: \(n.format(radix:16, width:8)) *** \(message)"<br class=""><br class="">"Something with leading zeroes: \(x.format(fill: zero, width:8))"<br class="">```<br class=""></div></div></blockquote><div><br class=""></div><div>Another thing that might limit adoption is the verbosity of this format. It works fine if I need to print one or two things, but it gets unwieldy very quickly.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div class=""><br class="">### C String Interop<br class=""><br class="">Our support for interoperation with nul-terminated C strings is scattered and<br class="">incoherent, with 6 ways to transform a C string into a `String` and four ways to<br class="">do the inverse. These APIs should be replaced with the following<br class=""><br class="">```swift<br class="">extension String {<br class=""> /// Constructs a `String` having the same contents as `nulTerminatedUTF8`.<br class=""> ///<br class=""> /// - Parameter nulTerminatedUTF8: a sequence of contiguous UTF-8 encoded <br class=""> /// bytes ending just before the first zero byte (NUL character).<br class=""> init(cString nulTerminatedUTF8: UnsafePointer<CChar>)<br class=""><br class=""> /// Constructs a `String` having the same contents as `nulTerminatedCodeUnits`.<br class=""> ///<br class=""> /// - Parameter nulTerminatedCodeUnits: a sequence of contiguous code units in<br class=""> /// the given `encoding`, ending just before the first zero code unit.<br class=""> /// - Parameter encoding: describes the encoding in which the code units<br class=""> /// should be interpreted.<br class=""> init<Encoding: UnicodeEncoding>(<br class=""> cString nulTerminatedCodeUnits: UnsafePointer<Encoding.CodeUnit>,<br class=""> encoding: Encoding)<br class=""><br class=""> /// Invokes the given closure on the contents of the string, represented as a<br class=""> /// pointer to a null-terminated sequence of UTF-8 code units.<br class=""> func withCString<Result>(<br class=""> _ body: (UnsafePointer<CChar>) throws -> Result) rethrows -> Result<br class="">}<br class="">```<br class=""><br class="">In both of the construction APIs, any invalid encoding sequence detected will<br class="">have its longest valid prefix replaced by U+FFFD, the Unicode replacement<br class="">character, per Unicode specification. This covers the common case. The<br class="">replacement is done *physically* in the underlying storage and the validity of<br class="">the result is recorded in the `String`'s `encoding` such that future accesses<br class="">need not be slowed down by possible error repair separately.<br class=""><br class="">Construction that is aborted when encoding errors are detected can be<br class="">accomplished using APIs on the `encoding`. String types that retain their<br class="">physical encoding even in the presence of errors and are repaired on-the-fly can<br class="">be built as different instances of the `Unicode` protocol.<br class=""><br class="">### Unicode 9 Conformance<br class=""><br class="">Unicode 9 (and MacOS 10.11) brought us support for family emoji, which changes<br class="">the process of properly identifying `Character` boundaries. We need to update<br class="">`String` to account for this change.<br class=""><br class="">### High-Performance String Processing<br class=""><br class="">Many strings are short enough to store in 64 bits, many can be stored using only<br class="">8 bits per unicode scalar, others are best encoded in UTF-16, and some come to<br class="">us already in some other encoding, such as UTF-8, that would be costly to<br class="">translate. Supporting these formats while maintaining usability for<br class="">general-purpose APIs demands that a single `String` type can be backed by many<br class="">different representations.<br class=""><br class="">That said, the highest performance code always requires static knowledge of the<br class="">data structures on which it operates, and for this code, dynamic selection of<br class="">representation comes at too high a cost. Heavy-duty text processing demands a<br class="">way to opt out of dynamism and directly use known encodings. Having this<br class="">ability can also make it easy to cleanly specialize code that handles dynamic<br class="">cases for maximal efficiency on the most common representations.<br class=""><br class="">To address this need, we can build models of the `Unicode` protocol that encode<br class="">representation information into the type, such as `NFCNormalizedUTF16String`.<br class=""><br class="">### Parsing ASCII Structure<br class=""><br class="">Although many machine-readable formats support the inclusion of arbitrary<br class="">Unicode text, it is also common that their fundamental structure lies entirely<br class="">within the ASCII subset (JSON, YAML, many XML formats). These formats are often<br class="">processed most efficiently by recognizing ASCII structural elements as ASCII,<br class="">and capturing the arbitrary sections between them in more-general strings. The<br class="">current String API offers no way to efficiently recognize ASCII and skip past<br class="">everything else without the overhead of full decoding into unicode scalars.<br class=""><br class="">For these purposes, strings should supply an `extendedASCII` view that is a<br class="">collection of `UInt32`, where values less than `0x80` represent the<br class="">corresponding ASCII character, and other values represent data that is specific<br class="">to the underlying encoding of the string.<br class=""></div></div></blockquote><div><br class=""></div><div>There are some things that are know to lie entirely with ASCII–are there any plans to add a way to work with them in a simple manner (subscripting, looping, etc.), possibly through the use of a Array<ASCIIChar>? property or whatever?</div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class="">## Language Support<br class=""><br class="">This proposal depends on two new features in the Swift language:<br class=""><br class="">1. **Generic subscripts**, to<br class=""> enable unified slicing syntax.<br class=""><br class="">2. **A subtype relationship** between<br class=""> `Substring` and `String`, enabling framework APIs to traffic solely in<br class=""> `String` while still making it possible to avoid copies by handling<br class=""> `Substring`s where necessary.<br class=""><br class="">Additionally, **the ability to nest types and protocols inside<br class="">protocols** could significantly shrink the footprint of this proposal<br class="">on the top-level Swift namespace.<br class=""><br class=""><br class="">## Open Questions<br class=""><br class="">### Must `String` be limited to storing UTF-16 subset encodings?<br class=""><br class="">- The ability to handle `UTF-8`-encoded strings (models of `Unicode`) is not in<br class=""> question here; this is about what encodings must be storable, without<br class=""> transcoding, in the common currency type called “`String`”.<br class="">- ASCII, Latin-1, UCS-2, and UTF-16 are UTF-16 subsets. UTF-8 is not.<br class="">- If we have a way to get at a `String`'s code units, we need a concrete type in<br class=""> which to express them in the API of `String`, which is a concrete type<br class="">- If String needs to be able to represent UTF-32, presumably the code units need<br class=""> to be `UInt32`.<br class="">- Not supporting UTF-32-encoded text seems like one reasonable design choice.<br class="">- Maybe we can allow UTF-8 storage in `String` and expose its code units as<br class=""> `UInt16`, just as we would for Latin-1.<br class="">- Supporting only UTF-16-subset encodings would imply that `String` indices can<br class=""> be serialized without recording the `String`'s underlying encoding.<br class=""><br class="">### Do we need a type-erasable base protocol for UnicodeEncoding?<br class=""><br class="">UnicodeEncoding has an associated type, but it may be important to be able to<br class="">traffic in completely dynamic encoding values, e.g. for “tell me the most<br class="">efficient encoding for this string.”<br class=""><br class="">### Should there be a string “facade?”<br class=""><br class="">One possible design alternative makes `Unicode` a vehicle for expressing<br class="">the storage and encoding of code units, but does not attempt to give it an API<br class="">appropriate for `String`. Instead, string APIs would be provided by a generic<br class="">wrapper around an instance of `Unicode`:<br class=""><br class="">```swift<br class="">struct StringFacade<U: Unicode> : BidirectionalCollection {<br class=""><br class=""> // ...APIs for high-level string processing here...<br class=""><br class=""> var unicode: U // access to lower-level unicode details<br class="">}<br class=""><br class="">typealias String = StringFacade<StringStorage><br class="">typealias Substring = StringFacade<StringStorage.SubSequence><br class="">```<br class=""><br class="">This design would allow us to de-emphasize lower-level `String` APIs such as<br class="">access to the specific encoding, by putting them behind a `.unicode` property.<br class="">A similar effect in a facade-less design would require a new top-level<br class="">`StringProtocol` playing the role of the facade with an an `associatedtype<br class="">Storage : Unicode`.<br class=""><br class="">An interesting variation on this design is possible if defaulted generic<br class="">parameters are introduced to the language:<br class=""><br class="">```swift<br class="">struct String<U: Unicode = StringStorage> <br class=""> : BidirectionalCollection {<br class=""><br class=""> // ...APIs for high-level string processing here...<br class=""><br class=""> var unicode: U // access to lower-level unicode details<br class="">}<br class=""><br class="">typealias Substring = String<StringStorage.SubSequence><br class="">```<br class=""><br class="">One advantage of such a design is that naïve users will always extend “the right<br class="">type” (`String`) without thinking, and the new APIs will show up on `Substring`,<br class="">`MyUTF8String`, etc. That said, it also has downsides that should not be<br class="">overlooked, not least of which is the confusability of the meaning of the word<br class="">“string.” Is it referring to the generic or the concrete type?<br class=""><br class="">### `TextOutputStream` and `TextOutputStreamable`<br class=""><br class="">`TextOutputStreamable` is intended to provide a vehicle for<br class="">efficiently transporting formatted representations to an output stream<br class="">without forcing the allocation of storage. Its use of `String`, a<br class="">type with multiple representations, at the lowest-level unit of<br class="">communication, conflicts with this goal. It might be sufficient to<br class="">change `TextOutputStream` and `TextOutputStreamable` to traffic in an<br class="">associated type conforming to `Unicode`, but that is not yet clear.<br class="">This area will require some design work.<br class=""><br class="">### `description` and `debugDescription`<br class=""><br class="">* Should these be creating localized or non-localized representations?<br class="">* Is returning a `String` efficient enough?<br class="">* Is `debugDescription` pulling the weight of the API surface area it adds?<br class=""><br class="">### `StaticString`<br class=""><br class="">`StaticString` was added as a byproduct of standard library developed and kept<br class="">around because it seemed useful, but it was never truly *designed* for client<br class="">programmers. We need to decide what happens with it. Presumably *something*<br class="">should fill its role, and that should conform to `Unicode`.<br class=""><br class="">## Footnotes<br class=""><br class=""><b id="f0">0</b> The integers rewrite currently underway is expected to<br class=""> substantially reduce the scope of `Int`'s API by using more<br class=""> generics. [↩](#a0)<br class=""><br class=""><b id="f1">1</b> In practice, these semantics will usually be tied to the<br class="">version of the installed [ICU](<a href="http://icu-project.org" class="">http://icu-project.org</a>) library, which<br class="">programmatically encodes the most complex rules of the Unicode Standard and its<br class="">de-facto extension, CLDR.[↩](#a1)<br class=""><br class=""><b id="f2">2</b><br class="">See<br class="">[<a href="http://unicode.org/reports/tr29/#Notation" class="">http://unicode.org/reports/tr29/#Notation</a>](<a href="http://unicode.org/reports/tr29/#Notation" class="">http://unicode.org/reports/tr29/#Notation</a>). Note<br class="">that inserting Unicode scalar values to prevent merging of grapheme clusters would<br class="">also constitute a kind of misbehavior (one of the clusters at the boundary would<br class="">not be found in the result), so would be relatively costly to implement, with<br class="">little benefit. [↩](#a2)<br class=""><br class=""><b id="f4">4</b> The use of non-UCA-compliant ordering is fully sanctioned by<br class=""> the Unicode standard for this purpose. In fact there's<br class=""> a [whole chapter](<a href="http://www.unicode.org/versions/Unicode9.0.0/ch05.pdf" class="">http://www.unicode.org/versions/Unicode9.0.0/ch05.pdf</a>)<br class=""> dedicated to it. In particular, §5.17 says:<br class=""><br class=""><blockquote type="cite" class="">When comparing text that is visible to end users, a correct linguistic sort<br class="">should be used, as described in _Section 5.16, Sorting and<br class="">Searching_. However, in many circumstances the only requirement is for a<br class="">fast, well-defined ordering. In such cases, a binary ordering can be used.<br class=""></blockquote><br class=""> [↩](#a4)<br class=""><br class=""><br class=""><b id="f5">5</b> The queries supported by `NSCharacterSet` map directly onto<br class="">properties in a table that's indexed by unicode scalar value. This table is<br class="">part of the Unicode standard. Some of these queries (e.g., “is this an<br class="">uppercase character?”) may have fairly obvious generalizations to grapheme<br class="">clusters, but exactly how to do it is a research topic and *ideally* we'd either<br class="">establish the existing practice that the Unicode committee would standardize, or<br class="">the Unicode committee would do the research and we'd implement their<br class="">result.[↩](#a5)<br class=""><br class="">_______________________________________________<br class="">swift-evolution mailing list<br class=""><a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-evolution<br class=""></div></div></blockquote></div><br class=""></div></body></html>