Huge thanks for putting this out... a lot to read and digest but great to see.<br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 19, 2017 at 6:56 PM Ben Cohen via swift-evolution <<a href="mailto:swift-evolution@swift.org">swift-evolution@swift.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br class="gmail_msg">
<br class="gmail_msg">
Below is our take on a design manifesto for Strings in Swift 4 and beyond.<br class="gmail_msg">
<br class="gmail_msg">
Probably best read in rendered markdown on GitHub:<br class="gmail_msg">
<a href="https://github.com/apple/swift/blob/master/docs/StringManifesto.md" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/apple/swift/blob/master/docs/StringManifesto.md</a><br class="gmail_msg">
<br class="gmail_msg">
We’re eager to hear everyone’s thoughts.<br class="gmail_msg">
<br class="gmail_msg">
Regards,<br class="gmail_msg">
Ben and Dave<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
# String Processing For Swift 4<br class="gmail_msg">
<br class="gmail_msg">
* Authors: [Dave Abrahams](<a href="https://github.com/dabrahams" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/dabrahams</a>), [Ben Cohen](<a href="https://github.com/airspeedswift" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/airspeedswift</a>)<br class="gmail_msg">
<br class="gmail_msg">
The goal of re-evaluating Strings for Swift 4 has been fairly ill-defined thus<br class="gmail_msg">
far, with just this short blurb in the<br class="gmail_msg">
[list of goals](<a href="https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160725/025676.html" rel="noreferrer" class="gmail_msg" target="_blank">https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160725/025676.html</a>):<br class="gmail_msg">
<br class="gmail_msg">
> **String re-evaluation**: String is one of the most important fundamental<br class="gmail_msg">
> types in the language. The standard library leads have numerous ideas of how<br class="gmail_msg">
> to improve the programming model for it, without jeopardizing the goals of<br class="gmail_msg">
> providing a unicode-correct-by-default model. Our goal is to be better at<br class="gmail_msg">
> string processing than Perl!<br class="gmail_msg">
<br class="gmail_msg">
For Swift 4 and beyond we want to improve three dimensions of text processing:<br class="gmail_msg">
<br class="gmail_msg">
1. Ergonomics<br class="gmail_msg">
2. Correctness<br class="gmail_msg">
3. Performance<br class="gmail_msg">
<br class="gmail_msg">
This document is meant to both provide a sense of the long-term vision<br class="gmail_msg">
(including undecided issues and possible approaches), and to define the scope of<br class="gmail_msg">
work that could be done in the Swift 4 timeframe.<br class="gmail_msg">
<br class="gmail_msg">
## General Principles<br class="gmail_msg">
<br class="gmail_msg">
### Ergonomics<br class="gmail_msg">
<br class="gmail_msg">
It's worth noting that ergonomics and correctness are mutually-reinforcing. An<br class="gmail_msg">
API that is easy to use—but incorrectly—cannot be considered an ergonomic<br class="gmail_msg">
success. Conversely, an API that's simply hard to use is also hard to use<br class="gmail_msg">
correctly. Acheiving optimal performance without compromising ergonomics or<br class="gmail_msg">
correctness is a greater challenge.<br class="gmail_msg">
<br class="gmail_msg">
Consistency with the Swift language and idioms is also important for<br class="gmail_msg">
ergonomics. There are several places both in the standard library and in the<br class="gmail_msg">
foundation additions to `String` where patterns and practices found elsewhere<br class="gmail_msg">
could be applied to improve usability and familiarity.<br class="gmail_msg">
<br class="gmail_msg">
### API Surface Area<br class="gmail_msg">
<br class="gmail_msg">
Primary data types such as `String` should have APIs that are easily understood<br class="gmail_msg">
given a signature and a one-line summary. Today, `String` fails that test. As<br class="gmail_msg">
you can see, the Standard Library and Foundation both contribute significantly to<br class="gmail_msg">
its overall complexity.<br class="gmail_msg">
<br class="gmail_msg">
**Method Arity** | **Standard Library** | **Foundation**<br class="gmail_msg">
---|:---:|:---:<br class="gmail_msg">
0: `ƒ()` | 5 | 7<br class="gmail_msg">
1: `ƒ(:)` | 19 | 48<br class="gmail_msg">
2: `ƒ(::)` | 13 | 19<br class="gmail_msg">
3: `ƒ(:::)` | 5 | 11<br class="gmail_msg">
4: `ƒ(::::)` | 1 | 7<br class="gmail_msg">
5: `ƒ(:::::)` | - | 2<br class="gmail_msg">
6: `ƒ(::::::)` | - | 1<br class="gmail_msg">
<br class="gmail_msg">
**API Kind** | **Standard Library** | **Foundation**<br class="gmail_msg">
---|:---:|:---:<br class="gmail_msg">
`init` | 41 | 18<br class="gmail_msg">
`func` | 42 | 55<br class="gmail_msg">
`subscript` | 9 | 0<br class="gmail_msg">
`var` | 26 | 14<br class="gmail_msg">
<br class="gmail_msg">
**Total: 205 APIs**<br class="gmail_msg">
<br class="gmail_msg">
By contrast, `Int` has 80 APIs, none with more than two parameters.[0] String processing is complex enough; users shouldn't have<br class="gmail_msg">
to press through physical API sprawl just to get started.<br class="gmail_msg">
<br class="gmail_msg">
Many of the choices detailed below contribute to solving this problem,<br class="gmail_msg">
including:<br class="gmail_msg">
<br class="gmail_msg">
* Restoring `Collection` conformance and dropping the `.characters` view.<br class="gmail_msg">
* Providing a more general, composable slicing syntax.<br class="gmail_msg">
* Altering `Comparable` so that parameterized<br class="gmail_msg">
(e.g. case-insensitive) comparison fits smoothly into the basic syntax.<br class="gmail_msg">
* Clearly separating language-dependent operations on text produced<br class="gmail_msg">
by and for humans from language-independent<br class="gmail_msg">
operations on text produced by and for machine processing.<br class="gmail_msg">
* Relocating APIs that fall outside the domain of basic string processing and<br class="gmail_msg">
discouraging the proliferation of ad-hoc extensions.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
### Batteries Included<br class="gmail_msg">
<br class="gmail_msg">
While `String` is available to all programs out-of-the-box, crucial APIs for<br class="gmail_msg">
basic string processing tasks are still inaccessible until `Foundation` is<br class="gmail_msg">
imported. While it makes sense that `Foundation` is needed for domain-specific<br class="gmail_msg">
jobs such as<br class="gmail_msg">
[linguistic tagging](<a href="https://developer.apple.com/reference/foundation/nslinguistictagger" rel="noreferrer" class="gmail_msg" target="_blank">https://developer.apple.com/reference/foundation/nslinguistictagger</a>),<br class="gmail_msg">
one should not need to import anything to, for example, do case-insensitive<br class="gmail_msg">
comparison.<br class="gmail_msg">
<br class="gmail_msg">
### Unicode Compliance and Platform Support<br class="gmail_msg">
<br class="gmail_msg">
The Unicode standard provides a crucial objective reference point for what<br class="gmail_msg">
constitutes correct behavior in an extremely complex domain, so<br class="gmail_msg">
Unicode-correctness is, and will remain, a fundamental design principle behind<br class="gmail_msg">
Swift's `String`. That said, the Unicode standard is an evolving document, so<br class="gmail_msg">
this objective reference-point is not fixed.[1] While<br class="gmail_msg">
many of the most important operations—e.g. string hashing, equality, and<br class="gmail_msg">
non-localized comparison—will be stable, the semantics<br class="gmail_msg">
of others, such as grapheme breaking and localized comparison and case<br class="gmail_msg">
conversion, are expected to change as platforms are updated, so programs should<br class="gmail_msg">
be written so their correctness does not depend on precise stability of these<br class="gmail_msg">
semantics across OS versions or platforms. Although it may be possible to<br class="gmail_msg">
imagine static and/or dynamic analysis tools that will help users find such<br class="gmail_msg">
errors, the only sure way to deal with this fact of life is to educate users.<br class="gmail_msg">
<br class="gmail_msg">
## Design Points<br class="gmail_msg">
<br class="gmail_msg">
### Internationalization<br class="gmail_msg">
<br class="gmail_msg">
There is strong evidence that developers cannot determine how to use<br class="gmail_msg">
internationalization APIs correctly. Although documentation could and should be<br class="gmail_msg">
improved, the sheer size, complexity, and diversity of these APIs is a major<br class="gmail_msg">
contributor to the problem, causing novices to tune out, and more experienced<br class="gmail_msg">
programmers to make avoidable mistakes.<br class="gmail_msg">
<br class="gmail_msg">
The first step in improving this situation is to regularize all localized<br class="gmail_msg">
operations as invocations of normal string operations with extra<br class="gmail_msg">
parameters. Among other things, this means:<br class="gmail_msg">
<br class="gmail_msg">
1. Doing away with `localizedXXX` methods<br class="gmail_msg">
2. Providing a terse way to name the current locale as a parameter<br class="gmail_msg">
3. Automatically adjusting defaults for options such<br class="gmail_msg">
as case sensitivity based on whether the operation is localized.<br class="gmail_msg">
4. Removing correctness traps like `localizedCaseInsensitiveCompare` (see<br class="gmail_msg">
guidance in the<br class="gmail_msg">
[Internationalization and Localization Guide](<a href="https://developer.apple.com/library/content/documentation/MacOSX/Conceptual/BPInternational/InternationalizingYourCode/InternationalizingYourCode.html" rel="noreferrer" class="gmail_msg" target="_blank">https://developer.apple.com/library/content/documentation/MacOSX/Conceptual/BPInternational/InternationalizingYourCode/InternationalizingYourCode.html</a>).<br class="gmail_msg">
<br class="gmail_msg">
Along with appropriate documentation updates, these changes will make localized<br class="gmail_msg">
operations more teachable, comprehensible, and approachable, thereby lowering a<br class="gmail_msg">
barrier that currently leads some developers to ignore localization issues<br class="gmail_msg">
altogether.<br class="gmail_msg">
<br class="gmail_msg">
#### The Default Behavior of `String`<br class="gmail_msg">
<br class="gmail_msg">
Although this isn't well-known, the most accessible form of many operations on<br class="gmail_msg">
Swift `String` (and `NSString`) are really only appropriate for text that is<br class="gmail_msg">
intended to be processed for, and consumed by, machines. The semantics of the<br class="gmail_msg">
operations with the simplest spellings are always non-localized and<br class="gmail_msg">
language-agnostic.<br class="gmail_msg">
<br class="gmail_msg">
Two major factors play into this design choice:<br class="gmail_msg">
<br class="gmail_msg">
1. Machine processing of text is important, so we should have first-class,<br class="gmail_msg">
accessible functions appropriate to that use case.<br class="gmail_msg">
<br class="gmail_msg">
2. The most general localized operations require a locale parameter not required<br class="gmail_msg">
by their un-localized counterparts. This naturally skews complexity towards<br class="gmail_msg">
localized operations.<br class="gmail_msg">
<br class="gmail_msg">
Reaffirming that `String`'s simplest APIs have<br class="gmail_msg">
language-independent/machine-processed semantics has the benefit of clarifying<br class="gmail_msg">
the proper default behavior of operations such as comparison, and allows us to<br class="gmail_msg">
make [significant optimizations](#collation-semantics) that were previously<br class="gmail_msg">
thought to conflict with Unicode.<br class="gmail_msg">
<br class="gmail_msg">
#### Future Directions<br class="gmail_msg">
<br class="gmail_msg">
One of the most common internationalization errors is the unintentional<br class="gmail_msg">
presentation to users of text that has not been localized, but regularizing APIs<br class="gmail_msg">
and improving documentation can go only so far in preventing this error.<br class="gmail_msg">
Combined with the fact that `String` operations are non-localized by default,<br class="gmail_msg">
the environment for processing human-readable text may still be somewhat<br class="gmail_msg">
error-prone in Swift 4.<br class="gmail_msg">
<br class="gmail_msg">
For an audience of mostly non-experts, it is especially important that naïve<br class="gmail_msg">
code is very likely to be correct if it compiles, and that more sophisticated<br class="gmail_msg">
issues can be revealed progressively. For this reason, we intend to<br class="gmail_msg">
specifically and separately target localization and internationalization<br class="gmail_msg">
problems in the Swift 5 timeframe.<br class="gmail_msg">
<br class="gmail_msg">
### Operations With Options<br class="gmail_msg">
<br class="gmail_msg">
There are three categories of common string operation that commonly need to be<br class="gmail_msg">
tuned in various dimensions:<br class="gmail_msg">
<br class="gmail_msg">
**Operation**|**Applicable Options**<br class="gmail_msg">
---|---<br class="gmail_msg">
sort ordering | locale, case/diacritic/width-insensitivity<br class="gmail_msg">
case conversion | locale<br class="gmail_msg">
pattern matching | locale, case/diacritic/width-insensitivity<br class="gmail_msg">
<br class="gmail_msg">
The defaults for case-, diacritic-, and width-insensitivity are different for<br class="gmail_msg">
localized operations than for non-localized operations, so for example a<br class="gmail_msg">
localized sort should be case-insensitive by default, and a non-localized sort<br class="gmail_msg">
should be case-sensitive by default. We propose a standard “language” of<br class="gmail_msg">
defaulted parameters to be used for these purposes, with usage roughly like this:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
x.compared(to: y, case: .sensitive, in: swissGerman)<br class="gmail_msg">
<br class="gmail_msg">
x.lowercased(in: .currentLocale)<br class="gmail_msg">
<br class="gmail_msg">
x.allMatches(<br class="gmail_msg">
somePattern, case: .insensitive, diacritic: .insensitive)<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
This usage might be supported by code like this:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
enum StringSensitivity {<br class="gmail_msg">
case sensitive<br class="gmail_msg">
case insensitive<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
extension Locale {<br class="gmail_msg">
static var currentLocale: Locale { ... }<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
extension Unicode {<br class="gmail_msg">
// An example of the option language in declaration context,<br class="gmail_msg">
// with nil defaults indicating unspecified, so defaults can be<br class="gmail_msg">
// driven by the presence/absence of a specific Locale<br class="gmail_msg">
func frobnicated(<br class="gmail_msg">
case caseSensitivity: StringSensitivity? = nil,<br class="gmail_msg">
diacritic diacriticSensitivity: StringSensitivity? = nil,<br class="gmail_msg">
width widthSensitivity: StringSensitivity? = nil,<br class="gmail_msg">
in locale: Locale? = nil<br class="gmail_msg">
) -> Self { ... }<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
### Comparing and Hashing Strings<br class="gmail_msg">
<br class="gmail_msg">
#### Collation Semantics<br class="gmail_msg">
<br class="gmail_msg">
What Unicode says about collation—which is used in `<`, `==`, and hashing— turns<br class="gmail_msg">
out to be quite interesting, once you pick it apart. The full Unicode Collation<br class="gmail_msg">
Algorithm (UCA) works like this:<br class="gmail_msg">
<br class="gmail_msg">
1. Fully normalize both strings<br class="gmail_msg">
2. Convert each string to a sequence of numeric triples to form a collation key<br class="gmail_msg">
3. “Flatten” the key by concatenating the sequence of first elements to the<br class="gmail_msg">
sequence of second elements to the sequence of third elements<br class="gmail_msg">
4. Lexicographically compare the flattened keys<br class="gmail_msg">
<br class="gmail_msg">
While step 1 can usually<br class="gmail_msg">
be [done quickly](<a href="http://unicode.org/reports/tr15/#Description_Norm" rel="noreferrer" class="gmail_msg" target="_blank">http://unicode.org/reports/tr15/#Description_Norm</a>) and<br class="gmail_msg">
incrementally, step 2 uses a collation table that maps matching *sequences* of<br class="gmail_msg">
unicode scalars in the normalized string to *sequences* of triples, which get<br class="gmail_msg">
accumulated into a collation key. Predictably, this is where the real costs<br class="gmail_msg">
lie.<br class="gmail_msg">
<br class="gmail_msg">
*However*, there are some bright spots to this story. First, as it turns out,<br class="gmail_msg">
string sorting (localized or not) should be done down to what's called<br class="gmail_msg">
the<br class="gmail_msg">
[“identical” level](<a href="http://unicode.org/reports/tr10/#Multi_Level_Comparison" rel="noreferrer" class="gmail_msg" target="_blank">http://unicode.org/reports/tr10/#Multi_Level_Comparison</a>),<br class="gmail_msg">
which adds a step 3a: append the string's normalized form to the flattened<br class="gmail_msg">
collation key. At first blush this just adds work, but consider what it does<br class="gmail_msg">
for equality: two strings that normalize the same, naturally, will collate the<br class="gmail_msg">
same. But also, *strings that normalize differently will always collate<br class="gmail_msg">
differently*. In other words, for equality, it is sufficient to compare the<br class="gmail_msg">
strings' normalized forms and see if they are the same. We can therefore<br class="gmail_msg">
entirely skip the expensive part of collation for equality comparison.<br class="gmail_msg">
<br class="gmail_msg">
Next, naturally, anything that applies to equality also applies to hashing: it<br class="gmail_msg">
is sufficient to hash the string's normalized form, bypassing collation keys.<br class="gmail_msg">
This should provide significant speedups over the current implementation.<br class="gmail_msg">
Perhaps more importantly, since comparison down to the “identical” level applies<br class="gmail_msg">
even to localized strings, it means that hashing and equality can be implemented<br class="gmail_msg">
exactly the same way for localized and non-localized text, and hash tables with<br class="gmail_msg">
localized keys will remain valid across current-locale changes.<br class="gmail_msg">
<br class="gmail_msg">
Finally, once it is agreed that the *default* role for `String` is to handle<br class="gmail_msg">
machine-generated and machine-readable text, the default ordering of `String`s<br class="gmail_msg">
need no longer use the UCA at all. It is sufficient to order them in any way<br class="gmail_msg">
that's consistent with equality, so `String` ordering can simply be a<br class="gmail_msg">
lexicographical comparison of normalized forms,[4]<br class="gmail_msg">
(which is equivalent to lexicographically comparing the sequences of grapheme<br class="gmail_msg">
clusters), again bypassing step 2 and offering another speedup.<br class="gmail_msg">
<br class="gmail_msg">
This leaves us executing the full UCA *only* for localized sorting, and ICU's<br class="gmail_msg">
implementation has apparently been very well optimized.<br class="gmail_msg">
<br class="gmail_msg">
Following this scheme everywhere would also allow us to make sorting behavior<br class="gmail_msg">
consistent across platforms. Currently, we sort `String` according to the UCA,<br class="gmail_msg">
except that—*only on Apple platforms*—pairs of ASCII characters are ordered by<br class="gmail_msg">
unicode scalar value.<br class="gmail_msg">
<br class="gmail_msg">
#### Syntax<br class="gmail_msg">
<br class="gmail_msg">
Because the current `Comparable` protocol expresses all comparisons with binary<br class="gmail_msg">
operators, string comparisons—which may require<br class="gmail_msg">
additional [options](#operations-with-options)—do not fit smoothly into the<br class="gmail_msg">
existing syntax. At the same time, we'd like to solve other problems with<br class="gmail_msg">
comparison, as outlined<br class="gmail_msg">
in<br class="gmail_msg">
[this proposal](<a href="https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e" rel="noreferrer" class="gmail_msg" target="_blank">https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e</a>)<br class="gmail_msg">
(implemented by changes at the head<br class="gmail_msg">
of<br class="gmail_msg">
[this branch](<a href="https://github.com/CodaFi/swift/commits/space-the-final-frontier)" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/CodaFi/swift/commits/space-the-final-frontier)</a>).<br class="gmail_msg">
We should adopt a modification of that proposal that uses a method rather than<br class="gmail_msg">
an operator `<=>`:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
enum SortOrder { case before, same, after }<br class="gmail_msg">
<br class="gmail_msg">
protocol Comparable : Equatable {<br class="gmail_msg">
func compared(to: Self) -> SortOrder<br class="gmail_msg">
...<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
This change will give us a syntactic platform on which to implement methods with<br class="gmail_msg">
additional, defaulted arguments, thereby unifying and regularizing comparison<br class="gmail_msg">
across the library.<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension String {<br class="gmail_msg">
func compared(to: Self) -> SortOrder<br class="gmail_msg">
<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
**Note:** `SortOrder` should bridge to `NSComparisonResult`. It's also possible<br class="gmail_msg">
that the standard library simply adopts Foundation's `ComparisonResult` as is,<br class="gmail_msg">
but we believe the community should at least consider alternate naming before<br class="gmail_msg">
that happens. There will be an opportunity to discuss the choices in detail<br class="gmail_msg">
when the modified<br class="gmail_msg">
[Comparison Proposal](<a href="https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e" rel="noreferrer" class="gmail_msg" target="_blank">https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e</a>) comes<br class="gmail_msg">
up for review.<br class="gmail_msg">
<br class="gmail_msg">
### `String` should be a `Collection` of `Character`s Again<br class="gmail_msg">
<br class="gmail_msg">
In Swift 2.0, `String`'s `Collection` conformance was dropped, because we<br class="gmail_msg">
convinced ourselves that its semantics differed from those of `Collection` too<br class="gmail_msg">
significantly.<br class="gmail_msg">
<br class="gmail_msg">
It was always well understood that if strings were treated as sequences of<br class="gmail_msg">
`UnicodeScalar`s, algorithms such as `lexicographicalCompare`, `elementsEqual`,<br class="gmail_msg">
and `reversed` would produce nonsense results. Thus, in Swift 1.0, `String` was<br class="gmail_msg">
a collection of `Character` (extended grapheme clusters). During 2.0<br class="gmail_msg">
development, though, we realized that correct string concatenation could<br class="gmail_msg">
occasionally merge distinct grapheme clusters at the start and end of combined<br class="gmail_msg">
strings.<br class="gmail_msg">
<br class="gmail_msg">
This quirk aside, every aspect of strings-as-collections-of-graphemes appears to<br class="gmail_msg">
comport perfectly with Unicode. We think the concatenation problem is tolerable,<br class="gmail_msg">
because the cases where it occurs all represent partially-formed constructs. The<br class="gmail_msg">
largest class—isolated combining characters such as ◌́ (U+0301 COMBINING ACUTE<br class="gmail_msg">
ACCENT)—are explicitly called out in the Unicode standard as<br class="gmail_msg">
“[degenerate](<a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries)" rel="noreferrer" class="gmail_msg" target="_blank">http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries)</a>” or<br class="gmail_msg">
“[defective](<a href="http://www.unicode.org/versions/Unicode9.0.0/ch03.pdf)" rel="noreferrer" class="gmail_msg" target="_blank">http://www.unicode.org/versions/Unicode9.0.0/ch03.pdf)</a>”. The other<br class="gmail_msg">
cases—such as a string ending in a zero-width joiner or half of a regional<br class="gmail_msg">
indicator—appear to be equally transient and unlikely outside of a text editor.<br class="gmail_msg">
<br class="gmail_msg">
Admitting these cases encourages exploration of grapheme composition and is<br class="gmail_msg">
consistent with what appears to be an overall Unicode philosophy that “no<br class="gmail_msg">
special provisions are made to get marginally better behavior for… cases that<br class="gmail_msg">
never occur in practice.”[2] Furthermore, it seems<br class="gmail_msg">
unlikely to disturb the semantics of any plausible algorithms. We can handle<br class="gmail_msg">
these cases by documenting them, explicitly stating that the elements of a<br class="gmail_msg">
`String` are an emergent property based on Unicode rules.<br class="gmail_msg">
<br class="gmail_msg">
The benefits of restoring `Collection` conformance are substantial:<br class="gmail_msg">
<br class="gmail_msg">
* Collection-like operations encourage experimentation with strings to<br class="gmail_msg">
investigate and understand their behavior. This is useful for teaching new<br class="gmail_msg">
programmers, but also good for experienced programmers who want to<br class="gmail_msg">
understand more about strings/unicode.<br class="gmail_msg">
<br class="gmail_msg">
* Extended grapheme clusters form a natural element boundary for Unicode<br class="gmail_msg">
strings. For example, searching and matching operations will always produce<br class="gmail_msg">
results that line up on grapheme cluster boundaries.<br class="gmail_msg">
<br class="gmail_msg">
* Character-by-character processing is a legitimate thing to do in many real<br class="gmail_msg">
use-cases, including parsing, pattern matching, and language-specific<br class="gmail_msg">
transformations such as transliteration.<br class="gmail_msg">
<br class="gmail_msg">
* `Collection` conformance makes a wide variety of powerful operations<br class="gmail_msg">
available that are appropriate to `String`'s default role as the vehicle for<br class="gmail_msg">
machine processed text.<br class="gmail_msg">
<br class="gmail_msg">
The methods `String` would inherit from `Collection`, where similar to<br class="gmail_msg">
higher-level string algorithms, have the right semantics. For example,<br class="gmail_msg">
grapheme-wise `lexicographicalCompare`, `elementsEqual`, and application of<br class="gmail_msg">
`flatMap` with case-conversion, produce the same results one would expect<br class="gmail_msg">
from whole-string ordering comparison, equality comparison, and<br class="gmail_msg">
case-conversion, respectively. `reverse` operates correctly on graphemes,<br class="gmail_msg">
keeping diacritics moored to their base characters and leaving emoji intact.<br class="gmail_msg">
Other methods such as `indexOf` and `contains` make obvious sense. A few<br class="gmail_msg">
`Collection` methods, like `min` and `max`, may not be particularly useful<br class="gmail_msg">
on `String`, but we don't consider that to be a problem worth solving, in<br class="gmail_msg">
the same way that we wouldn't try to suppress `min` and `max` on a<br class="gmail_msg">
`Set([UInt8])` that was used to store IP addresses.<br class="gmail_msg">
<br class="gmail_msg">
* Many of the higher-level operations that we want to provide for `String`s,<br class="gmail_msg">
such as parsing and pattern matching, should apply to any `Collection`, and<br class="gmail_msg">
many of the benefits we want for `Collections`, such<br class="gmail_msg">
as unified slicing, should accrue<br class="gmail_msg">
equally to `String`. Making `String` part of the same protocol hierarchy<br class="gmail_msg">
allows us to write these operations once and not worry about keeping the<br class="gmail_msg">
benefits in sync.<br class="gmail_msg">
<br class="gmail_msg">
* Slicing strings into substrings is a crucial part of the vocabulary of<br class="gmail_msg">
string processing, and all other sliceable things are `Collection`s.<br class="gmail_msg">
Because of its collection-like behavior, users naturally think of `String`<br class="gmail_msg">
in collection terms, but run into frustrating limitations where it fails to<br class="gmail_msg">
conform and are left to wonder where all the differences lie. Many simply<br class="gmail_msg">
“correct” this limitation by declaring a trivial conformance:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension String : BidirectionalCollection {}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Even if we removed indexing-by-element from `String`, users could still do<br class="gmail_msg">
this:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension String : BidirectionalCollection {<br class="gmail_msg">
subscript(i: Index) -> Character { return characters[i] }<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
It would be much better to legitimize the conformance to `Collection` and<br class="gmail_msg">
simply document the oddity of any concatenation corner-cases, than to deny<br class="gmail_msg">
users the benefits on the grounds that a few cases are confusing.<br class="gmail_msg">
<br class="gmail_msg">
Note that the fact that `String` is a collection of graphemes does *not* mean<br class="gmail_msg">
that string operations will necessarily have to do grapheme boundary<br class="gmail_msg">
recognition. See the Unicode protocol section for details.<br class="gmail_msg">
<br class="gmail_msg">
### `Character` and `CharacterSet`<br class="gmail_msg">
<br class="gmail_msg">
`Character`, which represents a<br class="gmail_msg">
Unicode<br class="gmail_msg">
[extended grapheme cluster](<a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries" rel="noreferrer" class="gmail_msg" target="_blank">http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries</a>),<br class="gmail_msg">
is a bit of a black box, requiring conversion to `String` in order to<br class="gmail_msg">
do any introspection, including interoperation with ASCII. To fix this, we should:<br class="gmail_msg">
<br class="gmail_msg">
- Add a `unicodeScalars` view much like `String`'s, so that the sub-structure<br class="gmail_msg">
of grapheme clusters is discoverable.<br class="gmail_msg">
- Add a failable `init` from sequences of scalars (returning nil for sequences<br class="gmail_msg">
that contain 0 or 2+ graphemes).<br class="gmail_msg">
- (Lower priority) expose some operations, such as `func uppercase() -><br class="gmail_msg">
String`, `var isASCII: Bool`, and, to the extent they can be sensibly<br class="gmail_msg">
generalized, queries of unicode properties that should also be exposed on<br class="gmail_msg">
`UnicodeScalar` such as `isAlphabetic` and `isGraphemeBase` .<br class="gmail_msg">
<br class="gmail_msg">
Despite its name, `CharacterSet` currently operates on the Swift `UnicodeScalar`<br class="gmail_msg">
type. This means it is usable on `String`, but only by going through the unicode<br class="gmail_msg">
scalar view. To deal with this clash in the short term, `CharacterSet` should be<br class="gmail_msg">
renamed to `UnicodeScalarSet`. In the longer term, it may be appropriate to<br class="gmail_msg">
introduce a `CharacterSet` that provides similar functionality for extended<br class="gmail_msg">
grapheme clusters.[5]<br class="gmail_msg">
<br class="gmail_msg">
### Unification of Slicing Operations<br class="gmail_msg">
<br class="gmail_msg">
Creating substrings is a basic part of String processing, but the slicing<br class="gmail_msg">
operations that we have in Swift are inconsistent in both their spelling and<br class="gmail_msg">
their naming:<br class="gmail_msg">
<br class="gmail_msg">
* Slices with two explicit endpoints are done with subscript, and support<br class="gmail_msg">
in-place mutation:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
s[i..<j].mutate()<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
* Slicing from an index to the end, or from the start to an index, is done<br class="gmail_msg">
with a method and does not support in-place mutation:<br class="gmail_msg">
```swift<br class="gmail_msg">
s.prefix(upTo: i).readOnly()<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Prefix and suffix operations should be migrated to be subscripting operations<br class="gmail_msg">
with one-sided ranges i.e. `s.prefix(upTo: i)` should become `s[..<i]`, as<br class="gmail_msg">
in<br class="gmail_msg">
[this proposal](<a href="https://github.com/apple/swift-evolution/blob/9cf2685293108ea3efcbebb7ee6a8618b83d4a90/proposals/0132-sequence-end-ops.md" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/apple/swift-evolution/blob/9cf2685293108ea3efcbebb7ee6a8618b83d4a90/proposals/0132-sequence-end-ops.md</a>).<br class="gmail_msg">
With generic subscripting in the language, that will allow us to collapse a wide<br class="gmail_msg">
variety of methods and subscript overloads into a single implementation, and<br class="gmail_msg">
give users an easy-to-use and composable way to describe subranges.<br class="gmail_msg">
<br class="gmail_msg">
Further extending this EDSL to integrate use-cases like `s.prefix(maxLength: 5)`<br class="gmail_msg">
is an ongoing research project that can be considered part of the potential<br class="gmail_msg">
long-term vision of text (and collection) processing.<br class="gmail_msg">
<br class="gmail_msg">
### Substrings<br class="gmail_msg">
<br class="gmail_msg">
When implementing substring slicing, languages are faced with three options:<br class="gmail_msg">
<br class="gmail_msg">
1. Make the substrings the same type as string, and share storage.<br class="gmail_msg">
2. Make the substrings the same type as string, and copy storage when making the substring.<br class="gmail_msg">
3. Make substrings a different type, with a storage copy on conversion to string.<br class="gmail_msg">
<br class="gmail_msg">
We think number 3 is the best choice. A walk-through of the tradeoffs follows.<br class="gmail_msg">
<br class="gmail_msg">
#### Same type, shared storage<br class="gmail_msg">
<br class="gmail_msg">
In Swift 3.0, slicing a `String` produces a new `String` that is a view into a<br class="gmail_msg">
subrange of the original `String`'s storage. This is why `String` is 3 words in<br class="gmail_msg">
size (the start, length and buffer owner), unlike the similar `Array` type<br class="gmail_msg">
which is only one.<br class="gmail_msg">
<br class="gmail_msg">
This is a simple model with big efficiency gains when chopping up strings into<br class="gmail_msg">
multiple smaller strings. But it does mean that a stored substring keeps the<br class="gmail_msg">
entire original string buffer alive even after it would normally have been<br class="gmail_msg">
released.<br class="gmail_msg">
<br class="gmail_msg">
This arrangement has proven to be problematic in other programming languages,<br class="gmail_msg">
because applications sometimes extract small strings from large ones and keep<br class="gmail_msg">
those small strings long-term. That is considered a memory leak and was enough<br class="gmail_msg">
of a problem in Java that they changed from substrings sharing storage to<br class="gmail_msg">
making a copy in 1.7.<br class="gmail_msg">
<br class="gmail_msg">
#### Same type, copied storage<br class="gmail_msg">
<br class="gmail_msg">
Copying of substrings is also the choice made in C#, and in the default<br class="gmail_msg">
`NSString` implementation. This approach avoids the memory leak issue, but has<br class="gmail_msg">
obvious performance overhead in performing the copies.<br class="gmail_msg">
<br class="gmail_msg">
This in turn encourages trafficking in string/range pairs instead of in<br class="gmail_msg">
substrings, for performance reasons, leading to API challenges. For example:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
foo.compare(bar, range: start..<end)<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Here, it is not clear whether `range` applies to `foo` or `bar`. This<br class="gmail_msg">
relationship is better expressed in Swift as a slicing operation:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
foo[start..<end].compare(bar)<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Not only does this clarify to which string the range applies, it also brings<br class="gmail_msg">
this sub-range capability to any API that operates on `String` "for free". So<br class="gmail_msg">
these other combinations also work equally well:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
// apply range on argument rather than target<br class="gmail_msg">
foo.compare(bar[start..<end])<br class="gmail_msg">
// apply range on both<br class="gmail_msg">
foo[start..<end].compare(bar[start1..<end1])<br class="gmail_msg">
// compare two strings ignoring first character<br class="gmail_msg">
foo.dropFirst().compare(bar.dropFirst())<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
In all three cases, an explicit range argument need not appear on the `compare`<br class="gmail_msg">
method itself. The implementation of `compare` does not need to know anything<br class="gmail_msg">
about ranges. Methods need only take range arguments when that was an<br class="gmail_msg">
integral part of their purpose (for example, setting the start and end of a<br class="gmail_msg">
user's current selection in a text box).<br class="gmail_msg">
<br class="gmail_msg">
#### Different type, shared storage<br class="gmail_msg">
<br class="gmail_msg">
The desire to share underlying storage while preventing accidental memory leaks<br class="gmail_msg">
occurs with slices of `Array`. For this reason we have an `ArraySlice` type.<br class="gmail_msg">
The inconvenience of a separate type is mitigated by most operations used on<br class="gmail_msg">
`Array` from the standard library being generic over `Sequence` or `Collection`.<br class="gmail_msg">
<br class="gmail_msg">
We should apply the same approach for `String` by introducing a distinct<br class="gmail_msg">
`SubSequence` type, `Substring`. Similar advice given for `ArraySlice` would apply to `Substring`:<br class="gmail_msg">
<br class="gmail_msg">
> Important: Long-term storage of `Substring` instances is discouraged. A<br class="gmail_msg">
> substring holds a reference to the entire storage of a larger string, not<br class="gmail_msg">
> just to the portion it presents, even after the original string's lifetime<br class="gmail_msg">
> ends. Long-term storage of a `Substring` may therefore prolong the lifetime<br class="gmail_msg">
> of large strings that are no longer otherwise accessible, which can appear<br class="gmail_msg">
> to be memory leakage.<br class="gmail_msg">
<br class="gmail_msg">
When assigning a `Substring` to a longer-lived variable (usually a stored<br class="gmail_msg">
property) explicitly of type `String`, a type conversion will be performed, and<br class="gmail_msg">
at this point the substring buffer is copied and the original string's storage<br class="gmail_msg">
can be released.<br class="gmail_msg">
<br class="gmail_msg">
A `String` that was not its own `Substring` could be one word—a single tagged<br class="gmail_msg">
pointer—without requiring additional allocations. `Substring`s would be a view<br class="gmail_msg">
onto a `String`, so are 3 words - pointer to owner, pointer to start, and a<br class="gmail_msg">
length. The small string optimization for `Substring` would take advantage of<br class="gmail_msg">
the larger size, probably with a less compressed encoding for speed.<br class="gmail_msg">
<br class="gmail_msg">
The downside of having two types is the inconvenience of sometimes having a<br class="gmail_msg">
`Substring` when you need a `String`, and vice-versa. It is likely this would<br class="gmail_msg">
be a significantly bigger problem than with `Array` and `ArraySlice`, as<br class="gmail_msg">
slicing of `String` is such a common operation. It is especially relevant to<br class="gmail_msg">
existing code that assumes `String` is the currency type. To ease the pain of<br class="gmail_msg">
type mismatches, `Substring` should be a subtype of `String` in the same way<br class="gmail_msg">
that `Int` is a subtype of `Optional<Int>`. This would give users an implicit<br class="gmail_msg">
conversion from `Substring` to `String`, as well as the usual implicit<br class="gmail_msg">
conversions such as `[Substring]` to `[String]` that other subtype<br class="gmail_msg">
relationships receive.<br class="gmail_msg">
<br class="gmail_msg">
In most cases, type inference combined with the subtype relationship should<br class="gmail_msg">
make the type difference a non-issue and users will not care which type they<br class="gmail_msg">
are using. For flexibility and optimizability, most operations from the<br class="gmail_msg">
standard library will traffic in generic models of<br class="gmail_msg">
[`Unicode`](#the--code-unicode--code--protocol).<br class="gmail_msg">
<br class="gmail_msg">
##### Guidance for API Designers<br class="gmail_msg">
<br class="gmail_msg">
In this model, **if a user is unsure about which type to use, `String` is always<br class="gmail_msg">
a reasonable default**. A `Substring` passed where `String` is expected will be<br class="gmail_msg">
implicitly copied. When compared to the “same type, copied storage” model, we<br class="gmail_msg">
have effectively deferred the cost of copying from the point where a substring<br class="gmail_msg">
is created until it must be converted to `String` for use with an API.<br class="gmail_msg">
<br class="gmail_msg">
A user who needs to optimize away copies altogether should use this guideline:<br class="gmail_msg">
if for performance reasons you are tempted to add a `Range` argument to your<br class="gmail_msg">
method as well as a `String` to avoid unnecessary copies, you should instead<br class="gmail_msg">
use `Substring`.<br class="gmail_msg">
<br class="gmail_msg">
##### The “Empty Subscript”<br class="gmail_msg">
<br class="gmail_msg">
To make it easy to call such an optimized API when you only have a `String` (or<br class="gmail_msg">
to call any API that takes a `Collection`'s `SubSequence` when all you have is<br class="gmail_msg">
the `Collection`), we propose the following “empty subscript” operation,<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension Collection {<br class="gmail_msg">
subscript() -> SubSequence {<br class="gmail_msg">
return self[startIndex..<endIndex]<br class="gmail_msg">
}<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
which allows the following usage:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
funcThatIsJustLooking(at: <a href="http://person.name" rel="noreferrer" class="gmail_msg" target="_blank">person.name</a>[]) // pass <a href="http://person.name" rel="noreferrer" class="gmail_msg" target="_blank">person.name</a> as Substring<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
The `[]` syntax can be offered as a fixit when needed, similar to `&` for an<br class="gmail_msg">
`inout` argument. While it doesn't help a user to convert `[String]` to<br class="gmail_msg">
`[Substring]`, the need for such conversions is extremely rare, can be done with<br class="gmail_msg">
a simple `map` (which could also be offered by a fixit):<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
takesAnArrayOfSubstring(arrayOfString.map { $0[] })<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
#### Other Options Considered<br class="gmail_msg">
<br class="gmail_msg">
As we have seen, all three options above have downsides, but it's possible<br class="gmail_msg">
these downsides could be eliminated/mitigated by the compiler. We are proposing<br class="gmail_msg">
one such mitigation—implicit conversion—as part of the the "different type,<br class="gmail_msg">
shared storage" option, to help avoid the cognitive load on developers of<br class="gmail_msg">
having to deal with a separate `Substring` type.<br class="gmail_msg">
<br class="gmail_msg">
To avoid the memory leak issues of a "same type, shared storage" substring<br class="gmail_msg">
option, we considered whether the compiler could perform an implicit copy of<br class="gmail_msg">
the underlying storage when it detects the string is being "stored" for long<br class="gmail_msg">
term usage, say when it is assigned to a stored property. The trouble with this<br class="gmail_msg">
approach is it is very difficult for the compiler to distinguish between<br class="gmail_msg">
long-term storage versus short-term in the case of abstractions that rely on<br class="gmail_msg">
stored properties. For example, should the storing of a substring inside an<br class="gmail_msg">
`Optional` be considered long-term? Or the storing of multiple substrings<br class="gmail_msg">
inside an array? The latter would not work well in the case of a<br class="gmail_msg">
`components(separatedBy:)` implementation that intended to return an array of<br class="gmail_msg">
substrings. It would also be difficult to distinguish intentional medium-term<br class="gmail_msg">
storage of substrings, say by a lexer. There does not appear to be an effective<br class="gmail_msg">
consistent rule that could be applied in the general case for detecting when a<br class="gmail_msg">
substring is truly being stored long-term.<br class="gmail_msg">
<br class="gmail_msg">
To avoid the cost of copying substrings under "same type, copied storage", the<br class="gmail_msg">
optimizer could be enhanced to to reduce the impact of some of those copies.<br class="gmail_msg">
For example, this code could be optimized to pull the invariant substring out<br class="gmail_msg">
of the loop:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
for _ in 0..<lots {<br class="gmail_msg">
someFunc(takingString: bigString[bigRange])<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
It's worth noting that a similar optimization is needed to avoid an equivalent<br class="gmail_msg">
problem with implicit conversion in the "different type, shared storage" case:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
let substring = bigString[bigRange]<br class="gmail_msg">
for _ in 0..<lots { someFunc(takingString: substring) }<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
However, in the case of "same type, copied storage" there are many use cases<br class="gmail_msg">
that cannot be optimized as easily. Consider the following simple definition of<br class="gmail_msg">
a recursive `contains` algorithm, which when substring slicing is linear makes<br class="gmail_msg">
the overall algorithm quadratic:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension String {<br class="gmail_msg">
func containsChar(_ x: Character) -> Bool {<br class="gmail_msg">
return !isEmpty && (first == x || dropFirst().containsChar(x))<br class="gmail_msg">
}<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
For the optimizer to eliminate this problem is unrealistic, forcing the user to<br class="gmail_msg">
remember to optimize the code to not use string slicing if they want it to be<br class="gmail_msg">
efficient (assuming they remember):<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension String {<br class="gmail_msg">
// add optional argument tracking progress through the string<br class="gmail_msg">
func containsCharacter(_ x: Character, atOrAfter idx: Index? = nil) -> Bool {<br class="gmail_msg">
let idx = idx ?? startIndex<br class="gmail_msg">
return idx != endIndex<br class="gmail_msg">
&& (self[idx] == x || containsCharacter(x, atOrAfter: index(after: idx)))<br class="gmail_msg">
}<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
#### Substrings, Ranges and Objective-C Interop<br class="gmail_msg">
<br class="gmail_msg">
The pattern of passing a string/range pair is common in several Objective-C<br class="gmail_msg">
APIs, and is made especially awkward in Swift by the non-interchangeability of<br class="gmail_msg">
`Range<String.Index>` and `NSRange`.<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
s2.find(s2, sourceRange: NSRange(j..<s2.endIndex, in: s2))<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
In general, however, the Swift idiom for operating on a sub-range of a<br class="gmail_msg">
`Collection` is to *slice* the collection and operate on that:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
s2.find(s2[j..<s2.endIndex])<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Therefore, APIs that operate on an `NSString`/`NSRange` pair should be imported<br class="gmail_msg">
without the `NSRange` argument. The Objective-C importer should be changed to<br class="gmail_msg">
give these APIs special treatment so that when a `Substring` is passed, instead<br class="gmail_msg">
of being converted to a `String`, the full `NSString` and range are passed to<br class="gmail_msg">
the Objective-C method, thereby avoiding a copy.<br class="gmail_msg">
<br class="gmail_msg">
As a result, you would never need to pass an `NSRange` to these APIs, which<br class="gmail_msg">
solves the impedance problem by eliminating the argument, resulting in more<br class="gmail_msg">
idiomatic Swift code while retaining the performance benefit. To help users<br class="gmail_msg">
manually handle any cases that remain, Foundation should be augmented to allow<br class="gmail_msg">
the following syntax for converting to and from `NSRange`:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
let nsr = NSRange(i..<j, in: s) // An NSRange corresponding to s[i..<j]<br class="gmail_msg">
let iToJ = Range(nsr, in: s) // Equivalent to i..<j<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
### The `Unicode` protocol<br class="gmail_msg">
<br class="gmail_msg">
With `Substring` and `String` being distinct types and sharing almost all<br class="gmail_msg">
interface and semantics, and with the highest-performance string processing<br class="gmail_msg">
requiring knowledge of encoding and layout that the currency types can't<br class="gmail_msg">
provide, it becomes important to capture the common “string API” in a protocol.<br class="gmail_msg">
Since Unicode conformance is a key feature of string processing in swift, we<br class="gmail_msg">
call that protocol `Unicode`:<br class="gmail_msg">
<br class="gmail_msg">
**Note:** The following assumes several features that are planned but not yet implemented in<br class="gmail_msg">
Swift, and should be considered a sketch rather than a final design.<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
protocol Unicode<br class="gmail_msg">
: Comparable, BidirectionalCollection where Element == Character {<br class="gmail_msg">
<br class="gmail_msg">
associatedtype Encoding : UnicodeEncoding<br class="gmail_msg">
var encoding: Encoding { get }<br class="gmail_msg">
<br class="gmail_msg">
associatedtype CodeUnits<br class="gmail_msg">
: RandomAccessCollection where Element == Encoding.CodeUnit<br class="gmail_msg">
var codeUnits: CodeUnits { get }<br class="gmail_msg">
<br class="gmail_msg">
associatedtype UnicodeScalars<br class="gmail_msg">
: BidirectionalCollection where Element == UnicodeScalar<br class="gmail_msg">
var unicodeScalars: UnicodeScalars { get }<br class="gmail_msg">
<br class="gmail_msg">
associatedtype ExtendedASCII<br class="gmail_msg">
: BidirectionalCollection where Element == UInt32<br class="gmail_msg">
var extendedASCII: ExtendedASCII { get }<br class="gmail_msg">
<br class="gmail_msg">
var unicodeScalars: UnicodeScalars { get }<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
extension Unicode {<br class="gmail_msg">
// ... define high-level non-mutating string operations, e.g. search ...<br class="gmail_msg">
<br class="gmail_msg">
func compared<Other: Unicode>(<br class="gmail_msg">
to rhs: Other,<br class="gmail_msg">
case caseSensitivity: StringSensitivity? = nil,<br class="gmail_msg">
diacritic diacriticSensitivity: StringSensitivity? = nil,<br class="gmail_msg">
width widthSensitivity: StringSensitivity? = nil,<br class="gmail_msg">
in locale: Locale? = nil<br class="gmail_msg">
) -> SortOrder { ... }<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
extension Unicode : RangeReplaceableCollection where CodeUnits :<br class="gmail_msg">
RangeReplaceableCollection {<br class="gmail_msg">
// Satisfy protocol requirement<br class="gmail_msg">
mutating func replaceSubrange<C : Collection>(_: Range<Index>, with: C)<br class="gmail_msg">
where C.Element == Element<br class="gmail_msg">
<br class="gmail_msg">
// ... define high-level mutating string operations, e.g. replace ...<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
The goal is that `Unicode` exposes the underlying encoding and code units in<br class="gmail_msg">
such a way that for types with a known representation (e.g. a high-performance<br class="gmail_msg">
`UTF8String`) that information can be known at compile-time and can be used to<br class="gmail_msg">
generate a single path, while still allowing types like `String` that admit<br class="gmail_msg">
multiple representations to use runtime queries and branches to fast path<br class="gmail_msg">
specializations.<br class="gmail_msg">
<br class="gmail_msg">
**Note:** `Unicode` would make a fantastic namespace for much of<br class="gmail_msg">
what's in this proposal if we could get the ability to nest types and<br class="gmail_msg">
protocols in protocols.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
### Scanning, Matching, and Tokenization<br class="gmail_msg">
<br class="gmail_msg">
#### Low-Level Textual Analysis<br class="gmail_msg">
<br class="gmail_msg">
We should provide convenient APIs processing strings by character. For example,<br class="gmail_msg">
it should be easy to cleanly express, “if this string starts with `"f"`, process<br class="gmail_msg">
the rest of the string as follows…” Swift is well-suited to expressing this<br class="gmail_msg">
common pattern beautifully, but we need to add the APIs. Here are two examples<br class="gmail_msg">
of the sort of code that might be possible given such APIs:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
if let firstLetter = input.droppingPrefix(alphabeticCharacter) {<br class="gmail_msg">
somethingWith(input) // process the rest of input<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
if let (number, restOfInput) = input.parsingPrefix(Int.self) {<br class="gmail_msg">
...<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
The specific spelling and functionality of APIs like this are TBD. The larger<br class="gmail_msg">
point is to make sure matching-and-consuming jobs are well-supported.<br class="gmail_msg">
<br class="gmail_msg">
#### Unified Pattern Matcher Protocol<br class="gmail_msg">
<br class="gmail_msg">
Many of the current methods that do matching are overloaded to do the same<br class="gmail_msg">
logical operations in different ways, with the following axes:<br class="gmail_msg">
<br class="gmail_msg">
- Logical Operation: `find`, `split`, `replace`, match at start<br class="gmail_msg">
- Kind of pattern: `CharacterSet`, `String`, a regex, a closure<br class="gmail_msg">
- Options, e.g. case/diacritic sensitivity, locale. Sometimes a part of<br class="gmail_msg">
the method name, and sometimes an argument<br class="gmail_msg">
- Whole string or subrange.<br class="gmail_msg">
<br class="gmail_msg">
We should represent these aspects as orthogonal, composable components,<br class="gmail_msg">
abstracting pattern matchers into a protocol like<br class="gmail_msg">
[this one](<a href="https://github.com/apple/swift/blob/master/test/Prototypes/PatternMatching.swift#L33" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/apple/swift/blob/master/test/Prototypes/PatternMatching.swift#L33</a>),<br class="gmail_msg">
that can allow us to define logical operations once, without introducing<br class="gmail_msg">
overloads, and massively reducing API surface area.<br class="gmail_msg">
<br class="gmail_msg">
For example, using the strawman prefix `%` syntax to turn string literals into<br class="gmail_msg">
patterns, the following pairs would all invoke the same generic methods:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
if let found = s.firstMatch(%"searchString") { ... }<br class="gmail_msg">
if let found = s.firstMatch(someRegex) { ... }<br class="gmail_msg">
<br class="gmail_msg">
for m in s.allMatches((%"searchString"), case: .insensitive) { ... }<br class="gmail_msg">
for m in s.allMatches(someRegex) { ... }<br class="gmail_msg">
<br class="gmail_msg">
let items = s.split(separatedBy: ", ")<br class="gmail_msg">
let tokens = s.split(separatedBy: CharacterSet.whitespace)<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Note that, because Swift requires the indices of a slice to match the indices of<br class="gmail_msg">
the range from which it was sliced, operations like `firstMatch` can return a<br class="gmail_msg">
`Substring?` in lieu of a `Range<String.Index>?`: the indices of the match in<br class="gmail_msg">
the string being searched, if needed, can easily be recovered as the<br class="gmail_msg">
`startIndex` and `endIndex` of the `Substring`.<br class="gmail_msg">
<br class="gmail_msg">
Note also that matching operations are useful for collections in general, and<br class="gmail_msg">
would fall out of this proposal:<br class="gmail_msg">
<br class="gmail_msg">
```<br class="gmail_msg">
// replace subsequences of contiguous NaNs with zero<br class="gmail_msg">
forces.replace(oneOrMore([Float.nan]), [0.0])<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
#### Regular Expressions<br class="gmail_msg">
<br class="gmail_msg">
Addressing regular expressions is out of scope for this proposal.<br class="gmail_msg">
That said, it is important that to note the pattern matching protocol mentioned<br class="gmail_msg">
above provides a suitable foundation for regular expressions, and types such as<br class="gmail_msg">
`NSRegularExpression` can easily be retrofitted to conform to it. In the<br class="gmail_msg">
future, support for regular expression literals in the compiler could allow for<br class="gmail_msg">
compile-time syntax checking and optimization.<br class="gmail_msg">
<br class="gmail_msg">
### String Indices<br class="gmail_msg">
<br class="gmail_msg">
`String` currently has four views—`characters`, `unicodeScalars`, `utf8`, and<br class="gmail_msg">
`utf16`—each with its own opaque index type. The APIs used to translate indices<br class="gmail_msg">
between views add needless complexity, and the opacity of indices makes them<br class="gmail_msg">
difficult to serialize.<br class="gmail_msg">
<br class="gmail_msg">
The index translation problem has two aspects:<br class="gmail_msg">
<br class="gmail_msg">
1. `String` views cannot consume one anothers' indices without a cumbersome<br class="gmail_msg">
conversion step. An index into a `String`'s `characters` must be translated<br class="gmail_msg">
before it can be used as a position in its `unicodeScalars`. Although these<br class="gmail_msg">
translations are rarely needed, they add conceptual and API complexity.<br class="gmail_msg">
2. Many APIs in the core libraries and other frameworks still expose `String`<br class="gmail_msg">
positions as `Int`s and regions as `NSRange`s, which can only reference a<br class="gmail_msg">
`utf16` view and interoperate poorly with `String` itself.<br class="gmail_msg">
<br class="gmail_msg">
#### Index Interchange Among Views<br class="gmail_msg">
<br class="gmail_msg">
String's need for flexible backing storage and reasonably-efficient indexing<br class="gmail_msg">
(i.e. without dynamically allocating and reference-counting the indices<br class="gmail_msg">
themselves) means indices need an efficient underlying storage type. Although<br class="gmail_msg">
we do not wish to expose `String`'s indices *as* integers, `Int` offsets into<br class="gmail_msg">
underlying code unit storage makes a good underlying storage type, provided<br class="gmail_msg">
`String`'s underlying storage supports random-access. We think random-access<br class="gmail_msg">
*code-unit storage* is a reasonable requirement to impose on all `String`<br class="gmail_msg">
instances.<br class="gmail_msg">
<br class="gmail_msg">
Making these `Int` code unit offsets conveniently accessible and constructible<br class="gmail_msg">
solves the serialization problem:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
clipboard.write(s.endIndex.codeUnitOffset)<br class="gmail_msg">
let offset = clipboard.read(Int.self)<br class="gmail_msg">
let i = String.Index(codeUnitOffset: offset)<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Index interchange between `String` and its `unicodeScalars`, `codeUnits`,<br class="gmail_msg">
and [`extendedASCII`](#parsing-ascii-structure) views can be made entirely<br class="gmail_msg">
seamless by having them share an index type (semantics of indexing a `String`<br class="gmail_msg">
between grapheme cluster boundaries are TBD—it can either trap or be forgiving).<br class="gmail_msg">
Having a common index allows easy traversal into the interior of graphemes,<br class="gmail_msg">
something that is often needed, without making it likely that someone will do it<br class="gmail_msg">
by accident.<br class="gmail_msg">
<br class="gmail_msg">
- `String.index(after:)` should advance to the next grapheme, even when the<br class="gmail_msg">
index points partway through a grapheme.<br class="gmail_msg">
<br class="gmail_msg">
- `String.index(before:)` should move to the start of the grapheme before<br class="gmail_msg">
the current position.<br class="gmail_msg">
<br class="gmail_msg">
Seamless index interchange between `String` and its UTF-8 or UTF-16 views is not<br class="gmail_msg">
crucial, as the specifics of encoding should not be a concern for most use<br class="gmail_msg">
cases, and would impose needless costs on the indices of other views. That<br class="gmail_msg">
said, we can make translation much more straightforward by exposing simple<br class="gmail_msg">
bidirectional converting `init`s on both index types:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
let u8Position = String.UTF8.Index(someStringIndex)<br class="gmail_msg">
let originalPosition = String.Index(u8Position)<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
#### Index Interchange with Cocoa<br class="gmail_msg">
<br class="gmail_msg">
We intend to address `NSRange`s that denote substrings in Cocoa APIs as<br class="gmail_msg">
described [later in this document](#substrings--ranges-and-objective-c-interop).<br class="gmail_msg">
That leaves the interchange of bare indices with Cocoa APIs trafficking in<br class="gmail_msg">
`Int`. Hopefully such APIs will be rare, but when needed, the following<br class="gmail_msg">
extension, which would be useful for all `Collections`, can help:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension Collection {<br class="gmail_msg">
func index(offset: IndexDistance) -> Index {<br class="gmail_msg">
return index(startIndex, offsetBy: offset)<br class="gmail_msg">
}<br class="gmail_msg">
func offset(of i: Index) -> IndexDistance {<br class="gmail_msg">
return distance(from: startIndex, to: i)<br class="gmail_msg">
}<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
Then integers can easily be translated into offsets into a `String`'s `utf16`<br class="gmail_msg">
view for consumption by Cocoa:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
let cocoaIndex = s.utf16.offset(of: String.UTF16Index(i))<br class="gmail_msg">
let swiftIndex = s.utf16.index(offset: cocoaIndex)<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
### Formatting<br class="gmail_msg">
<br class="gmail_msg">
A full treatment of formatting is out of scope of this proposal, but<br class="gmail_msg">
we believe it's crucial for completing the text processing picture. This<br class="gmail_msg">
section details some of the existing issues and thinking that may guide future<br class="gmail_msg">
development.<br class="gmail_msg">
<br class="gmail_msg">
#### Printf-Style Formatting<br class="gmail_msg">
<br class="gmail_msg">
`String.format` is designed on the `printf` model: it takes a format string with<br class="gmail_msg">
textual placeholders for substitution, and an arbitrary list of other arguments.<br class="gmail_msg">
The syntax and meaning of these placeholders has a long history in<br class="gmail_msg">
C, but for anyone who doesn't use them regularly they are cryptic and complex,<br class="gmail_msg">
as the `printf (3)` man page attests.<br class="gmail_msg">
<br class="gmail_msg">
Aside from complexity, this style of API has two major problems: First, the<br class="gmail_msg">
spelling of these placeholders must match up to the types of the arguments, in<br class="gmail_msg">
the right order, or the behavior is undefined. Some limited support for<br class="gmail_msg">
compile-time checking of this correspondence could be implemented, but only for<br class="gmail_msg">
the cases where the format string is a literal. Second, there's no reasonable<br class="gmail_msg">
way to extend the formatting vocabulary to cover the needs of new types: you are<br class="gmail_msg">
stuck with what's in the box.<br class="gmail_msg">
<br class="gmail_msg">
#### Foundation Formatters<br class="gmail_msg">
<br class="gmail_msg">
The formatters supplied by Foundation are highly capable and versatile, offering<br class="gmail_msg">
both formatting and parsing services. When used for formatting, though, the<br class="gmail_msg">
design pattern demands more from users than it should:<br class="gmail_msg">
<br class="gmail_msg">
* Matching the type of data being formatted to a formatter type<br class="gmail_msg">
* Creating an instance of that type<br class="gmail_msg">
* Setting stateful options (`currency`, `dateStyle`) on the type. Note: the<br class="gmail_msg">
need for this step prevents the instance from being used and discarded in<br class="gmail_msg">
the same expression where it is created.<br class="gmail_msg">
* Overall, introduction of needless verbosity into source<br class="gmail_msg">
<br class="gmail_msg">
These may seem like small issues, but the experience of Apple localization<br class="gmail_msg">
experts is that the total drag of these factors on programmers is such that they<br class="gmail_msg">
tend to reach for `String.format` instead.<br class="gmail_msg">
<br class="gmail_msg">
#### String Interpolation<br class="gmail_msg">
<br class="gmail_msg">
Swift string interpolation provides a user-friendly alternative to printf's<br class="gmail_msg">
domain-specific language (just write ordinary swift code!) and its type safety<br class="gmail_msg">
problems (put the data right where it belongs!) but the following issues prevent<br class="gmail_msg">
it from being useful for localized formatting (among other jobs):<br class="gmail_msg">
<br class="gmail_msg">
* [SR-2303](<a href="https://bugs.swift.org/browse/SR-2303" rel="noreferrer" class="gmail_msg" target="_blank">https://bugs.swift.org/browse/SR-2303</a>) We are unable to restrict<br class="gmail_msg">
types used in string interpolation.<br class="gmail_msg">
* [SR-1260](<a href="https://bugs.swift.org/browse/SR-1260" rel="noreferrer" class="gmail_msg" target="_blank">https://bugs.swift.org/browse/SR-1260</a>) String interpolation can't<br class="gmail_msg">
distinguish (fragments of) the base string from the string substitutions.<br class="gmail_msg">
<br class="gmail_msg">
In the long run, we should improve Swift string interpolation to the point where<br class="gmail_msg">
it can participate in most any formatting job. Mostly this centers around<br class="gmail_msg">
fixing the interpolation protocols per the previous item, and supporting<br class="gmail_msg">
localization.<br class="gmail_msg">
<br class="gmail_msg">
To be able to use formatting effectively inside interpolations, it needs to be<br class="gmail_msg">
both lightweight (because it all happens in-situ) and discoverable. One<br class="gmail_msg">
approach would be to standardize on `format` methods, e.g.:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
"Column 1: \(n.format(radix:16, width:8)) *** \(message)"<br class="gmail_msg">
<br class="gmail_msg">
"Something with leading zeroes: \(x.format(fill: zero, width:8))"<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
### C String Interop<br class="gmail_msg">
<br class="gmail_msg">
Our support for interoperation with nul-terminated C strings is scattered and<br class="gmail_msg">
incoherent, with 6 ways to transform a C string into a `String` and four ways to<br class="gmail_msg">
do the inverse. These APIs should be replaced with the following<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
extension String {<br class="gmail_msg">
/// Constructs a `String` having the same contents as `nulTerminatedUTF8`.<br class="gmail_msg">
///<br class="gmail_msg">
/// - Parameter nulTerminatedUTF8: a sequence of contiguous UTF-8 encoded<br class="gmail_msg">
/// bytes ending just before the first zero byte (NUL character).<br class="gmail_msg">
init(cString nulTerminatedUTF8: UnsafePointer<CChar>)<br class="gmail_msg">
<br class="gmail_msg">
/// Constructs a `String` having the same contents as `nulTerminatedCodeUnits`.<br class="gmail_msg">
///<br class="gmail_msg">
/// - Parameter nulTerminatedCodeUnits: a sequence of contiguous code units in<br class="gmail_msg">
/// the given `encoding`, ending just before the first zero code unit.<br class="gmail_msg">
/// - Parameter encoding: describes the encoding in which the code units<br class="gmail_msg">
/// should be interpreted.<br class="gmail_msg">
init<Encoding: UnicodeEncoding>(<br class="gmail_msg">
cString nulTerminatedCodeUnits: UnsafePointer<Encoding.CodeUnit>,<br class="gmail_msg">
encoding: Encoding)<br class="gmail_msg">
<br class="gmail_msg">
/// Invokes the given closure on the contents of the string, represented as a<br class="gmail_msg">
/// pointer to a null-terminated sequence of UTF-8 code units.<br class="gmail_msg">
func withCString<Result>(<br class="gmail_msg">
_ body: (UnsafePointer<CChar>) throws -> Result) rethrows -> Result<br class="gmail_msg">
}<br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
In both of the construction APIs, any invalid encoding sequence detected will<br class="gmail_msg">
have its longest valid prefix replaced by U+FFFD, the Unicode replacement<br class="gmail_msg">
character, per Unicode specification. This covers the common case. The<br class="gmail_msg">
replacement is done *physically* in the underlying storage and the validity of<br class="gmail_msg">
the result is recorded in the `String`'s `encoding` such that future accesses<br class="gmail_msg">
need not be slowed down by possible error repair separately.<br class="gmail_msg">
<br class="gmail_msg">
Construction that is aborted when encoding errors are detected can be<br class="gmail_msg">
accomplished using APIs on the `encoding`. String types that retain their<br class="gmail_msg">
physical encoding even in the presence of errors and are repaired on-the-fly can<br class="gmail_msg">
be built as different instances of the `Unicode` protocol.<br class="gmail_msg">
<br class="gmail_msg">
### Unicode 9 Conformance<br class="gmail_msg">
<br class="gmail_msg">
Unicode 9 (and MacOS 10.11) brought us support for family emoji, which changes<br class="gmail_msg">
the process of properly identifying `Character` boundaries. We need to update<br class="gmail_msg">
`String` to account for this change.<br class="gmail_msg">
<br class="gmail_msg">
### High-Performance String Processing<br class="gmail_msg">
<br class="gmail_msg">
Many strings are short enough to store in 64 bits, many can be stored using only<br class="gmail_msg">
8 bits per unicode scalar, others are best encoded in UTF-16, and some come to<br class="gmail_msg">
us already in some other encoding, such as UTF-8, that would be costly to<br class="gmail_msg">
translate. Supporting these formats while maintaining usability for<br class="gmail_msg">
general-purpose APIs demands that a single `String` type can be backed by many<br class="gmail_msg">
different representations.<br class="gmail_msg">
<br class="gmail_msg">
That said, the highest performance code always requires static knowledge of the<br class="gmail_msg">
data structures on which it operates, and for this code, dynamic selection of<br class="gmail_msg">
representation comes at too high a cost. Heavy-duty text processing demands a<br class="gmail_msg">
way to opt out of dynamism and directly use known encodings. Having this<br class="gmail_msg">
ability can also make it easy to cleanly specialize code that handles dynamic<br class="gmail_msg">
cases for maximal efficiency on the most common representations.<br class="gmail_msg">
<br class="gmail_msg">
To address this need, we can build models of the `Unicode` protocol that encode<br class="gmail_msg">
representation information into the type, such as `NFCNormalizedUTF16String`.<br class="gmail_msg">
<br class="gmail_msg">
### Parsing ASCII Structure<br class="gmail_msg">
<br class="gmail_msg">
Although many machine-readable formats support the inclusion of arbitrary<br class="gmail_msg">
Unicode text, it is also common that their fundamental structure lies entirely<br class="gmail_msg">
within the ASCII subset (JSON, YAML, many XML formats). These formats are often<br class="gmail_msg">
processed most efficiently by recognizing ASCII structural elements as ASCII,<br class="gmail_msg">
and capturing the arbitrary sections between them in more-general strings. The<br class="gmail_msg">
current String API offers no way to efficiently recognize ASCII and skip past<br class="gmail_msg">
everything else without the overhead of full decoding into unicode scalars.<br class="gmail_msg">
<br class="gmail_msg">
For these purposes, strings should supply an `extendedASCII` view that is a<br class="gmail_msg">
collection of `UInt32`, where values less than `0x80` represent the<br class="gmail_msg">
corresponding ASCII character, and other values represent data that is specific<br class="gmail_msg">
to the underlying encoding of the string.<br class="gmail_msg">
<br class="gmail_msg">
## Language Support<br class="gmail_msg">
<br class="gmail_msg">
This proposal depends on two new features in the Swift language:<br class="gmail_msg">
<br class="gmail_msg">
1. **Generic subscripts**, to<br class="gmail_msg">
enable unified slicing syntax.<br class="gmail_msg">
<br class="gmail_msg">
2. **A subtype relationship** between<br class="gmail_msg">
`Substring` and `String`, enabling framework APIs to traffic solely in<br class="gmail_msg">
`String` while still making it possible to avoid copies by handling<br class="gmail_msg">
`Substring`s where necessary.<br class="gmail_msg">
<br class="gmail_msg">
Additionally, **the ability to nest types and protocols inside<br class="gmail_msg">
protocols** could significantly shrink the footprint of this proposal<br class="gmail_msg">
on the top-level Swift namespace.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
## Open Questions<br class="gmail_msg">
<br class="gmail_msg">
### Must `String` be limited to storing UTF-16 subset encodings?<br class="gmail_msg">
<br class="gmail_msg">
- The ability to handle `UTF-8`-encoded strings (models of `Unicode`) is not in<br class="gmail_msg">
question here; this is about what encodings must be storable, without<br class="gmail_msg">
transcoding, in the common currency type called “`String`”.<br class="gmail_msg">
- ASCII, Latin-1, UCS-2, and UTF-16 are UTF-16 subsets. UTF-8 is not.<br class="gmail_msg">
- If we have a way to get at a `String`'s code units, we need a concrete type in<br class="gmail_msg">
which to express them in the API of `String`, which is a concrete type<br class="gmail_msg">
- If String needs to be able to represent UTF-32, presumably the code units need<br class="gmail_msg">
to be `UInt32`.<br class="gmail_msg">
- Not supporting UTF-32-encoded text seems like one reasonable design choice.<br class="gmail_msg">
- Maybe we can allow UTF-8 storage in `String` and expose its code units as<br class="gmail_msg">
`UInt16`, just as we would for Latin-1.<br class="gmail_msg">
- Supporting only UTF-16-subset encodings would imply that `String` indices can<br class="gmail_msg">
be serialized without recording the `String`'s underlying encoding.<br class="gmail_msg">
<br class="gmail_msg">
### Do we need a type-erasable base protocol for UnicodeEncoding?<br class="gmail_msg">
<br class="gmail_msg">
UnicodeEncoding has an associated type, but it may be important to be able to<br class="gmail_msg">
traffic in completely dynamic encoding values, e.g. for “tell me the most<br class="gmail_msg">
efficient encoding for this string.”<br class="gmail_msg">
<br class="gmail_msg">
### Should there be a string “facade?”<br class="gmail_msg">
<br class="gmail_msg">
One possible design alternative makes `Unicode` a vehicle for expressing<br class="gmail_msg">
the storage and encoding of code units, but does not attempt to give it an API<br class="gmail_msg">
appropriate for `String`. Instead, string APIs would be provided by a generic<br class="gmail_msg">
wrapper around an instance of `Unicode`:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
struct StringFacade<U: Unicode> : BidirectionalCollection {<br class="gmail_msg">
<br class="gmail_msg">
// ...APIs for high-level string processing here...<br class="gmail_msg">
<br class="gmail_msg">
var unicode: U // access to lower-level unicode details<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
typealias String = StringFacade<StringStorage><br class="gmail_msg">
typealias Substring = StringFacade<StringStorage.SubSequence><br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
This design would allow us to de-emphasize lower-level `String` APIs such as<br class="gmail_msg">
access to the specific encoding, by putting them behind a `.unicode` property.<br class="gmail_msg">
A similar effect in a facade-less design would require a new top-level<br class="gmail_msg">
`StringProtocol` playing the role of the facade with an an `associatedtype<br class="gmail_msg">
Storage : Unicode`.<br class="gmail_msg">
<br class="gmail_msg">
An interesting variation on this design is possible if defaulted generic<br class="gmail_msg">
parameters are introduced to the language:<br class="gmail_msg">
<br class="gmail_msg">
```swift<br class="gmail_msg">
struct String<U: Unicode = StringStorage><br class="gmail_msg">
: BidirectionalCollection {<br class="gmail_msg">
<br class="gmail_msg">
// ...APIs for high-level string processing here...<br class="gmail_msg">
<br class="gmail_msg">
var unicode: U // access to lower-level unicode details<br class="gmail_msg">
}<br class="gmail_msg">
<br class="gmail_msg">
typealias Substring = String<StringStorage.SubSequence><br class="gmail_msg">
```<br class="gmail_msg">
<br class="gmail_msg">
One advantage of such a design is that naïve users will always extend “the right<br class="gmail_msg">
type” (`String`) without thinking, and the new APIs will show up on `Substring`,<br class="gmail_msg">
`MyUTF8String`, etc. That said, it also has downsides that should not be<br class="gmail_msg">
overlooked, not least of which is the confusability of the meaning of the word<br class="gmail_msg">
“string.” Is it referring to the generic or the concrete type?<br class="gmail_msg">
<br class="gmail_msg">
### `TextOutputStream` and `TextOutputStreamable`<br class="gmail_msg">
<br class="gmail_msg">
`TextOutputStreamable` is intended to provide a vehicle for<br class="gmail_msg">
efficiently transporting formatted representations to an output stream<br class="gmail_msg">
without forcing the allocation of storage. Its use of `String`, a<br class="gmail_msg">
type with multiple representations, at the lowest-level unit of<br class="gmail_msg">
communication, conflicts with this goal. It might be sufficient to<br class="gmail_msg">
change `TextOutputStream` and `TextOutputStreamable` to traffic in an<br class="gmail_msg">
associated type conforming to `Unicode`, but that is not yet clear.<br class="gmail_msg">
This area will require some design work.<br class="gmail_msg">
<br class="gmail_msg">
### `description` and `debugDescription`<br class="gmail_msg">
<br class="gmail_msg">
* Should these be creating localized or non-localized representations?<br class="gmail_msg">
* Is returning a `String` efficient enough?<br class="gmail_msg">
* Is `debugDescription` pulling the weight of the API surface area it adds?<br class="gmail_msg">
<br class="gmail_msg">
### `StaticString`<br class="gmail_msg">
<br class="gmail_msg">
`StaticString` was added as a byproduct of standard library developed and kept<br class="gmail_msg">
around because it seemed useful, but it was never truly *designed* for client<br class="gmail_msg">
programmers. We need to decide what happens with it. Presumably *something*<br class="gmail_msg">
should fill its role, and that should conform to `Unicode`.<br class="gmail_msg">
<br class="gmail_msg">
## Footnotes<br class="gmail_msg">
<br class="gmail_msg">
<b id="f0">0</b> The integers rewrite currently underway is expected to<br class="gmail_msg">
substantially reduce the scope of `Int`'s API by using more<br class="gmail_msg">
generics. [↩](#a0)<br class="gmail_msg">
<br class="gmail_msg">
<b id="f1">1</b> In practice, these semantics will usually be tied to the<br class="gmail_msg">
version of the installed [ICU](<a href="http://icu-project.org" rel="noreferrer" class="gmail_msg" target="_blank">http://icu-project.org</a>) library, which<br class="gmail_msg">
programmatically encodes the most complex rules of the Unicode Standard and its<br class="gmail_msg">
de-facto extension, CLDR.[↩](#a1)<br class="gmail_msg">
<br class="gmail_msg">
<b id="f2">2</b><br class="gmail_msg">
See<br class="gmail_msg">
[<a href="http://unicode.org/reports/tr29/#Notation](http://unicode.org/reports/tr29/%23Notation)" rel="noreferrer" class="gmail_msg" target="_blank">http://unicode.org/reports/tr29/#Notation](http://unicode.org/reports/tr29/#Notation)</a>. Note<br class="gmail_msg">
that inserting Unicode scalar values to prevent merging of grapheme clusters would<br class="gmail_msg">
also constitute a kind of misbehavior (one of the clusters at the boundary would<br class="gmail_msg">
not be found in the result), so would be relatively costly to implement, with<br class="gmail_msg">
little benefit. [↩](#a2)<br class="gmail_msg">
<br class="gmail_msg">
<b id="f4">4</b> The use of non-UCA-compliant ordering is fully sanctioned by<br class="gmail_msg">
the Unicode standard for this purpose. In fact there's<br class="gmail_msg">
a [whole chapter](<a href="http://www.unicode.org/versions/Unicode9.0.0/ch05.pdf" rel="noreferrer" class="gmail_msg" target="_blank">http://www.unicode.org/versions/Unicode9.0.0/ch05.pdf</a>)<br class="gmail_msg">
dedicated to it. In particular, §5.17 says:<br class="gmail_msg">
<br class="gmail_msg">
> When comparing text that is visible to end users, a correct linguistic sort<br class="gmail_msg">
> should be used, as described in _Section 5.16, Sorting and<br class="gmail_msg">
> Searching_. However, in many circumstances the only requirement is for a<br class="gmail_msg">
> fast, well-defined ordering. In such cases, a binary ordering can be used.<br class="gmail_msg">
<br class="gmail_msg">
[↩](#a4)<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
<b id="f5">5</b> The queries supported by `NSCharacterSet` map directly onto<br class="gmail_msg">
properties in a table that's indexed by unicode scalar value. This table is<br class="gmail_msg">
part of the Unicode standard. Some of these queries (e.g., “is this an<br class="gmail_msg">
uppercase character?”) may have fairly obvious generalizations to grapheme<br class="gmail_msg">
clusters, but exactly how to do it is a research topic and *ideally* we'd either<br class="gmail_msg">
establish the existing practice that the Unicode committee would standardize, or<br class="gmail_msg">
the Unicode committee would do the research and we'd implement their<br class="gmail_msg">
result.[↩](#a5)<br class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
swift-evolution mailing list<br class="gmail_msg">
<a href="mailto:swift-evolution@swift.org" class="gmail_msg" target="_blank">swift-evolution@swift.org</a><br class="gmail_msg">
<a href="https://lists.swift.org/mailman/listinfo/swift-evolution" rel="noreferrer" class="gmail_msg" target="_blank">https://lists.swift.org/mailman/listinfo/swift-evolution</a><br class="gmail_msg">
</blockquote></div>