<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><br><br>Sent from my iPad</div><div><br>On Jan 21, 2017, at 3:49 AM, Brent Royal-Gordon <<a href="mailto:brent@architechies.com">brent@architechies.com</a>> wrote:<br><br></div><blockquote type="cite"><div><blockquote type="cite"><span>On Jan 19, 2017, at 6:56 PM, Ben Cohen via swift-evolution <<a href="mailto:swift-evolution@swift.org">swift-evolution@swift.org</a>> wrote:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Below is our take on a design manifesto for Strings in Swift 4 and beyond.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Probably best read in rendered markdown on GitHub:</span><br></blockquote><blockquote type="cite"><span><a href="https://github.com/apple/swift/blob/master/docs/StringManifesto.md">https://github.com/apple/swift/blob/master/docs/StringManifesto.md</a></span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>We’re eager to hear everyone’s thoughts.</span><br></blockquote><span></span><br><span>There is so, so much good stuff here. </span></div></blockquote><div><br></div>Right back atcha, Brent! Thanks for the detailed review!<div><br><blockquote type="cite"><div><span>I'm really looking forward to seeing how these ideas develop and enter the language.</span><br><span></span><br><blockquote type="cite"><span>#### Future Directions</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>One of the most common internationalization errors is the unintentional</span><br></blockquote><blockquote type="cite"><span>presentation to users of text that has not been localized, but regularizing APIs</span><br></blockquote><blockquote type="cite"><span>and improving documentation can go only so far in preventing this error.</span><br></blockquote><blockquote type="cite"><span>Combined with the fact that `String` operations are non-localized by default,</span><br></blockquote><blockquote type="cite"><span>the environment for processing human-readable text may still be somewhat</span><br></blockquote><blockquote type="cite"><span>error-prone in Swift 4.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>For an audience of mostly non-experts, it is especially important that naïve</span><br></blockquote><blockquote type="cite"><span>code is very likely to be correct if it compiles, and that more sophisticated</span><br></blockquote><blockquote type="cite"><span>issues can be revealed progressively. For this reason, we intend to</span><br></blockquote><blockquote type="cite"><span>specifically and separately target localization and internationalization</span><br></blockquote><blockquote type="cite"><span>problems in the Swift 5 timeframe.</span><br></blockquote><span></span><br><span>I am very glad to see this statement in a Swift design document. I have a few ideas about this, but they can wait until the next version.</span><br><span></span><br><blockquote type="cite"><span>At first blush this just adds work, but consider what it does</span><br></blockquote><blockquote type="cite"><span>for equality: two strings that normalize the same, naturally, will collate the</span><br></blockquote><blockquote type="cite"><span>same. But also, *strings that normalize differently will always collate</span><br></blockquote><blockquote type="cite"><span>differently*. In other words, for equality, it is sufficient to compare the</span><br></blockquote><blockquote type="cite"><span>strings' normalized forms and see if they are the same. We can therefore</span><br></blockquote><blockquote type="cite"><span>entirely skip the expensive part of collation for equality comparison.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Next, naturally, anything that applies to equality also applies to hashing: it</span><br></blockquote><blockquote type="cite"><span>is sufficient to hash the string's normalized form, bypassing collation keys.</span><br></blockquote><span></span><br><span>That's a great catch.</span><br><span></span><br><blockquote type="cite"><span>This leaves us executing the full UCA *only* for localized sorting, and ICU's</span><br></blockquote><blockquote type="cite"><span>implementation has apparently been very well optimized.</span><br></blockquote><span></span><br><span>Sounds good to me.</span><br><span></span><br><blockquote type="cite"><span>Because the current `Comparable` protocol expresses all comparisons with binary</span><br></blockquote><blockquote type="cite"><span>operators, string comparisons—which may require</span><br></blockquote><blockquote type="cite"><span>additional [options](#operations-with-options)—do not fit smoothly into the</span><br></blockquote><blockquote type="cite"><span>existing syntax. At the same time, we'd like to solve other problems with</span><br></blockquote><blockquote type="cite"><span>comparison, as outlined</span><br></blockquote><blockquote type="cite"><span>in</span><br></blockquote><blockquote type="cite"><span>[this proposal](<a href="https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e">https://gist.github.com/CodaFi/f0347bd37f1c407bf7ea0c429ead380e</a>)</span><br></blockquote><blockquote type="cite"><span>(implemented by changes at the head</span><br></blockquote><blockquote type="cite"><span>of</span><br></blockquote><blockquote type="cite"><span>[this branch](<a href="https://github.com/CodaFi/swift/commits/space-the-final-frontier">https://github.com/CodaFi/swift/commits/space-the-final-frontier</a>)).</span><br></blockquote><blockquote type="cite"><span>We should adopt a modification of that proposal that uses a method rather than</span><br></blockquote><blockquote type="cite"><span>an operator `<=>`:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>enum SortOrder { case before, same, after }</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>protocol Comparable : Equatable {</span><br></blockquote><blockquote type="cite"><span>func compared(to: Self) -> SortOrder</span><br></blockquote><blockquote type="cite"><span>...</span><br></blockquote><blockquote type="cite"><span>}</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>This change will give us a syntactic platform on which to implement methods with</span><br></blockquote><blockquote type="cite"><span>additional, defaulted arguments, thereby unifying and regularizing comparison</span><br></blockquote><blockquote type="cite"><span>across the library.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>extension String {</span><br></blockquote><blockquote type="cite"><span>func compared(to: Self) -> SortOrder</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>}</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><span></span><br><span>While it's great that `compared(to:case:etc.)` is parallel to `compared(to:)`, you don't actually want to *use* anything like `compared(to:)` if you can help it. Think about the clarity at the use site:</span><br><span></span><br><span> if foo.compared(to: bar, case: .insensitive, locale: .current) == .before { … }</span><br></div></blockquote><div><br></div><div>Right. We intend to keep the usual comparison operators.</div><div><br></div>Poor readability of "foo <=> bar == .before" is another reason we think that giving up on "<=>" is no great loss.<div><br></div><div><div><blockquote type="cite"><div><span></span><span>The operands and sense of the comparison are kind of lost in all this garbage. You really want to see `foo < bar` in this code somewhere, but you don't.</span><br></div></blockquote><div><br></div>Yeah, we thought about trying to build a DSL for that, but failed. I think the best possible option would be something like:</div><div><br></div><div> foo.comparison(case: .insensitive, locale: .current) < bar</div><div><br></div><div>The biggest problem is that you can build things like</div><div><br></div><div> <span style="background-color: rgba(255, 255, 255, 0);"> fu = foo.comparison(case: .insensitive, locale: .current)</span></div><div><span style="background-color: rgba(255, 255, 255, 0);"> br = bar.comparison(case: .sensitive)</span></div><div> fu < br // what does this mean?</div><div><br></div><div>We could even prevent such nonsense from compiling, but the cost in library API surface area is quite large.</div><div><br><blockquote type="cite"><div><span></span><span>I'm struggling a little with the naming and syntax, but as a general approach, I think we want people to use something more like this:</span><br><span></span><br><span> if StringOptions(case: .insensitive, locale: .current).compare(foo < bar) { … }</span><br></div></blockquote><div><br></div>Yeah, we can't do that without making </div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>let a = foo < bar</div><div><br></div><div>ambiguous</div><div><br><blockquote type="cite"><div><span>Which might have an implementation like:</span><br><span></span><br><span> // This protocol might actually be part of your `Unicode` protocol; I'm just breaking it out separately here.</span><br><span> protocol StringOptionsComparable {</span><br><span> func compare(to: Self, options: StringOptions) -> SortOrder</span><br><span> }</span><br><span> extension StringOptionsComparable {</span><br><span> static func < (lhs: Self, rhs: Self) -> (lhs: Self, rhs: Self, op: (SortOrder) -> Bool) {</span><br><span> return (lhs, rhs, { $0 == .before })</span><br><span> }</span><br><span> static func == (lhs: Self, rhs: Self) -> (lhs: Self, rhs: Self, op: (SortOrder) -> Bool) {</span><br><span> return (lhs, rhs, { $0 == .same })</span><br><span> }</span><br><span> static func > (lhs: Self, rhs: Self) -> (lhs: Self, rhs: Self, op: (SortOrder) -> Bool) {</span><br><span> return (lhs, rhs, { $0 == .after })</span><br><span> }</span><br><span> // etc.</span><br><span> }</span><br><span> </span><br><span> struct StringOptions {</span><br><span> // Obvious properties and initializers go here</span><br><span> </span><br><span> func compare<StringType: StringOptionsComparable>(_ expression: (lhs: StringType, rhs: StringType, op: (SortOrder) -> Bool)) -> Bool {</span><br><span> return expression.op( expression.lhs.compare(to: expression.rhs, options: self) )</span><br><span> }</span><br><span> }</span><br><span></span><br><span>You could also imagine much less verbose syntaxes using custom operators. Strawman example:</span><br><span></span><br><span> if foo < bar %% (case: .insensitive, locale: .current) { … }</span><br><span></span><br><span>I think this would make human-friendly comparisons much easier to write and understand than adding a bunch of options to a `compared(to:)` call.</span><br></div></blockquote><div><br></div>That one has the same problem with ambiguity of "a < b". There might be an answer here but it's not obvious and I feel solving it can wait a little.</div><div><br></div><div><blockquote type="cite"><div><blockquote type="cite"><span>This quirk aside, every aspect of strings-as-collections-of-graphemes appears to</span><br></blockquote><blockquote type="cite"><span>comport perfectly with Unicode. We think the concatenation problem is tolerable,</span><br></blockquote><blockquote type="cite"><span>because the cases where it occurs all represent partially-formed constructs. </span><br></blockquote><blockquote type="cite"><span>...</span><br></blockquote><blockquote type="cite"><span>Admitting these cases encourages exploration of grapheme composition and is</span><br></blockquote><blockquote type="cite"><span>consistent with what appears to be an overall Unicode philosophy that “no</span><br></blockquote><blockquote type="cite"><span>special provisions are made to get marginally better behavior for… cases that</span><br></blockquote><blockquote type="cite"><span>never occur in practice.”[2]</span><br></blockquote><span></span><br><span>This sounds good to me.</span><br><span></span><br><blockquote type="cite"><span>### Unification of Slicing Operations</span><br></blockquote><span></span><br><span>I think you know what I think about this. :^)</span><br><span></span><br><span>(By the way, I've at least partially let this proposal drop for the moment because it's so dependent on generic subscripts to really be an improvement. I do plan to pick it up when those arrive; ping me then if I don't notice.)</span><br></div></blockquote><div><br></div>Okeydoke.</div><div><br><blockquote type="cite"><div><span>A question, though. We currently have a couple of methods, mostly with `subrange` in their names, that can be thought of as slicing operations but aren't:</span><br><span></span><br><span> collection.removeSubrange(i..<j)</span><br><span> collection[i..<j].removeAll()</span><br><span> </span><br><span> collection.replaceSubrange(i..<j, with: others)</span><br><span> collection[i..<j].replaceAll(with: others) // hypothetically</span><br><span></span><br><span>Should these be changed, too? Can we make them efficient (in terms of e.g. copy-on-write) if we do?</span><br></div></blockquote><div><br></div><div>We could, once the ownership model is implemented. However, I'm not sure whether it's enough of an improvement to be worth doing. You could go all the way to</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>collection[i..<j] = EmptyCollection()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>collection[i..<j] = others</div><div><br></div><div>But for that we'd need to (at least) introduce write-only subscripts.</div></div><div><br></div><div><blockquote type="cite"><div><blockquote type="cite"><span>### Substrings</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>When implementing substring slicing, languages are faced with three options:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>1. Make the substrings the same type as string, and share storage.</span><br></blockquote><blockquote type="cite"><span>2. Make the substrings the same type as string, and copy storage when making the substring.</span><br></blockquote><blockquote type="cite"><span>3. Make substrings a different type, with a storage copy on conversion to string.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>We think number 3 is the best choice.</span><br></blockquote><span></span><br><span>I agree, and I think `Substring` is the right name for it: parallel to `SubSequence`, explains where it comes from, captures the trade-offs nicely. `StringSlice` is parallel to `ArraySlice`, but it strikes me as a "foolish consistency", as the saying goes; it avoids a term of art for little reason I can see.</span><br><span></span><br><span>However, is there a reason we're talking about using a separate `Substring` type at all, instead of using `Slice<String>`? </span></div></blockquote><div><br></div>Yes: we couldn't specialize its representation to store short substrings inline, at least not without introducing an undesirable level of complexity.</div><div> <br><blockquote type="cite"><div><span>Perhaps I'm missing something, but I *think* it does everything we need here. (Of course, you could say the same thing about `ArraySlice`, and yet we have that, too.)</span><br></div></blockquote><div><br></div>ArraySlice is doomed :-)</div><div><br></div><div><a href="https://bugs.swift.org/browse/SR-3631">https://bugs.swift.org/browse/SR-3631</a></div><div><br><blockquote type="cite"><div><blockquote type="cite"><span>The downside of having two types is the inconvenience of sometimes having a</span><br></blockquote><blockquote type="cite"><span>`Substring` when you need a `String`, and vice-versa. It is likely this would</span><br></blockquote><blockquote type="cite"><span>be a significantly bigger problem than with `Array` and `ArraySlice`, as</span><br></blockquote><blockquote type="cite"><span>slicing of `String` is such a common operation. It is especially relevant to</span><br></blockquote><blockquote type="cite"><span>existing code that assumes `String` is the currency type. To ease the pain of</span><br></blockquote><blockquote type="cite"><span>type mismatches, `Substring` should be a subtype of `String` in the same way</span><br></blockquote><blockquote type="cite"><span>that `Int` is a subtype of `Optional<Int>`.</span><br></blockquote><span></span><br><span>I've seen people struggle with the `Array`/`ArraySlice` issue when writing recursive algorithms, so personally, I'd like to see a more general solution that handles all `Collection`s.</span><br></div></blockquote><div><br></div>The more general solution is "extend Unicode" or "extend Collection" (and when a String <u>parameter</u> is needed, "make your method generic over Collection/Unicode").</div><div><br><blockquote type="cite"><div><span>Rather than having an implicit copying conversion from `String` to `Substring` (or `Array` to `ArraySlice`, or `Collection` to `Collection.SubSequence`), I wonder if implicitly converting in the other direction might be more useful, at least in some circumstances. Converting in this direction does *not* involve an implicit copy, merely calculating a range, so you won't have the same performance surprises. On the other hand, it's also useful in fewer situations.</span><br></div></blockquote><div><br></div>That's the problem, right there, combined with the fact that we don't have a terse syntax like s[] for going the other way. I think it would be a much more elegant design, personally, but I don't see the tradeoffs working out. If we can come up with a way to do it that works, we should. So far, Ben and I have failed.</div><div><br><blockquote type="cite"><div><span></span><span>(If we did go with consistently using `Slice<T>`, this might merely be a special-cased `T -> Slice<T>` conversion. One type, special-cased until we feel comfortable inventing a general mechanism.)</span><br><span></span><br><blockquote type="cite"><span>A user who needs to optimize away copies altogether should use this guideline:</span><br></blockquote><blockquote type="cite"><span>if for performance reasons you are tempted to add a `Range` argument to your</span><br></blockquote><blockquote type="cite"><span>method as well as a `String` to avoid unnecessary copies, you should instead</span><br></blockquote><blockquote type="cite"><span>use `Substring`.</span><br></blockquote><span></span><br><span>I do like this as a guideline, though. There's definitely room in the standard library for "a string and a range of that string to operate upon".</span><br></div></blockquote><div><br></div>I don't know what you mean. It's our intention that nothing but the lowest level operations (e.g. replaceRange) would work on ranges when they could instead be working on slices.</div><div><br><blockquote type="cite"><div><blockquote type="cite"><span>##### The “Empty Subscript”</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>To make it easy to call such an optimized API when you only have a `String` (or</span><br></blockquote><blockquote type="cite"><span>to call any API that takes a `Collection`'s `SubSequence` when all you have is</span><br></blockquote><blockquote type="cite"><span>the `Collection`), we propose the following “empty subscript” operation,</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I```swift</span><br></blockquote><blockquote type="cite"><span>extension Collection {</span><br></blockquote><blockquote type="cite"><span> subscript() -> SubSequence { </span><br></blockquote><blockquote type="cite"><span> return self[startIndex..<endIndex] </span><br></blockquote><blockquote type="cite"><span> }</span><br></blockquote><blockquote type="cite"><span>}</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>which allows the following usage:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>funcThatIsJustLooking(at: person.name[]) // pass person.name as Substring</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><span></span><br><span>That's a little bit funky, but I guess it might work.</span><br><span></span><br><blockquote type="cite"><span>Therefore, APIs that operate on an `NSString`/`NSRange` pair should be imported</span><br></blockquote><blockquote type="cite"><span>without the `NSRange` argument. The Objective-C importer should be changed to</span><br></blockquote><blockquote type="cite"><span>give these APIs special treatment so that when a `Substring` is passed, instead</span><br></blockquote><blockquote type="cite"><span>of being converted to a `String`, the full `NSString` and range are passed to</span><br></blockquote><blockquote type="cite"><span>the Objective-C method, thereby avoiding a copy.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>As a result, you would never need to pass an `NSRange` to these APIs, which</span><br></blockquote><blockquote type="cite"><span>solves the impedance problem by eliminating the argument, resulting in more</span><br></blockquote><blockquote type="cite"><span>idiomatic Swift code while retaining the performance benefit. To help users</span><br></blockquote><blockquote type="cite"><span>manually handle any cases that remain, Foundation should be augmented to allow</span><br></blockquote><blockquote type="cite"><span>the following syntax for converting to and from `NSRange`:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>let nsr = NSRange(i..<j, in: s) // An NSRange corresponding to s[i..<j]</span><br></blockquote><blockquote type="cite"><span>let iToJ = Range(nsr, in: s) // Equivalent to i..<j</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><span></span><br><span>I sort of like this, but note that if we use `String` -> `Substring` conversion instead of the other way around, there's less magic needed to get this effect: `NSString, NSRange` can be imported as `Substring`, which automatically converts from `String` in exactly the manner we want it to.</span><br></div></blockquote><div><br></div>Indeed.</div><div><br><blockquote type="cite"><div><span></span><br><blockquote type="cite"><span>Since Unicode conformance is a key feature of string processing in swift, we</span><br></blockquote><blockquote type="cite"><span>call that protocol `Unicode`:</span><br></blockquote><span></span><br><span>I'm sorry, I think the name is too clever by half. It sounds something like what `UnicodeCodec` actually is. Or maybe a type representing a version of the Unicode standard or something. I'd prefer something more prosaic like `StringProtocol`.</span><br></div></blockquote><div><br></div>It's an option we considered. So far I think Unicode is better (most especially if we end up with a "facade" design) but we should discuss it.</div><div> <br><blockquote type="cite"><div><span></span><br><blockquote type="cite"><span>**Note:** `Unicode` would make a fantastic namespace for much of</span><br></blockquote><blockquote type="cite"><span>what's in this proposal if we could get the ability to nest types and</span><br></blockquote><blockquote type="cite"><span>protocols in protocols.</span><br></blockquote><span></span><br><span>I mean, sure, but then you imagine it being used generically:</span><br><span></span><br><span> func parse<UnicodeType: Unicode>(_ source: UnicodeType) -> UnicodeType</span><br><span> // which concrete types can `source` be???</span><br></div></blockquote><div><br></div>All "string" types, including String, Substring, UTF8String, StaticString, etc.</div><div><br></div><div><blockquote type="cite"><div><blockquote type="cite"><span>We should provide convenient APIs processing strings by character. For example,</span><br></blockquote><blockquote type="cite"><span>it should be easy to cleanly express, “if this string starts with `"f"`, process</span><br></blockquote><blockquote type="cite"><span>the rest of the string as follows…” Swift is well-suited to expressing this</span><br></blockquote><blockquote type="cite"><span>common pattern beautifully, but we need to add the APIs. Here are two examples</span><br></blockquote><blockquote type="cite"><span>of the sort of code that might be possible given such APIs:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>if let firstLetter = input.droppingPrefix(alphabeticCharacter) {</span><br></blockquote><blockquote type="cite"><span> somethingWith(input) // process the rest of input</span><br></blockquote><blockquote type="cite"><span>}</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>if let (number, restOfInput) = input.parsingPrefix(Int.self) {</span><br></blockquote><blockquote type="cite"><span> ...</span><br></blockquote><blockquote type="cite"><span>}</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>The specific spelling and functionality of APIs like this are TBD. The larger</span><br></blockquote><blockquote type="cite"><span>point is to make sure matching-and-consuming jobs are well-supported.</span><br></blockquote><span></span><br><span>Yes.</span><br><span></span><br><blockquote type="cite"><span>#### Unified Pattern Matcher Protocol</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Many of the current methods that do matching are overloaded to do the same</span><br></blockquote><blockquote type="cite"><span>logical operations in different ways, with the following axes:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>- Logical Operation: `find`, `split`, `replace`, match at start</span><br></blockquote><blockquote type="cite"><span>- Kind of pattern: `CharacterSet`, `String`, a regex, a closure</span><br></blockquote><blockquote type="cite"><span>- Options, e.g. case/diacritic sensitivity, locale. Sometimes a part of</span><br></blockquote><blockquote type="cite"><span> the method name, and sometimes an argument</span><br></blockquote><blockquote type="cite"><span>- Whole string or subrange.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>We should represent these aspects as orthogonal, composable components,</span><br></blockquote><blockquote type="cite"><span>abstracting pattern matchers into a protocol like</span><br></blockquote><blockquote type="cite"><span>[this one](<a href="https://github.com/apple/swift/blob/master/test/Prototypes/PatternMatching.swift#L33">https://github.com/apple/swift/blob/master/test/Prototypes/PatternMatching.swift#L33</a>),</span><br></blockquote><blockquote type="cite"><span>that can allow us to define logical operations once, without introducing</span><br></blockquote><blockquote type="cite"><span>overloads, and massively reducing API surface area.</span><br></blockquote><span></span><br><span>*Very* yes.</span><br><span></span><br><blockquote type="cite"><span>For example, using the strawman prefix `%` syntax to turn string literals into</span><br></blockquote><blockquote type="cite"><span>patterns, the following pairs would all invoke the same generic methods:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>if let found = s.firstMatch(%"searchString") { ... }</span><br></blockquote><blockquote type="cite"><span>if let found = s.firstMatch(someRegex) { ... }</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>for m in s.allMatches((%"searchString"), case: .insensitive) { ... }</span><br></blockquote><blockquote type="cite"><span>for m in s.allMatches(someRegex) { ... }</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>let items = s.split(separatedBy: ", ")</span><br></blockquote><blockquote type="cite"><span>let tokens = s.split(separatedBy: CharacterSet.whitespace)</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><span></span><br><span>Very, *very* yes.</span><br><span></span><br><span>If we do this, rather than your `%` operator (or whatever it becomes), I wonder if we can have these extensions:</span><br><span></span><br><span> // Assuming a protocol like:</span><br><span> protocol Pattern {</span><br><span> associatedtype PatternElement</span><br><span> func matches<CollectionType: Collection>(…) -> … where CollectionType.Element == Element</span><br><span> }</span><br><span> extension Equatable: Pattern {</span><br><span> typealias PatternElement = Self</span><br><span> …</span><br><span> }</span><br><span> extension Collection: Pattern where Element: Equatable {</span><br><span> typealias PatternElement = Element</span><br><span> }</span><br><span></span><br><span>...although then `Collection` would conform to `Pattern` through both itself and (conditionally) `Equatable`. Hmm.</span><br><span></span><br><span>I suppose we faced this same problem elsewhere and ended up with things like:</span><br><span></span><br><span> mutating func append(_ element: Element)</span><br><span> mutating func append<Seq: Sequence>(contentsOf seq: Seq) where Seq.Iterator.Element == Element</span><br><span></span><br><span>So we could do things like:</span><br><span></span><br><span> str.firstMatch("x") // single element, so this is a Character</span><br><span> str.firstMatch(contentsOf("xy"))</span><br><span> str.firstMatch(anyOf(["x", "y"] as Set))</span><br></div></blockquote><div><br></div>I really, really want to explore these ideas further, and I really, really don't want to do it in this thread, if you don't mind. There are lots of ways to slice this particular cupcake.</div><div><br><blockquote type="cite"><div><span></span><br><blockquote type="cite"><span>#### Index Interchange Among Views</span><br></blockquote><span></span><br><span>I really, really, really want this.</span><br><span></span><br><blockquote type="cite"><span>We think random-access</span><br></blockquote><blockquote type="cite"><span>*code-unit storage* is a reasonable requirement to impose on all `String`</span><br></blockquote><blockquote type="cite"><span>instances.</span><br></blockquote><span></span><br><span>Wait, you do? Doesn't that mean either using UTF-32, inventing a UTF-24 to use, or using some kind of complicated side table that adjusts for all the multi-unit characters in a UTF-16 or UTF-8 string? None of these sound ideal.</span><br></div></blockquote><div><br></div>No; I'm not sure why you would think that.</div><div><br><blockquote type="cite"><div><span></span><blockquote type="cite"><span>Index interchange between `String` and its `unicodeScalars`, `codeUnits`,</span><br></blockquote><blockquote type="cite"><span>and [`extendedASCII`](#parsing-ascii-structure) views can be made entirely</span><br></blockquote><blockquote type="cite"><span>seamless by having them share an index type (semantics of indexing a `String`</span><br></blockquote><blockquote type="cite"><span>between grapheme cluster boundaries are TBD—it can either trap or be forgiving).</span><br></blockquote><span></span><br><span>I think it should be forgiving, and I think it should be forgiving in a very specific way: It should treat indexing in the middle of a cluster as though you indexed at the beginning.</span><br></div></blockquote><div><br></div>That's my intuition as well.</div><div><br><blockquote type="cite"><div><span></span><span>The reason is `AttributedString`. You can think of `AttributedString` as being a type which adds additional views to a `String`; these views are indexed by `String.Index`, just like `String`, `String.UnicodeScalarView`, et.al., and advancing an index with these views advances it to the beginning of the next run. But you can also just subscript these views with an arbitrary index in the middle of a run, and it'll work correctly.</span><br><span></span><br><span>I think it would be useful for this behavior to be consistent among all `String` views.</span><br><span></span><br><blockquote type="cite"><span>Having a common index allows easy traversal into the interior of graphemes,</span><br></blockquote><blockquote type="cite"><span>something that is often needed, without making it likely that someone will do it</span><br></blockquote><blockquote type="cite"><span>by accident.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>- `String.index(after:)` should advance to the next grapheme, even when the</span><br></blockquote><blockquote type="cite"><span> index points partway through a grapheme.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>- `String.index(before:)` should move to the start of the grapheme before</span><br></blockquote><blockquote type="cite"><span> the current position.</span><br></blockquote><span></span><br><span>Good.</span><br><span></span><br><blockquote type="cite"><span>Seamless index interchange between `String` and its UTF-8 or UTF-16 views is not</span><br></blockquote><blockquote type="cite"><span>crucial, as the specifics of encoding should not be a concern for most use</span><br></blockquote><blockquote type="cite"><span>cases, and would impose needless costs on the indices of other views.</span><br></blockquote><span></span><br><span>I don't know about this, at least for the UTF-16 view. Here's why:</span><br><span></span><br><blockquote type="cite"><span>That leaves the interchange of bare indices with Cocoa APIs trafficking in</span><br></blockquote><blockquote type="cite"><span>`Int`. Hopefully such APIs will be rare, but when needed, the following</span><br></blockquote><blockquote type="cite"><span>extension, which would be useful for all `Collections`, can help:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>extension Collection {</span><br></blockquote><blockquote type="cite"><span> func index(offset: IndexDistance) -> Index {</span><br></blockquote><blockquote type="cite"><span> return index(startIndex, offsetBy: offset)</span><br></blockquote><blockquote type="cite"><span> }</span><br></blockquote><blockquote type="cite"><span> func offset(of i: Index) -> IndexDistance {</span><br></blockquote><blockquote type="cite"><span> return distance(from: startIndex, to: i)</span><br></blockquote><blockquote type="cite"><span> }</span><br></blockquote><blockquote type="cite"><span>}</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Then integers can easily be translated into offsets into a `String`'s `utf16`</span><br></blockquote><blockquote type="cite"><span>view for consumption by Cocoa:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>let cocoaIndex = s.utf16.offset(of: String.UTF16Index(i))</span><br></blockquote><blockquote type="cite"><span>let swiftIndex = s.utf16.index(offset: cocoaIndex)</span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><span></span><br><span>I worry that this conversion will be too obscure.</span></div></blockquote><div><br></div>I very much hope it will be rare enough that it'll be OK, but if it isn't, we can always have</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>l<span style="background-color: rgba(255, 255, 255, 0);">et cocoaIndex = s.utf16Offset(of: i)</span></div><div><br></div><div>and/or take other measures to simplify it.</div><div><br><blockquote type="cite"><div><span>In Objective-C, you don't really think very much about what "character" means; it's just an index that points to a location inside the string. I don't think people will know to use the `utf16` view instead of the others—especially the plain `String` version, which would be the most obvious one to use.</span><br><span></span><br><span>I think I'd prefer to see the following:</span><br><span></span><br><span>1. UTF-16 is the storage format, at least for an "ordinary" `Swift.String`.</span><br></div></blockquote><div><br></div>It will be, in the common case, but <u>many</u> people seem to want plain String to be able to store UTF-8, and I'm not yet prepared to rule that out.</div><div><br><blockquote type="cite"><div><span></span><span>2. `String.Index` is used down to the `UTF16View`. It stores a UTF-16 offset.</span><br><span></span><br><span>3. With just the standard library imported, `String.Index` does not have any obvious way to convert to or from an `Int` offset; you use `index(_:offsetBy:)` on one of the views. `utf16`'s implementation is just faster than the others.</span><br></div></blockquote><div><br></div>This is roughly where we are today.</div><div><br><blockquote type="cite"><div><span>4. Foundation adds `init(_:)` methods to `String.Index` and `Int`, as well as `Range<String.Index>` and `NSRange`, which perform mutual conversions:</span><br><span></span><br><span> XCTAssertEqual(Int(String.Index(cocoaIndex)), cocoaIndex)</span><br><span> XCTAssertEqual(NSRange(Range<String.Index>(cocoaRange)), cocoaRange)</span><br><span></span><br><span>I think this would really help to guide people to the right APIs for the task.</span><br><span></span><br><span>(Also, it would make my `AttributedString` thing work better, too.)</span><br><span></span><br><blockquote type="cite"><span>### Formatting</span><br></blockquote><span></span><br><span>Briefly: I am, let's say, 95% on board with your plan to replace format strings with interpolation and format methods. The remaining 5% concern is that it we'll need an adequate replacement for the ability to load a format string dynamically and have it reorder or alter the formatting of interpolated values. Obviously dynamic format strings are dangerous and limited, but where you *can* use them, they're invaluable.\</span><br></div></blockquote><div><br></div>Yes. We have ideas, though they're far from baked.</div><div><br><blockquote type="cite"><div><blockquote type="cite"><span>#### String Interpolation</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Swift string interpolation provides a user-friendly alternative to printf's</span><br></blockquote><blockquote type="cite"><span>domain-specific language (just write ordinary swift code!) and its type safety</span><br></blockquote><blockquote type="cite"><span>problems (put the data right where it belongs!) but the following issues prevent</span><br></blockquote><blockquote type="cite"><span>it from being useful for localized formatting (among other jobs):</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span> * [SR-2303](<a href="https://bugs.swift.org/browse/SR-2303">https://bugs.swift.org/browse/SR-2303</a>) We are unable to restrict</span><br></blockquote><blockquote type="cite"><span> types used in string interpolation.</span><br></blockquote><blockquote type="cite"><span> * [SR-1260](<a href="https://bugs.swift.org/browse/SR-1260">https://bugs.swift.org/browse/SR-1260</a>) String interpolation can't</span><br></blockquote><blockquote type="cite"><span> distinguish (fragments of) the base string from the string substitutions.</span><br></blockquote><span></span><br><span>If I find some copious free time, I could try to develop proposals for one or both of these. Would there be interest in them at this point? (Feel free to contact me off-list about this, preferably in a new thread.)</span><br><span></span><br><span>(Okay, one random thought, because I can't resist: Perhaps the "\(…)" syntax can be translated directly into an `init(…)` on the type you're creating. That is, you can write:</span><br><span></span><br><span> let x: MyString = "foo \(bar) baz \(quux, radix: 16)"</span><br><span></span><br><span>And that translates to:</span><br><span></span><br><span> let x = MyString(stringInterpolationSegments:</span><br><span> MyString(stringLiteral: "foo "),</span><br><span> MyString(bar),</span><br><span> MyString(stringLiteral: " baz "),</span><br><span> MyString(quux, radix: 16)</span><br><span> )</span><br><span></span><br><span>That would require you to redeclare `String` initializers on your own string type, but you probably need some of your own logic anyway, right?)</span><br></div></blockquote><div><br></div>Let's go to a separate thread for this, as you suggested.</div><div><br><blockquote type="cite"><div><blockquote type="cite"><span>In the long run, we should improve Swift string interpolation to the point where</span><br></blockquote><blockquote type="cite"><span>it can participate in most any formatting job. Mostly this centers around</span><br></blockquote><blockquote type="cite"><span>fixing the interpolation protocols per the previous item, and supporting</span><br></blockquote><blockquote type="cite"><span>localization.</span><br></blockquote><span></span><br><span>For what it's worth, by using a hacky workaround for SR-1260, I've written (Swift 2.0) code that passes strings with interpolations through the Foundation localized string tables: <<a href="https://gist.github.com/brentdax/79fa038c0af0cafb52dd">https://gist.github.com/brentdax/79fa038c0af0cafb52dd</a>> Obviously that's just a start, but it is incredibly convenient.</span><br></div></blockquote><div><br></div>I know; it's an inspiration :-)</div><div><br><blockquote type="cite"><div><blockquote type="cite"><span>### C String Interop</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Our support for interoperation with nul-terminated C strings is scattered and</span><br></blockquote><blockquote type="cite"><span>incoherent, with 6 ways to transform a C string into a `String` and four ways to</span><br></blockquote><blockquote type="cite"><span>do the inverse. These APIs should be replaced with the following</span><br></blockquote><span></span><br><span>These APIs are much better than the status quo, but it's a shame that we can't have them handle non-nul-terminated data, too.</span><br></div></blockquote><div><br></div>We thought about unifying them with other transcoding APIs, but the pointer-to-nul-terminated-code-units case is sufficiently important that we think they deserve dedicated support.</div><div><br><blockquote type="cite"><div><span></span><span>Actually... (Begin shaggy dog story...)</span><br><span></span><br><span>Suppose you introduce an `UnsafeNulTerminatedBufferPointer` type. Then you could write a *very* high-level API which handles pretty much every conversion under the sun:</span><br><span></span><br><span> extension String {</span><br><span> /// Constructs a `String` from a sequence of `codeUnits` in an indicated `encoding`.</span><br><span> /// </span><br><span> /// - Parameter codeUnits: A sequence of code units in the given `encoding`.</span><br><span> /// - Parameter encoding: The encoding the code units are in.</span><br><span> init<CodeUnits: Sequence, Encoding: UnicodeEncoding>(_ codeUnits: CodeUnits, encoding: Encoding)</span><br><span> where CodeUnits.Iterator.Element == Encoding.CodeUnit</span><br><span> }</span><br></div></blockquote><div><br></div>Yes, we intend to support something like that.</div><div><br><blockquote type="cite"><div><span>For UTF-8, at least, that would cover reading from `Array`, `UnsafeBufferPointer`, `UnsafeRawBufferPointer`, `UnsafeNulTerminatedBufferPointer`, `Data`, you name it. Maybe we could have a second one that always takes something producing bytes, no matter the encoding used:</span><br><span></span><br><span> extension String {</span><br><span> /// Constructs a `String` from the code units contained in `bytes` in a given `encoding`.</span><br><span> /// </span><br><span> /// - Parameter bytes: A sequence of bytes expressing code units in the given `encoding`.</span><br><span> /// - Parameter encoding: The encoding the code units are in.</span><br><span> init<Bytes: Sequence, Encoding: UnicodeEncoding>(_ codeUnits: CodeUnits, encoding: Encoding)</span><br><span> where CodeUnits.Iterator.Element == UInt8</span><br><span> }</span><br><span></span><br><span>These two initializers would replace...um, something like eight existing ones, including ones from Foundation. On the other hand, this is *very* generic. And, unless we actually changed the way `char *` imported to `UnsafeNulTerminatedBufferPointer<CChar>`, the C string call sequence would be pretty complicated:</span><br><span></span><br><span> String(UnsafeNulTerminatedBufferPointer(start: cString), encoding: UTF8.self)</span><br><span></span><br><span>So you might end up having to wrap it in an `init(cString:)` anyway, just for convenience. Oh well, it was worth exploring.</span><br></div></blockquote><div><br></div>I think you ended up where we did.</div><div><br><blockquote type="cite"><div><span>Prototype of the above: <a href="https://gist.github.com/brentdax/8b71f46b424dc64abaa77f18556e607b">https://gist.github.com/brentdax/8b71f46b424dc64abaa77f18556e607b</a></span><br><span></span><br><span>(Hmm...maybe bridge `char *` to a type like this instead?</span><br><span></span><br><span> struct CCharPointer {</span><br><span> var baseAddress: UnsafePointer<CChar> { get }</span><br><span> var nulTerminated: UnsafeNulTerminatedBufferPointer<CChar> { get }</span><br><span> func ofLength(_ length: Int) -> UnsafeBufferPointer<CChar></span><br><span> }</span><br><span></span><br><span>Nah, probably not gonna happen...)</span><br><span></span><br><blockquote type="cite"><span> init(cString nulTerminatedUTF8: UnsafePointer<CChar>)</span><br></blockquote><span></span><br><span>By the way, I just noticed an impedance mismatch in current Swift: `CChar` is usually an `Int8`, but `UnicodeScalar` and `UTF8` currently want `UInt8`. It'd be nice to address this somehow, if only by adding some signed variants or something.</span><br></div></blockquote><div><br></div>We thought about that problem and landed on the proposed interface above as all that is needed to resolve it.</div><div><br><blockquote type="cite"><div><span></span><br><blockquote type="cite"><span>### High-Performance String Processing</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Many strings are short enough to store in 64 bits, many can be stored using only</span><br></blockquote><blockquote type="cite"><span>8 bits per unicode scalar, others are best encoded in UTF-16, and some come to</span><br></blockquote><blockquote type="cite"><span>us already in some other encoding, such as UTF-8, that would be costly to</span><br></blockquote><blockquote type="cite"><span>translate. Supporting these formats while maintaining usability for</span><br></blockquote><blockquote type="cite"><span>general-purpose APIs demands that a single `String` type can be backed by many</span><br></blockquote><blockquote type="cite"><span>different representations.</span><br></blockquote><span></span><br><span>Just putting a pin in this, because I'll want to discuss it a little later.</span><br><span></span><br><blockquote type="cite"><span>### Parsing ASCII Structure</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Although many machine-readable formats support the inclusion of arbitrary</span><br></blockquote><blockquote type="cite"><span>Unicode text, it is also common that their fundamental structure lies entirely</span><br></blockquote><blockquote type="cite"><span>within the ASCII subset (JSON, YAML, many XML formats). These formats are often</span><br></blockquote><blockquote type="cite"><span>processed most efficiently by recognizing ASCII structural elements as ASCII,</span><br></blockquote><blockquote type="cite"><span>and capturing the arbitrary sections between them in more-general strings. The</span><br></blockquote><blockquote type="cite"><span>current String API offers no way to efficiently recognize ASCII and skip past</span><br></blockquote><blockquote type="cite"><span>everything else without the overhead of full decoding into unicode scalars.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>For these purposes, strings should supply an `extendedASCII` view that is a</span><br></blockquote><blockquote type="cite"><span>collection of `UInt32`, where values less than `0x80` represent the</span><br></blockquote><blockquote type="cite"><span>corresponding ASCII character, and other values represent data that is specific</span><br></blockquote><blockquote type="cite"><span>to the underlying encoding of the string.</span><br></blockquote><span></span><br><span>This sounds interesting, but:</span><br><span></span><br><span>1. It doesn't sound like you anticipate there being any way to compare an element of the `extendedASCII` view to a character literal. That seems like it'd be really useful.</span><br></div></blockquote><div><br></div>We don't have character literals :-)</div><div><br></div><div>However, I agree that there needs to be a way to do it. The thing would be to make it easy to construct a UInt8 from a string literal. There are a few possibilities; <span style="background-color: rgba(255, 255, 255, 0);">I'm a little nervous about making this work:</span></div><div><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div><span style="background-color: rgba(255, 255, 255, 0);"><span class="Apple-tab-span" style="white-space: pre;"> </span>if c == "X" { ... }</span></div><div><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div>but maybe I should just get over it. The cleanest alternative I can think of is:</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if c == ascii("X") { ... }</div><div><br></div><div>where "X" is required by the compiler to be a single ascii character.</div><div><br></div><div>I guess another possibility is to introduce an ASCII type and overload operators so it can be compared with all the Ints:</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if c == "X" as ASCII { ... }</div><div><br></div><div><blockquote type="cite"><div><span>2. I don't really understand how you envision using the "data specific to the underlying encoding" sections. Presumably you'll want to convert that data into a string eventually, right?</span><br></div></blockquote><div><br></div>It already <u>is</u> in a string. The point is that we have a way to scan the string looking for ASCII patterns without transcoding it.</div><div><br><blockquote type="cite"><div><span>Do you have pseudocode or something lying around that might help us understand how you think this might be used?</span><br></div></blockquote><div><br></div>Not exactly. The pattern matching prototype you referred to earlier would be enhanced to use the extendedASCII view when it was available and the pattern being matched was suitably restricted. How, exactly, that works is still a bit of a research project though.</div><div><br><blockquote type="cite"><div><span></span><blockquote type="cite"><span>### Do we need a type-erasable base protocol for UnicodeEncoding?</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>UnicodeEncoding has an associated type, but it may be important to be able to</span><br></blockquote><blockquote type="cite"><span>traffic in completely dynamic encoding values, e.g. for “tell me the most</span><br></blockquote><blockquote type="cite"><span>efficient encoding for this string.”</span><br></blockquote><span></span><br><span>As long as you're here, we haven't talked about `UnicodeEncoding` much. I assume this is a slightly modified version of `UnicodeCodec`? Anything to say about it?</span><br></div></blockquote><div><br></div>That's basically right. You can see a first cut at it in the unicode-rethink branch on GitHub.</div><div><br><blockquote type="cite"><div><span></span><span>If it *is* similar to `UnicodeCodec`, one thing I will note is that the way `UnicodeCodec` works in code units is rather annoying for I/O. It may make sense to have some sort of type-erasing wrapper around `UnicodeCodec` which always uses bytes. (You then have to worry about endianness, of course...)</span><br></div></blockquote><div><br></div>Take a look at the branch and let me know how this looks like it would work for I/O.</div><div><br></div><div>By the way, I think I/O really needs a special kind of collection: a sort of deque built out of I/O buffer-sized chunks that are filled on demand from a Sequence. That is part, at least, of how I justify UnicodeEncoding having a Collection-based interface where UnicodeCodec used Iterator.</div><div><br><blockquote type="cite"><div><span></span><br><blockquote type="cite"><span>### Should there be a string “facade?”</span><br></blockquote><blockquote type="cite"><span>…</span><br></blockquote><blockquote type="cite"><span>An interesting variation on this design is possible if defaulted generic</span><br></blockquote><blockquote type="cite"><span>parameters are introduced to the language:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>```swift</span><br></blockquote><blockquote type="cite"><span>struct String<U: Unicode = StringStorage> </span><br></blockquote><blockquote type="cite"><span> : BidirectionalCollection {</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span> // ...APIs for high-level string processing here...</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span> var unicode: U // access to lower-level unicode details</span><br></blockquote><blockquote type="cite"><span>}</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>typealias Substring = String<StringStorage.SubSequence></span><br></blockquote><blockquote type="cite"><span>```</span><br></blockquote><span></span><br><span>I think this is a very, very interesting idea. A few notes:</span><br><span></span><br><span>* Earlier, I said I didn't like `Unicode` as a protocol name. If we go this route, I think `StringStorage` is a good name for that protocol. The default storage might be something like `UTF16StringStorage`, or just, you know, `DefaultStringStorage`.</span><br><span></span><br><span>* Earlier, you mentioned the tension between using multiple representations for flexibility and pinning down one representation for speed. One way to handle this might be to have `String`'s default `StringStorage` be a superclass or type-erased wrapper or something.</span></div></blockquote><div><br></div>Yes, that's the idea.</div><div><br><blockquote type="cite"><div><span> That way, if you just write `String`, you get something flexible; if you write `String<NFCNormalizedUTF16StringStorage>`, you get something fast.</span><br></div></blockquote><div><br></div>This only works in the "facade" variant where you have a defaulted generic parameter feature, but yes, that's the idea of that variant.</div><div><br><blockquote type="cite"><div><span></span><span>* Could `NSString` be a `StringStorage`, or support a trivial wrapper that converts it into a `StringStorage`? Would that be helpful at all?</span><br></div></blockquote><div><br></div>Yes, that's part of the idea.</div><div><br><blockquote type="cite"><div><span></span><span>* If we do this, does `String.Index` become a type-specific thing? That is, might `String<UTF8Storage>.Index` be different from `String<UTF16Storage>.Index`? </span></div></blockquote><div><br></div>Yes.</div><div><br><blockquote type="cite"><div><span>What does that mean for `String.Index` unification?</span><br></div></blockquote><div><br></div>Not much. We never intended for indices to be interchangeable among different specific string types (other than a string and its SubSequence).</div><div><br><blockquote type="cite"><div><blockquote type="cite"><span>### `description` and `debugDescription`</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>* Should these be creating localized or non-localized representations?</span><br></blockquote><span></span><br><span>`debugDescription`, I think, is non-localized; it's something helpful for the programmer, and the programmer's language is not the user's. It's also usually something you don't want to put *too* much effort into, other than to dump a lot of data about the instance.</span><br><span></span><br><span>`description` would have to change to be localizable. (Specifically, it would have to take a locale.) This is doable, of course, but it hasn't been done yet.</span><br></div></blockquote><div><br></div>Well, it could use the current locale. These things are supposed to remain lightweight.</div><div><br><blockquote type="cite"><div><span></span><blockquote type="cite"><span>* Is returning a `String` efficient enough?</span><br></blockquote><span></span><br><span>I'm not sure how important efficiency is for `description`, honestly.</span><br></div></blockquote><div><br></div>It depends how intimately this is tied into interpolation and formatting, I think.</div><div><br><blockquote type="cite"><div><blockquote type="cite"><span>* Is `debugDescription` pulling the weight of the API surface area it adds?</span><br></blockquote><span></span><br><span>Maybe? Or maybe it's better off as part of the `Mirror` instead of a property on the instance itself.</span><br></div></blockquote><div><br></div>That's a very interesting thought!</div><div><br><blockquote type="cite"><div><span></span><blockquote type="cite"><span>### `StaticString`</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>`StaticString` was added as a byproduct of standard library developed and kept</span><br></blockquote><blockquote type="cite"><span>around because it seemed useful, but it was never truly *designed* for client</span><br></blockquote><blockquote type="cite"><span>programmers. We need to decide what happens with it. Presumably *something*</span><br></blockquote><blockquote type="cite"><span>should fill its role, and that should conform to `Unicode`.</span><br></blockquote><span></span><br><span>Maybe. One complication there is that `Unicode` presumably supports mutation, which `StaticString` doesn't.</span><br></div></blockquote><div><br></div>No, Unicode doesn't support mutation. A mutable Unicode will usually conform to Unicode and RangeReplaceableCollection (but not MutableCollection, because replacing a grapheme is not an O(1) operation).</div><div><br><blockquote type="cite"><div><span>Another possibility I've discussed in the past is renaming `StaticString` to `StringLiteral` and using it largely as a way to initialize `String`. (I mentioned that in a thread about the need for public integer and floating-point literal types that are more expressive now that we're supporting larger integer/float types.) </span></div></blockquote><div><br></div>Yes, a broad redesign of all literals is crucial. However, there are other sources of static string data than literals and those need to be accommodated.</div><div><br><blockquote type="cite"><div><span>It could have just enough API surface to access it as a buffer of UTF-8 bytes and thereby build a `String` or `Data` from it.</span><br><span></span><br><span>Well, that's it for this massive email. You guys are doing a hell of a job on this.</span><br></div></blockquote><div><br></div>Thanks for all the feedback, and the encouragement!</div><div><br><blockquote type="cite"><div><span>Hope this helps,</span><br><span>-- </span><br><span>Brent Royal-Gordon</span><br><span>Architechies</span><br><span></span><br></div></blockquote></div></div></div></body></html>