<font size=2 face="sans-serif">Thanks for the clarifications.</font><br><font size=2 face="sans-serif">More comments below.</font><br><br><tt><font size=2>dabrahams@apple.com wrote on 01/24/2017 05:50:59 PM:<br><br>> Maybe it wasn't clear from the document, but the intention is that<br>> String would be able to use any model of Unicode as a backing store,
and<br>> that you could easily build unsafe models of Unicode... but also that<br>> you could use your unsafe model of Unicode directly, in string-ish
ways.<br></font></tt><br><tt><font size=2>I see. If I understand correctly, it will be possible
for instance to implement an unsafe model of Unicode with a UInt8 code
unit and a maxLengthOfEncodedScalar equal to 1 by only keeping the 8 lowest
bits of Unicode scalars.</font></tt><br><br><br><br><tt><font size=2>> > A lot of machine processing of strings continues
to deal with 8-bit <br>> > quantities (even 7-bit quantities, not UTF-8).<br>> > Swift strings are not very good at that. I see progress in the
manifesto <br>> > but nothing to really close the performance gap with C.<br>> > That's where "unsafe" mechanisms could come into play.<br>> <br>> extendedASCII is supposed to address that. Given a smart enough<br>> optimizer, it should be possible to become competitive with C even<br>> without using unsafe constructs. However, we recognize the importance<br>> of being able to squeeze out that last bit of performance by dropping<br>> down to unsafe storage.<br></font></tt><br><tt><font size=2>I doubt a 32-bit encoding can bridge the performance
gap with C in particular because wire protocols will continue to favor
compact encodings. Incoming strings will have to be expanded to the extendedASCII
representation before processing and probably compacted afterwards. So
while this may address the needs of computationally intensive string processing
tasks, this does not help simple parsing tasks on simple strings.</font></tt><br><br><br><tt><font size=2><br>> > To guarantee Unicode correctness, a C string must be validated
or <br>> > transformed to be considered a Swift string.<br>> <br>> Not really. You can do error-correction on the fly. However,
I think<br>> pre-validation is often worthwhile because once you know something
is<br>> valid it's much cheaper to decode correctly (especially for UTF-8).<br></font></tt><br><tt><font size=2>Sure. Eager vs. lazy validation is a valuable distinction,
but what I am after here is side-stepping validation altogether. I understand
now that user-defined encodings will make side-stepping validation possible.</font></tt><br><br><br><tt><font size=2><br>> > If I understand the C String interop section correctly, in Swift
4,<br>> > this should not force a copy, but traversing the string is still<br>> > required. <br>> <br>> *What* should not force a copy?</font></tt><br><br><tt><font size=2>I would like to have a constructor that takes a pointer
to a null-terminated sequence of bytes (or a sequence of bytes and a length)
and turns it into a Swift string without allocation of a new backing store
for the string and without copying the bytes in the sequence from one place
in memory to another. I understand this may require the programmer to handle
memory management for the backing store.</font></tt><br><br><br><br><tt><font size=2>> > I hope I am correct about the no-copy thing,
and I would also like to<br>> > permit promoting C strings to Swift strings without validation.
This<br>> > is obviously unsafe in general, but I know my strings... and
I care<br>> > about performance. ;)<br>> <br>> We intend to support that use-case. That's part of the reason
for the<br>> ValidUTF8 and ValidUTF16 encodings you see here:<br>> </font></tt><a href="https://github.com/apple/swift/blob/unicode-rethink/stdlib/public/"><tt><font size=2>https://github.com/apple/swift/blob/unicode-rethink/stdlib/public/</font></tt></a><tt><font size=2><br>> core/Unicode2.swift#L598<br>> and here:<br>> </font></tt><a href="https://github.com/apple/swift/blob/unicode-rethink/stdlib/public/"><tt><font size=2>https://github.com/apple/swift/blob/unicode-rethink/stdlib/public/</font></tt></a><tt><font size=2><br>> core/Unicode2.swift#L862<br></font></tt><br><tt><font size=2>OK</font></tt><br><br><br><tt><font size=2><br>> > More importantly, it is not possible to mutate bytes in a Swift
string<br>> > at will. Again it makes sense from the point of view of
always<br>> > correct Unicode sequences. But it does not for machine
processing of<br>> > C strings with C-like performance. Today, I can cheat using
a<br>> > "_public" API for this, i.e., myString._core. _baseAddress!.
This<br>> > should be doable from an official "unsafe" API.<br>> <br>> We intend to support that use-case.<br>> <br>> > Memory safety is also at play here, as well as ownership. A
proper<br>> > API could guarantee the backing store is writable for instance,
that<br>> > it is not shared. A memory-safe but not unicode-safe API
could do<br>> > bounds checks.<br>> ><br>> > While low-level C string processing can be done using unsafe
memory<br>> > buffers with performance, the lack of bridging with "real"
Swift<br>> > strings kills the deal. No literals syntax (or costly coercions),<br>> > none of the many useful string APIs.<br>> ><br>> > To illustrate these points here is a simple experiment: code
written<br>> > to synthesize an http date string from a bunch of integers. There
are<br>> > four versions of the code going from nice high-level Swift code
to<br>> > low-level C-like code. (Some of this code is also about
avoiding ARC<br>> > overheads, and string interpolation overheads, hence the four<br>> > versions.)<br>> ><br>> > On my macbook pro (swiftc -O), the performance is as follows:<br>> ><br>> > interpolation + func: 2.303032365s<br>> > interpolation + array: 1.224858418s<br>> > append: 0.918512377s<br>> > memcpy: 0.182104674s<br>> ><br>> > While the benchmarking could be done more carefully, I think
the main<br>> > observation is valid. The nice code is more than 10x slower
than the<br>> > C-like code. Moreover, the ugly-but-still-valid-Swift code
is still<br>> > about 5x slower than the C like code. For some applications,
e.g. web<br>> > servers, this kind of numbers matter...<br>> ><br>> > Some of the proposed improvements would help with this, e.g.,
small<br>> > strings optimization, and maybe changes to the concatenation<br>> > semantics. But it seems to me that a big performance gap
will remain.<br>> > (Concatenation even with strncat is significantly slower than
memcpy<br>> > for fixed-size strings.)<br>> ><br>> > I believe there is a need and an opportunity for a fast "less
safe"<br>> > String API. I hope it will be on the roadmap soon.<br>> <br>> I think it's already in the roadmap...the one that's in my head. If
you<br>> want to submit a PR with amendments to the manifesto, that'd be great.<br>> Also thanks very much for the example below; we'll definitely<br>> be referring to it as we proceed forward.<br></font></tt><br><tt><font size=2>Here is a gist for the example code:</font></tt><br><a href=https://gist.github.com/tardieu/b6a9c4d53d56d089c58089ba8f6274b5><tt><font size=2 color=blue>https://gist.github.com/tardieu/b6a9c4d53d56d089c58089ba8f6274b5</font></tt></a><br><br><tt><font size=2>I can sketch key elements of an unsafe String API
and some motivating arguments in a pull request. Is this what you are asking
for?</font></tt><br><br><tt><font size=2>Best,</font></tt><br><br><tt><font size=2>Olivier</font></tt><BR>