<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 13, 2017, at 6:55 PM, Taylor Swift via swift-evolution <<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><br class="Apple-interchange-newline"><br class=""><div class="gmail_quote">On Thu, Jul 13, 2017 at 6:56 PM, Andrew Trick<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:atrick@apple.com" target="_blank" class="">atrick@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><blockquote type="cite" class=""><span class=""><div class="">On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution <<a href="mailto:swift-evolution@swift.org" target="_blank" class="">swift-evolution@swift.org</a>> wrote:</div><br class="m_7627022841191771449Apple-interchange-newline"></span><div class=""><div class="h5"><div class=""><div dir="ltr" class=""><p class="">Hi all, I’ve written up a proposal to modify the unsafe pointer API for greater consistency, safety, and ease of use.</p><p class="">~~~<br class=""></p><p class="">Swift currently offers two sets of pointer types — singular pointers such as<span class="Apple-converted-space"> </span><code class="">UnsafeMutablePointer</code>, and vector (buffer) pointers such as<span class="Apple-converted-space"> </span><code class="">UnsafeMutable</code><b class=""><code class="">Buffer</code></b><code class="">Pointer</code>. This implies a natural separation of tasks the two kinds of pointers are meant to do. For example, buffer pointers implement<span class="Apple-converted-space"> </span><code class="">Collection</code><span class="Apple-converted-space"> </span>conformance, while singular pointers do not.</p><p class="">However, some aspects of the pointer design contradict these implied roles. It is possible to allocate an arbitrary number of instances from a type method on a singular pointer, but not from a buffer pointer. The result of such an operation returns a singular pointer, even though a buffer pointer would be more appropriate to capture the information about the<span class="Apple-converted-space"> </span><i class="">number</i><span class="Apple-converted-space"> </span>of instances allocated. It’s possible to subscript into a singular pointer, even though they are not real<span class="Apple-converted-space"> </span><code class="">Collection</code>s. Some parts of the current design turn UnsafePointers into downright<span class="Apple-converted-space"> </span><i class="">Dangerous</i>Pointers, leading users to believe that they have allocated or freed memory when in fact, they have not.</p><p class="">This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.</p><p class=""><<a href="https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06" target="_blank" class="">https://gist.github.com/<wbr class="">kelvin13/<wbr class="">a9c033193a28b1d4960a89b25fbffb<wbr class="">06</a>></p><p class="">~~~<br class=""></p></div></div></div></div></blockquote><br class=""></div><div class=""><div class="">Thanks for taking time to write this up.</div><div class=""><br class=""></div><div class="">General comments:</div><div class=""><br class=""></div><div class="">UnsafeBufferPointer is an API layer on top of UnsafePointer. The role</div><div class="">of UnsafeBufferPointer is direct memory access sans lifetime</div><div class="">management with Collection semantics. The role of UnsafePointer is</div><div class="">primarily C interop. Those C APIs should be wrapped in Swift APIs that</div><div class="">take UnsafeBufferPointer whenever the pointer represents a C array. I</div><div class="">suppose making UnsafePointer less convenient would push developers</div><div class="">toward UnsafeBufferPointer. I don't think that's worth outright</div><div class="">breaking source, but gradual deprecation of convenience methods, like</div><div class="">`susbscript` might be acceptable.</div></div></div></blockquote><br class=""></div><div class="gmail_quote">Gradual deprecation is exactly what I am proposing. As the<span class="Apple-converted-space"> </span><a href="https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06#proposed-solution" class="">document states</a>, the only methods which should be marked immediately as unavailable are the `<span style="font-family: monospace, monospace;" class="">deallocate(capacity:)</span>` methods, for safety and source compatibility reasons. Removing `<span style="font-family: monospace, monospace;" class="">deallocate(capacity:)</span>` now and forcing a loud compiler error prevents catastrophic *silent* source breakage in the future, or worse, from having to *support our own bug*.<br class=""></div><div class="gmail_quote"> <blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">I have mixed feelings about stripping UnsafePointer of basic</div><div class="">functionality. Besides breaking source, doing that would be</div><div class="">inconsistent with its role as a lower API layer. The advantage would</div><div class="">just be descreasing API surface area and forcing developers to use a</div><div class="">higher-level API.</div></div></div></blockquote><div class=""><br class=""></div><div class="">UnsafePointer is as much a high level API as UnsafeBufferPointer is. You wouldn’t create a buffer pointer of length 1 just so you can “stick with the high level API”. UnsafePointer and UnsafeBufferPointer are two tools that do related but different things and they can exist at whatever abstract level you need them at. After all, UnsafeBufferPointer is nothing but an UnsafePointer? with a length value attached to it. If you’re allocating more than one instance of memory, you almost certainly need to track the length of the buffer anyway.<br class=""> <br class=""></div></div></div></div></div></blockquote><div><br class=""></div><div>I disagree. UnsafePointer can be viewed as the low level representation of Swift’s memory model. UnsafeBufferPointer is a convenient facility for programming at this level, as illustrated by e.g. RandomAccessCollection conformance. One could drop UnsafeBufferPointer from the standard library without compromising Swift’s core capabilities, it would just be a large convenience regression.</div><div><br class=""></div><div>UnsafePointer is a low level necessity; UnsafeBufferPointer is a low level convenience.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">The additive changes you propose are fairly obvious. See [SR-3088]</div><div class="">UnsafeMutableBufferPointer doesn't have an allocating init.</div><div class=""><br class=""></div><div class="">I haven't wanted to waste review cycles on small additive</div><div class="">changes. It may make sense to batch them up into one coherent</div><div class="">proposal. Here are a few more to consider.</div><div class=""><br class=""></div><div class="">- [SR-3929] UnsafeBufferPointer should have init from mutable</div><div class="">- [SR-4340] UnsafeBufferPointer needs a withMemoryRebound method</div><div class="">- [SR-3087] No way to arbitrarily initialise an Array's storage</div></div></div></blockquote><div class=""><br class=""></div><div class="">The feature requests you mention are all very valuable, however with Michael’s point about fixing the memorystate API’s, the size of this proposal has already grown to encompass dozens of methods in five types. I think this says a lot about just how broken the current system is</div></div></div></div></div></blockquote><div><br class=""></div><div>I fail to see how the current system is broken. It accurately reflects Swift’s memory model while providing the facilities to stay in the bounds of defined behavior. I do agree that it would be far more convenient for UnsafeBufferPointer to provide allocation/deallocation and initialization/deinitialization convenience functionality as well, as I’ve often wanted it myself.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">, but I think it’s better to try to fix one class of problems at a time, and save the less closely-related issues for separate proposals.<br class=""></div></div></div></div></div></blockquote><div><br class=""></div><div>These seem very closely related. Isn’t the overall goal of this proposal to have API parity for UnsafeBufferPointer?</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">Point by point:</div><div class=""><br class=""></div><div class="">> drop the capacity parameter from UnsafeMutablePointer.allocate(<wbr class="">) and deallocate().</div><div class=""><br class=""></div><div class="">I do not agree with removing the capacity parameter and adding a</div><div class="">single-instance allocation API. UnsafePointer was not designed for</div><div class="">single instances, it was primarily designed for C-style arrays. I</div><div class="">don't see the value in providing a different unsafe API for single</div><div class="">vs. multiple values.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Although it’s common to *receive* Unsafe__Pointers from C API’s, it’s rare to *create* them from the Swift side. 95% of the time your Swift data lives in a Swift Array, and you use withUnsafePointer(_:) to send them to the C API, or just pass them directly with Array bridging.<span class="Apple-converted-space"> </span><br class=""><br class=""></div><div class="">The only example I can think of where I had to allocate memory from the Swift side to pass to a C API is when I was using the Cairo C library and I wanted the Swift code to own the image buffer backing the Cairo C structs and I wanted to manage the memory manually to prevent the buffer backing from getting deallocated prematurely. I think I ended up using UnsafeMutableBufferPointer and extracting baseAddresses to manage the memory. This proposal tries to mitigate that pain of extracting baseAddresses by giving buffer pointers their own memory management methods.<br class=""></div><div class=""><br class=""></div></div></div></div></div></blockquote><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">As for the UnsafePointers you get from C APIs, they almost always come with a size (or you specify it beforehand with a parameter) so you’re probably going to be turning them into UnsafeBufferPointers anyway.<br class=""><br class=""></div><div class="">I also have to say it’s not common to deallocate something in Swift that you didn’t previously allocate in Swift.<span class="Apple-converted-space"> </span><br class=""></div><div class=""><br class=""></div></div></div></div></div></blockquote><div><br class=""></div><div>I’m not sure it is even defined to use `deallocate` on memory that wasn’t `allocated` in Swift. Andy, thoughts here? Perhaps more clarity in deallocate’s documentation is needed?</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">I agree the primary allocation API should be</div><div class="">UnsafeMutableBufferPointer.<wbr class="">allocate(capacity:). There is an argument</div><div class="">to be made for removing UnsafeMutablePointer.allocate(<wbr class="">capacity:)</div><div class="">entirely. But, as Michael Ilseman pointed out, that would involve</div><div class="">reevaluating several other members of the UnsafePointer API. I think</div><div class="">it's reasonable for UnsafePointer to retain all its functionality as a</div><div class="">lower level API.</div><div class=""><br class=""></div></div></div></blockquote><div class=""><br class=""></div><div class="">I think duplication of functionality is something to be avoided if possible.<br class=""></div><div class=""> </div></div></div></div></div></blockquote><div><br class=""></div><div>You have to weigh duplication against convenience. If avoiding duplication is more important, then we should keep the APIs only on UnsafePointer. But, I would prefer convenience in this scenario.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""></div><div class="">I don't understand what is misleading about</div><div class="">UnsafePointer.deallocate(<wbr class="">capacity:). It *is* inconvenienent for the</div><div class="">user to keep track of memory capacity. Presumably that was done so</div><div class="">either the implementation can move away from malloc/free or some sort</div><div class="">of memory tracking can be implemented on the standard library</div><div class="">side. Obviously, UnsafeBufferPointer.<wbr class="">deallocate() would be cleaner in</div><div class="">most cases.</div></div></div></blockquote><div class=""><br class=""></div><div class="">It’s misleading because it plain doesn’t deallocate `capacity` instances. It deletes the whole memory block regardless of what you pass in the capacity argument. If the implementation is ever “fixed” so that it actually deallocates `capacity` instances, suddenly every source that uses `deallocate(capacity:)` will break, and *no one will know* until their app starts mysteriously crashing. If the method is not removed, we will have to support this behavior to avoid breaking sources, and basically say “yes the argument label says it deallocates a capacity, but what it *really* does is free the whole block and we can’t fix it because existing code assumes this behavior”.<br class=""></div><div class=""> </div></div></div></div></div></blockquote><div><br class=""></div><div>Note that many of these things are basically programming at the memory semantics level, not necessarily at a execution level.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class=""><br class=""></div><div class="">> add an allocate(count:) type method to UnsafeMutableBufferPointer</div><div class=""><br class=""></div><div class="">`capacity` should be used for allocating uninitialized memory not</div><div class="">`count`. `count` should only refer to a number of initialized objects!</div></div></div></blockquote><div class=""><br class=""></div><div class="">We can decide on what the correct term should be, but the current state of Swift pointers is that *neither* convention is being followed. Just look at the API for UnsafeMutableRawPointer. It’s a mess. This proposal at the minimum establishes a consistent convention. It can be revised if you feel `capacity` is more appropriate than `count`. If what you mean is that it’s important to maintain the distinction between “initialized counts” and “uninitialized counts”, well that can be revised in too.<br class=""></div><div class=""> </div></div></div></div></div></blockquote><div><br class=""></div><div>So I just looked over all of the APIs, and they are very consistent in their use of `count` and `capacity`. Where are you seeing a violation?</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">> add a deallocate() instance method to UnsafeMutableBufferPointer</div><div class=""><br class=""></div><div class="">Yes, of course! I added a mention of that in SR-3088.</div><div class=""><br class=""></div><div class="">> remove subscripts from UnsafePointer and UnsafeMutablePointer</div><div class=""><br class=""></div><div class="">It's often more clear to perform arithmetic on C array indices rather</div><div class="">than pointers. That said, I'm happy to push developers to use</div><div class="">UnsafeBufferPointer whenever that have a known capacity. To me, this</div><div class="">is a question of whether the benefit of making a dangerous thing less</div><div class="">convenient is worth breaking source compatibility.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Again, I think this is more about what the real use patterns are. If you are subscripting into a C array with integers, then UnsafeBufferPointer is the tool for the job, since it give you Collection conformance. If you can’t make an UnsafeBufferPointer, it’s probably because you don’t know the length of the array, and so you’re probably iterating through it one element at a time. UnsafeMutablePointer.successor() is perfect for this job. If you want to extract or set fields at fixed but irregular offsets, UnsafeRawPointer is the tool for the job. But I’m hard-pressed to think of a use case for random access into a singular typed pointer.<br class=""></div></div><br class=""></div></div></div></blockquote><div><br class=""></div><div><div><div>I’m also very interested in more convenience facilities for low level programming and I view UnsafeBufferPointer as the first low level convenience construct. The point of the very careful dance that the Unsafe*Pointers go through is to help the programmer avoid unwittingly falling into undefined behavior.</div><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"></div></div></div></blockquote></div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="" style="word-wrap: break-word;"><div class=""></div></div></blockquote></div></div></div></blockquote></div><br class=""><blockquote type="cite" class=""><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">swift-evolution mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:swift-evolution@swift.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">swift-evolution@swift.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://lists.swift.org/mailman/listinfo/swift-evolution" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.swift.org/mailman/listinfo/swift-evolution</a></div></blockquote></div><br class=""></body></html>