<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On May 19, 2016, at 10:35 PM, Russ Bishop <<a href="mailto:xenadu@gmail.com" class="">xenadu@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><h1 style="box-sizing: border-box; font-size: 2.25em; margin-right: 0px; margin-bottom: 16px; margin-left: 0px; line-height: 1.2; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; background-color: rgb(255, 255, 255); margin-top: 0px !important;" class="">UnsafeBytePointer API for In-Memory Layout</h1><p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; background-color: rgb(255, 255, 255);" class=""><code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">UnsafePointer</code> and <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">UnsafeMutable</code> refer to a typed region of memory, and the compiler must be able to assume that <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">UnsafePointer</code> element (<code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">Pointee</code>) type is consistent with other access to the same memory. See <a href="https://github.com/atrick/swift/blob/type-safe-mem-docs/docs/TypeSafeMemory.rst" style="box-sizing: border-box; background-color: transparent; -webkit-text-decoration-skip: objects; color: rgb(64, 120, 192); text-decoration: none;" class="">proposed Type Safe Memory Access documentation</a>. Consequently, conversion between <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">UnsafePointer</code> element types exposes an easy way to abuse the type system.</p></div></div></blockquote><div class=""><br class=""></div><div class="">I don’t necessarily disagree with the proposal but I think we should clearly answer the following question:</div></div></div></div></blockquote><div><br class=""></div>I think these are two questions:</div><div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class="">Why doesn’t UnsafePointer<T>(_: UnsafePointer<U>) read as UnsafePointer<T>(_: UnsafePointer<Void>). That is to say you can only “type pun” through a Void pointer.</div></div></div></div></blockquote><div><br class=""></div><div>That’s a reasonable request and would definitely ease migration. However, it still communicates to the user that type punning is a normal, expected use of UnsafePointer. Also, it doesn’t allow all uses of potentially type punning to be identified through code inspection. I know how helpful that feature is because I’ve been auditing code for potential undefined behavior.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div class=""> A convenience method could be offered, something like UnsafePointer.reinterpretBytes<U>(_ ptr: UnsafePointer<U>, as: U.Type) -> U so all valid cases of type punning can be explicit.</div></div></div></div></blockquote><div><br class=""></div><div>Maybe you mean UnsafePointer.reinterpretBytes<U>(as: U.Type) -> U </div><div><br class=""></div><div>That’s a possibility. It’s slightly reminiscent of my first attempt to deal with this problem.</div><div><br class=""></div><div>However, simply as a convenience it’s too similar to unsafeBitCast(p[0], to: U.self) assuming you know the types are layout compatible.</div><div><br class=""></div><div>With my proposal you could now do UnsafeBytePointer(p).load(U.self), which is overall a much more clear, safer design.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; background-color: rgb(255, 255, 255);" class="">As motivation for such an API, consider that an <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">UnsafePointer<Void></code> or <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">OpaquePointer</code> may be currently be obtained from an external API. However, the developer may know the memory layout and may want to read or write elements whose types are compatible with that layout. This a reasonable use case, but unless the developer can guarantee that all accesses to the same memory location have the same type, then they cannot use <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">UnsafePointer</code> to access the memory without risking undefined behavior.</p></div></div></blockquote><div class="">IMHO if we had a @packed attribute a lot of this nonsense could be made explicit by defining a Swift struct that had the appropriate memory layout. This is how a lot of “PInvoke” stuff was done in the C# world. It also gives you an “out” if you need a very specific layout in memory for some other reason.</div></div></div></div></blockquote><div><br class=""></div><div>I think think that’s complementary and addresses the usability of doing manual layout.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; background-color: rgb(255, 255, 255);" class="">Just as with <code class="" style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;">unsafeBitCast</code>, although the destination of the cast can usually be inferred, we want the developer to explicitly state the intended destination type, both because type inferrence can be surprising, and because it's important to the reader for code comprehension.</p></div></div></blockquote><div class="">I’d definitely prefer a labelled initializer, especially one with an uncommon name. IMHO It should immediately stand out in code reviews.</div></div></div></div></blockquote><div><br class=""></div><div>Well, I agree with that sentiment. I would even be fine with a freestanding function to make it really clear, but that defies convention. I’m looking for people to weigh in.</div><div><br class=""></div><div>Once of the reasons I finished migrating the stdlib (multiple times), is so that proposal reviewers can look at my branch see the real effects of the proposal. My first implementation was something like UnsafePointer.init(unsafePointerCast: p). I’ve actually gotten strong feedback to force the destination type to be spelled, and shorten the label. I could probably dig up an earlier version of the changes.</div><div><br class=""></div><div>I have to admit though that the UnsafePointer(p, to: U.self) syntax tends to read better and at least there’s an easy regex that can pick up on it.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; background-color: rgb(255, 255, 255);" class="">Note: For API clarity we could consider a typealias for <code class="" style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;">VoidPointer</code>. A separate <code class="" style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;">VoidPointer</code> type would not be very useful--there's no danger that <code class="" style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;">UnsafeBytePointer</code> will be casually dereferenced, and no danger in allowing pointer arithmetic since the only reasonable interpretation is that of a byte-addressable memory.</p></div></div></blockquote><div class="">Agreed; even today messing with UnsafeMutablePointer<Void> requires you to understand that the size corresponds to bytes which is not intuitive.</div></div></div></div></blockquote><div><br class=""></div><div>Ah, that’s good feedback in favor of replacing UnsafePointer<Void> and its status as imported type!</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 16px; background-color: rgb(255, 255, 255);" class="">Loading from and storing to memory via an <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">Unsafe[Mutable]BytePointer</code> is safe independent of the type of value being loaded or stored and independent of the memory's allocated type as long as layout guarantees are met (per the ABI). This allows legal type punning within Swift and allows Swift code to access a common region of memory that may be shared across an external interface that does not provide type safety guarantees. Accessing type punned memory directly through a designated <code style="box-sizing: border-box; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 14px; padding: 0.2em 0px; margin: 0px; background-color: rgba(0, 0, 0, 0.0392157); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">Unsafe[Mutable]BytePointer</code> type provides sound basis for compiler implementation of strict aliasing. This is in contrast with the approach of simply providing a special unsafe pointer cast operation for bypassing type safety, which cannot be reliably implemented.</p></div></div></blockquote></div><br class=""><div class="">I’m not sure how to word it but I feel like some of this might help if it were included at the very beginning so people understand why this is a problem. I also think the stdlib docs should have a lot more to say about the rules, undefined behavior, and the consequences thereof. That will be all that a lot of developers ever bother to learn on the subject (a shame but out of scope for a swift evolution proposal :) )</div></div></div></blockquote><br class=""></div><div>I thought that including the link to <a href="https://github.com/atrick/swift/blob/type-safe-mem-docs/docs/TypeSafeMemory.rst" class="">proposed Type Safe Memory Access documentation</a> in the first paragraph was sufficient. But reviewers seem to be skipping over it!</div><div><br class=""></div><div>Andy</div><br class=""></body></html>