[swift-evolution] [RFC] UnsafeBytePointer API for In-Memory Layout

[Andrew Trick]

> On May 19, 2016, at 10:35 PM, Russ Bishop <xenadu at gmail.com> wrote:
> 
> 
>> 
>> UnsafeBytePointer API for In-Memory Layout
>> 
>> UnsafePointer and UnsafeMutable refer to a typed region of memory, and the compiler must be able to assume that UnsafePointer element (Pointee) type is consistent with other access to the same memory. See proposed Type Safe Memory Access documentation <https://github.com/atrick/swift/blob/type-safe-mem-docs/docs/TypeSafeMemory.rst>. Consequently, conversion between UnsafePointer element types exposes an easy way to abuse the type system.
>> 
> 
> I don’t necessarily disagree with the proposal but I think we should clearly answer the following question:

I think these are two questions:

> Why doesn’t UnsafePointer<T>(_: UnsafePointer<U>) read as UnsafePointer<T>(_: UnsafePointer<Void>). That is to say you can only “type pun” through a Void pointer.

That’s a reasonable request and would definitely ease migration. However, it still communicates to the user that type punning is a normal, expected use of UnsafePointer. Also, it doesn’t allow all uses of potentially type punning to be identified through code inspection. I know how helpful that feature is because I’ve been auditing code for potential undefined behavior.

> A convenience method could be offered, something like UnsafePointer.reinterpretBytes<U>(_ ptr: UnsafePointer<U>, as: U.Type) -> U so all valid cases of type punning can be explicit.

Maybe you mean UnsafePointer.reinterpretBytes<U>(as: U.Type) -> U 

That’s a possibility. It’s slightly reminiscent of my first attempt to deal with this problem.

However, simply as a convenience it’s too similar to unsafeBitCast(p[0], to: U.self) assuming you know the types are layout compatible.

With my proposal you could now do UnsafeBytePointer(p).load(U.self), which is overall a much more clear, safer design.

>> As motivation for such an API, consider that an UnsafePointer<Void> or OpaquePointer may be currently be obtained from an external API. However, the developer may know the memory layout and may want to read or write elements whose types are compatible with that layout. This a reasonable use case, but unless the developer can guarantee that all accesses to the same memory location have the same type, then they cannot use UnsafePointer to access the memory without risking undefined behavior.
>> 
> IMHO if we had a @packed attribute a lot of this nonsense could be made explicit by defining a Swift struct that had the appropriate memory layout. This is how a lot of “PInvoke” stuff was done in the C# world. It also gives you an “out” if you need a very specific layout in memory for some other reason.

I think think that’s complementary and addresses the usability of doing manual layout.

>> Just as with unsafeBitCast, although the destination of the cast can usually be inferred, we want the developer to explicitly state the intended destination type, both because type inferrence can be surprising, and because it's important to the reader for code comprehension.
>> 
> I’d definitely prefer a labelled initializer, especially one with an uncommon name. IMHO It should immediately stand out in code reviews.

Well, I agree with that sentiment. I would even be fine with a freestanding function to make it really clear, but that defies convention. I’m looking for people to weigh in.

Once of the reasons I finished migrating the stdlib (multiple times), is so that proposal reviewers can look at my branch see the real effects of the proposal. My first implementation was something like UnsafePointer.init(unsafePointerCast: p). I’ve actually gotten strong feedback to force the destination type to be spelled, and shorten the label. I could probably dig up an earlier version of the changes.

I have to admit though that the UnsafePointer(p, to: U.self) syntax tends to read better and at least there’s an easy regex that can pick up on it.

>> Note: For API clarity we could consider a typealias for VoidPointer. A separate VoidPointer type would not be very useful--there's no danger that UnsafeBytePointer will be casually dereferenced, and no danger in allowing pointer arithmetic since the only reasonable interpretation is that of a byte-addressable memory.
>> 
> Agreed; even today messing with UnsafeMutablePointer<Void> requires you to understand that the size corresponds to bytes which is not intuitive.

Ah, that’s good feedback in favor of replacing UnsafePointer<Void> and its status as imported type!

>> Loading from and storing to memory via an Unsafe[Mutable]BytePointer is safe independent of the type of value being loaded or stored and independent of the memory's allocated type as long as layout guarantees are met (per the ABI). This allows legal type punning within Swift and allows Swift code to access a common region of memory that may be shared across an external interface that does not provide type safety guarantees. Accessing type punned memory directly through a designated Unsafe[Mutable]BytePointer type provides sound basis for compiler implementation of strict aliasing. This is in contrast with the approach of simply providing a special unsafe pointer cast operation for bypassing type safety, which cannot be reliably implemented.
>> 
> 
> I’m not sure how to word it but I feel like some of this might help if it were included at the very beginning so people understand why this is a problem. I also think the stdlib docs should have a lot more to say about the rules, undefined behavior, and the consequences thereof. That will be all that a lot of developers ever bother to learn on the subject (a shame but out of scope for a swift evolution proposal :) )

I thought that including the link to proposed Type Safe Memory Access documentation <https://github.com/atrick/swift/blob/type-safe-mem-docs/docs/TypeSafeMemory.rst> in the first paragraph was sufficient. But reviewers seem to be skipping over it!

Andy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160520/736bec27/attachment.html>