[swift-evolution] [RFC] UnsafeBytePointer API for In-Memory Layout

[Geordie Jay]

I read this proposal and I'm a bit unsure what its purpose would be:

Basically you want to prevent UnsafePointer<XYZ>(UnsafePointer<Void>)
conversions and/or vice-versa? And you'd achieve this by replacing
UnsafePointer<Void> with UnsafeBytePointer that has no bound pointer type?

In one sense the change seems fine to me, but as someone who uses a lot of
C APIs and a lot of CoreAudio/CoreMIDI in Swift already I can't really see
what benefit it'd bring. Presumably we'd still want an option of converting
UnsafeBytePointer to UnsafePointer<SomeActualType> for things like C
function pointer callback "context"/"userInfo" uses, so it's not like we'd
be preventing programmer error in that way.

Call me conservative but to me the current system seems to work as well as
it can. If anything it's already enough boilerplate going through hoops
converting an UnsafeMutablePointer<Void> into a [Float] even when I know
and the C API knows perfectly well what it actually contains... Would
happily be convinced otherwise about this proposal though, I'm pretty new
at all this.

Geordie
Andrew Trick via swift-evolution <swift-evolution at swift.org> schrieb am
Mo., 9. Mai 2016 um 20:15:

> Hello Swift evolution,
>
> I sent this to swift-dev last week. Sorry to post on two lists!
>
> Swift does a great job of protecting against undefined behavior--as long
> as you avoid "unsafe" APIs, that is. However, unsafe APIs are important for
> giving developers control over implementation details and performance.
> Naturally, the contract between unsafe APIs and the optimizer is crucial.
> When a developer uses an unsafe API, the rules governing safe, well-defined
> behavior must be clear. On the opposite end, the optimizer must know which
> assumptions it can make based on those rules. Simply saying that anything
> goes because "unsafe" is in the name is not helpful to this effort.
>
> For a long time, I've wanted these rules nailed down. We have more users
> taking advantage of advanced features, and more optimizations that take
> advantage of assumptions guided by the type system. This seems like a
> particularly good time to resolve UnsafePointer semantics, considering the
> type system and UnsafePointer work that's been going on recently. Strict
> aliasing is something I would like addressed. If we do nothing here, then
> we will end up by default inheriting C/C++ semantics, as with any language
> that relies on a C/C++ backend. In other words, developers will be forced
> to write code with technically undefined behavior and rely on the compiler
> to be smart enough to recognize and recover from common patterns. Or we can
> take advantage of this opportunity and instead adopt a sound memory model
> with respect to aliasing.
>
> This proposal is only an RFC at this point. I'm sending it out now to
> allow for plenty of time for discussion (or advance warning). Keep in mind
> that it could change considerably before it goes up for review.
>
> -Andy
>
>
> UnsafeBytePointer API for In-Memory Layout
>
>    - Proposal: SE-NNNN
>    <https://github.com/atrick/swift-evolution/blob/voidpointer/proposals/XXXX-unsafebytepointer.md>
>    - Author(s): Andrew Trick <https://github.com/atrick>
>    - Status: Awaiting review
>    <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#rationale>
>    - Review manager: TBD
>
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#introduction>
> Introduction
>
> UnsafePointer and UnsafeMutable refer to a typed region of memory, and
> the compiler must be able to assume that UnsafePointer element (Pointee)
> type is consistent with other access to the same memory. See proposed
> Type Safe Memory Access documentation
> <https://github.com/atrick/swift/blob/type-safe-mem-docs/docs/TypeSafeMemory.rst>.
> Consequently, inferred conversion between UnsafePointer element types
> exposes an easy way to abuse the type system. No alternative currently
> exists for manual memory layout and direct access to untyped memory, and
> that leads to an overuse of UnsafePointer. These uses of UnsafePointer,
> which depend on pointer type conversion, make accidental type punning
> likely. Type punning via UnsafePointer is semantically undefined behavior
> and de facto undefined behavior given the optimizer's long-time treatment
> of UnsafePointer.
>
> In this document, all mentions of UnsafePointer also apply to
> UnsafeMutablePointer.
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#motivation>
> Motivation
>
> To avoid accidental type punning, we should prohibit inferred conversion
> between UnsafePointer<T> and UnsafePointer<U> unless the target of the
> conversion is an untyped or nondereferenceable pointer (currently
> represented as UnsafePointer<Void>).
>
> To support this change we should introduce a new pointer type that does
> not bind the type of its Pointee. Such a new pointer type would provide
> an ideal foundation for an API that allows byte-wise pointer arithmetic and
> a legal, well-defined means to access an untyped region of memory.
>
> As motivation for such an API, consider that an UnsafePointer<Void> or
> OpaquePointer may be currently be obtained from an external API. However,
> the developer may know the memory layout and may want to read or write
> elements whose types are compatible with that layout. This a reasonable use
> case, but unless the developer can guarantee that all accesses to the same
> memory location have the same type, then they cannot use UnsafePointer to
> access the memory without risking undefined behavior.
>
> An UnsafeBytePointer example, using a new proposed API is included below.
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#proposed-solution>Proposed
> solution
>
> Introduce an UnsafeBytePointer type along with an API for obtaining a
> UnsafeBytePointer value at a relative byte offset and loading and storing
> arbitrary types at that location.
>
> Statically prohibit inferred UnsafePointer conversion while allowing
> inferred UnsafePointer to UnsafeBytePointerconversion.
>
> UnsafeBytePointer meets multiple requirements:
>
>    1. An untyped pointer to memory
>    2. Pointer arithmetic within byte-addressable memory
>    3. Type-unsafe access to memory (legal type punning)
>
> UnsafeBytePointer will replace UnsafeMutablePointer<Void> as the
> representation for untyped memory. For API clarify we could consider a
> typealias for VoidPointer. I don't think a separate VoidPointer type
> would be useful--there's no danger that UnsafeBytePointer will be
> casually dereferenced, and don't see the danger in allowing pointer
> arithmetic since the only reasonable interpretation is that of a
> byte-addressable memory.
>
> Providing an API for type-unsafe memory access would not serve a purpose
> without the ability to compute byte offsets. Of course, we could require
> users to convert back and forth using bitPatterns, but I think that would
> be awkward and only obscure the purpose of the UnsafeBytePointer type.
>
> In this proposal, UnsafeBytePointer does not specify mutability. Adding an
> UnsafeMutableBytePointer would be straightforward, but adding another
> pointer type needs strong justification. I expect to get input from the
> community on this. If we agree that the imported type for const void* should
> be UnsafeBytePointer, then we probably need UnsafeMutablePointer to handle
> interoperability.
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#detailed-design>Detailed
> design
>
> The public API is shown here. For details and comments, see the unsafeptr_convert
> branch <https://github.com/atrick/swift/commits/unsafeptr_convert>.
>
> struct UnsafeBytePointer : Hashable, _Pointer {
>
>   let _rawValue: Builtin.RawPointer
>
>   var hashValue: Int {...}
>
>   init<T>(_ : UnsafePointer<T>)
>   init<T>(_ : UnsafeMutablePointer<T>)
>   init?<T>(_ : UnsafePointer<T>?)
>   init?<T>(_ : UnsafeMutablePointer<T>?)
>
>   init<T>(_ : OpaquePointer<T>)
>   init?<T>(_ : OpaquePointer<T>?)
>
>   init?(bitPattern: Int)
>   init?(bitPattern: UInt)
>
>   func load<T>(_ : T.Type) -> T
>
>   @warn_unused_result
>   init(allocatingBytes size: Int, alignedTo: Int)
>
>   @warn_unused_result
>   init<T>(allocatingCapacity count: Int, of: T.Type)
>
>   func deallocateBytes(_ size: Int, alignedTo: Int)
>
>   func deallocateCapacity<T>(_ num: Int, of: T.Type)
>
>   // Returns a pointer one byte after the initialized memory.
>   func initialize<T>(with newValue: T, count: Int = 1) -> UnsafeBytePointer
>
>   // Returns a pointer one byte after the initialized memory.
>   func initialize<T>(from: UnsafePointer<T>, count: Int) -> UnsafeBytePointer
>
>   func initializeBackward<T>(from source: UnsafePointer<T>, count: Int)
>
>   func deinitialize<T>(_ : T.Type, count: Int = 1)
> }
> extension OpaquePointer {
>   init(_ : UnsafeBytePointer)
> }
> extension Int {
>   init(bitPattern: UnsafeBytePointer)
> }
> extension UInt {
>   init(bitPattern: UnsafeBytePointer)
> }
> extension UnsafeBytePointer : RandomAccessIndex {
>   typealias Distance = Int
>
>   func successor() -> UnsafeBytePointer
>   func predecessor() -> UnsafeBytePointer
>   func distance(to : UnsafeBytePointer) -> Int
>   func advanced(by : Int) -> UnsafeBytePointer
> }
> func == (lhs: UnsafeBytePointer, rhs: UnsafeBytePointer) -> Bool
> func < (lhs: UnsafeBytePointer, rhs: UnsafeBytePointer) -> Bool
> func + (lhs: UnsafeBytePointer, rhs: Int) -> UnsafeBytePointer
> func + (lhs: Int, rhs: UnsafeBytePointer) -> UnsafeBytePointer
> func - (lhs: UnsafeBytePointer, rhs: Int) -> UnsafeBytePointer
> func - (lhs: UnsafeBytePointer, rhs: UnsafeBytePointer) -> Int
> func += (lhs: inout UnsafeBytePointer, rhs: Int)
> func -= (lhs: inout UnsafeBytePointer, rhs: Int)
>
> Occasionally, we need to convert from a UnsafeBytePointer to an
> UnsafePointer. This should only be done in very rare circumstances when
> the author understands the compiler's strict type rules for UnsafePointer.
> Although this could be done by casting through an OpaquePointer, an
> explicit, designated unsafe pointer cast API would makes the risks more
> obvious and self-documenting. For example:
>
> extension UnsafePointer {
>   init(_ from: UnsafeBytePointer, toPointee: Pointee.type)
> }extension UnsafeMutablePointer {
>   init(_ from: UnsafeBytePointer, toPointee: Pointee.type)
> }
>
> Similarly, conversion between UnsafePointer types must now be spelled
> with an explicitly Pointee type:
>
> extension UnsafePointer {
>   init<U>(_ from: UnsafePointer<U>, toPointee: Pointee.Type)
>   init<U>(_ from: UnsafeMutablePointer<U>, toPointee: Pointee.Type)
> }extension UnsafeMutablePointer {
>   init<U>(_ from: UnsafeMutablePointer<U>, toPointee: Pointee.Type)
> }
>
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#impact-on-existing-code>Impact
> on existing code
>
> The largest impact of this change is that void* and const void* are
> imported as UnsafeBytePointer. This impacts many public APIs, but with
> implicit argument conversion should not affect typical uses of those APIs.
>
> Any Swift projects that rely on type inference to convert between
> UnsafePointer types will need to take action. The developer needs to
> determine whether type punning is necessary. If so, they must migrate to
> the UnsafeBytePointer API. Otherwise, they can work around the new
> restriction by using a toPointee, or mutating label.
>
> Disallowing inferred UnsafePointer direct conversion requires some
> standard library code to use an explicit toPointeelabel for unsafe
> conversions that may violate strict aliasing.
>
> All occurrences of Unsafe[Mutable]Pointer<Void> in the standard library
> are converted to UnsafeBytePointer. e.g. unsafeAddress() now returns
> UnsafeBytePointer, not UnsafePointer<Void>.
>
> Some occurrences of Unsafe[Mutable]Pointer<Pointee> in the standard
> library are replaced with UnsafeBytePointer, either because the code was
> playing too loosely with strict aliasing rules, or because the code
> actually wanted to perform pointer arithmetic on byte-addresses.
>
> StringCore.baseAddress changes from OpaquePointer to UnsafeBytePointer because
> it is computing byte offsets and accessing the memory. OpaquePointer is
> meant for bridging, but should be truly opaque; that is, nondereferenceable
> and not involved in address computation.
>
> The StringCore implementation does a considerable amount of casting
> between different views of the String storage. The current implementation
> already demonstrates some awareness of strict aliasing rules. The rules are
> generally followed by ensuring that the StringBuffer only be accessed
> using the appropriate CodeUnit within Swift code. For interoperability
> and optimization, String buffers frequently need to be cast to and from
> CChar. This is valid as long access to the buffer from Swift is guarded
> by dynamic checks of the encoding type. These unsafe, but dynamically legal
> conversion points will now be labeled with toPointee.
>
> CoreAudio utilities now use an UnsafeBytePointer.
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#implementation-status>Implementation
> status
>
> On my unsafeptr_convert branch
> <https://github.com/atrick/swift/commits/unsafeptr_convert>, I've made
> most of the necessary changes to support the addition of UnsafeBytePointerand
> the removal of inferred UnsafePointer conversion.
>
> There are a several things going on here in order to make it possible to
> build the standard library with the changes:
>
>    -
>
>    A new UnsafeBytePointer type is defined.
>    -
>
>    The type system imports void* as UnsafeBytePointer.
>    -
>
>    The type system handles implicit conversions to UnsafeBytePointer.
>    -
>
>    UnsafeBytePointer replaces both UnsafePointer<Void> and
>    UnsafeMutablePointer<Void>.
>    -
>
>    The standard library was relying on inferred UnsafePointer conversion
>    in over 100 places. Most of these conversions now either take an explicit
>    label, such as 'toPointee', 'mutating'. Some have been rewritten.
>    -
>
>    Several places in the standard library that were playing loosely with
>    strict aliasing or doing bytewise pointer arithmetic now use
>    UnsafeBytePointer instead.
>    -
>
>    Explicit labeled Unsafe[Mutable]Pointer initializers are added.
>    -
>
>    The inferred Unsafe[Mutable]Pointer conversion is removed.
>
> TODO:
>
> Once this proposal is accepted, and the rules for casting between pointers
> types have been decided, we need to finish implementing the type system
> support. The current implementation (intentionally) breaks a few tests in
> pointer_conversion.swift. We also need to ensure that interoperability
> requirements are met. Currently, many argument casts to be explicitly
> labeled. The current implementation also makes it easy for users to hit an
> "ambiguous use of 'init'" error when relying on implicit argument
> conversion.
>
> Additionally:
>
>    -
>
>    A name mangled abbreviation needs to be created for UnsafeBytePointer.
>    -
>
>    The StringAPI tests should probably be rewritten with UnsafeBytePointer
>    .
>    -
>
>    The NSStringAPI utilities and tests may need to be ported to
>    UnsafeBytePointer
>    -
>
>    The CoreAudio utilities and tests may need to be ported to
>    UnsafeBytePointer.
>
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#alternatives-considered>Alternatives
> considered
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#existing-workaround>Existing
> workaround
>
> In some cases, developers can safely reinterpret values to achieve the
> same effect as type punning:
>
> let ptrI32 = UnsafeMutablePointer<Int32>(allocatingCapacity: 1)
> ptrI32[0] = Int32()let u = unsafeBitCast(ptrI32[0], to: UInt32.self)
>
> Note that all access to the underlying memory is performed with the same
> element type. This is perfectly legitimate, but simply isn't a complete
> solution. It also does not eliminate the inherent danger in declaring a
> typed pointer and expecting it to point to values of a different type.
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#discarded-alternatives>Discarded
> alternatives
>
> We considered adding a typePunnedMemory property to the existing
> Unsafe[Mutabale]Pointer API. This would provide a legal way to access a
> potentially type punned Unsafe[Mutabale]Pointer. However, it would
> certainly cause confusion without doing much to reduce likelihood of
> programmer error. Furthermore, there are no good use cases for such a
> property evident in the standard library.
>
> The opaque _RawByte struct is a technique that allows for
> byte-addressable buffers while hiding the dangerous side effects of type
> punning (a _RawByte could be loaded but it's value cannot be directly
> inspected). UnsafePointer<_RawByte> is a clever alternative to
> UnsafeBytePointer. However, it doesn't do enough to prevent undefined
> behavior. The loaded _RawByte would naturally be accessed via
> unsafeBitCast, which would mislead the author into thinking that they
> have legally bypassed the type system. In actuality, this API blatantly
> violates strict aliasing. It theoretically results in undefined behavior as
> it stands, and may actually exhibit undefined behavior if the user recovers
> the loaded value.
>
> To solve the safety problem with UnsafePointer<_RawByte>, the compiler
> could associate special semantics with a UnsafePointer bound to this
> concrete generic parameter type. Statically enforcing casting rules would
> be difficult if not impossible without new language features. It would also
> be impossible to distinguish between typed and untyped pointer APIs. For
> example, UnsafePointer<T>.load<U> would be a nonsensical vestige.
>
> <https://github.com/atrick/swift-evolution/tree/voidpointer/proposals#alternate-proposal-for-void-type>Alternate
> proposal for void* type
>
> Changing the imported type for void* will be somewhat disruptive.
> Furthermore, this proposal currently drops the distinction between void*
>  and const void*--an obvious loss of API information.
>
> We could continue to import void* as UnsafeMutablePointer<Void> and const
> void* as UnsafePointer<Void>, which will continue to serve as an "opaque"
> untyped pointer. Converting to UnsafeBytePointer would be necesarry to
> perform pointer arithmetic or to conservatively handle possible type
> punning.
>
> This alternative is *much* less disruptive, but we are left with two
> forms of untyped pointer, one of which (UnsafePointer) the type system
> somewhat conflates with typed pointers.
>
> Given the current restrictions of the language, it's not clear how to
> statically enforce the necessary rules for castingUnsafePointer<Void> once
> general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160509/4042023d/attachment-0001.html>