[swift-evolution] Opaque Pointers in Swift

Johannes Weiß johannesweiss at apple.com
Tue Oct 24 12:23:24 CDT 2017 

Hi Cory,

I think we're dealing with two separate issues here.

1) that all forward declared struct pointers get imported as an OpaquePointer which makes us lose all type-safety
2) that it's a fairly frequent case that C libraries evolve from 'pointers to fully declared structs' to 'pointers to forward declared structs'


Regarding 1)
------------
I fully agree that is pretty bad and I believe I have an idea what the Swift importer should do for that case:

For the following C types

    struct foo_s;
    typedef foo_s *foo;

    struct bar_s;
    typedef bar_s *bar;

the C compiler today imports both `foo` and `bar` as `OpaquePointer` which isn't very helpful. Instead I believe the importer should declare to phantom types

    enum foo_s {}
    enum bar_s {}

and import `foo` as `typealias foo = OpaquePointer<foo_s>` and `bar` as `typealias bar = OpaquePointer<bar_s>`.

That seems to preserve the right semantics:

 - we can't have any values of `foo_s` or `bar_s` as they're phantom
 - we can however have pointers to them
 - in this example the pointers are `OpaquePointer<foo_s>` and `OpaquePointer<bar_s>`, deliberately not `UnsafePointer<foo_s>` and `UnsafePointer<bar_s>` so we don't have an issue that the user can do 'unsafePtr.pointee'

How do people think about this proposed change?

Regarding 2)
------------

I feel you'd prefer if we'd import the above as `typealias foo = UnsafePointer<foo_s>` which I would be happy with but I can also see why `OpaquePointer` exists: It stops us from dereferencing the pointer at compile time (just like C does).

So yes, I agree we should fix (1). For the time being I believe I have something that might work for you use-case even today (and would work even nicer if the above changes are implemented):

Today, we already have the following initialisers

struct UnsafePointer<T> {
    init?(_ ptr: OpaquePointer)
}

and

struct OpaquePointer {
    init?<T>(_ ptr: UnsafePointer<T>)
}

by just adding the following two trivial ones:

extension UnsafePointer<T> {
    init?(_ ptr: UnsafePointer<T>) { return ptr }
}
extension OpaquePointer {
    init?<T>(_ ptr: OpaquePointer) { return ptr }
}

this should solve your problem:

- when you receive a pointer from the C library, store it as OpaquePointer
    let myOpaquePointer: OpaquePointer = OpaquePointer(c_library_create_function())
  which now should work regardless of whether you're linking the old or the new version of the C library
- when you pass a pointer to the C library:
    c_library_consuming_function(.init(myOpaquePointer))
  which should (modulo it doesn't right now https://bugs.swift.org/browse/SR-6211) select the right initialiser for you, only it doesn't 😲

But fortunately we can work around it:

--- SNIP ---
extension UnsafePointer {
    static func make(_ ptr: UnsafePointer<Pointee>) -> UnsafePointer<Pointee> {
        return ptr
    }
    static func make(_ ptr: OpaquePointer) -> UnsafePointer<Pointee> {
        return UnsafePointer(ptr)
    }
}
extension OpaquePointer {
    static func make<T>(_ ptr: UnsafePointer<T>) -> OpaquePointer {
        return OpaquePointer(ptr)!
    }
    static func make(_ ptr: OpaquePointer) -> OpaquePointer {
        return ptr
    }
}

func mockCLibraryCreateOld() -> UnsafePointer<Int> {
    return UnsafePointer(UnsafeMutablePointer<Int>.allocate(capacity: 1))
}

func mockCLibraryCreateNew() -> OpaquePointer {
    return OpaquePointer(mockCLibraryCreateOld())
}

func mockCLibraryConsumeOld(_ x: UnsafePointer<Int>) {}
func mockCLibraryConsumeNew(_ x: OpaquePointer) {}


let fromCold: OpaquePointer = .make(mockCLibraryCreateOld())
let fromCnew: OpaquePointer = .make(mockCLibraryCreateNew())

mockCLibraryConsumeOld(.make(fromCold))
mockCLibraryConsumeNew(.make(fromCnew))
mockCLibraryConsumeOld(.make(fromCnew))
mockCLibraryConsumeNew(.make(fromCold))
--- SNAP ---

HTH

-- Johannes

    

> On 24 Oct 2017, at 9:14 am, Cory Benfield via swift-evolution <swift-evolution at swift.org> wrote:
> 
> I wanted to discuss a recent difficulty I’ve encountered while writing a Swift program that uses a C library that has recently changed its API to use opaque pointers, with an eye towards asking whether there are suggestions for ways to tackle the problem that I haven’t considered, or whether some enhancement to Swift should be proposed to provide a solution.
> 
> A common trend in modern C code is to encapsulate application data by using pointers to “opaque” data structures: that is, data structures whose complete definition is not available in the header files for the library. This has many benefits from the perspective of library developers, mostly notably because it limits the ABI of the library, making it easier to change the internals without requiring recompilation or breaking changes. Pointers to these structures are translated into Swift code in the form of the OpaquePointer type.
> 
> Older C libraries frequently have non-opaque structures: that is, the structure definition is available in the header files for the library. When using code like this from Swift, pointers to these structures are translated to Unsafe[Mutable]Pointer<T>.
> 
> Both of these cases are well-handled by Swift today: opaque pointers correctly can do absolutely nothing, whereas typed pointers have the option of having behaviour based on knowing about the size of the data structure to which they point. All very good.
> 
> A problem arises if a C dependency chooses to transition from non-opaque to opaque structures. This is a transition that well-maintained C libraries are strongly incentivised to make, but if you want to write Swift code that will compile against both the old and new version of the library you run into substantial issues. To illustrate the issue I’ll construct a small problem based on the most widely-used library to recently make this transition, OpenSSL.
> 
> In OpenSSL 1.1.0 almost all of the previously-open data structures were made opaque, including the heavily used SSL_CTX structure. In terms of C code, the header file declaration changed from 
> 
> 
> 	struct ssl_ctx_st {
>    		const SSL_METHOD *method;
> 		// snip 250 lines of structure declaration
> 	}
> 	typedef struct ssl_ctx_st SSL_CTX;
> 
> to
> 
> 	typedef struct ssl_ctx_st SSL_CTX;
> 
> 
> At an API level, any function that worked on the SSL_CTX structure that existed before this change was unaffected. For example, the function SSL_CTX_use_certificate has the same C API in both versions:
> 
> 	int SSL_CTX_use_certificate(SSL_CTX *ctx, X509 *x);
> 
> Unfortunately, in Swift the API for this function changes dramatically, from 
> 
> 	func SSL_CTX_use_certificate(_ ctx: UnsafeMutablePointer<SSL_CTX>!,
> 	                                                  _ x: UnsafeMutablePointer<X509>!) -> Int32
> 
> to
> 
> 	func SSL_CTX_use_certificate(_ ctx: OpaquePointer!,
> 	                                                  _ x: OpaquePointer!) -> Int32
> 
> The reason this is problematic is that there is no implicit cast in either direction between UnsafeMutablePointer<T> and OpaquePointer. This means the API here has changed in an incompatible way: types that are valid before the structure was made opaque are not valid afterwards. This adds a pretty substantial burden to supporting multiple versions of the same library from Swift code.
> 
> So far I have thought of the following solutions to this problem that I can implement today:
> 
> 1. Write a C wrapper library that exposes a third, consistent type that is the same on all versions. Most likely this would be done by re-exposing all these methods with arguments that take `void *` and performing the cast in C code. This, unfortunately, loses some of the Swift compiler’s ability to enforce type safety, as all these arguments will now be UnsafeRawPointer. This is not any worse than OpaquePointer, but it’s objectively worse than the un-opaqued version.
> 
> 2. Write a C wrapper library that embeds these pointers in single, non-opaque structures with separate types. This allows us to keep the type safety at the cost of verbosity and an additional layer of indirection.
> 
> 3. Write two different Swift wrappers for each of these versions that expose the same outer types, and transform them internally. Not ideal: the conditional compilation story here isn’t good and distributing this library via Swift PM would be hard.
> 
> I’d be really interested in hearing whether there is a solution I’m missing that can be implemented today. If there is *not* such a solution, is there interest in attempting to tackle this problem in Swift more directly? There are plenty of language changes that could be made to solve this solution (e.g. changing OpaquePointer to OpaquePointer<T> but making it impossible to dereference, then treating UnsafePointer<T> as a subclass of OpaquePointer<T>, or making OpaquePointer a protocol implemented by UnsafePointer<T>, or all kinds of other things), but I wanted to hear from the community about suggested approach.
> 
> I’d love not to have to manually maintain a C wrapper just to escape Swift’s type system here.
> 
> Thanks,
> 
> Cory
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

More information about the swift-evolution mailing list