[swift-evolution] Opaque Pointers in Swift

[Cory Benfield]

I wanted to discuss a recent difficulty I’ve encountered while writing a Swift program that uses a C library that has recently changed its API to use opaque pointers, with an eye towards asking whether there are suggestions for ways to tackle the problem that I haven’t considered, or whether some enhancement to Swift should be proposed to provide a solution.

A common trend in modern C code is to encapsulate application data by using pointers to “opaque” data structures: that is, data structures whose complete definition is not available in the header files for the library. This has many benefits from the perspective of library developers, mostly notably because it limits the ABI of the library, making it easier to change the internals without requiring recompilation or breaking changes. Pointers to these structures are translated into Swift code in the form of the OpaquePointer type.

Older C libraries frequently have non-opaque structures: that is, the structure definition is available in the header files for the library. When using code like this from Swift, pointers to these structures are translated to Unsafe[Mutable]Pointer<T>.

Both of these cases are well-handled by Swift today: opaque pointers correctly can do absolutely nothing, whereas typed pointers have the option of having behaviour based on knowing about the size of the data structure to which they point. All very good.

A problem arises if a C dependency chooses to transition from non-opaque to opaque structures. This is a transition that well-maintained C libraries are strongly incentivised to make, but if you want to write Swift code that will compile against both the old and new version of the library you run into substantial issues. To illustrate the issue I’ll construct a small problem based on the most widely-used library to recently make this transition, OpenSSL.

In OpenSSL 1.1.0 almost all of the previously-open data structures were made opaque, including the heavily used SSL_CTX structure. In terms of C code, the header file declaration changed from 


	struct ssl_ctx_st {
    		const SSL_METHOD *method;
		// snip 250 lines of structure declaration
	}
	typedef struct ssl_ctx_st SSL_CTX;

to

	typedef struct ssl_ctx_st SSL_CTX;


At an API level, any function that worked on the SSL_CTX structure that existed before this change was unaffected. For example, the function SSL_CTX_use_certificate has the same C API in both versions:

	int SSL_CTX_use_certificate(SSL_CTX *ctx, X509 *x);

Unfortunately, in Swift the API for this function changes dramatically, from 

	func SSL_CTX_use_certificate(_ ctx: UnsafeMutablePointer<SSL_CTX>!,
	                                                  _ x: UnsafeMutablePointer<X509>!) -> Int32

to

	func SSL_CTX_use_certificate(_ ctx: OpaquePointer!,
	                                                  _ x: OpaquePointer!) -> Int32

The reason this is problematic is that there is no implicit cast in either direction between UnsafeMutablePointer<T> and OpaquePointer. This means the API here has changed in an incompatible way: types that are valid before the structure was made opaque are not valid afterwards. This adds a pretty substantial burden to supporting multiple versions of the same library from Swift code.

So far I have thought of the following solutions to this problem that I can implement today:

1. Write a C wrapper library that exposes a third, consistent type that is the same on all versions. Most likely this would be done by re-exposing all these methods with arguments that take `void *` and performing the cast in C code. This, unfortunately, loses some of the Swift compiler’s ability to enforce type safety, as all these arguments will now be UnsafeRawPointer. This is not any worse than OpaquePointer, but it’s objectively worse than the un-opaqued version.

2. Write a C wrapper library that embeds these pointers in single, non-opaque structures with separate types. This allows us to keep the type safety at the cost of verbosity and an additional layer of indirection.

3. Write two different Swift wrappers for each of these versions that expose the same outer types, and transform them internally. Not ideal: the conditional compilation story here isn’t good and distributing this library via Swift PM would be hard.

I’d be really interested in hearing whether there is a solution I’m missing that can be implemented today. If there is *not* such a solution, is there interest in attempting to tackle this problem in Swift more directly? There are plenty of language changes that could be made to solve this solution (e.g. changing OpaquePointer to OpaquePointer<T> but making it impossible to dereference, then treating UnsafePointer<T> as a subclass of OpaquePointer<T>, or making OpaquePointer a protocol implemented by UnsafePointer<T>, or all kinds of other things), but I wanted to hear from the community about suggested approach.

I’d love not to have to manually maintain a C wrapper just to escape Swift’s type system here.

Thanks,

Cory