[swift-users] Communicating with dynamically loaded swift library
Daniel Dunbar
daniel_dunbar at apple.com
Sun Oct 8 00:49:43 CDT 2017
Is it possible for you to make a small test package that shows the problem, and file a bug in bugs.swift.org? There may be something we need to fix in SwiftPM before this can work (because of our linking model).
- Daniel
> On Oct 7, 2017, at 10:42 PM, Ján Kosa via swift-users <swift-users at swift.org> wrote:
>
> That is exactly what I did. The only package that depends on the protobuf is the PluginInterface. Both MyPlugin and and PluginConsumer depend on the PluginInterface and not on the protobuf itself. I had to shuffle around my dependencies a bit, which resulted in smaller number of dependencies but they don't make much sense now (as in, some target had to depend on PluginInterface even if they don't need to, just to get access to protobuf). I could live with that if it solved the issue, but it didn't.
>
> I am adding my Package.swift files in case I missed something:
>
> PluginInterface:
>
> ```swift
> let package = Package(
> name: "PluginInterface",
>
> products: [ .library(name: "PluginInterface", type: .dynamic, targets: ["PluginInterface"]) ],
>
> dependencies: [ .package(url: "https://github.com/apple/swift-protobuf.git <https://github.com/apple/swift-protobuf.git>", from: "0.0.0") ],
>
> targets: [ .target(name: "PluginInterface", dependencies: ["SwiftProtobuf"]) ]
>
> )```
>
>
>
> MyPlugin:
>
> ```swift
>
> let package = Package(
>
> name: "MyPlugin",
>
> products: [ .library(name: "MyPlugin", type: .dynamic, targets: ["PluginImpl"]) ],
>
> dependencies: [
>
> .package(url: "path/to/PluginInterface.git", from: "0.0.0"),
>
> ],
>
> targets: [
>
> .target(name: "PluginImpl", dependencies: ["ProtoBufMessages"]),
>
> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>
> ]
>
>
> )```
>
>
>
> PluginConsumer:
>
> ```swift
>
> let package = Package(
>
> name: "PluginConsumer",
>
> dependencies: [
>
> .package(url: "https://github.com/PerfectlySoft/Perfect-WebSockets.git <https://github.com/PerfectlySoft/Perfect-WebSockets.git>", from: "3.0.0"),
>
> .package(url: "https://github.com/PerfectlySoft/Perfect-HTTPServer.git <https://github.com/PerfectlySoft/Perfect-HTTPServer.git>", from: "3.0.0"),
>
> .package(url: "path/to/PluginInterface", from: "0.0.0"),
>
> .package(url: "https://github.com/krzyzanowskim/CryptoSwift.git <https://github.com/krzyzanowskim/CryptoSwift.git>", from: "0.0.0")
>
> ],
>
> targets: [
>
> .target(name: "AppMaster", dependencies: ["Shared", "CryptoSwift"]),
>
> .target(name: "PluginConsumer", dependencies: ["Shared", "CryptoSwift"]),
>
> .target(name: "Shared", dependencies: ["ProtoBufMessages", "PerfectHTTPServer", "PerfectWebSockets"]),
>
> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>
> ]
>
>
> )```
>
>
>
> App master is separate executable that shares some functionality with PluginConsumer, but it doesn't link against it in any way. I guess it could be omitted, but I wanted to give you whole thing as it is
>
>
> On 7 October 2017 at 18:33, Geordie Jay <geojay at gmail.com <mailto:geojay at gmail.com>> wrote:
>
> Ján Kosa <lopenka at gmail.com <mailto:lopenka at gmail.com>> schrieb am Sa. 7. Okt. 2017 um 15:27:
> I tried to use @_exported and it helped somewhat. While I still have same warnings, size of the PluginInterface library went down by 6mb (to 120kb) so it looks like Protobuf is no longer statically linked to it. However, size of PluginConsumer executable went up by same 6mb, it looks like it is linked there twice now.
>
> To be clear: take protobuf out of the PluginConsumer dependencies. Actually, I’m not sure which is which, but protobuf should only be listed as a dependency of one package, where it is imported as @_exported. After that, your other modules depend on the package that imports / exports protobuf.
>
> I also noticed interesting thing. If I build executable using `swift build` the size is around 17mb, when I generate xcode project and build it using that, size is around 200kb, but I get same warnings using both approaches
>
> On 7 October 2017 at 15:44, Geordie Jay <geojay at gmail.com <mailto:geojay at gmail.com>> wrote:
>
> Ján Kosa <lopenka at gmail.com <mailto:lopenka at gmail.com>> schrieb am Sa. 7. Okt. 2017 um 13:34:
> I tried swift package clean, but it didn't help
>
> "Try to ensure the plugin provider module (libA) is (only) being compiled into its standalone shared library file."
> How do I go about this? It is 3rd party module, it doesn't define any products (https://github.com/apple/swift-protobuf.git <https://github.com/apple/swift-protobuf.git>). Is there something I can do in my Package files to make sure it is loaded dynamically?
>
> When you compile a package depending on protobuf, all the relevant symbols end up in your package’s library file. So here’s something you might try:
>
> import protobuf into your own “PluginProvider” module (package), which has a shared library product like this: ‘@_exported import Protobuf’ in some compiled swift file. Then from the other dependent modules “import PluginProvider” - the protobuf symbols should be available, and all from one (nonconflicting) source.
>
> Geordie
>
>
>
> On 6 October 2017 at 22:52, Geordie Jay <geojay at gmail.com <mailto:geojay at gmail.com>> wrote:
> I think SwiftPM is (incorrectly) compiling A.XYZ <http://a.xyz/> into each of the modules that depend on it, as well as into your intended libA.so file.
>
> Try to ensure the plugin provider module (libA) is (only) being compiled into its standalone shared library file. Try cleaning the swiftpm build for one (swift package clean) and ensure the Package.swift files are correctly set up to output the shared library.
>
> Sorry I can’t be more specific, I’ve had these same kinds of issues before but I’m not 100% what they were.
>
> Geordie
>
>
> Ján Kosa via swift-users <swift-users at swift.org <mailto:swift-users at swift.org>> schrieb am Fr. 6. Okt. 2017 um 14:41:
> It worked! Took me a while to iron out details, but it is working now. Huge thanks sir, I will name my firstborn after you.
> Thanks for the @_cdecl("initializePlugin") tip as well, I didn't know about it and it will be very useful.
>
> I am having slightly related problem now (it was there before, but I ignored it for the time being), not sure if I should start a new thread?
>
> The PluginInterface module has one external dependency on module A, PluginConsumer has the dependency on module B which has dependency on same module A that the PluginInterface uses. When I load the plugin library, I get bunch of errors like:
>
> Class A.XYZ <http://a.xyz/> is implemented in both libPluginInterface.dylib and libMyPlugin.dylib
>
> I know why it is there, but I don't know how to get rid of it. I can't just remove dependency from PluginConsumer and use the one from PluginInterface (if that would even work?) because PluginConsumer does not depend on it directly, but it is going through module B first
>
> Cheers,
> Lope
>
> On 4 October 2017 at 22:17, Daniel Dunbar <daniel_dunbar at apple.com <mailto:daniel_dunbar at apple.com>> wrote:
> The way that I have done this in the past is pass a protocol as an unsafe pointer to an exposed entry point:
> ```swift
> let entryPoint = dlsym(handle, “initializePlugin”)
> guard entryPoint != nil else {
> fatalError("missing plugin entry point: \(pluginPath)")
> }
> typealias PluginInitializationFunc = @convention(c) (UnsafeRawPointer) -> ()
> let f = unsafeBitCast(entryPoint, to: PluginInitializationFunc.self)
> f(Unmanaged.passUnretained(self).toOpaque())
> ```
>
> and then in the plugin convert back to the appropriate type:
>
> ```
> @_cdecl("initializePlugin")
> public func initializePlugin(_ ptr: UnsafeRawPointer) {
> let manager = Unmanaged<PluginManager>.fromOpaque(ptr).takeUnretainedValue()
> ```
>
> HTH,
> - Daniel
>
>> On Oct 4, 2017, at 11:02 AM, Ján Kosa via swift-users <swift-users at swift.org <mailto:swift-users at swift.org>> wrote:
>>
>> Hello folks,
>>
>> I have been toying with dynamic libraries, trying to implement plugin functionality. I was able to get to the point where I can call simple function in loaded library, but I am having troubles starting more sophisticated communication channel.
>>
>> There are 3 projects
>> - PluginConsumer is an app that loads plugin libraries
>> - MyPlugin is a plugin implementation, output is dynamic library that PluginConsumer loads
>> - PluginInterface is common interface that both MyPlugin and PluginConsumer use, so that they know how to communicate
>>
>> My first idea was to have PluginInterface be a simple SPM project with single file where the bare-bones PluginInterface class would be:
>>
>>
>> open class PluginInterface {
>>
>> open func sayHi()
>>
>> }
>>
>>
>>
>> Package.swift file:
>>
>>
>>
>> // swift-tools-version:4.0
>>
>> import PackageDescription
>>
>> let package = Package(
>>
>> name: "PluginInterface",
>>
>> products: [ .library(name: "PluginInterface", type: .dynamic, targets: ["PluginInterface"]) ],
>>
>> targets: [ .target(name: "PluginInterface") ]
>>
>>
>> )
>>
>>
>>
>>
>>
>> UserPlugin is also very simple project containing only one file:
>>
>>
>>
>> public func getPlugin() -> AnyObject {
>>
>> return MyPlugin()
>>
>>
>> }
>>
>>
>>
>> class MyPlugin: PluginInterface {
>>
>> override func sayHi() {
>>
>> print("Hi from my plugin")
>>
>> }
>>
>> }
>>
>> Package.swift:
>>
>>
>>
>> // swift-tools-version:4.0
>>
>> import PackageDescription
>>
>> let package = Package(
>>
>> name: "MyPlugin",
>>
>> products: [ .library(name: "MyPlugin", type: .dynamic, targets: ["MyPlugin"]) ],
>>
>> dependencies: [ .package(url: "url_to_PluginInterface", from: "0.0.0"), ],
>>
>> targets: [
>>
>> .target(name: "PluginInterface", dependencies: ["PluginInterface"]),
>>
>> .target(name: "MyPlugin", dependencies: ["PluginInterface"]),
>>
>> ]
>>
>>
>> )
>>
>>
>>
>> The PluginConsumer is bit more complicated, but here is relevant part (lib loading and function calling):
>>
>>
>>
>> typealias InitFunction = @convention(c) () -> AnyObject
>>
>>
>>
>> let openRes = dlopen(pathToLib, RTLD_NOW|RTLD_LOCAL)
>>
>> if openRes != nil {
>>
>> defer {
>>
>> dlclose(openRes)
>>
>> }
>>
>> let symbolName = "mangled_symbol_name"
>>
>> let sym = dlsym(openRes, symbolName)
>>
>>
>>
>> if sym != nil {
>>
>> let f: InitFunction = unsafeBitCast(sym, to: InitFunction.self)
>>
>> let plugin = f() as? PluginInterface
>>
>> }
>>
>>
>> }
>>
>> Package.swift file:
>>
>> // swift-tools-version:4.0
>>
>> import PackageDescription
>>
>> let package = Package(
>>
>> name: "PluginConsumer",
>>
>> dependencies: [ .package(url: "path_to_plugin_interface", from: "0.0.0") ],
>>
>> targets: [ .target(name: "PluginConsumer", dependencies: ["PluginConsumer"]) ]
>>
>>
>> )
>>
>>
>>
>> This all compiles nicely, MyPlugin project creates dylib file that executable created by PluginConsumer can load, but the problem is with following line:
>>
>> let plugin = f() as? PluginInterface
>>
>> Type of the plugin is MyPlugin, but from the consumer's view, it doesn't inherit from PluginInterface so I can't call sayHi() method. I assume this is because there is no relation between PluginInterface class that compiler uses for MyPlugin project one that it uses for PluginConsumer project. After library is loaded, they are two completely different classes that happen to share same name. Is my assumption correct and how do I go about fixing it?
>>
>> I had an idea I could make PluginInterface emit dynamic library that would be dynamically linked by both MyPlugin and PluginConsumer, thus making them share same PluginInterface class, but I can't figure out how to do that (or if it's right way of doing this).
>>
>>
>>
>> Any help appreciated :)
>>
>> Lope
>>
>> _______________________________________________
>> swift-users mailing list
>> swift-users at swift.org <mailto:swift-users at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-users <https://lists.swift.org/mailman/listinfo/swift-users>
>
>
> _______________________________________________
> swift-users mailing list
> swift-users at swift.org <mailto:swift-users at swift.org>
> https://lists.swift.org/mailman/listinfo/swift-users <https://lists.swift.org/mailman/listinfo/swift-users>
>
>
>
> _______________________________________________
> swift-users mailing list
> swift-users at swift.org
> https://lists.swift.org/mailman/listinfo/swift-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-users/attachments/20171007/0e4c81d1/attachment.html>
More information about the swift-users
mailing list