[swift-users] Communicating with dynamically loaded swift library
Ján Kosa
lopenka at gmail.com
Sun Oct 8 12:02:05 CDT 2017
I filed bug report: https://bugs.swift.org/browse/SR-6091
On 8 October 2017 at 20:31, Daniel Dunbar <daniel_dunbar at apple.com> wrote:
> Can you file this on bugs.swift.org? We are likely to lose track of it if
> it just an email thread
>
> - Daniel
>
> On Oct 8, 2017, at 8:27 AM, Ján Kosa <lopenka at gmail.com> wrote:
>
> Hey guys,
>
> I have been able to create simplest possible setup that exhibits this
> problem, the protobuf module wasn't necessary. All I import is
> PluginInterface and I get the error:
>
> objc[66212]: Class _TtC15PluginInterface15PluginInterface is implemented
> in both /Users/Lope/Dev/swift/SwiftPlugins/PluginConsumer/.
> build/x86_64-apple-macosx10.10/debug/libPluginInterface.dylib
> (0x10fc7b668) and /Users/Lope/Dev/swift/SwiftPlugins/
> PluginImplementation/.build/x86_64-apple-macosx10.10/debug/libPluginInterface.dylib
> (0x1107dd668). One of the two will be used. Which one is undefined.
>
>
> You can find modules on my github:
>
> https://github.com/Lopdo/SwiftPlugins-PluginInterface
>
> https://github.com/Lopdo/SwiftPlugins-PluginImplementation
>
> https://github.com/Lopdo/SwiftPlugins-PluginConsumer
>
>
> Could you have a quick look if I didn't mess up or missed something and I
> will create bug report. If you want to run the code for yourself, you will
> have to change the path to dylib in the PluginConsumer (I don't know how to
> use relative path and didn't have time to find out yet)
>
> On 8 October 2017 at 09:51, Ján Kosa via swift-users <
> swift-users at swift.org> wrote:
>
>> I was afraid it will come to that :) I will try to make something
>> tonight, either it will help you fix it, or I will find out what I did wrong
>>
>> On 8 October 2017 at 09:49, Daniel Dunbar <daniel_dunbar at apple.com>
>> wrote:
>>
>>> Is it possible for you to make a small test package that shows the
>>> problem, and file a bug in bugs.swift.org? There may be something we
>>> need to fix in SwiftPM before this can work (because of our linking model).
>>>
>>> - Daniel
>>>
>>>
>>> On Oct 7, 2017, at 10:42 PM, Ján Kosa via swift-users <
>>> swift-users at swift.org> wrote:
>>>
>>> That is exactly what I did. The only package that depends on the
>>> protobuf is the PluginInterface. Both MyPlugin and and PluginConsumer
>>> depend on the PluginInterface and not on the protobuf itself. I had to
>>> shuffle around my dependencies a bit, which resulted in smaller number of
>>> dependencies but they don't make much sense now (as in, some target had to
>>> depend on PluginInterface even if they don't need to, just to get access to
>>> protobuf). I could live with that if it solved the issue, but it didn't.
>>>
>>> I am adding my Package.swift files in case I missed something:
>>>
>>> PluginInterface:
>>>
>>> ```swift
>>> let package = Package(
>>>
>>> name: "PluginInterface",
>>>
>>> products: [ .library(name: "PluginInterface", type: .dynamic, targets: [
>>> "PluginInterface"]) ],
>>>
>>> dependencies: [ .package(url: "https://github.com/apple/swif
>>> t-protobuf.git", from: "0.0.0") ],
>>>
>>> targets: [ .target(name: "PluginInterface", dependencies: [
>>> "SwiftProtobuf"]) ]
>>>
>>> )```
>>>
>>>
>>> MyPlugin:
>>>
>>> ```swift
>>>
>>> let package = Package(
>>>
>>> name: "MyPlugin",
>>>
>>> products: [ .library(name: "MyPlugin", type: .dynamic, targets: [
>>> "PluginImpl"]) ],
>>>
>>> dependencies: [
>>>
>>> .package(url: "path/to/PluginInterface.git", from: "0.0.0"),
>>>
>>> ],
>>>
>>> targets: [
>>>
>>> .target(name: "PluginImpl", dependencies: ["ProtoBufMessages"]),
>>>
>>> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>>>
>>> ]
>>>
>>> )```
>>>
>>>
>>> PluginConsumer:
>>>
>>> ```swift
>>>
>>> let package = Package(
>>>
>>> name: "PluginConsumer",
>>>
>>> dependencies: [
>>>
>>> .package(url: "https://github.com/PerfectlySoft/Perfect-WebSockets.git",
>>> from: "3.0.0"),
>>>
>>> .package(url: "https://github.com/PerfectlySoft/Perfect-HTTPServer.git",
>>> from: "3.0.0"),
>>>
>>> .package(url: "path/to/PluginInterface", from: "0.0.0"),
>>>
>>> .package(url: "https://github.com/krzyzanowskim/CryptoSwift.git", from:
>>> "0.0.0")
>>>
>>> ],
>>>
>>> targets: [
>>>
>>> .target(name: "AppMaster", dependencies: ["Shared", "CryptoSwift"]),
>>>
>>> .target(name: "PluginConsumer", dependencies: ["Shared", "CryptoSwift"
>>> ]),
>>>
>>> .target(name: "Shared", dependencies: ["ProtoBufMessages",
>>> "PerfectHTTPServer", "PerfectWebSockets"]),
>>>
>>> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>>>
>>> ]
>>>
>>> )```
>>>
>>>
>>> App master is separate executable that shares some functionality with
>>> PluginConsumer, but it doesn't link against it in any way. I guess it could
>>> be omitted, but I wanted to give you whole thing as it is
>>>
>>> On 7 October 2017 at 18:33, Geordie Jay <geojay at gmail.com> wrote:
>>>
>>>>
>>>> Ján Kosa <lopenka at gmail.com> schrieb am Sa. 7. Okt. 2017 um 15:27:
>>>>
>>>>> I tried to use @_exported and it helped somewhat. While I still have
>>>>> same warnings, size of the PluginInterface library went down by 6mb (to
>>>>> 120kb) so it looks like Protobuf is no longer statically linked to it.
>>>>> However, size of PluginConsumer executable went up by same 6mb, it looks
>>>>> like it is linked there twice now.
>>>>>
>>>>
>>>> To be clear: take protobuf out of the PluginConsumer dependencies.
>>>> Actually, I’m not sure which is which, but protobuf should only be listed
>>>> as a dependency of one package, where it is imported as @_exported. After
>>>> that, your other modules depend on the package that imports / exports
>>>> protobuf.
>>>>
>>>> I also noticed interesting thing. If I build executable using `swift
>>>>> build` the size is around 17mb, when I generate xcode project and build it
>>>>> using that, size is around 200kb, but I get same warnings using both
>>>>> approaches
>>>>>
>>>>> On 7 October 2017 at 15:44, Geordie Jay <geojay at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> Ján Kosa <lopenka at gmail.com> schrieb am Sa. 7. Okt. 2017 um 13:34:
>>>>>>
>>>>>>> I tried swift package clean, but it didn't help
>>>>>>>
>>>>>>> "Try to ensure the plugin provider module (libA) is (only) being
>>>>>>> compiled into its standalone shared library file."
>>>>>>> How do I go about this? It is 3rd party module, it doesn't define
>>>>>>> any products (https://github.com/apple/swift-protobuf.git). Is
>>>>>>> there something I can do in my Package files to make sure it is loaded
>>>>>>> dynamically?
>>>>>>>
>>>>>>
>>>>>> When you compile a package depending on protobuf, all the relevant
>>>>>> symbols end up in your package’s library file. So here’s something you
>>>>>> might try:
>>>>>>
>>>>>> import protobuf into your own “PluginProvider” module (package),
>>>>>> which has a shared library product like this: ‘@_exported import Protobuf’
>>>>>> in some compiled swift file. Then from the other dependent modules “import
>>>>>> PluginProvider” - the protobuf symbols should be available, and all from
>>>>>> one (nonconflicting) source.
>>>>>>
>>>>>> Geordie
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 6 October 2017 at 22:52, Geordie Jay <geojay at gmail.com> wrote:
>>>>>>>
>>>>>>>> I think SwiftPM is (incorrectly) compiling A.XYZ <http://a.xyz/>
>>>>>>>> into each of the modules that depend on it, as well as into your intended
>>>>>>>> libA.so file.
>>>>>>>>
>>>>>>>> Try to ensure the plugin provider module (libA) is (only) being
>>>>>>>> compiled into its standalone shared library file. Try cleaning the swiftpm
>>>>>>>> build for one (swift package clean) and ensure the Package.swift files are
>>>>>>>> correctly set up to output the shared library.
>>>>>>>>
>>>>>>>> Sorry I can’t be more specific, I’ve had these same kinds of issues
>>>>>>>> before but I’m not 100% what they were.
>>>>>>>>
>>>>>>>> Geordie
>>>>>>>>
>>>>>>>>
>>>>>>>> Ján Kosa via swift-users <swift-users at swift.org> schrieb am Fr. 6.
>>>>>>>> Okt. 2017 um 14:41:
>>>>>>>>
>>>>>>>>> It worked! Took me a while to iron out details, but it is working
>>>>>>>>> now. Huge thanks sir, I will name my firstborn after you.
>>>>>>>>> Thanks for the @_cdecl("initializePlugin") tip as well, I didn't
>>>>>>>>> know about it and it will be very useful.
>>>>>>>>>
>>>>>>>>> I am having slightly related problem now (it was there before, but
>>>>>>>>> I ignored it for the time being), not sure if I should start a new thread?
>>>>>>>>>
>>>>>>>>> The PluginInterface module has one external dependency on module
>>>>>>>>> A, PluginConsumer has the dependency on module B which has dependency on
>>>>>>>>> same module A that the PluginInterface uses. When I load the plugin
>>>>>>>>> library, I get bunch of errors like:
>>>>>>>>>
>>>>>>>>> Class A.XYZ <http://a.xyz/> is implemented in
>>>>>>>>> both libPluginInterface.dylib and libMyPlugin.dylib
>>>>>>>>>
>>>>>>>>> I know why it is there, but I don't know how to get rid of it. I
>>>>>>>>> can't just remove dependency from PluginConsumer and use the one from
>>>>>>>>> PluginInterface (if that would even work?) because PluginConsumer does not
>>>>>>>>> depend on it directly, but it is going through module B first
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Lope
>>>>>>>>>
>>>>>>>>> On 4 October 2017 at 22:17, Daniel Dunbar <daniel_dunbar at apple.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> The way that I have done this in the past is pass a protocol as
>>>>>>>>>> an unsafe pointer to an exposed entry point:
>>>>>>>>>> ```swift
>>>>>>>>>> let entryPoint = dlsym(handle, “initializePlugin”)
>>>>>>>>>> guard entryPoint != nil else {
>>>>>>>>>> fatalError("missing plugin entry point:
>>>>>>>>>> \(pluginPath)")
>>>>>>>>>> }
>>>>>>>>>> typealias PluginInitializationFunc = @convention(c)
>>>>>>>>>> (UnsafeRawPointer) -> ()
>>>>>>>>>> let f = unsafeBitCast(entryPoint, to:
>>>>>>>>>> PluginInitializationFunc.self)
>>>>>>>>>> f(Unmanaged.passUnretained(self).toOpaque())
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> and then in the plugin convert back to the appropriate type:
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> @_cdecl("initializePlugin")
>>>>>>>>>> public func initializePlugin(_ ptr: UnsafeRawPointer) {
>>>>>>>>>> let manager = Unmanaged<PluginManager>.fromO
>>>>>>>>>> paque(ptr).takeUnretainedValue()
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> HTH,
>>>>>>>>>> - Daniel
>>>>>>>>>>
>>>>>>>>>> On Oct 4, 2017, at 11:02 AM, Ján Kosa via swift-users <
>>>>>>>>>> swift-users at swift.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello folks,
>>>>>>>>>>
>>>>>>>>>> I have been toying with dynamic libraries, trying to implement
>>>>>>>>>> plugin functionality. I was able to get to the point where I can call
>>>>>>>>>> simple function in loaded library, but I am having troubles starting more
>>>>>>>>>> sophisticated communication channel.
>>>>>>>>>>
>>>>>>>>>> There are 3 projects
>>>>>>>>>> - PluginConsumer is an app that loads plugin libraries
>>>>>>>>>> - MyPlugin is a plugin implementation, output is dynamic library
>>>>>>>>>> that PluginConsumer loads
>>>>>>>>>> - PluginInterface is common interface that both MyPlugin and
>>>>>>>>>> PluginConsumer use, so that they know how to communicate
>>>>>>>>>>
>>>>>>>>>> My first idea was to have PluginInterface be a simple SPM project
>>>>>>>>>> with single file where the bare-bones PluginInterface class would be:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> open class PluginInterface {
>>>>>>>>>>
>>>>>>>>>> open func sayHi()
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Package.swift file:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> // swift-tools-version:4.0
>>>>>>>>>>
>>>>>>>>>> import PackageDescription
>>>>>>>>>>
>>>>>>>>>> let package = Package(
>>>>>>>>>>
>>>>>>>>>> name: "PluginInterface",
>>>>>>>>>>
>>>>>>>>>> products: [ .library(name: "PluginInterface", type:
>>>>>>>>>> .dynamic, targets: ["PluginInterface"]) ],
>>>>>>>>>>
>>>>>>>>>> targets: [ .target(name: "PluginInterface") ]
>>>>>>>>>>
>>>>>>>>>> )
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> UserPlugin is also very simple project containing only one file:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> public func getPlugin() -> AnyObject {
>>>>>>>>>>
>>>>>>>>>> return MyPlugin()
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> class MyPlugin: PluginInterface {
>>>>>>>>>>
>>>>>>>>>> override func sayHi() {
>>>>>>>>>>
>>>>>>>>>> print("Hi from my plugin")
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Package.swift:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> // swift-tools-version:4.0
>>>>>>>>>>
>>>>>>>>>> import PackageDescription
>>>>>>>>>>
>>>>>>>>>> let package = Package(
>>>>>>>>>>
>>>>>>>>>> name: "MyPlugin",
>>>>>>>>>>
>>>>>>>>>> products: [ .library(name: "MyPlugin", type: .dynamic,
>>>>>>>>>> targets: ["MyPlugin"]) ],
>>>>>>>>>>
>>>>>>>>>> dependencies: [ .package(url: "url_to_PluginInterface",
>>>>>>>>>> from: "0.0.0"), ],
>>>>>>>>>>
>>>>>>>>>> targets: [
>>>>>>>>>>
>>>>>>>>>> .target(name: "PluginInterface", dependencies: [
>>>>>>>>>> "PluginInterface"]),
>>>>>>>>>>
>>>>>>>>>> .target(name: "MyPlugin", dependencies: [
>>>>>>>>>> "PluginInterface"]),
>>>>>>>>>>
>>>>>>>>>> ]
>>>>>>>>>>
>>>>>>>>>> )
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The PluginConsumer is bit more complicated, but here is relevant
>>>>>>>>>> part (lib loading and function calling):
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> typealias InitFunction = @convention(c) () -> AnyObject
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> let openRes = dlopen(pathToLib, RTLD_NOW|RTLD_LOCAL)
>>>>>>>>>>
>>>>>>>>>> if openRes != nil {
>>>>>>>>>>
>>>>>>>>>> defer {
>>>>>>>>>>
>>>>>>>>>> dlclose(openRes)
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> let symbolName = "mangled_symbol_name"
>>>>>>>>>>
>>>>>>>>>> let sym = dlsym(openRes, symbolName)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> if sym != nil {
>>>>>>>>>>
>>>>>>>>>> let f: InitFunction = unsafeBitCast(sym, to: InitFunction
>>>>>>>>>> .self)
>>>>>>>>>>
>>>>>>>>>> let plugin = f() as? PluginInterface
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Package.swift file:
>>>>>>>>>>
>>>>>>>>>> // swift-tools-version:4.0
>>>>>>>>>>
>>>>>>>>>> import PackageDescription
>>>>>>>>>>
>>>>>>>>>> let package = Package(
>>>>>>>>>>
>>>>>>>>>> name: "PluginConsumer",
>>>>>>>>>>
>>>>>>>>>> dependencies: [ .package(url: "path_to_plugin_interface",
>>>>>>>>>> from: "0.0.0") ],
>>>>>>>>>>
>>>>>>>>>> targets: [ .target(name: "PluginConsumer", dependencies: [
>>>>>>>>>> "PluginConsumer"]) ]
>>>>>>>>>>
>>>>>>>>>> )
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This all compiles nicely, MyPlugin project creates dylib file
>>>>>>>>>> that executable created by PluginConsumer can load, but the problem is with
>>>>>>>>>> following line:
>>>>>>>>>>
>>>>>>>>>> let plugin = f() as? PluginInterface
>>>>>>>>>>
>>>>>>>>>> Type of the plugin is MyPlugin, but from the consumer's view, it
>>>>>>>>>> doesn't inherit from PluginInterface so I can't call sayHi() method. I
>>>>>>>>>> assume this is because there is no relation between PluginInterface class
>>>>>>>>>> that compiler uses for MyPlugin project one that it uses for PluginConsumer
>>>>>>>>>> project. After library is loaded, they are two completely different classes
>>>>>>>>>> that happen to share same name. Is my assumption correct and how do I go
>>>>>>>>>> about fixing it?
>>>>>>>>>>
>>>>>>>>>> I had an idea I could make PluginInterface emit dynamic library
>>>>>>>>>> that would be dynamically linked by both MyPlugin and PluginConsumer, thus
>>>>>>>>>> making them share same PluginInterface class, but I can't figure out how to
>>>>>>>>>> do that (or if it's right way of doing this).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Any help appreciated :)
>>>>>>>>>>
>>>>>>>>>> Lope
>>>>>>>>>> _______________________________________________
>>>>>>>>>> swift-users mailing list
>>>>>>>>>> swift-users at swift.org
>>>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> swift-users mailing list
>>>>>>>>> swift-users at swift.org
>>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>> _______________________________________________
>>> swift-users mailing list
>>> swift-users at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>
>>>
>>>
>>
>> _______________________________________________
>> swift-users mailing list
>> swift-users at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-users/attachments/20171008/52499bea/attachment.html>
More information about the swift-users
mailing list