[swift-users] Communicating with dynamically loaded swift library
Ján Kosa
lopenka at gmail.com
Sun Oct 8 10:27:18 CDT 2017
Hey guys,
I have been able to create simplest possible setup that exhibits this
problem, the protobuf module wasn't necessary. All I import is
PluginInterface and I get the error:
objc[66212]: Class _TtC15PluginInterface15PluginInterface is implemented in
both
/Users/Lope/Dev/swift/SwiftPlugins/PluginConsumer/.build/x86_64-apple-macosx10.10/debug/libPluginInterface.dylib
(0x10fc7b668) and
/Users/Lope/Dev/swift/SwiftPlugins/PluginImplementation/.build/x86_64-apple-macosx10.10/debug/libPluginInterface.dylib
(0x1107dd668). One of the two will be used. Which one is undefined.
You can find modules on my github:
https://github.com/Lopdo/SwiftPlugins-PluginInterface
https://github.com/Lopdo/SwiftPlugins-PluginImplementation
https://github.com/Lopdo/SwiftPlugins-PluginConsumer
Could you have a quick look if I didn't mess up or missed something and I
will create bug report. If you want to run the code for yourself, you will
have to change the path to dylib in the PluginConsumer (I don't know how to
use relative path and didn't have time to find out yet)
On 8 October 2017 at 09:51, Ján Kosa via swift-users <swift-users at swift.org>
wrote:
> I was afraid it will come to that :) I will try to make something tonight,
> either it will help you fix it, or I will find out what I did wrong
>
> On 8 October 2017 at 09:49, Daniel Dunbar <daniel_dunbar at apple.com> wrote:
>
>> Is it possible for you to make a small test package that shows the
>> problem, and file a bug in bugs.swift.org? There may be something we
>> need to fix in SwiftPM before this can work (because of our linking model).
>>
>> - Daniel
>>
>>
>> On Oct 7, 2017, at 10:42 PM, Ján Kosa via swift-users <
>> swift-users at swift.org> wrote:
>>
>> That is exactly what I did. The only package that depends on the protobuf
>> is the PluginInterface. Both MyPlugin and and PluginConsumer depend on the
>> PluginInterface and not on the protobuf itself. I had to shuffle around my
>> dependencies a bit, which resulted in smaller number of dependencies but
>> they don't make much sense now (as in, some target had to depend on
>> PluginInterface even if they don't need to, just to get access to
>> protobuf). I could live with that if it solved the issue, but it didn't.
>>
>> I am adding my Package.swift files in case I missed something:
>>
>> PluginInterface:
>>
>> ```swift
>> let package = Package(
>>
>> name: "PluginInterface",
>>
>> products: [ .library(name: "PluginInterface", type: .dynamic, targets: [
>> "PluginInterface"]) ],
>>
>> dependencies: [ .package(url: "https://github.com/apple/swif
>> t-protobuf.git", from: "0.0.0") ],
>>
>> targets: [ .target(name: "PluginInterface", dependencies: [
>> "SwiftProtobuf"]) ]
>>
>> )```
>>
>>
>> MyPlugin:
>>
>> ```swift
>>
>> let package = Package(
>>
>> name: "MyPlugin",
>>
>> products: [ .library(name: "MyPlugin", type: .dynamic, targets: [
>> "PluginImpl"]) ],
>>
>> dependencies: [
>>
>> .package(url: "path/to/PluginInterface.git", from: "0.0.0"),
>>
>> ],
>>
>> targets: [
>>
>> .target(name: "PluginImpl", dependencies: ["ProtoBufMessages"]),
>>
>> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>>
>> ]
>>
>> )```
>>
>>
>> PluginConsumer:
>>
>> ```swift
>>
>> let package = Package(
>>
>> name: "PluginConsumer",
>>
>> dependencies: [
>>
>> .package(url: "https://github.com/PerfectlySoft/Perfect-WebSockets.git",
>> from: "3.0.0"),
>>
>> .package(url: "https://github.com/PerfectlySoft/Perfect-HTTPServer.git",
>> from: "3.0.0"),
>>
>> .package(url: "path/to/PluginInterface", from: "0.0.0"),
>>
>> .package(url: "https://github.com/krzyzanowskim/CryptoSwift.git", from:
>> "0.0.0")
>>
>> ],
>>
>> targets: [
>>
>> .target(name: "AppMaster", dependencies: ["Shared", "CryptoSwift"]),
>>
>> .target(name: "PluginConsumer", dependencies: ["Shared", "CryptoSwift"]),
>>
>> .target(name: "Shared", dependencies: ["ProtoBufMessages",
>> "PerfectHTTPServer", "PerfectWebSockets"]),
>>
>> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>>
>> ]
>>
>> )```
>>
>>
>> App master is separate executable that shares some functionality with
>> PluginConsumer, but it doesn't link against it in any way. I guess it could
>> be omitted, but I wanted to give you whole thing as it is
>>
>> On 7 October 2017 at 18:33, Geordie Jay <geojay at gmail.com> wrote:
>>
>>>
>>> Ján Kosa <lopenka at gmail.com> schrieb am Sa. 7. Okt. 2017 um 15:27:
>>>
>>>> I tried to use @_exported and it helped somewhat. While I still have
>>>> same warnings, size of the PluginInterface library went down by 6mb (to
>>>> 120kb) so it looks like Protobuf is no longer statically linked to it.
>>>> However, size of PluginConsumer executable went up by same 6mb, it looks
>>>> like it is linked there twice now.
>>>>
>>>
>>> To be clear: take protobuf out of the PluginConsumer dependencies.
>>> Actually, I’m not sure which is which, but protobuf should only be listed
>>> as a dependency of one package, where it is imported as @_exported. After
>>> that, your other modules depend on the package that imports / exports
>>> protobuf.
>>>
>>> I also noticed interesting thing. If I build executable using `swift
>>>> build` the size is around 17mb, when I generate xcode project and build it
>>>> using that, size is around 200kb, but I get same warnings using both
>>>> approaches
>>>>
>>>> On 7 October 2017 at 15:44, Geordie Jay <geojay at gmail.com> wrote:
>>>>
>>>>>
>>>>> Ján Kosa <lopenka at gmail.com> schrieb am Sa. 7. Okt. 2017 um 13:34:
>>>>>
>>>>>> I tried swift package clean, but it didn't help
>>>>>>
>>>>>> "Try to ensure the plugin provider module (libA) is (only) being
>>>>>> compiled into its standalone shared library file."
>>>>>> How do I go about this? It is 3rd party module, it doesn't define any
>>>>>> products (https://github.com/apple/swift-protobuf.git). Is there
>>>>>> something I can do in my Package files to make sure it is loaded
>>>>>> dynamically?
>>>>>>
>>>>>
>>>>> When you compile a package depending on protobuf, all the relevant
>>>>> symbols end up in your package’s library file. So here’s something you
>>>>> might try:
>>>>>
>>>>> import protobuf into your own “PluginProvider” module (package), which
>>>>> has a shared library product like this: ‘@_exported import Protobuf’ in
>>>>> some compiled swift file. Then from the other dependent modules “import
>>>>> PluginProvider” - the protobuf symbols should be available, and all from
>>>>> one (nonconflicting) source.
>>>>>
>>>>> Geordie
>>>>>
>>>>>
>>>>>
>>>>>> On 6 October 2017 at 22:52, Geordie Jay <geojay at gmail.com> wrote:
>>>>>>
>>>>>>> I think SwiftPM is (incorrectly) compiling A.XYZ <http://a.xyz/>
>>>>>>> into each of the modules that depend on it, as well as into your intended
>>>>>>> libA.so file.
>>>>>>>
>>>>>>> Try to ensure the plugin provider module (libA) is (only) being
>>>>>>> compiled into its standalone shared library file. Try cleaning the swiftpm
>>>>>>> build for one (swift package clean) and ensure the Package.swift files are
>>>>>>> correctly set up to output the shared library.
>>>>>>>
>>>>>>> Sorry I can’t be more specific, I’ve had these same kinds of issues
>>>>>>> before but I’m not 100% what they were.
>>>>>>>
>>>>>>> Geordie
>>>>>>>
>>>>>>>
>>>>>>> Ján Kosa via swift-users <swift-users at swift.org> schrieb am Fr. 6.
>>>>>>> Okt. 2017 um 14:41:
>>>>>>>
>>>>>>>> It worked! Took me a while to iron out details, but it is working
>>>>>>>> now. Huge thanks sir, I will name my firstborn after you.
>>>>>>>> Thanks for the @_cdecl("initializePlugin") tip as well, I didn't
>>>>>>>> know about it and it will be very useful.
>>>>>>>>
>>>>>>>> I am having slightly related problem now (it was there before, but
>>>>>>>> I ignored it for the time being), not sure if I should start a new thread?
>>>>>>>>
>>>>>>>> The PluginInterface module has one external dependency on module A,
>>>>>>>> PluginConsumer has the dependency on module B which has dependency on same
>>>>>>>> module A that the PluginInterface uses. When I load the plugin library, I
>>>>>>>> get bunch of errors like:
>>>>>>>>
>>>>>>>> Class A.XYZ <http://a.xyz/> is implemented in
>>>>>>>> both libPluginInterface.dylib and libMyPlugin.dylib
>>>>>>>>
>>>>>>>> I know why it is there, but I don't know how to get rid of it. I
>>>>>>>> can't just remove dependency from PluginConsumer and use the one from
>>>>>>>> PluginInterface (if that would even work?) because PluginConsumer does not
>>>>>>>> depend on it directly, but it is going through module B first
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Lope
>>>>>>>>
>>>>>>>> On 4 October 2017 at 22:17, Daniel Dunbar <daniel_dunbar at apple.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The way that I have done this in the past is pass a protocol as an
>>>>>>>>> unsafe pointer to an exposed entry point:
>>>>>>>>> ```swift
>>>>>>>>> let entryPoint = dlsym(handle, “initializePlugin”)
>>>>>>>>> guard entryPoint != nil else {
>>>>>>>>> fatalError("missing plugin entry point:
>>>>>>>>> \(pluginPath)")
>>>>>>>>> }
>>>>>>>>> typealias PluginInitializationFunc = @convention(c)
>>>>>>>>> (UnsafeRawPointer) -> ()
>>>>>>>>> let f = unsafeBitCast(entryPoint, to:
>>>>>>>>> PluginInitializationFunc.self)
>>>>>>>>> f(Unmanaged.passUnretained(self).toOpaque())
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> and then in the plugin convert back to the appropriate type:
>>>>>>>>>
>>>>>>>>> ```
>>>>>>>>> @_cdecl("initializePlugin")
>>>>>>>>> public func initializePlugin(_ ptr: UnsafeRawPointer) {
>>>>>>>>> let manager = Unmanaged<PluginManager>.fromO
>>>>>>>>> paque(ptr).takeUnretainedValue()
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> HTH,
>>>>>>>>> - Daniel
>>>>>>>>>
>>>>>>>>> On Oct 4, 2017, at 11:02 AM, Ján Kosa via swift-users <
>>>>>>>>> swift-users at swift.org> wrote:
>>>>>>>>>
>>>>>>>>> Hello folks,
>>>>>>>>>
>>>>>>>>> I have been toying with dynamic libraries, trying to implement
>>>>>>>>> plugin functionality. I was able to get to the point where I can call
>>>>>>>>> simple function in loaded library, but I am having troubles starting more
>>>>>>>>> sophisticated communication channel.
>>>>>>>>>
>>>>>>>>> There are 3 projects
>>>>>>>>> - PluginConsumer is an app that loads plugin libraries
>>>>>>>>> - MyPlugin is a plugin implementation, output is dynamic library
>>>>>>>>> that PluginConsumer loads
>>>>>>>>> - PluginInterface is common interface that both MyPlugin and
>>>>>>>>> PluginConsumer use, so that they know how to communicate
>>>>>>>>>
>>>>>>>>> My first idea was to have PluginInterface be a simple SPM project
>>>>>>>>> with single file where the bare-bones PluginInterface class would be:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> open class PluginInterface {
>>>>>>>>>
>>>>>>>>> open func sayHi()
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Package.swift file:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> // swift-tools-version:4.0
>>>>>>>>>
>>>>>>>>> import PackageDescription
>>>>>>>>>
>>>>>>>>> let package = Package(
>>>>>>>>>
>>>>>>>>> name: "PluginInterface",
>>>>>>>>>
>>>>>>>>> products: [ .library(name: "PluginInterface", type: .dynamic,
>>>>>>>>> targets: ["PluginInterface"]) ],
>>>>>>>>>
>>>>>>>>> targets: [ .target(name: "PluginInterface") ]
>>>>>>>>>
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> UserPlugin is also very simple project containing only one file:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> public func getPlugin() -> AnyObject {
>>>>>>>>>
>>>>>>>>> return MyPlugin()
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> class MyPlugin: PluginInterface {
>>>>>>>>>
>>>>>>>>> override func sayHi() {
>>>>>>>>>
>>>>>>>>> print("Hi from my plugin")
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Package.swift:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> // swift-tools-version:4.0
>>>>>>>>>
>>>>>>>>> import PackageDescription
>>>>>>>>>
>>>>>>>>> let package = Package(
>>>>>>>>>
>>>>>>>>> name: "MyPlugin",
>>>>>>>>>
>>>>>>>>> products: [ .library(name: "MyPlugin", type: .dynamic,
>>>>>>>>> targets: ["MyPlugin"]) ],
>>>>>>>>>
>>>>>>>>> dependencies: [ .package(url: "url_to_PluginInterface", from:
>>>>>>>>> "0.0.0"), ],
>>>>>>>>>
>>>>>>>>> targets: [
>>>>>>>>>
>>>>>>>>> .target(name: "PluginInterface", dependencies: [
>>>>>>>>> "PluginInterface"]),
>>>>>>>>>
>>>>>>>>> .target(name: "MyPlugin", dependencies: ["PluginInterface"
>>>>>>>>> ]),
>>>>>>>>>
>>>>>>>>> ]
>>>>>>>>>
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The PluginConsumer is bit more complicated, but here is relevant
>>>>>>>>> part (lib loading and function calling):
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> typealias InitFunction = @convention(c) () -> AnyObject
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> let openRes = dlopen(pathToLib, RTLD_NOW|RTLD_LOCAL)
>>>>>>>>>
>>>>>>>>> if openRes != nil {
>>>>>>>>>
>>>>>>>>> defer {
>>>>>>>>>
>>>>>>>>> dlclose(openRes)
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> let symbolName = "mangled_symbol_name"
>>>>>>>>>
>>>>>>>>> let sym = dlsym(openRes, symbolName)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> if sym != nil {
>>>>>>>>>
>>>>>>>>> let f: InitFunction = unsafeBitCast(sym, to: InitFunction.
>>>>>>>>> self)
>>>>>>>>>
>>>>>>>>> let plugin = f() as? PluginInterface
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Package.swift file:
>>>>>>>>>
>>>>>>>>> // swift-tools-version:4.0
>>>>>>>>>
>>>>>>>>> import PackageDescription
>>>>>>>>>
>>>>>>>>> let package = Package(
>>>>>>>>>
>>>>>>>>> name: "PluginConsumer",
>>>>>>>>>
>>>>>>>>> dependencies: [ .package(url: "path_to_plugin_interface",
>>>>>>>>> from: "0.0.0") ],
>>>>>>>>>
>>>>>>>>> targets: [ .target(name: "PluginConsumer", dependencies: [
>>>>>>>>> "PluginConsumer"]) ]
>>>>>>>>>
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This all compiles nicely, MyPlugin project creates dylib file that
>>>>>>>>> executable created by PluginConsumer can load, but the problem is with
>>>>>>>>> following line:
>>>>>>>>>
>>>>>>>>> let plugin = f() as? PluginInterface
>>>>>>>>>
>>>>>>>>> Type of the plugin is MyPlugin, but from the consumer's view, it
>>>>>>>>> doesn't inherit from PluginInterface so I can't call sayHi() method. I
>>>>>>>>> assume this is because there is no relation between PluginInterface class
>>>>>>>>> that compiler uses for MyPlugin project one that it uses for PluginConsumer
>>>>>>>>> project. After library is loaded, they are two completely different classes
>>>>>>>>> that happen to share same name. Is my assumption correct and how do I go
>>>>>>>>> about fixing it?
>>>>>>>>>
>>>>>>>>> I had an idea I could make PluginInterface emit dynamic library
>>>>>>>>> that would be dynamically linked by both MyPlugin and PluginConsumer, thus
>>>>>>>>> making them share same PluginInterface class, but I can't figure out how to
>>>>>>>>> do that (or if it's right way of doing this).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any help appreciated :)
>>>>>>>>>
>>>>>>>>> Lope
>>>>>>>>> _______________________________________________
>>>>>>>>> swift-users mailing list
>>>>>>>>> swift-users at swift.org
>>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> swift-users mailing list
>>>>>>>> swift-users at swift.org
>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> _______________________________________________
>> swift-users mailing list
>> swift-users at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-users
>>
>>
>>
>
> _______________________________________________
> swift-users mailing list
> swift-users at swift.org
> https://lists.swift.org/mailman/listinfo/swift-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-users/attachments/20171008/186068d0/attachment.html>
More information about the swift-users
mailing list