[swift-users] Communicating with dynamically loaded swift library
Ján Kosa
lopenka at gmail.com
Sun Oct 8 00:51:17 CDT 2017
I was afraid it will come to that :) I will try to make something tonight,
either it will help you fix it, or I will find out what I did wrong
On 8 October 2017 at 09:49, Daniel Dunbar <daniel_dunbar at apple.com> wrote:
> Is it possible for you to make a small test package that shows the
> problem, and file a bug in bugs.swift.org? There may be something we need
> to fix in SwiftPM before this can work (because of our linking model).
>
> - Daniel
>
>
> On Oct 7, 2017, at 10:42 PM, Ján Kosa via swift-users <
> swift-users at swift.org> wrote:
>
> That is exactly what I did. The only package that depends on the protobuf
> is the PluginInterface. Both MyPlugin and and PluginConsumer depend on the
> PluginInterface and not on the protobuf itself. I had to shuffle around my
> dependencies a bit, which resulted in smaller number of dependencies but
> they don't make much sense now (as in, some target had to depend on
> PluginInterface even if they don't need to, just to get access to
> protobuf). I could live with that if it solved the issue, but it didn't.
>
> I am adding my Package.swift files in case I missed something:
>
> PluginInterface:
>
> ```swift
> let package = Package(
>
> name: "PluginInterface",
>
> products: [ .library(name: "PluginInterface", type: .dynamic, targets: [
> "PluginInterface"]) ],
>
> dependencies: [ .package(url: "https://github.com/apple/swift-protobuf.git
> ", from: "0.0.0") ],
>
> targets: [ .target(name: "PluginInterface", dependencies: ["SwiftProtobuf"])
> ]
>
> )```
>
>
> MyPlugin:
>
> ```swift
>
> let package = Package(
>
> name: "MyPlugin",
>
> products: [ .library(name: "MyPlugin", type: .dynamic, targets: [
> "PluginImpl"]) ],
>
> dependencies: [
>
> .package(url: "path/to/PluginInterface.git", from: "0.0.0"),
>
> ],
>
> targets: [
>
> .target(name: "PluginImpl", dependencies: ["ProtoBufMessages"]),
>
> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>
> ]
>
> )```
>
>
> PluginConsumer:
>
> ```swift
>
> let package = Package(
>
> name: "PluginConsumer",
>
> dependencies: [
>
> .package(url: "https://github.com/PerfectlySoft/Perfect-WebSockets.git",
> from: "3.0.0"),
>
> .package(url: "https://github.com/PerfectlySoft/Perfect-HTTPServer.git",
> from: "3.0.0"),
>
> .package(url: "path/to/PluginInterface", from: "0.0.0"),
>
> .package(url: "https://github.com/krzyzanowskim/CryptoSwift.git", from:
> "0.0.0")
>
> ],
>
> targets: [
>
> .target(name: "AppMaster", dependencies: ["Shared", "CryptoSwift"]),
>
> .target(name: "PluginConsumer", dependencies: ["Shared", "CryptoSwift"]),
>
> .target(name: "Shared", dependencies: ["ProtoBufMessages",
> "PerfectHTTPServer", "PerfectWebSockets"]),
>
> .target(name: "ProtoBufMessages", dependencies: ["PluginInterface"])
>
> ]
>
> )```
>
>
> App master is separate executable that shares some functionality with
> PluginConsumer, but it doesn't link against it in any way. I guess it could
> be omitted, but I wanted to give you whole thing as it is
>
> On 7 October 2017 at 18:33, Geordie Jay <geojay at gmail.com> wrote:
>
>>
>> Ján Kosa <lopenka at gmail.com> schrieb am Sa. 7. Okt. 2017 um 15:27:
>>
>>> I tried to use @_exported and it helped somewhat. While I still have
>>> same warnings, size of the PluginInterface library went down by 6mb (to
>>> 120kb) so it looks like Protobuf is no longer statically linked to it.
>>> However, size of PluginConsumer executable went up by same 6mb, it looks
>>> like it is linked there twice now.
>>>
>>
>> To be clear: take protobuf out of the PluginConsumer dependencies.
>> Actually, I’m not sure which is which, but protobuf should only be listed
>> as a dependency of one package, where it is imported as @_exported. After
>> that, your other modules depend on the package that imports / exports
>> protobuf.
>>
>> I also noticed interesting thing. If I build executable using `swift
>>> build` the size is around 17mb, when I generate xcode project and build it
>>> using that, size is around 200kb, but I get same warnings using both
>>> approaches
>>>
>>> On 7 October 2017 at 15:44, Geordie Jay <geojay at gmail.com> wrote:
>>>
>>>>
>>>> Ján Kosa <lopenka at gmail.com> schrieb am Sa. 7. Okt. 2017 um 13:34:
>>>>
>>>>> I tried swift package clean, but it didn't help
>>>>>
>>>>> "Try to ensure the plugin provider module (libA) is (only) being
>>>>> compiled into its standalone shared library file."
>>>>> How do I go about this? It is 3rd party module, it doesn't define any
>>>>> products (https://github.com/apple/swift-protobuf.git). Is there
>>>>> something I can do in my Package files to make sure it is loaded
>>>>> dynamically?
>>>>>
>>>>
>>>> When you compile a package depending on protobuf, all the relevant
>>>> symbols end up in your package’s library file. So here’s something you
>>>> might try:
>>>>
>>>> import protobuf into your own “PluginProvider” module (package), which
>>>> has a shared library product like this: ‘@_exported import Protobuf’ in
>>>> some compiled swift file. Then from the other dependent modules “import
>>>> PluginProvider” - the protobuf symbols should be available, and all from
>>>> one (nonconflicting) source.
>>>>
>>>> Geordie
>>>>
>>>>
>>>>
>>>>> On 6 October 2017 at 22:52, Geordie Jay <geojay at gmail.com> wrote:
>>>>>
>>>>>> I think SwiftPM is (incorrectly) compiling A.XYZ <http://a.xyz/>
>>>>>> into each of the modules that depend on it, as well as into your intended
>>>>>> libA.so file.
>>>>>>
>>>>>> Try to ensure the plugin provider module (libA) is (only) being
>>>>>> compiled into its standalone shared library file. Try cleaning the swiftpm
>>>>>> build for one (swift package clean) and ensure the Package.swift files are
>>>>>> correctly set up to output the shared library.
>>>>>>
>>>>>> Sorry I can’t be more specific, I’ve had these same kinds of issues
>>>>>> before but I’m not 100% what they were.
>>>>>>
>>>>>> Geordie
>>>>>>
>>>>>>
>>>>>> Ján Kosa via swift-users <swift-users at swift.org> schrieb am Fr. 6.
>>>>>> Okt. 2017 um 14:41:
>>>>>>
>>>>>>> It worked! Took me a while to iron out details, but it is working
>>>>>>> now. Huge thanks sir, I will name my firstborn after you.
>>>>>>> Thanks for the @_cdecl("initializePlugin") tip as well, I didn't
>>>>>>> know about it and it will be very useful.
>>>>>>>
>>>>>>> I am having slightly related problem now (it was there before, but I
>>>>>>> ignored it for the time being), not sure if I should start a new thread?
>>>>>>>
>>>>>>> The PluginInterface module has one external dependency on module A,
>>>>>>> PluginConsumer has the dependency on module B which has dependency on same
>>>>>>> module A that the PluginInterface uses. When I load the plugin library, I
>>>>>>> get bunch of errors like:
>>>>>>>
>>>>>>> Class A.XYZ <http://a.xyz/> is implemented in
>>>>>>> both libPluginInterface.dylib and libMyPlugin.dylib
>>>>>>>
>>>>>>> I know why it is there, but I don't know how to get rid of it. I
>>>>>>> can't just remove dependency from PluginConsumer and use the one from
>>>>>>> PluginInterface (if that would even work?) because PluginConsumer does not
>>>>>>> depend on it directly, but it is going through module B first
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Lope
>>>>>>>
>>>>>>> On 4 October 2017 at 22:17, Daniel Dunbar <daniel_dunbar at apple.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The way that I have done this in the past is pass a protocol as an
>>>>>>>> unsafe pointer to an exposed entry point:
>>>>>>>> ```swift
>>>>>>>> let entryPoint = dlsym(handle, “initializePlugin”)
>>>>>>>> guard entryPoint != nil else {
>>>>>>>> fatalError("missing plugin entry point:
>>>>>>>> \(pluginPath)")
>>>>>>>> }
>>>>>>>> typealias PluginInitializationFunc = @convention(c)
>>>>>>>> (UnsafeRawPointer) -> ()
>>>>>>>> let f = unsafeBitCast(entryPoint, to:
>>>>>>>> PluginInitializationFunc.self)
>>>>>>>> f(Unmanaged.passUnretained(self).toOpaque())
>>>>>>>> ```
>>>>>>>>
>>>>>>>> and then in the plugin convert back to the appropriate type:
>>>>>>>>
>>>>>>>> ```
>>>>>>>> @_cdecl("initializePlugin")
>>>>>>>> public func initializePlugin(_ ptr: UnsafeRawPointer) {
>>>>>>>> let manager = Unmanaged<PluginManager>.fromO
>>>>>>>> paque(ptr).takeUnretainedValue()
>>>>>>>> ```
>>>>>>>>
>>>>>>>> HTH,
>>>>>>>> - Daniel
>>>>>>>>
>>>>>>>> On Oct 4, 2017, at 11:02 AM, Ján Kosa via swift-users <
>>>>>>>> swift-users at swift.org> wrote:
>>>>>>>>
>>>>>>>> Hello folks,
>>>>>>>>
>>>>>>>> I have been toying with dynamic libraries, trying to implement
>>>>>>>> plugin functionality. I was able to get to the point where I can call
>>>>>>>> simple function in loaded library, but I am having troubles starting more
>>>>>>>> sophisticated communication channel.
>>>>>>>>
>>>>>>>> There are 3 projects
>>>>>>>> - PluginConsumer is an app that loads plugin libraries
>>>>>>>> - MyPlugin is a plugin implementation, output is dynamic library
>>>>>>>> that PluginConsumer loads
>>>>>>>> - PluginInterface is common interface that both MyPlugin and
>>>>>>>> PluginConsumer use, so that they know how to communicate
>>>>>>>>
>>>>>>>> My first idea was to have PluginInterface be a simple SPM project
>>>>>>>> with single file where the bare-bones PluginInterface class would be:
>>>>>>>>
>>>>>>>>
>>>>>>>> open class PluginInterface {
>>>>>>>>
>>>>>>>> open func sayHi()
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Package.swift file:
>>>>>>>>
>>>>>>>>
>>>>>>>> // swift-tools-version:4.0
>>>>>>>>
>>>>>>>> import PackageDescription
>>>>>>>>
>>>>>>>> let package = Package(
>>>>>>>>
>>>>>>>> name: "PluginInterface",
>>>>>>>>
>>>>>>>> products: [ .library(name: "PluginInterface", type: .dynamic,
>>>>>>>> targets: ["PluginInterface"]) ],
>>>>>>>>
>>>>>>>> targets: [ .target(name: "PluginInterface") ]
>>>>>>>>
>>>>>>>> )
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> UserPlugin is also very simple project containing only one file:
>>>>>>>>
>>>>>>>>
>>>>>>>> public func getPlugin() -> AnyObject {
>>>>>>>>
>>>>>>>> return MyPlugin()
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> class MyPlugin: PluginInterface {
>>>>>>>>
>>>>>>>> override func sayHi() {
>>>>>>>>
>>>>>>>> print("Hi from my plugin")
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> Package.swift:
>>>>>>>>
>>>>>>>>
>>>>>>>> // swift-tools-version:4.0
>>>>>>>>
>>>>>>>> import PackageDescription
>>>>>>>>
>>>>>>>> let package = Package(
>>>>>>>>
>>>>>>>> name: "MyPlugin",
>>>>>>>>
>>>>>>>> products: [ .library(name: "MyPlugin", type: .dynamic,
>>>>>>>> targets: ["MyPlugin"]) ],
>>>>>>>>
>>>>>>>> dependencies: [ .package(url: "url_to_PluginInterface", from:
>>>>>>>> "0.0.0"), ],
>>>>>>>>
>>>>>>>> targets: [
>>>>>>>>
>>>>>>>> .target(name: "PluginInterface", dependencies: [
>>>>>>>> "PluginInterface"]),
>>>>>>>>
>>>>>>>> .target(name: "MyPlugin", dependencies: ["PluginInterface"
>>>>>>>> ]),
>>>>>>>>
>>>>>>>> ]
>>>>>>>>
>>>>>>>> )
>>>>>>>>
>>>>>>>>
>>>>>>>> The PluginConsumer is bit more complicated, but here is relevant
>>>>>>>> part (lib loading and function calling):
>>>>>>>>
>>>>>>>>
>>>>>>>> typealias InitFunction = @convention(c) () -> AnyObject
>>>>>>>>
>>>>>>>>
>>>>>>>> let openRes = dlopen(pathToLib, RTLD_NOW|RTLD_LOCAL)
>>>>>>>>
>>>>>>>> if openRes != nil {
>>>>>>>>
>>>>>>>> defer {
>>>>>>>>
>>>>>>>> dlclose(openRes)
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> let symbolName = "mangled_symbol_name"
>>>>>>>>
>>>>>>>> let sym = dlsym(openRes, symbolName)
>>>>>>>>
>>>>>>>>
>>>>>>>> if sym != nil {
>>>>>>>>
>>>>>>>> let f: InitFunction = unsafeBitCast(sym, to: InitFunction.
>>>>>>>> self)
>>>>>>>>
>>>>>>>> let plugin = f() as? PluginInterface
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> Package.swift file:
>>>>>>>>
>>>>>>>> // swift-tools-version:4.0
>>>>>>>>
>>>>>>>> import PackageDescription
>>>>>>>>
>>>>>>>> let package = Package(
>>>>>>>>
>>>>>>>> name: "PluginConsumer",
>>>>>>>>
>>>>>>>> dependencies: [ .package(url: "path_to_plugin_interface",
>>>>>>>> from: "0.0.0") ],
>>>>>>>>
>>>>>>>> targets: [ .target(name: "PluginConsumer", dependencies: [
>>>>>>>> "PluginConsumer"]) ]
>>>>>>>>
>>>>>>>> )
>>>>>>>>
>>>>>>>>
>>>>>>>> This all compiles nicely, MyPlugin project creates dylib file that
>>>>>>>> executable created by PluginConsumer can load, but the problem is with
>>>>>>>> following line:
>>>>>>>>
>>>>>>>> let plugin = f() as? PluginInterface
>>>>>>>>
>>>>>>>> Type of the plugin is MyPlugin, but from the consumer's view, it
>>>>>>>> doesn't inherit from PluginInterface so I can't call sayHi() method. I
>>>>>>>> assume this is because there is no relation between PluginInterface class
>>>>>>>> that compiler uses for MyPlugin project one that it uses for PluginConsumer
>>>>>>>> project. After library is loaded, they are two completely different classes
>>>>>>>> that happen to share same name. Is my assumption correct and how do I go
>>>>>>>> about fixing it?
>>>>>>>>
>>>>>>>> I had an idea I could make PluginInterface emit dynamic library
>>>>>>>> that would be dynamically linked by both MyPlugin and PluginConsumer, thus
>>>>>>>> making them share same PluginInterface class, but I can't figure out how to
>>>>>>>> do that (or if it's right way of doing this).
>>>>>>>>
>>>>>>>>
>>>>>>>> Any help appreciated :)
>>>>>>>>
>>>>>>>> Lope
>>>>>>>> _______________________________________________
>>>>>>>> swift-users mailing list
>>>>>>>> swift-users at swift.org
>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> swift-users mailing list
>>>>>>> swift-users at swift.org
>>>>>>> https://lists.swift.org/mailman/listinfo/swift-users
>>>>>>>
>>>>>>
>>>>>
>>>
> _______________________________________________
> swift-users mailing list
> swift-users at swift.org
> https://lists.swift.org/mailman/listinfo/swift-users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-users/attachments/20171008/702e02aa/attachment.html>
More information about the swift-users
mailing list