[swift-users] [swift-build-dev] Importing C system libraries

Ankit Aggarwal ankit_aggarwal at apple.com
Wed Mar 29 14:07:20 CDT 2017 

Yep, we need to have better support for system packages. Unfortunately it is not covered in the Swift 4 Package Manager roadmap <https://lists.swift.org/pipermail/swift-evolution-announce/2017-January/000307.html>, but community driven proposal are always welcomed!

> On 30-Mar-2017, at 12:25 AM, Kelvin Ma <kelvinsthirteen at gmail.com> wrote:
> 
> I agree that portability is valuable and that it’s something that’s lacking in the c/makefile workflow. I just don’t think empty git repositories are the right solution. Perhaps we can get the best of both worlds with something like this in the SwiftPM:
> 
> 
>     .Package(include: "cairo.h", link: "cairo" version: "1.1.4", remote: "git://anongit.freedesktop.org/git/cairo <http://anongit.freedesktop.org/git/cairo>")
> 
> It would search for the cairo lib on local machine in /usr/lib and /usr/local/lib , and if it can’t find it, or if the version doesn’t match, it will then download the C sources from the Cairo project’s *official* git repository and build it for you. Options could be added to specify alternative search paths or user prompts.
> 
> On Wed, Mar 29, 2017 at 12:42 PM, Ankit Aggarwal <ankit_aggarwal at apple.com <mailto:ankit_aggarwal at apple.com>> wrote:
> I agree that libressl isn't a good example because of the security questions it raises, but libYAML is a good self contained package IMO. With custom targets layout proposal, the forks can be replaced by submodules. The problem with apt-get approach is, it takes away the "portability" from the package and it doesn't work in cases when you don't have full access to the system. That said, we do support this approach upto a certain degree. For e.g., swiftpm allows system package authors to declare hints for installing a system package dependency (via brew or apt-get). The /usr/include, /usr/lib etc are searched by default and non-standard paths are supported using pkg config. See: https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#require-system-libraries <https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#require-system-libraries>
> 
>> On 29-Mar-2017, at 10:55 PM, Kelvin Ma <kelvinsthirteen at gmail.com <mailto:kelvinsthirteen at gmail.com>> wrote:
>> 
>> I don’t think this is a good approach, the libressl <https://github.com/vapor/clibressl> repo is pretty much just the source code of the C library copied and pasted into a Swift module. That’s not a good thing™. The linux build paradigm is, the library maintains its own *official* source repository, the OS package distributors build it, users install the built dependencies with `sudo apt-get install libwhatever-dev` and the project source just builds and links to the library managed by the system.
>> 
>> Here you have to keep the forked library source up to date with the real library source, download it from the internet and build the entire project plus the libraries of which there could be many.
>> 
>> IMO the ideal way to import system libs would be for the user to install the dependency with `apt`, just as you would for any C program, have the `include` statements within the project source (like you would for any C program), and then have the paths to `/usr/include` and `/usr/lib` and `/usr/local/include` etc in a Makefile (or Package.swift). Usually it’s only “specialty” libraries like libspiro that people download and build manually, and even then, it’s downloaded from the Spiro project’s own official repository, not a third party fork. That’s the “accepted” way to do things, that linux ecosystems are designed around. Of course, this is very similar to the modulemap system that currently works in Swift. I just wish modulemaps could be streamlined a little, maybe combined with the top level Package.swift.
>> 
>> On Wed, Mar 29, 2017 at 1:03 AM, Ankit Aggarwal <ankit_aggarwal at apple.com <mailto:ankit_aggarwal at apple.com>> wrote:
>> 
>>> On 29-Mar-2017, at 11:22 AM, kelvinsthirteen at gmail.com <mailto:kelvinsthirteen at gmail.com> wrote:
>>> 
>>> I figured that was the intention, but we can’t be too surprised that everyone is maintaining personal modulemap repositories (and polluting search results — just try googling for a Swift PNG library!), especially when this central repo still doesn’t exist yet.
>> 
>> Yeah thats unfortunate, maybe this will improve once we have an index.
>> 
>>> If Swift ever comes on par with C in terms of being usage and the lingua franca of the FOSS world, I can see linux distributions shipping standard modulemaps the same way they ship C headers, but you have to admit this is years (decades?) away at best.
>>> 
>>> On the flip side of it, it does wonders in terms of motivating people (me at least) to start writing pure Swift replacements for some of these C libraries (like libpng)…
>> 
>> I think its better to reuse existing libraries than writing from scratch (unless really needed). A good approach that works is "porting" these libraries to build with SwiftPM, see: libYAML <https://github.com/jpsim/Yams>, libressl <https://github.com/vapor/clibressl>. However, this porting can be difficult to do right but it should become much easier once we have build settings support and custom targets layout <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170320/034469.html>.
>> 
>>> 
>>> On Mar 29, 2017, at 12:42 AM, Ankit Aggarwal <ankit_aggarwal at apple.com <mailto:ankit_aggarwal at apple.com>> wrote:
>>> 
>>>> I think the idea was that there will be one such repository which other packages can use, that too only until the system libraries start shipping their standard modulemap. I thought we had this written down in our documentation somewhere but couldn't find it.
>>>> 
>>>> Maybe Daniel can expand on this.
>>>> 
>>>> On Wed, Mar 29, 2017 at 10:48 AM, Kelvin Ma via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>>> This worked! Thanks! But why is having empty git repositories strewn about the “correct” way? System libraries should be imported from within the project, as they are in C. You have to admit it’s getting quite silly that Swift devs keep repositories like these <https://github.com/kelvin13/swift-zlib> on our github accounts. That zlib repository contains exactly ten lines of code. I used to have 6 or 7 repos like that one up there before I got rid of them and switched to local repos. 
>>>> 
>>>> On Wed, Mar 29, 2017 at 12:03 AM, Ankit Aggarwal <ankit_aggarwal at apple.com <mailto:ankit_aggarwal at apple.com>> wrote:
>>>> In this case, these are just umbrella headers. If your modulemap contains absolute path to the header, then you don't need the header files, but SwiftPM will probably warn about this. Note that this is a "hack" to have system packages inside a single repository. The correct way is to have system package as a separate published package which you only need to do once.
>>>> 
>>>>> On 29-Mar-2017, at 10:26 AM, Kelvin Ma <kelvinsthirteen at gmail.com <mailto:kelvinsthirteen at gmail.com>> wrote:
>>>>> 
>>>>> I will try this, but why are the header files inside the Sources directory? System headers should live in /usr/include…
>>>>> 
>>>>> On Tue, Mar 28, 2017 at 11:48 PM, Ankit Aggarwal <ankit_aggarwal at apple.com <mailto:ankit_aggarwal at apple.com>> wrote:
>>>>> Hi,
>>>>> 
>>>>> Apologies for not replying to this earlier. 
>>>>> 
>>>>> You can have multiple targets in a single package. Each target can either be Swift or C-family. The type of target is determined by the sources contained in it (*.c/*.cpp etc means C target, *.swift means Swift target). So if you want to create multiple C targets, this layout should work:
>>>>> 
>>>>> Package.swift
>>>>> Sources/
>>>>>     Bitmap
>>>>>     Cubify
>>>>>     Cairo/anchor.c <---- This is just an empty file to tell SwiftPM that this is a C target.
>>>>>     Cairo/include/Cairo.h
>>>>>     Cairo/include/module.modulemap
>>>>>     GLFW/anchor.c
>>>>>     GLFW/include/GLFW.h
>>>>>     GLFW/include/module.modulemap
>>>>> 
>>>>> The modulemap is automatically generated, if not provided. This is a package which contains two targets (one C and one Swift): https://github.com/jpsim/Yams <https://github.com/jpsim/Yams>
>>>>> 
>>>>> If you need to pass a bunch of compiler flags, you can use SwiftPM's pkgConfig feature but that will require you to have a separate repository for Cario and GLFW. You can experiment without creating tags using the edit feature <https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#editable-packages>.
>>>>> 
>>>>> PS: You can join SwiftPM slack channel for quicker turn around time: https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html>
>>>>> 
>>>>> Thanks,
>>>>> Ankit
>>>>> 
>>>>> 
>>>>> On Wed, Mar 29, 2017 at 6:06 AM, Michael Ilseman via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>>>>> This is into uncharted territory for me, but it seems you’re building with SwiftPM. You’ll probably want to configure extra compiler flags if that’s possible. You could also bite the bullet and build your C libraries with SwiftPM as well. Hopefully someone on swift-build-dev can help you out.
>>>>> 
>>>>> CC-ing Ankit
>>>>> 
>>>>> 
>>>>>> On Mar 28, 2017, at 5:09 PM, Kelvin Ma <kelvinsthirteen at gmail.com <mailto:kelvinsthirteen at gmail.com>> wrote:
>>>>>> 
>>>>>> How do I compile a project with many modules? My tree looks like this:
>>>>>> 
>>>>>> <Selection_001.png>
>>>>>> ​
>>>>>> 
>>>>>> On Tue, Mar 28, 2017 at 12:47 PM, Michael Ilseman <milseman at apple.com <mailto:milseman at apple.com>> wrote:
>>>>>> Sure! In this example, I have built libgit2. I have a directory called Git, and inside that I have the following module map:
>>>>>> 
>>>>>> module Git [system] {
>>>>>>        header "<my path>/libgit2/include/git2.h"
>>>>>>        export *
>>>>>> }
>>>>>> 
>>>>>> When I run, I use:
>>>>>> 
>>>>>> swift -I <path-to-“Git”-directory> -L <path-to-built-libgit2> -lgit2 foo.swift
>>>>>> 
>>>>>> inside foo.swift I can:
>>>>>> 
>>>>>> import Git
>>>>>> // … use libGit2
>>>>>> 
>>>>>> 
>>>>>> Read more about how to write a more appropriate module.map file for your purposes at https://clang.llvm.org/docs/Modules.html <https://clang.llvm.org/docs/Modules.html>. For example, you might be able to define link flags inside the module.map, use umbrella directories, submodules, etc.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Mar 28, 2017, at 6:27 AM, Kelvin Ma <kelvinsthirteen at gmail.com <mailto:kelvinsthirteen at gmail.com>> wrote:
>>>>>>> 
>>>>>>> Can you give an example?
>>>>>>> 
>>>>>>> On Mon, Mar 27, 2017 at 3:59 PM, Michael Ilseman <milseman at apple.com <mailto:milseman at apple.com>> wrote:
>>>>>>> Sure. At a low level, you can create a module.map file and use -L/-l flags in your invocation of Swift. If you want to do so at a higher level, then perhaps SwiftPM can. CCing swift-build-dev for the SwiftPM part.
>>>>>>> 
>>>>>>> 
>>>>>>> > On Mar 26, 2017, at 3:20 PM, Kelvin Ma via swift-users <swift-users at swift.org <mailto:swift-users at swift.org>> wrote:
>>>>>>> >
>>>>>>> > Idk if this has been asked before, but is there a way to import C libraries into a Swift project without creating a local git repo? Preferably something similar to C where you can just `#include` headers and then specify the link flags (in Package.swift?)
>>>>>>> >
>>>>>>> > It’s getting very cumbersome to make a bunch of empty git repos just to use libglfw or libcairo.
>>>>>>> > _______________________________________________
>>>>>>> > swift-users mailing list
>>>>>>> > swift-users at swift.org <mailto:swift-users at swift.org>
>>>>>>> > https://lists.swift.org/mailman/listinfo/swift-users <https://lists.swift.org/mailman/listinfo/swift-users>
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> swift-build-dev mailing list
>>>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> swift-build-dev mailing list
>>>> swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>
>>>> https://lists.swift.org/mailman/listinfo/swift-build-dev <https://lists.swift.org/mailman/listinfo/swift-build-dev>
>>>> 
>>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-users/attachments/20170330/16ee4f8d/attachment.html>

More information about the swift-users mailing list