[swift-dev] Dockerized Swift, Build Times and Other Notes Regarding CI

[swizzlr]

Hi all,

I’ve been working on running Swift applications and building the Swift toolchain in docker containers. I’d like to share some thoughts and notes I’ve made, since they may be of interest to others. Especially to those of you stubbornly clinging to 10.10 on 2013 era Macbook Airs which have all the horsepower of a donkey.

For those of us who are unfamiliar with docker, it’s essentially an alternative to virtual machines, where the host shares its kernel with the container. Consequently the VM overhead is removed entirely, but you have a controlled environment that is versionable, space efficient and full of awesome, basically.

Let’s start with the headline:

Renting resources from Amazon using docker-machine to build Swift.

The magic sauce is:

> docker-machine create --driver amazonec2 --amazonec2-instance-type c4.8xlarge --amazonec2-request-spot-instance --amazonec2-spot-price 1 --amazonec2-root-size 120 --amazonec2-access-key BLANK --amazonec2-zone e --amazonec2-secret-key BLANK --amazonec2-vpc-id BLANK aws


Docker-machine manages the underlying connection between the `docker` command line tool and the daemon running on the host. In this case, we can provision a machine in the cloud with 36 cores, 10gbis fibre and an enormous SSD. It will cost no more than a dollar an hour (amazon spot pricing lets you bid on capacity with a max bid, here set to $1). All our commands will execute there, entirely removing bandwidth, capacity or power issues on our end. (Yes, it’s the 1960s again, thin clients etc).

If you’re willing to spend the aforementioned 50 cents or so, you can achieve the following result (clean build):

> time (./swift/utils/build-script --assertions --no-swift-stdlib-assertions --llbuild --swiftpm --xctest --build-subdir=buildbot_linux --lldb --release --foundation -- --swift-enable-ast-verifier=0 --install-swift --install-lldb --install-llbuild --install-swiftpm --install-xctest --install-prefix=/usr '--swift-install-components=compiler;clang-builtin-headers;stdlib;sdk-overlay;dev' --build-swift-static-stdlib=1 --install-destdir=/ --install-foundation --reconfigure > /dev/null 2>&1 )
> 
> real	8m54.418s
> user	156m25.255s
> sys	9m34.455s

Which is, all things being equal, pretty nice. I’m curious what most others face. Nightly untested builds would be quite doable, and I think using Docker for the Linux builds and development would be of significant interest to many, including whoever at Apple is co-ordinating the CI efforts. Docker is very stable, simple and powerful in its ability to control for many factors when working on such large projects as Swift.

(For those curious, my use case here was that withVarArgs is not yet available in the snapshot from 10 Dec, but it is in master.)

Quickly trying out Swift

In the same manner as IBM’s Swift sandbox (https://developer.ibm.com/swift/2015/12/03/introducing-the-ibm-swift-sandbox/ <https://developer.ibm.com/swift/2015/12/03/introducing-the-ibm-swift-sandbox/>), it’s now trivial to provision something a bit smaller in the cloud or locally and type 

> `docker pull swiftdocker/swift`

With the image downloaded (warning, it’s a couple of gigs, so I’d recommend using a fat pipe or a cloud machine), you can proceed to 

> `docker run --privileged -it swiftdocker/swift swift` 

and immediately get a Swift interpreter! The privileged flag is some kind of weird voodoo to get LLDB to work properly, since it needs to think it has access to the underlying hardware. Any suggestions on patches to fix this issue would be most welcome.

Hacking on Swift

Just as it’s possible to build Swift in a docker container so too is it possible to actively develop the language in a Linux environment, while still benefitting from any local affordances you may have. If there is interest from the core team, I would be happy to contribute a meta-repo of Swift, submoduleing all the repos and supplying a docker-compose file which would enable developers to provision a development machine while being able to mirror local changes over to the machine. Similarly, “preheated” images could be built which would eliminate the need for a fresh build when first starting on the project.


Deploying cloud applications in Swift

With the above infrastructure already in place and growing, deploying an application using docker is as simple as writing your Swift application, and then a Dockerfile describing how to build and start it:

> FROM swiftdocker/swift
> ADD . /code
> WORKDIR /code
> RUN swift build
> CMD .build/Debug/code

The CMD line is the statement executed when the container is started. Many services already exist to support automatically running and scaling dockerized infrastructure, and it’s my hope that having Swift running in Docker will contribute to adoption by a wider community of developers.

Testing Swift applications on CI as a service providers

Whether or not you choose to deploy an application in Docker, services like Travis CI use docker extensively to automate their test machine provisioning infrastructure. If your Swift library or application can build on Linux (command line tools are an obvious target for the latter), you would be able to take advantage of a Docker environment to quickly achieve low build times, as opposed to the generally expensive and slower OS X machines that are optimised for Xcode and tend to lack the open source toolchain.

—

I hope some of you may find this information useful. I’ve posted it on dev since I believe it’s more relevant to the developers than the users, right now. Please let me know if you have any questions about Docker or how it might relate to Swift!

Oh, and Merry Christmas/have a great winter holiday!

Tom


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20151218/756550dc/attachment.html>