[swift-dev] [swift-build-dev] [RFC] Toolchain based build process

[Daniel Dunbar]

Hi Karl,

On Fri, Jun 10, 2016 at 2:38 AM, Karl via swift-build-dev <
swift-build-dev at swift.org> wrote:

> We can’t separate building for ‘Build’ and building for an arbitrary
> ‘Host’. For those who don’t know, this is how cross-compiling works with
> Swift (this isn’t off-topic, I’m going to bring it back firmly to this
> proposal):
>
> Currently, when you install the compiler, the standard library is in
> (prefix)/lib/swift/(Target)/(arch). On OSX, we cross-compile the standard
> library for the iOS targets, so you’ll see subfolders for ‘macosx’,
> ‘iphoneos’, etc in there. All you need to cross-compile in Swift is the
> standard library; swiftmodules don’t need ‘ar’, so libraries don’t even
> need any binutils to cross-compile, which is refreshing. A cross-linker
> *is* unfortunately still required for executables (come on, LLD!).
>
> Grab the standard library from a linux machine and try:
> echo 'print("hello, world")' | ./swiftc —target=x86-unknown-linux-gnu
> -resource-dir {PATH_TO_LINUX_STDLIB} -emit-module -
>
> So when we’re cross-compiling the swift portions of the standard-library,
> we’re using the swift compiler we just made for Build, telling it to use
> Host’s lib/swift directory, and there it will find the correct subdirectory
> for Target. My native compiler for my OSX Build machine knows not to look
> for the Linux host’s own Target in swift-macosx-x86_64/lib/swift, but in
> swift-linux-armv7/lib/swift.
>
> —
>
> Okay, so that’s out of the way.
>
> There is an actual problem with how we build/configure the target
> libraries (Foundation, XCTest and libdispatch). We currently configure them
> like every other product (for each host), instead of for each target. A
> consequence of this is, for example, that we won’t try to build Foundation
> for cross-compile targets such as Android. What we should do is this:
>
> For each host:
> -> Build cmark/llvm/swift
> -> For each target of this host:
> -> -> Build Foundation/XCTest/libdispatch
> -> Build swiftpm
>
> When you look at it like this, from a tools // target-libraries
> perspective (swiftpm is an exception; it’s a host tool, but it depends on
> Foundation), I came up with a different idea (possibly, I’m not sure), that
> might also help the dependency issue, although I don’t know the specifics
> of it: I would suggest that after building swift, we install it, then build
> the target libraries and install them as we do so.
>
> That would give us a stable standard-library (lib/swift) location which we
> could treat like an installed system. So when Foundation builds (for each
> Target now), it sets -resource-dir to the installed Host copy of swift, and
> the Build swift compiler can find Target’s standard library (yeah, I know).
> It installs itself in there. Then XCTest comes along, it sets the same
> -resource-dir, so now it can find Foundation just with ‘import Foundation’
> and nothing else. In other words: every product after the swift compiler
> can assume its dependencies are like they’ve been installed on the system.
>
> I’m not sure if that’s what you’re referring to with a ‘toolchain-based’
> build process or not. The reason I think it might be is because I don’t
> think this would require big changes. We’d just have to install products
> after swift, then install the target libraries as we go. Foundation could
> even keep on generating build.ninja in-tree, sadly.
>

Yup, this is exactly what I was referring to (although for different
reasons).

 - Daniel


> There isn’t much difference between supporting one cross-host or
> supporting n cross-hosts. I would prefer ’n’, if we could get a nice
> argument syntax for it, because it’s more convenient to integrate with
> other tools, but it’s not a big deal. Actually, I have an idea for a better
> build-script argument syntax which would scale to NxM cross-compiled hosts
> and targets elegantly, but that’s a topic for another day.
>
> Karl
>
>
> On 3 Jun 2016, at 22:31, Daniel Dunbar via swift-dev <swift-dev at swift.org>
> wrote:
>
>
> On Jun 3, 2016, at 1:14 PM, Saleem Abdulrasool <compnerd at compnerd.org>
> wrote:
>
> On Wed, Jun 1, 2016 at 2:18 PM, Daniel Dunbar via swift-dev <
> swift-dev at swift.org> wrote:
>
>> Hi all,
>>
>> The current build process for the overall Swift project (i.e., the
>> compiler + associated projects like Foundation, XCTest, and SwiftPM) relies
>> on each project having dependencies on the built artifacts of previously
>> built projects. Those dependencies are currently communicated to each
>> project in the chain through an ad hoc set of arguments to the individual
>> project's build process. This has been painful to maintain, and makes it
>> hard to reason about the environment that each of the associated projects
>> are building within.
>>
>> Instead, I would like to move towards what I have been calling a
>> "toolchain-based" build process.
>>
>> In this model:
>>
>> 1. The entire build process will be organized as a sequential,
>> incremental construction of a complete toolchain.
>>         - Each individual project will build and copy its products into
>> a staging area.
>> 2. The build script will always create a composed toolchain.
>>         - It will start with an empty toolchain, and merge in the
>> content from each project as it builds.
>>         - This will use rsync & hard-links to avoid needing to stay fast.
>> 3. Each individual project build will just use the composed toolchain to
>> build.
>>         - This will replace the grab bag of options we use to
>> communicate between the projects.
>> 4. At the end of the build, we will have constructed a complete toolchain
>> which can be installed (or used with Xcode).
>>
>> Aside from simplifying the overall build process, this has a couple very
>> nice upsides:
>>
>> 1. At the end of the build, the user has a complete toolchain. They can
>> install it, or use it as they would a distributed snapshot. This is very
>> beneficial for people who are only building the Swift project to get a more
>> recent version of the compiler.
>>
>> 2. Each individual project can be built using the "official" build
>> process with a downloaded snapshot (assuming it is of a new enough
>> version). This is very beneficial for easing contribution to projects like
>> Foundation, XCTest, and SwiftPM which are pure Swift and have fast build
>> times, but which currently build in Swift CI using a very different process
>> from the snapshot-based workflow.
>>
>> Concrete details:
>>
>> 1. I do not plan to change the actual install process in the short term.
>> The actual install process used today relies on the CMake-based install
>> process for some projects (most importantly Swift) and isn't suitable for
>> use in this fashion (where incremental development speed is of high
>> importance). Instead, my plan is to teach the build script itself how to
>> assemble the staging area for each individual project, with the long term
>> goal of using that for the official install process instead of the
>> CMake-based process.
>>
>> 2. My plan is that the build script would only support building one
>> "primary product" (i.e. toolchain). That product may itself be a complete
>> cross compiler toolchain with support for multiple platforms, but the
>> expectation is that users would invoke the build script multiple times if
>> building multiple toolchains. However, to support Canadian Cross [1] build
>> scenarios the build script may need to manage the construction of two
>> products, the primary toolchain and the intermediate (cross compiling)
>> toolchain used to build the artifacts in the primary toolchain.
>>
>> 3. As a concrete example of a problem this solves, SwiftPM currently
>> doesn't test its `swift-test` product in Swift CI, because that product
>> requires using all of the toolchain stack (swift, Foundation, and XCTest),
>> but the product itself is only intended to be used as part of a concrete
>> toolchain. As such, it doesn't know how to operate correctly when given the
>> piecemeal build products for each of those projects, and teaching it to do
>> so would be very cumbersome for us to maintain.
>>
>> Feedback welcome!
>>
>
> Overall, I think that this would be a great change.
>
> I am however slightly confused on why the build script needs to be
> concerned about candian cross at all if it can be taught how to use a
> specified toolchain.
>
> Because these (standard at least as per auto tools) terms can be confusing:
>
> [1] build - the platform where things are being built
> [2] host - the platform where the generated binaries will be run
> [3] target - the platform where the binaries generated by the compiler on
> the host platform will run
>
> It should be possible to shift the burden on the user.
>
>
> Why would we _want_ to shift the burden to the user?
>
> From the user's perspective, they want to end up with a toolchain that
> compiles `Host -> Target`... I see it as an implementation detail (of the
> `build-script`) that producing what the user wants requires us to build the
> intermediate `Build -> Target` toolchain.
>
>   They would be responsible for the first stage of the canadian cross (to
> build a toolchain capable of generating binaries for the host platform).
> Then invoke the build script a second time with the toolchain they just
> built to cross-compile the cross-compiling toolchain.
>
>
> The tricky part of the canadian cross situation isn't the toolchain
> capable of generating binaries for the host, it is the toolchain capable of
> generating binaries for the target, but which needs to run on the build
> host. This is needed when you need to produce runtime libraries that will
> execute on the Target, but you are simultaneously trying to construct a
> cross compiler for a Host which you cannot actually execute. This is when
> you need the extra `Build -> Target` toolchain, and is the situation I was
> referring to.
>
>  - Daniel
>
>
> Or did I completely misunderstand your detail point 2 and this is the
> expectation?
>
>
>
>  - Daniel
>
>
>  - Daniel
>>
>> [1] https://en.wikipedia.org/wiki/Cross_compiler#Canadian_Cross
>>
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-dev
>
>
> --
> Saleem Abdulrasool
> compnerd (at) compnerd (dot) org
>
>
> _______________________________________________
> swift-dev mailing list
> swift-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev
>
>
>
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160610/6aa2efb2/attachment.html>