[swift-dev] [RFC] Toolchain based build process

[Saleem Abdulrasool]

On Wed, Jun 1, 2016 at 2:18 PM, Daniel Dunbar via swift-dev <
swift-dev at swift.org> wrote:

> Hi all,
>
> The current build process for the overall Swift project (i.e., the
> compiler + associated projects like Foundation, XCTest, and SwiftPM) relies
> on each project having dependencies on the built artifacts of previously
> built projects. Those dependencies are currently communicated to each
> project in the chain through an ad hoc set of arguments to the individual
> project's build process. This has been painful to maintain, and makes it
> hard to reason about the environment that each of the associated projects
> are building within.
>
> Instead, I would like to move towards what I have been calling a
> "toolchain-based" build process.
>
> In this model:
>
> 1. The entire build process will be organized as a sequential, incremental
> construction of a complete toolchain.
>         - Each individual project will build and copy its products into a
> staging area.
> 2. The build script will always create a composed toolchain.
>         - It will start with an empty toolchain, and merge in the content
> from each project as it builds.
>         - This will use rsync & hard-links to avoid needing to stay fast.
> 3. Each individual project build will just use the composed toolchain to
> build.
>         - This will replace the grab bag of options we use to communicate
> between the projects.
> 4. At the end of the build, we will have constructed a complete toolchain
> which can be installed (or used with Xcode).
>
> Aside from simplifying the overall build process, this has a couple very
> nice upsides:
>
> 1. At the end of the build, the user has a complete toolchain. They can
> install it, or use it as they would a distributed snapshot. This is very
> beneficial for people who are only building the Swift project to get a more
> recent version of the compiler.
>
> 2. Each individual project can be built using the "official" build process
> with a downloaded snapshot (assuming it is of a new enough version). This
> is very beneficial for easing contribution to projects like Foundation,
> XCTest, and SwiftPM which are pure Swift and have fast build times, but
> which currently build in Swift CI using a very different process from the
> snapshot-based workflow.
>
> Concrete details:
>
> 1. I do not plan to change the actual install process in the short term.
> The actual install process used today relies on the CMake-based install
> process for some projects (most importantly Swift) and isn't suitable for
> use in this fashion (where incremental development speed is of high
> importance). Instead, my plan is to teach the build script itself how to
> assemble the staging area for each individual project, with the long term
> goal of using that for the official install process instead of the
> CMake-based process.
>
> 2. My plan is that the build script would only support building one
> "primary product" (i.e. toolchain). That product may itself be a complete
> cross compiler toolchain with support for multiple platforms, but the
> expectation is that users would invoke the build script multiple times if
> building multiple toolchains. However, to support Canadian Cross [1] build
> scenarios the build script may need to manage the construction of two
> products, the primary toolchain and the intermediate (cross compiling)
> toolchain used to build the artifacts in the primary toolchain.
>
> 3. As a concrete example of a problem this solves, SwiftPM currently
> doesn't test its `swift-test` product in Swift CI, because that product
> requires using all of the toolchain stack (swift, Foundation, and XCTest),
> but the product itself is only intended to be used as part of a concrete
> toolchain. As such, it doesn't know how to operate correctly when given the
> piecemeal build products for each of those projects, and teaching it to do
> so would be very cumbersome for us to maintain.
>
> Feedback welcome!
>

Overall, I think that this would be a great change.

I am however slightly confused on why the build script needs to be
concerned about candian cross at all if it can be taught how to use a
specified toolchain.

Because these (standard at least as per auto tools) terms can be confusing:

[1] build - the platform where things are being built
[2] host - the platform where the generated binaries will be run
[3] target - the platform where the binaries generated by the compiler on
the host platform will run

It should be possible to shift the burden on the user.  They would be
responsible for the first stage of the canadian cross (to build a toolchain
capable of generating binaries for the host platform).  Then invoke the
build script a second time with the toolchain they just built to
cross-compile the cross-compiling toolchain.

Or did I completely misunderstand your detail point 2 and this is the
expectation?

 - Daniel
>
> [1] https://en.wikipedia.org/wiki/Cross_compiler#Canadian_Cross
>
> _______________________________________________
> swift-dev mailing list
> swift-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev


-- 
Saleem Abdulrasool
compnerd (at) compnerd (dot) org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160603/e0ef6e85/attachment.html>