[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Rick Ballard]

> On Jan 6, 2016, at 11:52 AM, Max Howell <max.howell at apple.com> wrote:
>>>> 
>>>>>> Another real problem with this type of design, is that is makes more advanced build systems very complicated to make. For example, distributed builds that bring together all of the compiled bits to run tests on get blocked because the build command starts to expect certain intermediate output. This is a real problem my previous team still has to this day with xcodebuild and actively prevents us from doing this exact thing with standard tools from Apple, so we have to basically roll our own. I see this design following in the exact same footsteps.
>>>>> 
>>>>> I’m not sure I understand.
>>>> 
>>>> Every time you run “swift build” it’s going to trigger a build. This is should never be required to run tests. Even if you say, “we’ll only re-build on incremental builds”, you are assuming that you are building and running tests within the same environment and that the environment is a build-capable environment. This is not a valid assumption. This is what xcodebuild test does today, it tries to smartly only trigger builds when necessary, but since we build on one set of boxes and run tests on other machines, this simply doesn’t work.
>>>> 
>>>> We have the opportunity to build a proper toolset that can be correctly used within many different workflows. However, I already see a lot of the coupling starting to happen today that has happened within xcodebuild already, and that really concerns me. Basically, it means that these tools aren’t going to be usable outside of the smaller-scale apps, and teams like mine are going to have to continue investing in more robust and flexible tools to enable our scenarios.
>>> 
>>> Well, it is not too late to uncouple these two pieces. Can you provide justification as to why we should decouple them? The engineering is not terrible to decouple them, but it would make things more complicated. Currently a good deal of the knowledge about where built products are and how they relate to targets does not need to be communicated outside of a single process, and simpler systems have less bugs, so I’d prefer not to complicate things without good rationale.
>> 
>> I don’t know what other examples to really provide here… the ability to build is completely orthogonal to the ability to test. Given a set of binaries, I should be able to run the tests. The tests are not always statically linked to the testing target. Often times they are dynamically injected or used to drive the program externally. 
>> 
>> Maybe this provides some better context: some of our apps take multiple hours to perform full builds. These apps are tested across multiple OS installations, hardware configurations, and locales. The apps are not built on all of these target machines. All of these machines should be able to run the same tests available on a developer’s machine, and there is no fundamental reason why they cannot be run in the same way (this is historically been the case with Xcode projects because of arbitrary decisions and unnecessary coupling).
>> 
>> With this proposal, how do I run these tests without triggering a build? Is “swift build --test” only going to run the tests without attempting to build first? That is not how I read the proposal.
> 
> A fast null-build negates this. But if it turns out our null-builds are not fast enough for large projects we will happily accept a patch to allow you to skip the build and just run the tests.
> 
> Notably: currently the tests compile to executables, so once they are built you can run them independently. I intend to keep this design, if possible.
> 
>> I spent a good deal of my time when I was on the Office for Mac team at Microsoft working around all of limitations that made it extremely challenging to actually make use of many of the tools coming out of Apple’s Xcode team. I’d like to see the Swift tools not go down the same path and be tightly coupled in each of the phases so that integration because a huge pain, especially when integration needs to happen with non-Swift tools.
> 
> Thanks for voicing your concerns. I believe we are doing a pretty thorough job already with `swift build` of decoupling it from the rest of the system. As the product evolves and if you feel we are not taking the right approach I’ll gladly receive the criticism and try to respond to it with engineering.

One thing I'll add here: Unless I misunderstand, I don't think that this is a design/architectural limitation, but rather just a limitation of what functionality is currently exposed. In your example with xcodebuild, architecturally we do have the ability to test without building, and that functionality is even exposed in the Xcode.app Product menu, we've just never implemented a flag to xcodebuild for it. But we haven't boxed ourselves into an architectural corner there; there's no improper coupling at a technical level. Likewise, while this initial proposal doesn't specify the ability to test without building, I think that would be easy to add later on when we want it. Does that address your concern here?

	- Rick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160107/d3ee6430/attachment.html>