[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Rick Ballard]

> On Jan 7, 2016, at 11:18 AM, David Owens II <david at owensd.io> wrote:
> 
>> One thing I'll add here: Unless I misunderstand, I don't think that this is a design/architectural limitation, but rather just a limitation of what functionality is currently exposed. In your example with xcodebuild, architecturally we do have the ability to test without building, and that functionality is even exposed in the Xcode.app Product menu, we've just never implemented a flag to xcodebuild for it. But we haven't boxed ourselves into an architectural corner there; there's no improper coupling at a technical level. Likewise, while this initial proposal doesn't specify the ability to test without building, I think that would be easy to add later on when we want it. Does that address your concern here?
>> 
>> 	- Rick
> 
> No, it doesn’t. While on the Office for Mac team, we repeatedly asked for many of the items in Xcode and xcodebuild to be made less coupled with each of the steps in the process. We were told no in most cases and given hacky workarounds in others that were prone to break with each incremental update to Xcode. Most of these breaks were found out because we picked up the latest beta build and attempted to run our automation tools only to find that something subtle had changed in our workaround or the basis of the workaround is no longer there.
> 
> Sure, SPM is “open source”, however, it’s unrealistic for people/teams to maintain forks of the tools to support features they need. Maybe the changes get accepted into a pull request, maybe not.
> 
> Honestly, I’m a bit shocked that the separation of building and executing tests is such a controversial thing. There should be three high-level tasks:
> 
> 1. build
> 2. run-tests
> 3. build + run-tests
> 
> What I’m saying is that by not supporting #2, you’re creating an adoption blocker for any team that cannot or does not want to build every time it runs their tests. There are many reasons for this. I’ve already given several, however, another reason is instrumentation for things like code-coverage or product code markers, often happen as an intermediate step outside of the normal build operation. This is especially true in the cases of bug investigation. Saying that I cannot trigger the running of these tests without a build step baffling to me.

I am not saying that we won't support this; I expect that we will, for the very good reasons that you've raised. I'm just saying that this isn't some fundamental architectural limitation that needs to be addressed by this proposal; it's just a matter of what options we expose in the (command-line) UI, and is fairly easy to change. (We should make sure that that's the case – that this is implemented as separable build and test steps – but I expect we'd be doing that anyway).

I don't want to derail this thread with discussion of Xcode's behavior, which is outside the scope of this project, but if you want to send me (off-thread) the radar # for the bug(s) you've filed about supporting xcodebuild test-without-build, I'd be happy to take a look at what happened there.

	 -Rick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160107/13fa55a3/attachment.html>