[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Drew Crawford]

> On Jan 8, 2016, at 12:51 PM, Brian Pratt <brian at pratt.io> wrote:
> 
>  A simple function that can throw an error (and *maybe* returns a simple status of pass/fail?) is probably enough to facilitate the vast majority of testing frameworks, and it *doesn't* come with lots of baggage of protocols and implementation details that we can't assume will be required or even useful for said framework. We should consider what we can leave out instead.

This is not the case, for reasons you can find in the third-party test frameworks proposal; but shortly: it is not innovation to have a protocol which merely allows 10 competing implementations of an XCTest-like framework.

Testing is not a one-size-fits-all concept.  The role of SwiftPM is IMO:

To have a simple way to run XCTest suites, which is the way most people (including SwiftPM & Foundation & many many codebases) write their tests now, and
To push innovation into third-party test frameworks that are free to explore the problem space in a more unrestricted way that is not bound by the constraints of XCTest, source compatibility, or the swift-evolution proposal process

This proposal is mostly targeted at the first situation.  I do believe it is the "minimum viable proposal" to address that first situation.  Perhaps the assumption that we are studying (only) the first situation needs to be more clearly stated.

XCTest compatibility is a feature many people require.  I understand folks who find it heavyweight for new code.  But I think the best way forward is to kick the design problem for other models out to third parties, because this mailing list will not scale to designing a novel testing framework one API at a time.  Let's let the ecosystem go to work on this problem.

We need a proposal to do that, and it is currently being drafted on swift-build-dev.  Please help.

> than continue to debate a proposal that itself admits that it amounts to a temporary solution

I don't think this is the case at all; the proposal admits to being one solution among many, and one that will need further extensions.  But people have XCTest test suites, they want to test them, we should support it.  There's nothing temporary about that.

> On Jan 8, 2016, at 11:13 AM, David Owens II <david at owensd.io> wrote:

> For example, before any meaningful work can be done to support different types of test runners, the work is going to need to done to break the build & run tests coupling. In this way, I think the proposal does a disservice because it’s not laying a good foundation to build on, it’s laying a foundation that we already know needs to be broken up.


I don't believe this is the case either.  Under this proposal, building and running is coupled at the CLI, but introducing new CLI arguments is not any great hardship. 

It may help to clearly identify the coupling you mean, and how this proposal very specifically establishes it.  Everybody agrees that "coupling is bad, kids" but not everyone understands how this proposal brings it into being.

I wonder if your objection is simply that the XCTest proposal hit review first, rather than the third-party frameworks proposal.  That is only because the XCTest folks are simply better motivated to draft it: there is tons of existing code that can benefit, the test framework actually exists, etc.  The third-party frameworks are not as well-motivated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160108/a717a52a/attachment.html>