[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Drew Crawford]

As I've taken some more time to evaluate the criticisms people have raised (several off-list), my view of the proposal has changed.  There is enough substantive dissent here that we should send this back for more cooking.

A summary of the criticisms that convinced me are:

1.  Whether testing and SwiftPM should be coupled at all.  This isn't obviously clear to everybody that SwiftPM is the right place to solve the problem and a reasoned justification in the document is necessary.  The only treatment of this coupling at all is that it "will help ensure a stable and reliable packaging ecosystem", but there is no discussion of why this is so.  IMO this is a critical omission.

2.  Whether privileging XCTest is in the best interest of the ecosystem.  My informal poll reveals much more debate than I had expected about whether people even want to use XCTest for their programs, and so I think an approach that starts with defining the interface first and wiring XCTest into it second is better for the long-term health of the ecosystem than an approach that starts with XCTest first and invents a separate system for third-party later which we may or may not retrofit XCTest back into.  I think a proposal that takes the latter position should include a reasoned technical justification for that decision.

3.  The proposal's treatment of various topics "in passing" is frustrating.  For example, the proposal says

> We expect that such an implementation would take the form of a Swift protocol

If we are right now deciding to use a protocol (over say a UNIX process model) then this is not the kind of detailed discussion we need to justify that decision–and I say that as a person in favor of doing it.

Or if we are not right now deciding that, why is it in the proposal? So I think the proposal needs to be a lot clearer about what is an essential part of what is being proposed and what is nonessential, and right now it tries to cut both ways on several points.  The protocol system, JUnit XML, Testability for debug builds, opting out of additional compilation, all exist in this grey area where I am not sure if they are passing ideas or binding precedent that we aren't defending well (or at all in some cases).

4.  I think all reviewers supporting the proposal have found "irregularities" in the document.  Paul Cantrell said it best that this was a "vision and first draft" but "not a releasable product".  I have argued that we should use a different standard of evidence for greenfield proposals.

The fact is, though, there is no "official" explanation for why we should +1 a document that we all agree has deficiencies.  I have tried to argue that it is a stopgap solution that is necessary but I find my own argument poor.

 In another context of SwiftPM, Daniel Dunbar said:

> the approach we want to take for developing and adding features is to be in a good place *at the time of release*...  With that mindset, it doesn't usually make sense to add features that we consider actively/potentially harmful, even if that solves some current short term pain.

(emphasis in original)

It seems like the same principle should apply here.  If we have consensus that this proposal is potentially harmful (and there are some details that I would characterize that way) then let's send it back and get a better document.

5.  The "Alternatives considered" section admits that it considers no alternatives, but I think there are clear alternatives to each of the 4 numbered criticisms above.  Specifically each of these are potential alternatives to the proposal:

    1.  Do testing from a completely separate tool that isn't SwiftPM (probably frontended through swift-multitool)
    1a.  Or building the test frontend inside the XCTest project instead
    2.  Start with defining interface for the testing framework first, instead of potentially having one codepath for XCTest and another codepath for third party
    3.  UNIX process model instead of protocol, JSON format instead of JUnit XML, compiling without testability for debug builds by default, etc.
    4.  Doing nothing, until we have a "final draft" proposal that reviewers can endorse with fewer reservations.

To be clear, I'm not particularly advocating in favor of doing any of these, but I do think we can do a lot better than not imagining any alternatives, which is the present text.

While I don't advance these as arguments to reject, I think we should take the opportunity to fix them:

1.  Being more careful about not assuming compilation times are free, see Paul Cantrell and David Owens's excellent arguments
2.  There is some kind of documentation snafu in the swift-evolution process as it relates to this proposal.  The process document <https://github.com/apple/swift-evolution/blob/master/process.md#review-process> says the Review Manager is a member of the core team, but our RM  is not listed there <https://swift.org/community/#community-structure>.  I'm sure there is a reasonable explanation, but we should fix the docs.

I want testing as bad as all of you, but I see enough substance in the criticism that I now believe we should reject the proposal as it presently appears before us.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160111/5910a895/attachment.html>