<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">As I've taken some more time to evaluate the criticisms people have raised (several off-list), my view of the proposal has changed. There is enough substantive dissent here that we should send this back for more cooking.<div class=""><br class=""></div><div class="">A summary of the criticisms that convinced me are:</div><div class=""><br class=""></div><div class="">1. Whether testing and SwiftPM should be coupled at all. This isn't obviously clear to everybody that SwiftPM is the right place to solve the problem and a reasoned justification in the document is necessary. The only treatment of this coupling at all is that it "will help ensure a stable and reliable packaging ecosystem", but there is no discussion of why this is so. IMO this is a critical omission.</div><div class=""><br class=""></div><div class="">2. Whether privileging XCTest is in the best interest of the ecosystem. My informal poll reveals much more debate than I had expected about whether people even want to use XCTest for their programs, and so I think an approach that starts with defining the interface first and wiring XCTest into it second is better for the long-term health of the ecosystem than an approach that starts with XCTest first and invents a separate system for third-party later which we may or may not retrofit XCTest back into. I think a proposal that takes the latter position should include a reasoned technical justification for that decision.</div><div class=""><br class=""></div><div class="">3. The proposal's treatment of various topics "in passing" is frustrating. For example, the proposal says</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class="">We expect that such an implementation would take the form of a Swift protocol</blockquote></div><br class=""><div class="">If we are right now deciding to use a protocol (over say a UNIX process model) then this is not the kind of detailed discussion we need to justify that decision–and I say that as a person in favor of doing it.</div><div class=""><br class=""></div><div class="">Or if we are not right now deciding that, why is it in the proposal? So I think the proposal needs to be a lot clearer about what is an essential part of what is being proposed and what is nonessential, and right now it tries to cut both ways on several points. The protocol system, JUnit XML, Testability for debug builds, opting out of additional compilation, all exist in this grey area where I am not sure if they are passing ideas or binding precedent that we aren't defending well (or at all in some cases).</div><div class=""><br class=""></div><div class="">4. I think all reviewers supporting the proposal have found "irregularities" in the document. Paul Cantrell said it best that this was a "<span style="font-family: HelveticaNeue;" class="">vision and first draft" but "not a releasable product". I have argued that we should use a different standard of evidence for greenfield proposals.</span></div><div class=""><span style="font-family: HelveticaNeue;" class=""><br class=""></span></div><div class=""><span style="font-family: HelveticaNeue;" class="">The fact is, though, there is no "official" explanation for why we should +1 a document that we all agree has deficiencies. I have tried to argue that it is a stopgap solution that is necessary but I find my own argument poor.</span></div><div class=""><span style="font-family: HelveticaNeue;" class=""><br class=""></span></div><div class=""><span style="font-family: HelveticaNeue;" class=""> In another context of SwiftPM, Daniel Dunbar said:</span></div><div class=""><span style="font-family: HelveticaNeue;" class=""><br class=""></span></div><div class=""><font face="HelveticaNeue" class=""><blockquote type="cite" class="">the approach we want to take for developing and adding features is to be in a good place *<i class="">at the time of release*</i>... With that mindset, it doesn't usually make sense to add features that we consider actively/potentially harmful, even if that solves some current short term pain.</blockquote><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div>(emphasis in original)<br class=""><br class=""></font></div><div class=""><font face="HelveticaNeue" class="">It seems like the same principle should apply here. If we have consensus that this proposal is potentially harmful (and there are some details that I would characterize that way) then let's send it back and get a better document.</font></div><div class=""><br class=""></div><div class=""><font face="HelveticaNeue" class="">5. The "Alternatives considered" section admits that it considers no alternatives, but I think there are clear alternatives to each of the 4 numbered criticisms above. Specifically each of these are potential alternatives to the proposal:</font></div><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div><div class=""><font face="HelveticaNeue" class=""> 1. Do testing from a completely separate tool that isn't SwiftPM (probably frontended through swift-multitool)</font></div><div class=""><font face="HelveticaNeue" class=""> 1a. Or building the test frontend inside the XCTest project instead</font></div><div class=""><font face="HelveticaNeue" class=""> 2. Start with defining interface for the testing framework first, instead of potentially having one codepath for XCTest and another codepath for third party</font></div><div class=""><font face="HelveticaNeue" class=""> 3. UNIX process model instead of protocol, JSON format instead of JUnit XML, compiling without testability for debug builds by default, etc.</font></div><div class=""><font face="HelveticaNeue" class=""> 4. Doing nothing, until we have a "final draft" proposal that reviewers can endorse with fewer reservations.</font></div><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div><div class=""><font face="HelveticaNeue" class="">To be clear, I'm not particularly advocating in favor of doing any of these, but I do think we can do a lot better than not imagining <i class="">any</i> alternatives, which is the present text.</font></div><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div><div class=""><font face="HelveticaNeue" class="">While I don't advance these as arguments to reject, I think we should take the opportunity to fix them:</font></div><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div><div class=""><font face="HelveticaNeue" class="">1. Being more careful about not assuming compilation times are free, see Paul Cantrell and David Owens's excellent arguments</font></div><div class=""><font face="HelveticaNeue" class="">2. There is some kind of documentation snafu in the swift-evolution process as it relates to this proposal. The </font><a href="https://github.com/apple/swift-evolution/blob/master/process.md#review-process" class="">process document</a><font face="HelveticaNeue" class=""> says the Review Manager is a member of the core team, but our RM </font><a href="https://swift.org/community/#community-structure" class=""> is not listed there</a><font face="HelveticaNeue" class="">. I'm sure there is a reasonable explanation, but we should fix the docs.</font></div><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div><div class=""><font face="HelveticaNeue" class="">I want testing as bad as all of you, but I see enough substance in the criticism that I now believe we should reject the proposal as it presently appears before us.</font></div><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div><div class=""><font face="HelveticaNeue" class=""><br class=""></font></div></body></html>