<div dir="ltr"><div>> This is not the case, for reasons you can find in the third-party
test frameworks proposal; but shortly: it is not innovation to have a
protocol which merely allows 10 competing implementations of an
XCTest-like framework.<br><br></div>I don't really think innovation is something worth sacrificing *absolute* flexibility for. Not in a tool that's primarily meant to enable developers to distribute and consume shared libraries of code. There are going to be many ways that developers prefer to write tests and something as generic as a package manager should, in my mind, allow for all of that. While I again agree that XCTest is the most sensible default, I think there's value in expressing the idea (one not called out by either this proposal or the one for third-party testing frameworks) that the build tool should not have concrete dependencies on XCTest but rather XCTest should integrate as one of many potential generic implementations usable with the build tool in the most minimal interface possible. <br><br>That is to say, the build tool shouldn't take an opinion at all how tests are run, just about the general result. Enforcing any kind of uniformity is counter-intuitive as people may end up with fragmentation in their build tools themselves rather than just in their testing frameworks. There are already more than a handful of Swift testing frameworks that aren't XCTest-like, even some that are more Quickcheck-inspired, and these prescriptive testing approaches within an opinionated build tool hinder the community's ability to innovate on their own. The language and its tools should give users flexibility to allow for innovation, not try to prescribe anything about how testing should be done in order to enforce uniformity. That's a loss.<br><div><br><br>> To have a simple way to run XCTest suites, which is the way most
people (including SwiftPM & Foundation & many many codebases)
write their tests now, and<br>> To push innovation into third-party test frameworks that are free to
explore the problem space in a more unrestricted way that is not bound
by the constraints of XCTest, source compatibility, or the
swift-evolution proposal process. <br><br>Right, so, we agree here -- just wanted to make sure I register a concern that rushing to implement XCTest integration with the build tool feels like a mistake, especially before the community can meaningfully contribute to the relatively new third-party frameworks proposal as it stands. Making the build tool XCTest dependent is probably more difficult than making it generic, and it seems like we'd backtrack quickly.<br><br>Why not just let me tell my Package object that I want a specific executable target for testing, one that swift-build will build and then run linking whatever packages I specify from `testDependencies`. Boom, XCTest works, and so do many other frameworks, and all authors have to do is give the proper exit status. This mirrors the UNIX philosophy of small, composable components and I think that's a good approach to take with Swift and its generic tooling. Further, this makes SwiftPM work for a lot of generic cases until the third-party frameworks proposal is properly ironed out. That protocol may be the ultimate long-term win, but I see no reason to make a short-term sacrifice (XCTest-specific integration) in order to facilitate it when something generic and simple can achieve basically the exact same result from a user experience perspective?<br><br></div><div>Again, it's good for the language to be prescriptive, but it's better to be flexible and let the community evolve their individual tooling as they wish.<br></div><div><br><br>> It may help to clearly identify the coupling you mean, and how this
proposal very specifically establishes it. Everybody agrees that
"coupling is bad, kids" but not everyone understands how this proposal
brings it into being.<br><br><br>You're absolutely right -- the nature of the complaints has been very high-level at this point, so, details: I think, in particular, the prescription of testing output creates an unnecessarily large overlap between potential (and existing) third-party frameworks that should be avoided. It means that SwiftPM would have to implement test reporters that might make certain assumptions about how the tests are run (synchronously?) and what output is deemed important. It might make assumptions about how tests fail and how they report those failures to the user. And, having used XCTest as distributed by XCode, it would be frustrating to have very little control over your test output because you have to maintain compliance with a standard that most certainly -is- bound by the constraints of the swift-evolution process. If the stated goal is to iterate, it makes sense to keep the components flexible and light-weight so that iteration can happen more quickly and outside of the bounds of language review / release cycle.<br><br>I understand that supporting XCTest is the most pragmatic move, short-term, but I also think the existing proposals convey an overly specific approach to testing that doesn't feel like it belongs in an otherwise generic build / packaging tool.<br><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 8, 2016 at 7:54 PM, Drew Crawford <span dir="ltr"><<a href="mailto:drew@sealedabstract.com" target="_blank">drew@sealedabstract.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><span class=""><br><div><blockquote type="cite"><div>On Jan 8, 2016, at 12:51 PM, Brian Pratt <<a href="mailto:brian@pratt.io" target="_blank">brian@pratt.io</a>> wrote:</div><br><div><span style="font-family:HelveticaNeue;font-size:13px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important"><span> </span>A simple function that can throw an error (and *maybe* returns a simple status of pass/fail?) is probably enough to facilitate the vast majority of testing frameworks, and it *doesn't* come with lots of baggage of protocols and implementation details that we can't assume will be required or even useful for said framework. We should consider what we can leave out instead.</span></div></blockquote></div><br></span><div>This is not the case, for reasons you can find in the third-party test frameworks proposal; but shortly: it is not innovation to have a protocol which merely allows 10 competing implementations of an XCTest-like framework.</div><div><br></div><div>Testing is not a one-size-fits-all concept. The role of SwiftPM is IMO:</div><div><br></div><div><ul><li>To have a simple way to run XCTest suites, which is the way most people (including SwiftPM & Foundation & many many codebases) write their tests now, and</li><li>To push innovation into third-party test frameworks that are free to explore the problem space in a more unrestricted way that is not bound by the constraints of XCTest, source compatibility, or the swift-evolution proposal process</li></ul><div><br></div></div><div><b>This proposal is mostly targeted at the first situation.</b> I do believe it is the "minimum viable proposal" to address that first situation. Perhaps the assumption that we are studying (only) the first situation needs to be more clearly stated.</div><div><br></div><div>XCTest compatibility is a feature many people require. I understand folks who find it heavyweight for new code. But I think the best way forward is to kick the design problem for other models out to third parties, because this mailing list will not scale to designing a novel testing framework one API at a time. Let's let the ecosystem go to work on this problem.</div><div><br></div><div>We need a proposal to do that, and it is currently being drafted on swift-build-dev. Please help.</div><span class=""><div><br></div><div><blockquote type="cite"><span style="font-family:HelveticaNeue">than continue to debate a proposal that itself admits that it amounts to a temporary solution</span></blockquote><br></div></span><div>I don't think this is the case at all; the proposal admits to being one solution among many, and one that will need further extensions. But people have XCTest test suites, they want to test them, we should support it. There's nothing temporary about that.</div><span class=""><div><br></div><div><blockquote type="cite">On Jan 8, 2016, at 11:13 AM, David Owens II <<a href="mailto:david@owensd.io" target="_blank">david@owensd.io</a>> wrote:</blockquote></div><div><blockquote type="cite"><span style="font-family:HelveticaNeue">For example, before any meaningful work can be done to support different types of test runners, the work is going to need to done to break the build & run tests coupling. In this way, I think the proposal does a disservice because it’s not laying a good foundation to build on, it’s laying a foundation that we already know needs to be broken up.</span></blockquote></div><div><br></div></span><div>I don't believe this is the case either. Under this proposal, building and running is coupled at the CLI, but introducing new CLI arguments is not any great hardship. </div><div><br></div><div>It may help to clearly identify the coupling you mean, and how this proposal very specifically establishes it. Everybody agrees that "coupling is bad, kids" but not everyone understands how this proposal brings it into being.</div><div><br></div><div>I wonder if your objection is simply that the XCTest proposal hit review first, rather than the third-party frameworks proposal. That is only because the XCTest folks are simply better motivated to draft it: there is tons of existing code that can benefit, the test framework actually exists, etc. The third-party frameworks are not as well-motivated.</div></div></blockquote></div><br></div>