[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Drew Crawford]

> On Jan 8, 2016, at 9:17 PM, Brian Pratt <brian at pratt.io> wrote:
> 
> I don't really think innovation is something worth sacrificing *absolute* flexibility for. Not in a tool that's primarily meant to enable developers to distribute and consume shared libraries of code. There are going to be many ways that developers prefer to write tests and something as generic as a package manager should, in my mind, allow for all of that. While I again agree that XCTest is the most sensible default, I think there's value in expressing the idea (one not called out by either this proposal or the one for third-party testing frameworks) that the build tool should not have concrete dependencies on XCTest but rather XCTest should integrate as one of many potential generic implementations usable with the build tool in the most minimal interface possible. 
> 
> That is to say, the build tool shouldn't take an opinion at all how tests are run, just about the general result. Enforcing any kind of uniformity is counter-intuitive as people may end up with fragmentation in their build tools themselves rather than just in their testing frameworks. There are already more than a handful of Swift testing frameworks that aren't XCTest-like, even some that are more Quickcheck-inspired, and these prescriptive testing approaches within an opinionated build tool hinder the community's ability to innovate on their own. The language and its tools should give users flexibility to allow for innovation, not try to prescribe anything about how testing should be done in order to enforce uniformity. That's a loss.

This is a nice philosophy, but what does any of it mean?

We already have "*absolute* flexibility"–SwiftPM and Foundation tests are simply an ad-hoc script.  The script can literally do anything, and report its outputs any way.  It can use any testing framework.  It can live anywhere in your repository and be named anything, and can require any number of system-installed dependencies (or not), can be written in any language that you may or may not have installed (Python seems popular, some debate if 2 or 3 <https://github.com/apple/swift-package-manager/pull/108>...), it can take any number of arguments, it can build and test together or separately, it can play music in your iTunes library.  A proposal that provided "absolute" flexibility would simply be:

> # Motivation
> 
> We need absolute flexibility in tests.
> 
> # Proposed solution
> 
> Anyone can write tests however the fuck they want.

Any longer proposal is *necessarily* going to define *some* limits to use of any proposed testing feature.  The question is whether some particular limit is sense or nonsense, not whether there should be anything standardized at all.  And we need to have a specific conversation around the exact technical details of the interface.

That conversation is currently ongoing on swift-build-dev.  If you think a UNIX process model (which does limit flexibility!) is superior to a protocol-based approach, I would love to read an argument for that on the thread.  It has some drawbacks we have been talking about (such as no common reporting format for CI / GUI / Xcode to consume) but maybe you have ideas for how we can solve that problem within a UNIX process model interface.

In any case, SE-0019's treatment of the interface is so incidental that I do not see why the interface question (which is in my mind still very debatable at this moment) would be a defining issue in this review.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160108/1e980e59/attachment.html>