[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Brian Pratt]

It's not the approach (I'll save my commentary on that for the right
thread) it's the fact that I think the build tool shouldn't be concerned
with things like test output and GUI reporting. That feels like a behavior
that belongs in the testing tools themselves, doesn't it?

On Friday, January 8, 2016, Drew Crawford <drew at sealedabstract.com> wrote:

>
> On Jan 8, 2016, at 9:17 PM, Brian Pratt <brian at pratt.io
> <javascript:_e(%7B%7D,'cvml','brian at pratt.io');>> wrote:
>
> I don't really think innovation is something worth sacrificing *absolute*
> flexibility for. Not in a tool that's primarily meant to enable developers
> to distribute and consume shared libraries of code. There are going to be
> many ways that developers prefer to write tests and something as generic as
> a package manager should, in my mind, allow for all of that. While I again
> agree that XCTest is the most sensible default, I think there's value in
> expressing the idea (one not called out by either this proposal or the one
> for third-party testing frameworks) that the build tool should not have
> concrete dependencies on XCTest but rather XCTest should integrate as one
> of many potential generic implementations usable with the build tool in the
> most minimal interface possible.
>
> That is to say, the build tool shouldn't take an opinion at all how tests
> are run, just about the general result. Enforcing any kind of uniformity is
> counter-intuitive as people may end up with fragmentation in their build
> tools themselves rather than just in their testing frameworks. There are
> already more than a handful of Swift testing frameworks that aren't
> XCTest-like, even some that are more Quickcheck-inspired, and these
> prescriptive testing approaches within an opinionated build tool hinder the
> community's ability to innovate on their own. The language and its tools
> should give users flexibility to allow for innovation, not try to prescribe
> anything about how testing should be done in order to enforce uniformity.
> That's a loss.
>
>
> This is a nice philosophy, but what does any of it mean?
>
> We already have "*absolute* flexibility"–SwiftPM and Foundation tests are
> simply an ad-hoc script.  The script can literally do anything, and report
> its outputs any way.  It can use any testing framework.  It can live
> anywhere in your repository and be named anything, and can require any
> number of system-installed dependencies (or not), can be written in any
> language that you may or may not have installed (Python seems popular, some
> debate if 2 or 3 <https://github.com/apple/swift-package-manager/pull/108>...),
> it can take any number of arguments, it can build and test together or
> separately, it can play music in your iTunes library.  A proposal that
> provided "absolute" flexibility would simply be:
>
> # Motivation
>
>
> We need absolute flexibility in tests.
>
>
> # Proposed solution
>
> Anyone can write tests however the fuck they want.
>
>
> Any longer proposal is *necessarily* going to define *some* limits to use
> of any proposed testing feature.  The question is whether some particular
> limit is sense or nonsense, not whether there should be anything
> standardized at all.  And we need to have a specific conversation around
> the *exact* technical details of the interface.
>
> That conversation is currently ongoing on swift-build-dev.  If you think a
> UNIX process model (which does limit flexibility!) is superior to a
> protocol-based approach, I would love to read an argument for that on the
> thread.  It has some drawbacks we have been talking about (such as no
> common reporting format for CI / GUI / Xcode to consume) but maybe you have
> ideas for how we can solve that problem within a UNIX process model
> interface.
>
> In any case, SE-0019's treatment of the interface is so incidental that I
> do not see why the interface question (which is in my mind still very
> debatable at this moment) would be a defining issue in this review.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160109/91a3702e/attachment.html>