[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Brian Pratt]

> A tightly-coupled test harness is an incremental step towards a
loosely-coupled test harness.

While going from specific to generic is usually the right approach, in this
case I think going specific is going to create a lot of work that could be
avoided until there's a more finalized version of what the long-term goal
is going to be. Beyond that I'm not even sure how the build tool would
integrate with XCTest in a way that amounts to more than "build it as an
executable" and "run the executable." The language in the proposal raises a
coupling concern, but technically speaking, it's not really going to be
much of a concern.

The more big-picture point I want to make clear is that the build tool,
IMO, shouldn't take an opinion on how tests are run, what their output is,
etc. By just running a binary testing target with a main.swift and checking
the exit status, we can push a lot of the specifics down into those tools
which enables the community to develop testing tools more freely without
having to wait for this discussion to be 100% finished (or for any sort of
proposal/review process from the language/tooling itself, for that matter).
Adopting more detailed standards, at this point, from a build-tool level,
feels extremely premature.

So, between the facts that:
- there's a perfectly acceptable intermediate step for almost all user
experiences (including the "default" one of using XCTest),
- this proposal is very prescriptive in some parts, and very broad and
open-ended in others,
- the proposal for integrating third-party testing tools is still under
discussion,
- some of the details here might be better off living in XCTest

I just think this idea needs some more time to bake, from a technical
perspective... and I think swift-build-dev is a good place to continue that
discussion. But as others mentioned the proposal is open-ended enough that
I don't think doubling down on talking implementation details invalidates
any of it -- the proposal as worded feels more like a declaration of
direction than anything, and in general I like the direction! But I'd
prefer some of the specifics (e.g, around testing output) be brought up to
those working on XCTest instead of integrated with the build tool itself.

So: proposal seems okay from a big-picture perspective, except for a few
things, but I don't think this is ready to have work begin on it yet in
earnest. And because of that, I'd say a simple, working version of testing
should be implemented first, just to make the build tool more usable for
many developers (including me).

- Brian

On Mon, Jan 11, 2016 at 5:14 AM, Drew Crawford <drew at sealedabstract.com>
wrote:

> > I don't think the situations are the same. Quickly skimming that thread,
> it seems like people are arguing that the proposed feature is a step in the
> wrong direction. (I don't know enough to say if I agree with them; I'm
> simply characterizing their position.)
>
>
> I argue that this proposal is a step in the wrong direction.  I suspect
> David and Brian might argue that as well (I don't want to put words in
> their mouth though).  So that is the sense in which the situations are
> similar: people are arguing the proposal is a step in the wrong direction.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160111/c01aea43/attachment.html>