[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Brian Pratt]

> This is not the case, for reasons you can find in the third-party test
frameworks proposal; but shortly: it is not innovation to have a protocol
which merely allows 10 competing implementations of an XCTest-like
framework.

I don't really think innovation is something worth sacrificing *absolute*
flexibility for. Not in a tool that's primarily meant to enable developers
to distribute and consume shared libraries of code. There are going to be
many ways that developers prefer to write tests and something as generic as
a package manager should, in my mind, allow for all of that. While I again
agree that XCTest is the most sensible default, I think there's value in
expressing the idea (one not called out by either this proposal or the one
for third-party testing frameworks) that the build tool should not have
concrete dependencies on XCTest but rather XCTest should integrate as one
of many potential generic implementations usable with the build tool in the
most minimal interface possible.

That is to say, the build tool shouldn't take an opinion at all how tests
are run, just about the general result. Enforcing any kind of uniformity is
counter-intuitive as people may end up with fragmentation in their build
tools themselves rather than just in their testing frameworks. There are
already more than a handful of Swift testing frameworks that aren't
XCTest-like, even some that are more Quickcheck-inspired, and these
prescriptive testing approaches within an opinionated build tool hinder the
community's ability to innovate on their own. The language and its tools
should give users flexibility to allow for innovation, not try to prescribe
anything about how testing should be done in order to enforce uniformity.
That's a loss.


> To have a simple way to run XCTest suites, which is the way most people
(including SwiftPM & Foundation & many many codebases) write their tests
now, and
> To push innovation into third-party test frameworks that are free to
explore the problem space in a more unrestricted way that is not bound by
the constraints of XCTest, source compatibility, or the swift-evolution
proposal process.

Right, so, we agree here -- just wanted to make sure I register a concern
that rushing to implement XCTest integration with the build tool feels like
a mistake, especially before the community can meaningfully contribute to
the relatively new third-party frameworks proposal as it stands. Making the
build tool XCTest dependent is probably more difficult than making it
generic, and it seems like we'd backtrack quickly.

Why not just let me tell my Package object that I want a specific
executable target for testing, one that swift-build will build and then run
linking whatever packages I specify from `testDependencies`. Boom, XCTest
works, and so do many other frameworks, and all authors have to do is give
the proper exit status. This mirrors the UNIX philosophy of small,
composable components and I think that's a good approach to take with Swift
and its generic tooling. Further, this makes SwiftPM work for a lot of
generic cases until the third-party frameworks proposal is properly ironed
out. That protocol may be the ultimate long-term win, but I see no reason
to make a short-term sacrifice (XCTest-specific integration) in order to
facilitate it when something generic and simple can achieve basically the
exact same result from a user experience perspective?

Again, it's good for the language to be prescriptive, but it's better to be
flexible and let the community evolve their individual tooling as they wish.


> It may help to clearly identify the coupling you mean, and how this
proposal very specifically establishes it.  Everybody agrees that "coupling
is bad, kids" but not everyone understands how this proposal brings it into
being.


You're absolutely right -- the nature of the complaints has been very
high-level at this point, so, details: I think, in particular, the
prescription of testing output creates an unnecessarily large overlap
between potential (and existing) third-party frameworks that should be
avoided. It means that SwiftPM would have to implement test reporters that
might make certain assumptions about how the tests are run (synchronously?)
and what output is deemed important. It might make assumptions about how
tests fail and how they report those failures to the user. And, having used
XCTest as distributed by XCode, it would be frustrating to have very little
control over your test output because you have to maintain compliance with
a standard that most certainly -is- bound by the constraints of the
swift-evolution process. If the stated goal is to iterate, it makes sense
to keep the components flexible and light-weight so that iteration can
happen more quickly and outside of the bounds of language review / release
cycle.

I understand that supporting XCTest is the most pragmatic move, short-term,
but I also think the existing proposals convey an overly specific approach
to testing that doesn't feel like it belongs in an otherwise generic build
/ packaging tool.



On Fri, Jan 8, 2016 at 7:54 PM, Drew Crawford <drew at sealedabstract.com>
wrote:

>
> On Jan 8, 2016, at 12:51 PM, Brian Pratt <brian at pratt.io> wrote:
>
>  A simple function that can throw an error (and *maybe* returns a simple
> status of pass/fail?) is probably enough to facilitate the vast majority of
> testing frameworks, and it *doesn't* come with lots of baggage of protocols
> and implementation details that we can't assume will be required or even
> useful for said framework. We should consider what we can leave out instead.
>
>
> This is not the case, for reasons you can find in the third-party test
> frameworks proposal; but shortly: it is not innovation to have a protocol
> which merely allows 10 competing implementations of an XCTest-like
> framework.
>
> Testing is not a one-size-fits-all concept.  The role of SwiftPM is IMO:
>
>
>    - To have a simple way to run XCTest suites, which is the way most
>    people (including SwiftPM & Foundation & many many codebases) write their
>    tests now, and
>    - To push innovation into third-party test frameworks that are free to
>    explore the problem space in a more unrestricted way that is not bound by
>    the constraints of XCTest, source compatibility, or the swift-evolution
>    proposal process
>
>
> *This proposal is mostly targeted at the first situation.*  I do believe
> it is the "minimum viable proposal" to address that first situation.
> Perhaps the assumption that we are studying (only) the first situation
> needs to be more clearly stated.
>
> XCTest compatibility is a feature many people require.  I understand folks
> who find it heavyweight for new code.  But I think the best way forward is
> to kick the design problem for other models out to third parties, because
> this mailing list will not scale to designing a novel testing framework one
> API at a time.  Let's let the ecosystem go to work on this problem.
>
> We need a proposal to do that, and it is currently being drafted on
> swift-build-dev.  Please help.
>
> than continue to debate a proposal that itself admits that it amounts to a
> temporary solution
>
>
> I don't think this is the case at all; the proposal admits to being one
> solution among many, and one that will need further extensions.  But people
> have XCTest test suites, they want to test them, we should support it.
> There's nothing temporary about that.
>
> On Jan 8, 2016, at 11:13 AM, David Owens II <david at owensd.io> wrote:
>
> For example, before any meaningful work can be done to support different
> types of test runners, the work is going to need to done to break the build
> & run tests coupling. In this way, I think the proposal does a disservice
> because it’s not laying a good foundation to build on, it’s laying a
> foundation that we already know needs to be broken up.
>
>
> I don't believe this is the case either.  Under this proposal, building
> and running is coupled at the CLI, but introducing new CLI arguments is not
> any great hardship.
>
> It may help to clearly identify the coupling you mean, and how this
> proposal very specifically establishes it.  Everybody agrees that "coupling
> is bad, kids" but not everyone understands how this proposal brings it into
> being.
>
> I wonder if your objection is simply that the XCTest proposal hit review
> first, rather than the third-party frameworks proposal.  That is only
> because the XCTest folks are simply better motivated to draft it: there is
> tons of existing code that can benefit, the test framework actually exists,
> etc.  The third-party frameworks are not as well-motivated.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160108/603f10ea/attachment.html>