[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Brian Pratt]

However what is sufficient? Is it mandatory for us to output JUnit-esque XML
for all possible test frameworks? Would that restrict what innovations are
possible by testing frameworks?

"for all possible test frameworks"? I don't think you can do something like
that unless the build-tool provides an entire output shell that testing
frameworks can hook into for reporting at runtime... and in that case, yes,
I personally think it would be overly restrictive on testing frameworks, at
least from the perspective of a *build tool*. The build
tool's responsibility is that it has to compile and run the tests, and then
allow said tests to be configured to provide feedback in whatever way I can
configure them to. The opinionated output proposed in this document would
seem to overreach those responsibilities unless, as Max and Brian
mentioned, it lived in XCTest or similar.


Is it enough to simply report success or failure for the invocation? CI
services
would then know if the tests ran to success or not and could simply provide
the output from the test framework as logs.


I'd say yes: Most CI tools I know of will just check the exit status of the
commands they're provided, in the simplest cases... So, if the test
framework provides a zero/non-zero status for pass/fail, that's enough for
most CI services I've used (Travis, Wercker, Jenkins, to name a few). Yes,
you can get a lot more detailed, but the implementations would probably
differ for each service. Exit statuses are something they all understand,
and something the build tool understands already too.


If we are to provide XML/JSON output as part of this initial implementation
then we must parse XCTest output or figure out some other way to integrate
with XCTest so that the output format can be modified.

Parsing the output feels like a fragile integration, and one that could
handcuff what XCTest can change in the future. While it'd be possible to
integrate with XCTest more directly (via a reporter registration system
that I'm sure Brian can explain better than I can), I don't think the value
of that has been defended very well from the perspective of the build tool:
The win of getting uniform test output from a build tool is not worth the
loss of simple testing integration, whether it be provided by a runtime
library defining a set of output protocols, or by intercepting other tools'
output and parsing them to transform it, or anything else I've seen
mentioned.

We would like to get testing up to speed as soon as possible.

If this is true, then we should make it clear our intention to push those
details out to XCTest and allow it to innovate and report on its own terms.
Anything else is going to slow things down. Right now, given that the
output part of the proposal is getting pushback both from developers
working on XCTest and developers who have worked on third-party testing
frameworks, I'd say that's just too much scope for a build tool to be
taking on.

Beyond that, there's a very low-effort way that's already included in the
tool to get testing up-to-speed, and this would do that: let me define a
target (or a list of targets) in my Package.swift that are for tests.
Compile them, link them with my testDependencies, and run them, checking
exit statuses for results. This achieves:
- the ability to build and run tests from the build-tool
- it works with XCTest. Today. Not in the future when they have
registration for reporters or when the build-tool writes some test wrapper
- a simple integration point for third-party frameworks
- support for almost every single CI system in existence (if it can run
Swift)
- a quick turnaround to put this in the hands of developers (who want it)

More notably, it leaves the concerns of actual testing output to the
testing frameworks, where, as so far all parties who have commented on this
review agree, those concerns actually belong. This would even fit in the
confines of this proposal and could "kick the can down the road" for
third-party testing protocols for quite some time as the build tool
evolves. Anything else is going to:
- limit the flexibility of test framework developers (including those who
work on XCTest)
- require more development (in the case of creating an XCTest
formatter/reporter, or a parser/transformer, or a protocol that XCTest
itself will then have to implement)
- be more complex

All of those things are net losses, all for something that we can't even
agree is *even philosophically a responsibility of the build tool*? Let's
just drop the output idea, even if just for now, and start delivering
first-class testability with the tools that already exist, please.

On Tue, Jan 19, 2016 at 1:50 PM, Max Howell via swift-evolution <
swift-evolution at swift.org> wrote:

> My only reservation is in regards to the following:
>
> Executing a test from the terminal will produce user-readable output. This
> should incorporate colorization and other formatting similar to other
> testing tools to indicate the success and failure of different tests. [...]
> An additional option may be passed to the testing command to output
> JUnit-style XML or other formats that can be integrated with continuous
> integration (CI) and other systems.
>
>
> swift-corelibs-xctest is currently responsible for its own output--it
> prints test results to stdout. How do the authors of the proposal plan
> to implement colorization and formatting?
>
>
> This is a good point.
>
> Personally, I believe that the testing framework itself--in this case
> swift-corelibs-xctest--should be responsible for colorization and
> output formatting. I believe swift-corelibs-xctest will soon provide
> an observation API similar to Apple XCTest (adding it has been
> discussed several times; see:
> https://github.com/apple/swift-corelibs-xctest/pull/40,
>
> https://lists.swift.org/pipermail/swift-corelibs-dev/2015-December/000034.html
> ).
> Once it does, any user will be able to register whichever output
> formatter they wish.
>
>
> The proposal is not specific about who is providing the output. I
> personally
> expected the testing-framework to provide the default Terminal output
>
> I am concerned to find this proposal for swift-package-manager mention
> output formatting, since I it doesn't seem like output formatters
> belong in swift-package-manager itself, and the API for
> swift-corelibs-xctest hasn't been implemented yet. I suggest deferring
> the discussion on output formatting to a future proposal--it seems
> like something that would be merely nice to have, whereas this
> proposal states "we would like to get testing up to speed as soon as
> possible.”
>
>
> I think the proposal is OK. We’re not clear about who provides the output,
> so nothing is fixed.
>
> When we work on the alternative testing framework proposal we will go into
> more detail on this matter.
>
> So TL;DR: for now XCTest will provide the output, colorization can be added
> immediately with some output parsing but is probably better left for the
> next
> proposal.
>
> One part of the proposal cannot be met immediately:
>
> An additional option may be passed to the testing command to output
> JUnit-style XML or other formats that can be integrated with continuous
> integration (CI) and other systems.
>
>
> It is essential that `swift test` itself can provide some kind of
> standardized
> output so other tools can use `swift test` as part of larger CI systems.
>
> However what is sufficient? Is it mandatory for us to output JUnit-esque
> XML
> for all possible test frameworks? Would that restrict what innovations are
> possible by testing frameworks?
>
> Is it enough to simply report success or failure for the invocation? CI
> services
> would then know if the tests ran to success or not and could simply provide
> the output from the test framework as logs.
>
> If we are to provide XML/JSON output as part of this initial implementation
> then we must parse XCTest output or figure out some other way to integrate
> with XCTest so that the output format can be modified.
>
> Really the proposal should be modified and this part removed pending the
> next proposal, but it would be a shame to delay implementation by a further
> week.
>
> Max
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160119/8cad9b89/attachment.html>