[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Max Howell]

> My only reservation is in regards to the following:
> 
>> Executing a test from the terminal will produce user-readable output. This should incorporate colorization and other formatting similar to other testing tools to indicate the success and failure of different tests. [...] An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.
> 
> swift-corelibs-xctest is currently responsible for its own output--it
> prints test results to stdout. How do the authors of the proposal plan
> to implement colorization and formatting?

This is a good point.

> Personally, I believe that the testing framework itself--in this case
> swift-corelibs-xctest--should be responsible for colorization and
> output formatting. I believe swift-corelibs-xctest will soon provide
> an observation API similar to Apple XCTest (adding it has been
> discussed several times; see:
> https://github.com/apple/swift-corelibs-xctest/pull/40,
> https://lists.swift.org/pipermail/swift-corelibs-dev/2015-December/000034.html).
> Once it does, any user will be able to register whichever output
> formatter they wish.

The proposal is not specific about who is providing the output. I personally
expected the testing-framework to provide the default Terminal output

> I am concerned to find this proposal for swift-package-manager mention
> output formatting, since I it doesn't seem like output formatters
> belong in swift-package-manager itself, and the API for
> swift-corelibs-xctest hasn't been implemented yet. I suggest deferring
> the discussion on output formatting to a future proposal--it seems
> like something that would be merely nice to have, whereas this
> proposal states "we would like to get testing up to speed as soon as
> possible.”

I think the proposal is OK. We’re not clear about who provides the output,
so nothing is fixed.

When we work on the alternative testing framework proposal we will go into
more detail on this matter.

So TL;DR: for now XCTest will provide the output, colorization can be added
immediately with some output parsing but is probably better left for the next
proposal.

One part of the proposal cannot be met immediately:

> An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.

It is essential that `swift test` itself can provide some kind of standardized
output so other tools can use `swift test` as part of larger CI systems.

However what is sufficient? Is it mandatory for us to output JUnit-esque XML
for all possible test frameworks? Would that restrict what innovations are
possible by testing frameworks?

Is it enough to simply report success or failure for the invocation? CI services
would then know if the tests ran to success or not and could simply provide
the output from the test framework as logs.

If we are to provide XML/JSON output as part of this initial implementation
then we must parse XCTest output or figure out some other way to integrate
with XCTest so that the output format can be modified.

Really the proposal should be modified and this part removed pending the
next proposal, but it would be a shame to delay implementation by a further
week.

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160119/95e47151/attachment.html>