[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Brian Gesiak]

On Fri, Jan 15, 2016 at 2:34 PM, Rick Ballard via swift-build-dev
<swift-build-dev at swift.org> wrote
>         * What is your evaluation of the proposal?

I think this is an excellent proposal, and I am excited to see testing
support added to swift-package-manager.

My only reservation is in regards to the following:

> Executing a test from the terminal will produce user-readable output. This should incorporate colorization and other formatting similar to other testing tools to indicate the success and failure of different tests. [...] An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.

swift-corelibs-xctest is currently responsible for its own output--it
prints test results to stdout. How do the authors of the proposal plan
to implement colorization and formatting?

Personally, I believe that the testing framework itself--in this case
swift-corelibs-xctest--should be responsible for colorization and
output formatting. I believe swift-corelibs-xctest will soon provide
an observation API similar to Apple XCTest (adding it has been
discussed several times; see:
https://github.com/apple/swift-corelibs-xctest/pull/40,
https://lists.swift.org/pipermail/swift-corelibs-dev/2015-December/000034.html).
Once it does, any user will be able to register whichever output
formatter they wish.

So I would recommend that swift-package-manager use this API. Ideally,
swift-package-manager would provide default formatters, and those
could be customized via command-line arguments to `swift test`.

Another potential solution would be to borrow a page from third-party
utilities like xcpretty (https://github.com/supermarin/xcpretty),
which parses textual output from XCTest and formats it. This is
suboptimal, however, because it (1) assumes XCTest output is not
formatted via custom observers, and (2) establishes a dependency upon
a very particular output from XCTest, which is then hard to change.

I am concerned to find this proposal for swift-package-manager mention
output formatting, since I it doesn't seem like output formatters
belong in swift-package-manager itself, and the API for
swift-corelibs-xctest hasn't been implemented yet. I suggest deferring
the discussion on output formatting to a future proposal--it seems
like something that would be merely nice to have, whereas this
proposal states "we would like to get testing up to speed as soon as
possible."

On Fri, Jan 15, 2016 at 2:34 PM, Rick Ballard via swift-build-dev
<swift-build-dev at swift.org> wrote
>         * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I have read the proposal thoroughly.

- Brian Gesiak