[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Paul Cantrell]

> On Jan 8, 2016, at 1:57 PM, Max Howell <max.howell at apple.com> wrote:
> 
> We will iterate.
> …
> We can reverse decisions. For example, building tests by default is controversial, and perhaps it is unwise. We’ll find out in due course, I for one have no problem with reversing a decision provided it is proved incorrect.

That’s good to know.

Really, my reaction to this whole proposal is that I won’t really know until I try it, so let’s try it!

>> lack of clarity about how to run individual test modules from the command line,
> 
> I imagine it would be `swift test foo` where foo is the test module name.

Disambiguating between test module names and test names would need some thought then, but probably that’s easily solved.

>> lack of clarity about which test modules are run by default with “swift test” (“all of them” is probably the wrong answer),
> 
> Why is all of them the wrong answer? It seems to be the only sensible default to me. Clearly there should be a way to configure that in Package.swift, but that is for another proposal.

It depends on what all those different test modules are.

One possibility is that they’re arbitrary functional groupings: Tests/Validation, Tests/Computation, Tests/Formatting. In that case, “all of them” is right. But I don’t know anybody who organizes test modules at the top level like that.

Another possibility is that they correspond to targets/submodules under Sources: Tests/Core, Tests/FancyAddOn, Tests/UIKitExtensions, Tests/WatchKitExtensions. In that case, the default would be to build & run the tests for whatever source module(s) you’re building.

A third possibility is that they correspond to different _kinds_ of testing: Tests/Regression, Tests/Integration, Tests/Performance. In this example, the integration tests might require some kind of environment setup, the performance tests might take 60 seconds to run, and so you’d only want the regression tests to run by default.

OK, so maybe “all of them” is the default and it’s overridable … but I tend to think that there should be a default of Sources/Foo running only Tests/Foo by default. Something along those lines.

Cheers, P

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160108/66ad6959/attachment.html>