[swift-evolution] [Review] SE-0019 Swift Testing (Package Manager)

[Max Howell]

> On Build Configuration/Testability
> This seems to be a fairly narrow view, but maybe it's enough.
> I think release and debug are not the only configurations, though likely the most common set.

As yet SwiftPM doesn’t support other configurations, that would have to be another proposal.

> Perhaps some sensible defaults should be implicit (written automatically in a PM file) but customizable? 
> (apologies if this exists and I missed it.)

Sorry, I don’t understand this.

> Additionally, this takes a narrow view on the nature of testing. 
> Unit tests are not the only game in town and I would like to see this left open to other kinds of automated testing.
> You might have UI testing among others. XCUITest is not the only idea here.
> You may need to test interactions and flows between systems.

The proposal details future intention to support other test frameworks and styles.

> Is there a proper place here somewhere for ensuring a home for test resources? 
> (a standardized test/ subdirectory for static/standard test data )

Not as part of the proposal. Constructing paths with __FILE__ works for SwiftPM itself currently and is enough for now IMO.

> Lovely open ended callouts on the test reporting section.
> Important for that to be malleable and replaceable. Might be JSON would be an obvious addition there…?

We can certainly build it so it’s the output engine is a protocol underneath.

JSON makes sense, but I don’t think it needs to hold up review though, a PR to add JSON after the proposal is implemented is adequate IMO.

> Lastly, should there be any built-in facility for bug/issue tracking relationships?
> Perhaps too much minutia for this proposal, but seems a good thing to include.
> In particular for reporting, any ticket/tracking numbers referring to regression tests are always good callouts in manager friendly reporting.

Sounds like an interesting and valuable addition. But again I think it can be a PR. The proposal is mostly to ensure a solid foundation that we can build on in future. We must ensure there is nothing missing or of the wrong direction at the start.

> Lastly as an addition, would it make sense to include some kind of test/code-coverage element here?

Certainly, but I think this is outside the scope of this proposal.

Thanks!

Max