[swift-build-dev] protocol for third-party testing frameworks

[Max Howell]

> I have some interest in this problem since I have problems such that it isn't clear whether they can be solved within the mandate of XCTest.  So I am potentially a test framework author, if only as an insurance policy against fixing XCTest to not suck.
> 
> I see some places for test framework to innovate, notably:
> 
> Parallelization of tests.  Swift language tests use all cores, which is AWESOME.  XCTest does not, which is sad.  But a general-purpose threading model for all software is not clear.  This is a good problem for third-party frameworks to study and try out different things.

Sounds great.

> Configuring what tests are to be performed.  Group them by suites?  Burn-in for 10 minutes, regardless of the number of tests?  Turn off and on tests in the source code?

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

> Complex pass/fail conditions.  XCTest is notoriously bad at failing an entire run because of one performance outlier.

Sounds good. Would fit into a “protocol” design easily enough.

> Data reporting.  Performance tests might have various unusual metrics etc. that should be reported.  For example, I have tests that have metrics measured in "bytes copied”.

Sounds good.

> Based on those requirements, I propose the following API as a starting point of discussion.
> 
> protocol TestFramework {
>     /**This function is called by SwiftPM to begin all tests
>     A TestFramework discovers tests and reports them to the delegate.
>     It performs the tests, keeping the delegate updated in realtime.
>     Finally, it calls delegate.finish() */
>     func begin()
>     
>     ///A test framework reports back data to SwiftPM (or other data gatherer) here
>     var delegate: TestFrameworkDelegate { get set }
> }
> 
> protocol Test {
>     /**A unique name.  Reverse-domain notation is encouraged, and XCTest should use these to represent suites. */
>     var name : String { get }
>     
>     /**The current status of the test */
>     var status: TestStatus { get }
>     
>     /**Metrics that are standardized by this specification.  These metrics are well-known and interoperable between tools */
>     var metrics: [StandardMetric] { get }
>     
>     /**A test may have other metrics, defined by third parties. */
>     var userMetrics: [UserMetric] { get }
> }
> 
> protocol TestFrameworkDelegate {
>     /**Call this function to report an update to the test.
>      
> The implementation of this fuction should scan the ivars of the Test for a new name / status / metrics and update UX or reporting appropriately.
> - warning: The implementation of this function must be threadsafe.
>     */
>     func updateTest(test: Test)
>     
>     /**Create a log event associated with the test, which is appended to the event log.  
>      
> SwiftPM uses this information to collate logs in the case that multiple tests are executing simultaneously.
> - warning: The implementation of this function must be threadsafe.  In the case that the same test logs multiple events simultaneously, the order in which they are appended to the log is undefined.
> */
>     func log(test: Test, event: String)
>     
>     /**Call this function to indicate that a test framework has discovered all tests and they have been updated to Planned status.  Novel tests will no longer be discovered.
> 
> The use of this method is optional, since some test frameworks may not plan a fixed number of tests.
> This function can be used by UIs that want a complete list of tests for some reason (e.g. to put them in a UITableView). */
>     func allTestsPlanned()
>     
>     /**Indicates that testing is complete and no more delegate callbacks will be made.
> - parameter status: The status of the overall testing framework.  In general this is .Failed if any of the underlying tests failed, but a test framework may choose to apply custom logic to rule on its underlying tests.
>      */
>     func finish(status: TestStatus)
> }
> 
> enum TestStatus {
>     case Planned ///The test has been discovered
>     case Running ///The test is currently executing
>     case Passed ///The test finished successfully
>     case Failed(ErrorType) ///The test failed for the specified reason
>     case Skipped(String) ///The test will not execute.  The string contains more information (e.g. wrong suite, ran out of time, etc.)
>     case Indeterminate ///The test executed, and ended in a state that is neither a pass nor a failure
>     case Orange ///The test ended in a state that  may warrant further investigation but is not automatically ruled a failure.  https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor <https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor>
> }
> 
> enum StandardMetric {
>     /// a list of test executions, with the number of seconds of each execution.  In XCTest's situation, this is 10 doubles, one for each run.
>     case Runtime([Double])
>     
>     ///A unique string that identifies the "performance class" of the machine, such as "iPhone 6s".  Equal strings mean test results are from a machine in the same performance class and so are directly comparable.  An unequal string means they are not comparable.
>     case PerformanceClass(String)
> 
> }
> 
> protocol UserMetric {}
> 
> There is an extremely straightforward implementation of this API for XCTest.  Simply loop over the tests and update their status.

My initial glance thinks this could be simpler, but maybe not. I’d have to give more time to this and I don’t right now have it. Certainly I think it is a great start.

> However this API is much more powerful than XCTest and would allow considerable experimentation for third-party frameworks on matters of parallelization, test discovery, pass/fail logic, etc., and so I think it is more agnostic and free of XCTest assumptions, which Kyle Fuller expressed concerns about, and I share.
> 
> Meanwhile the reporting APIs here makes sure we have parseable tests in a uniform format that UX tools can work with as Rick Ballard was concerned about.
> 
> I certainly think there are more ideas to consider here, but this is a reasonable first draft to have a conversation about.

I’d like us to try and map existing popular testing styles onto this protocol before we right up the full proposal.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160104/2233b449/attachment.html>