[swift-build-dev] protocol for third-party testing frameworks

[Drew Crawford]

Rick Ballard has asked for a proposal for third-party testing frameworks:

> Would you mind starting a thread on swift-build-dev at swift.org about your proposal here? I'd love for us to put together a concrete proposal for how we'll support other test frameworks sooner rather than later – it sounds like you and others would like to get that support added soon!
> 
> FWIW, the main concern I have with the lightweight "just use a main.swift that runs whatever test code you want" approach is that it doesn't lead to unified test behavior for swift packages. I'd really like package users to be able to rely on being able to run the tests for all their dependencies without needing to manually install other test support first (re: @bppr 's concern), and to get parseable test output in a uniform format, so that tools like a CI dashboard, or a package index that checks package health, can easily be built on top.
> 
> Let's discuss on-list what sort of protocol we'd need to define to let you easily get up and running with a different test framework while a) making it possible for swiftPM to get all the components it needs to run your tests automatically and b) providing the test output in a uniform format. Thanks!

It doesn't look like there is a thread yet.

I have some interest in this problem since I have problems such that it isn't clear whether they can be solved within the mandate of XCTest.  So I am potentially a test framework author, if only as an insurance policy against fixing XCTest to not suck.

I see some places for test framework to innovate, notably:

Parallelization of tests.  Swift language tests use all cores, which is AWESOME.  XCTest does not, which is sad.  But a general-purpose threading model for all software is not clear.  This is a good problem for third-party frameworks to study and try out different things.
Configuring what tests are to be performed.  Group them by suites?  Burn-in for 10 minutes, regardless of the number of tests?  Turn off and on tests in the sourcecode?
Complex pass/fail conditions.  XCTest is notoriously bad at failing an entire run because of one performance outlier.
Data reporting.  Performance tests might have various unusual metrics etc. that should be reported.  For example, I have tests that have metrics measured in "bytes copied".

Based on those requirements, I propose the following API as a starting point of discussion.

protocol TestFramework {
    /**This function is called by SwiftPM to begin all tests
    A TestFramework discovers tests and reports them to the delegate.
    It performs the tests, keeping the delegate updated in realtime.
    Finally, it calls delegate.finish() */
    func begin()
    
    ///A test framework reports back data to SwiftPM (or other data gatherer) here
    var delegate: TestFrameworkDelegate { get set }
}

protocol Test {
    /**A unique name.  Reverse-domain notation is encouraged, and XCTest should use these to represent suites. */
    var name : String { get }
    
    /**The current status of the test */
    var status: TestStatus { get }
    
    /**Metrics that are standardized by this specification.  These metrics are well-known and interoperable between tools */
    var metrics: [StandardMetric] { get }
    
    /**A test may have other metrics, defined by third parties. */
    var userMetrics: [UserMetric] { get }
}

protocol TestFrameworkDelegate {
    /**Call this function to report an update to the test.
     
The implementation of this fuction should scan the ivars of the Test for a new name / status / metrics and update UX or reporting appropriately.
- warning: The implementation of this function must be threadsafe.
    */
    func updateTest(test: Test)
    
    /**Create a log event associated with the test, which is appended to the event log.  
     
SwiftPM uses this information to collate logs in the case that multiple tests are executing simultaneously.
- warning: The implementation of this function must be threadsafe.  In the case that the same test logs multiple events simultaneously, the order in which they are appended to the log is undefined.
*/
    func log(test: Test, event: String)
    
    /**Call this function to indicate that a test framework has discovered all tests and they have been updated to Planned status.  Novel tests will no longer be discovered.

The use of this method is optional, since some test frameworks may not plan a fixed number of tests.
This function can be used by UIs that want a complete list of tests for some reason (e.g. to put them in a UITableView). */
    func allTestsPlanned()
    
    /**Indicates that testing is complete and no more delegate callbacks will be made.
- parameter status: The status of the overall testing framework.  In general this is .Failed if any of the underlying tests failed, but a test framework may choose to apply custom logic to rule on its underlying tests.
     */
    func finish(status: TestStatus)
}

enum TestStatus {
    case Planned ///The test has been discovered
    case Running ///The test is currently executing
    case Passed ///The test finished successfully
    case Failed(ErrorType) ///The test failed for the specified reason
    case Skipped(String) ///The test will not execute.  The string contains more information (e.g. wrong suite, ran out of time, etc.)
    case Indeterminate ///The test executed, and ended in a state that is neither a pass nor a failure
    case Orange ///The test ended in a state that  may warrant further investigation but is not automatically ruled a failure.  https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor
}

enum StandardMetric {
    /// a list of test executions, with the number of seconds of each execution.  In XCTest's situation, this is 10 doubles, one for each run.
    case Runtime([Double])
    
    ///A unique string that identifies the "performance class" of the machine, such as "iPhone 6s".  Equal strings mean test results are from a machine in the same performance class and so are directly comparable.  An unequal string means they are not comparable.
    case PerformanceClass(String)

}

protocol UserMetric {}

There is an extremely straightforward implementation of this API for XCTest.  Simply loop over the tests and update their status.

However this API is much more powerful than XCTest and would allow considerable experimentation for third-party frameworks on matters of parallelization, test discovery, pass/fail logic, etc., and so I think it is more agnostic and free of XCTest assumptions, which Kyle Fuller expressed concerns about, and I share.

Meanwhile the reporting APIs here makes sure we have parseable tests in a uniform format that UX tools can work with as Rick Ballard was concerned about.

I certainly think there are more ideas to consider here, but this is a reasonable first draft to have a conversation about.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20151229/e368d8c3/attachment.html>