<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Rick Ballard has asked for a proposal for third-party testing frameworks:<div class=""><br class=""></div><div class=""><blockquote type="cite" class="">Would you mind starting a thread on <a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a> about your proposal here? I'd love for us to put together a concrete proposal for how we'll support other test frameworks sooner rather than later – it sounds like you and others would like to get that support added soon!<br class=""><br class="">FWIW, the main concern I have with the lightweight "just use a main.swift that runs whatever test code you want" approach is that it doesn't lead to unified test behavior for swift packages. I'd really like package users to be able to rely on being able to run the tests for all their dependencies without needing to manually install other test support first (re: @bppr 's concern), and to get parseable test output in a uniform format, so that tools like a CI dashboard, or a package index that checks package health, can easily be built on top.<br class=""><br class="">Let's discuss on-list what sort of protocol we'd need to define to let you easily get up and running with a different test framework while a) making it possible for swiftPM to get all the components it needs to run your tests automatically and b) providing the test output in a uniform format. Thanks!</blockquote><br class=""></div><div class="">It doesn't look like there is a thread yet.</div><div class=""><br class=""></div><div class="">I have some interest in this problem since I have problems such that it isn't clear whether they can be solved within the mandate of XCTest. So I am potentially a test framework author, if only as an insurance policy against fixing XCTest to not suck.</div><div class=""><br class=""></div><div class="">I see some places for test framework to innovate, notably:</div><div class=""><br class=""></div><div class=""><ul class="MailOutline"><li class="">Parallelization of tests. Swift language tests use all cores, which is AWESOME. XCTest does not, which is sad. But a general-purpose threading model for all software is not clear. This is a good problem for third-party frameworks to study and try out different things.</li><li class="">Configuring what tests are to be performed. Group them by suites? Burn-in for 10 minutes, regardless of the number of tests? Turn off and on tests in the sourcecode?</li><li class="">Complex pass/fail conditions. XCTest is notoriously bad at failing an entire run because of one performance outlier.</li><li class="">Data reporting. Performance tests might have various unusual metrics etc. that should be reported. For example, I have tests that have metrics measured in "bytes copied".</li></ul><div class=""><br class=""></div></div><div class="">Based on those requirements, I propose the following API as a starting point of discussion.</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">protocol</span> TestFramework {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**This function is called by SwiftPM to begin all tests</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> A TestFramework discovers tests and reports them to the delegate.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> It performs the tests, keeping the delegate updated in realtime.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> Finally, it calls delegate.finish() */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> begin()</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>///A test framework reports back data to SwiftPM (or other data gatherer) here</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> delegate: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">TestFrameworkDelegate</span> { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">set</span> }</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(187, 44, 162);" class="">protocol<span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Test {</span></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**A unique name. Reverse-domain notation is encouraged, and XCTest should use these to represent suites. */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> name : <span style="font-variant-ligatures: no-common-ligatures; color: #703daa" class="">String</span> { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**The current status of the test */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> status: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">TestStatus</span> { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**Metrics that are standardized by this specification. These metrics are well-known and interoperable between tools */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> metrics: [<span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">StandardMetric</span>] { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**A test may have other metrics, defined by third parties. */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> userMetrics: [<span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">UserMetric</span>] { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">protocol</span> TestFrameworkDelegate {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**Call this function to report an update to the test.</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">The implementation of this fuction should scan the ivars of the Test for a new name / status / metrics and update UX or reporting appropriately.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">- warning: The implementation of this function must be threadsafe.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> updateTest(test: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">Test</span>)</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**Create a log event associated with the test, which is appended to the event log. </div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">SwiftPM uses this information to collate logs in the case that multiple tests are executing simultaneously.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">- warning: The implementation of this function must be threadsafe. In the case that the same test logs multiple events simultaneously, the order in which they are appended to the log is undefined.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">*/</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> log(test: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">Test</span>, event: <span style="font-variant-ligatures: no-common-ligatures; color: #703daa" class="">String</span>)</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**Call this function to indicate that a test framework has discovered all tests and they have been updated to Planned status. Novel tests will no longer be discovered.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">The use of this method is optional, since some test frameworks may not plan a fixed number of tests.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">This function can be used by UIs that want a complete list of tests for some reason (e.g. to put them in a UITableView). */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> allTestsPlanned()</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/**Indicates that testing is complete and no more delegate callbacks will be made.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">- parameter status: The status of the overall testing framework. In general this is .Failed if any of the underlying tests failed, but a test framework may choose to apply custom logic to rule on its underlying tests.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> finish(status: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">TestStatus</span>)</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">enum</span> TestStatus {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Planned </span>///The test has been discovered</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Running </span>///The test is currently executing</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Passed </span>///The test finished successfully</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Failed(ErrorType) </span>///The test failed for the specified reason</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Skipped(String) </span>///The test will not execute. The string contains more information (e.g. wrong suite, ran out of time, etc.)</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Indeterminate </span>///The test executed, and ended in a state that is neither a pass nor a failure</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> Orange </span>///The test ended in a state that may warrant further investigation but is not automatically ruled a failure. <a href="https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor" class="">https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor</a></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">enum</span> StandardMetric {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>/// a list of test executions, with the number of seconds of each execution. In XCTest's situation, this is 10 doubles, one for each run.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span> Runtime([Double])</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> </p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #000000" class=""> </span>///A unique string that identifies the "performance class" of the machine, such as "iPhone 6s". Equal strings mean test results are from a machine in the same performance class and so are directly comparable. An unequal string means they are not comparable.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span> PerformanceClass(String)</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">protocol</span> UserMetric {}</div></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><br class=""></div></blockquote>There is an extremely straightforward implementation of this API for XCTest. Simply loop over the tests and update their status.<div class=""><br class=""></div><div class="">However this API is much more powerful than XCTest and would allow considerable experimentation for third-party frameworks on matters of parallelization, test discovery, pass/fail logic, etc., and so I think it is more agnostic and free of XCTest assumptions, which Kyle Fuller expressed concerns about, and I share.</div><div class=""><br class=""></div><div class="">Meanwhile the reporting APIs here makes sure we have parseable tests in a uniform format that UX tools can work with as Rick Ballard was concerned about.</div><div class=""><br class=""></div><div class="">I certainly think there are more ideas to consider here, but this is a reasonable first draft to have a conversation about.</div></body></html>