<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div><blockquote type="cite" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I have some interest in this problem since I have problems such that it isn't clear whether they can be solved within the mandate of XCTest. So I am potentially a test framework author, if only as an insurance policy against fixing XCTest to not suck.</div><div class=""><br class=""></div><div class="">I see some places for test framework to innovate, notably:</div><div class=""><br class=""></div><div class=""><ul class="MailOutline"><li class="">Parallelization of tests. Swift language tests use all cores, which is AWESOME. XCTest does not, which is sad. But a general-purpose threading model for all software is not clear. This is a good problem for third-party frameworks to study and try out different things.</li></ul></div></div></blockquote><div><br class=""></div><div>Sounds great.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><ul class="MailOutline"><li class="">Configuring what tests are to be performed. Group them by suites? Burn-in for 10 minutes, regardless of the number of tests? Turn off and on tests in the source code?</li></ul></div></div></div></blockquote><div><br class=""></div><div>I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><ul class="MailOutline"><li class="">Complex pass/fail conditions. XCTest is notoriously bad at failing an entire run because of one performance outlier.</li></ul></div></div></div></blockquote><div><br class=""></div><div>Sounds good. Would fit into a “protocol” design easily enough.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><ul class="MailOutline"><li class="">Data reporting. Performance tests might have various unusual metrics etc. that should be reported. For example, I have tests that have metrics measured in "bytes copied”.</li></ul></div></div></div></blockquote><div><br class=""></div><div>Sounds good.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Based on those requirements, I propose the following API as a starting point of discussion.</div><div class=""><br class=""></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">protocol</span> TestFramework {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**This function is called by SwiftPM to begin all tests</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> A TestFramework discovers tests and reports them to the delegate.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> It performs the tests, keeping the delegate updated in realtime.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> Finally, it calls delegate.finish() */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> begin()</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>///A test framework reports back data to SwiftPM (or other data gatherer) here</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> delegate: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">TestFrameworkDelegate</span> { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">set</span> }</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(187, 44, 162);" class="">protocol<span style="font-variant-ligatures: no-common-ligatures;" class=""> Test {</span></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**A unique name. Reverse-domain notation is encouraged, and XCTest should use these to represent suites. */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> name : <span style="font-variant-ligatures: no-common-ligatures; color: #703daa" class="">String</span> { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**The current status of the test */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> status: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">TestStatus</span> { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**Metrics that are standardized by this specification. These metrics are well-known and interoperable between tools */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> metrics: [<span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">StandardMetric</span>] { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**A test may have other metrics, defined by third parties. */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">var</span> userMetrics: [<span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">UserMetric</span>] { <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">get</span> }</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">protocol</span> TestFrameworkDelegate {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**Call this function to report an update to the test.</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">The implementation of this fuction should scan the ivars of the Test for a new name / status / metrics and update UX or reporting appropriately.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">- warning: The implementation of this function must be threadsafe.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> updateTest(test: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">Test</span>)</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**Create a log event associated with the test, which is appended to the event log. </div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">SwiftPM uses this information to collate logs in the case that multiple tests are executing simultaneously.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">- warning: The implementation of this function must be threadsafe. In the case that the same test logs multiple events simultaneously, the order in which they are appended to the log is undefined.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">*/</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> log(test: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">Test</span>, event: <span style="font-variant-ligatures: no-common-ligatures; color: #703daa" class="">String</span>)</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**Call this function to indicate that a test framework has discovered all tests and they have been updated to Planned status. Novel tests will no longer be discovered.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">The use of this method is optional, since some test frameworks may not plan a fixed number of tests.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">This function can be used by UIs that want a complete list of tests for some reason (e.g. to put them in a UITableView). */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> allTestsPlanned()</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/**Indicates that testing is complete and no more delegate callbacks will be made.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class="">- parameter status: The status of the overall testing framework. In general this is .Failed if any of the underlying tests failed, but a test framework may choose to apply custom logic to rule on its underlying tests.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""> */</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">func</span> finish(status: <span style="font-variant-ligatures: no-common-ligatures; color: #4f8187" class="">TestStatus</span>)</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">enum</span> TestStatus {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> Planned </span>///The test has been discovered</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> Running </span>///The test is currently executing</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> Passed </span>///The test finished successfully</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> Failed(ErrorType) </span>///The test failed for the specified reason</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> Skipped(String) </span>///The test will not execute. The string contains more information (e.g. wrong suite, ran out of time, etc.)</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> Indeterminate </span>///The test executed, and ended in a state that is neither a pass nor a failure</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span><span style="font-variant-ligatures: no-common-ligatures;" class=""> Orange </span>///The test ended in a state that may warrant further investigation but is not automatically ruled a failure. <a href="https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor" class="">https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor</a></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">enum</span> StandardMetric {</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>/// a list of test executions, with the number of seconds of each execution. In XCTest's situation, this is 10 doubles, one for each run.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span> Runtime([Double])</div></div><div class=""><p style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""> <br class="webkit-block-placeholder"></p></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> </span>///A unique string that identifies the "performance class" of the machine, such as "iPhone 6s". Equal strings mean test results are from a machine in the same performance class and so are directly comparable. An unequal string means they are not comparable.</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""> <span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">case</span> PerformanceClass(String)</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class="">}</div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; min-height: 13px;" class=""><br class=""></div></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><span style="font-variant-ligatures: no-common-ligatures; color: #bb2ca2" class="">protocol</span> UserMetric {}</div></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo;" class=""><br class=""></div></blockquote>There is an extremely straightforward implementation of this API for XCTest. Simply loop over the tests and update their status.</div></div></blockquote><div><br class=""></div><div>My initial glance thinks this could be simpler, but maybe not. I’d have to give more time to this and I don’t right now have it. Certainly I think it is a great start.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">However this API is much more powerful than XCTest and would allow considerable experimentation for third-party frameworks on matters of parallelization, test discovery, pass/fail logic, etc., and so I think it is more agnostic and free of XCTest assumptions, which Kyle Fuller expressed concerns about, and I share.</div><div class=""><br class=""></div><div class="">Meanwhile the reporting APIs here makes sure we have parseable tests in a uniform format that UX tools can work with as Rick Ballard was concerned about.</div><div class=""><br class=""></div><div class="">I certainly think there are more ideas to consider here, but this is a reasonable first draft to have a conversation about.</div></div></div></blockquote><br class=""></div><div>I’d like us to try and map existing popular testing styles onto this protocol before we right up the full proposal.</div><div><br class=""></div><div>I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.</div><br class=""></body></html>