[swift-corelibs-dev] [xctest] Who tests the tests?

[Daniel Dunbar]

Hi Brian,

> On Dec 3, 2015, at 3:45 PM, Brian Gesiak <modocache at gmail.com> wrote:
> 
> Hello! This is in reference to https://github.com/apple/swift-corelibs-xctest/pull/3 <https://github.com/apple/swift-corelibs-xctest/pull/3>. That pull request contains a commit that attempts to refactor XCTest such that it is more "unit-testable".
> 
> To do so, it gives XCTMain an additional parameter: a list of objects conforming to the Reporter protocol. I think of this as a minimal, corelibs equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because Reporter only defines Reporter.log(), whereas XCTestObserver has one method for each kind of test event (started, failed, finished, etc.).
> 
> These reporters are, for now, storied in a global array. In the future, I'd like to discuss moving XCTest to a model in which all tests are (optionally) run in sub-processes, each of which may (optionally) run in parallel. This global array most certainly won't work for such a change, but for now, I simply want to have regression tests on the project. It's hard to send pull requests without knowing everything still works!
> 
> Besides this approach, which modifies XCTest in order to test it, it may be more prudent to add tests *without* changing XCTest at all. To do so, I could add tests that run programs that call XCTMain(), then verify what's printed to stdout. This could be done using a Python script (which would go well with the build script, also in Python).

It should be possible to use an out-of-process model that still uses XCTest itself to run the tests. For example, in the package manager we have some tests which spawn the package manager in order to test the end-to-end behavior. Ideally we would only do this for a small number of tests that really need this level of testing, and use unit testing for the rest.

To be more concrete, what I am proposing here is that we have test cases which amount to:
--
class SomeTestCase : XCSelfTestCase {
  func testFailure() {
     withTestCase("Inputs/failing-test.swift") { results in
       ... check the results
     }
  }
}
--
where "Inputs/failing-test.swift" would be a checked in input file which has some expected behavior, and withTestCase() would be infrastructure (in XCSelfTestCase) which spawns a process to build and run that test, and then capture the output into a results object which could have further assertions run against it.

 - Daniel

> 
> I'd love input on which of these approaches sounds more viable. Other ideas are also, of course, welcome!
> 
> - Brian Gesiak
>  _______________________________________________
> swift-corelibs-dev mailing list
> swift-corelibs-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-corelibs-dev/attachments/20151203/85d05828/attachment.html>