[swift-corelibs-dev] [xctest] Who tests the tests?

[Brian Gesiak]

Hello! This is in reference to
https://github.com/apple/swift-corelibs-xctest/pull/3. That pull request
contains a commit that attempts to refactor XCTest such that it is more
"unit-testable".

To do so, it gives XCTMain an additional parameter: a list of objects
conforming to the Reporter protocol. I think of this as a minimal, corelibs
equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because
Reporter only defines Reporter.log(), whereas XCTestObserver has one method
for each kind of test event (started, failed, finished, etc.).

These reporters are, for now, storied in a global array. In the future, I'd
like to discuss moving XCTest to a model in which all tests are
(optionally) run in sub-processes, each of which may (optionally) run in
parallel. This global array most certainly won't work for such a change,
but for now, I simply want to have regression tests on the project. It's
hard to send pull requests without knowing everything still works!

Besides this approach, which modifies XCTest in order to test it, it may be
more prudent to add tests *without* changing XCTest at all. To do so, I
could add tests that run programs that call XCTMain(), then verify what's
printed to stdout. This could be done using a Python script (which would go
well with the build script, also in Python).

I'd love input on which of these approaches sounds more viable. Other ideas
are also, of course, welcome!

- Brian Gesiak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-corelibs-dev/attachments/20151203/bf770f6b/attachment.html>