[swift-evolution] Solving the issue of unit-testing precondition with the Standard Library?

[Brent Royal-Gordon]

> The forked-process approach also has the advantage of catching crashes for any other unexpected reason (eg. division by zero), and of ensuring that test executions are hermetically sealed without data leaking between test instances.

If by "forked-process approach" you mean putting code like this in XCTest:

	func XCTAssertPreconditionsPass(_ expression: Void -> Void, _ message: String) {
		let childPid = fork()
		switch childPid {
		case -1:
			XCTFail("Forking during test \(message) failed")
		case 0:
			expression()
			exit(0)
		default:
			var status: Int32 = 0
			guard waitpid(childPid, &status, 0) > 0 else {
				fatalError("XCTAssertPreconditionsPass waitpid() failed!")
			}
			if status != 0 {
				XCTFail(message)
			}
		}
	}

I don't think that's going to work in practice. Apple Foundation (and I believe CF as well) don't support forking unless you immediately exec() <http://lists.apple.com/archives/cocoa-dev/2007/Oct/msg01237.html>. Even if XCTest could communicate with the subprocesses without using Foundation, many of the tests themselves would use Foundation; having tests fail because of the test harness's implementation is probably not acceptable.

It might be possible to run whole tests, rather than individual assertions, by fork/exec-ing our harness with a command-line flag indicating the test to run, but we would need a way to mark the tests which ought to be treated this way. (Or we could run every test in a sub-process, but that seems like it would introduce significant overhead.)

With the assistance of the compiler, we might be able to have precondition() and friends throw C++ exceptions in builds with testability enabled, and then have XCTest call into some C++ glue to wrap a try/catch block around the expression. This would leave a trail of leaked objects and broken invariants in its wake, but the damage might not matter for many simple tests.

Ultimately, I think what this proposal is about is catching universal errors <https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst#universal-errors>. There's no mechanism to do that in Swift right now, but there *is* a general understanding that we'll probably need something eventually. We may not have a better option than to defer precondition testing until we have those.

-- 
Brent Royal-Gordon
Architechies