[swift-evolution] "Universal Error" testing method

[Joshua Marner]

I propose adding some kind of mechanism that allows better reporting in
unit tests dealing with Universal Errors
<https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst#universal-errors>
(and Logic Failures
<https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst#logic-failures>).
Currently the attitude is very RTFM-esq concerning implicit unwrap
operators and other runtime exceptions (eg, "the programmer should know
better than to use ! incorrectly"). This is very unhelpful for running
acceptance tests for large projects with many contributors, as the current
behavior will actually crash the entire test running process resulting in
no opportunities for reporting the error and logging it as a test failure
if a programmer made a mistake. This is made even more frustrating by the
fact that these are often simple fixes once they are located, but are
impossible to locate by reviewing test results or even console output in
many cases. At best, we get the name of the failing test buried in the
console output.

My proposal is to add an optional mechanism by which implicit, runtime
errors can be propagated on a "best effort" basis (starting with implicit
unwraps, asserts and array out-of-bounds array accesses) to allow calling
code to report the error. For example, we could use a keyword like "unsafe
do/catch" or something similar to enable this behavior on all code within
the block, or we could extend the "@testable import" keyword to also enable
unwinding runtime errors for the import in question. This would preserve
behavior in existing code by making it completely opt-in.

Using the "@testable import" keyword should prevent abuse of this feature
in normal applications. It also has the side benefit of encouraging the
programmer of writing tests if they get better error reporting in the
context of a testing framework.

The rollout of such a feature will be nice in that it doesn't have to be
completely implemented in order to be useful, since we are only talking
about affecting test results.

Any concerns? This feature suggestion has been borne out of much
frustration, so there are undoubtedly aspects that I missed.

~Joshua Marner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20170420/39c6b4d2/attachment.html>