[swift-evolution] Proposal: XCTest Support for Swift Error Handling

[Chris Hanson]

We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.

  -- Chris Hanson (chanson at apple.com <mailto:chanson at apple.com>)


XCTest Support for Swift Error Handling
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Chris Hanson <https://github.com/eschaton>
Status: Review
Review manager: TBD
Introduction
Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.

Motivation
Currently, if a developer wants to use a call that may throw an error in a test, they need to use Swift's do..catch construct in their test because tests are not themselves allowed to throw errors.

As an example, a vending machine object that has had insufficient funds deposited may throw an error if asked to vend an item. A test for that situation could reasonably use the do..catchconstruct to check that this occurs as expected. However, that means all other tests also need to use either a do..catch or try! construct — and the failure of a try! is catastrophic, so do..catch would be preferred simply for better reporting within tests.

func testVendingOneItem() {
    do {
        vendingMachine.deposit(5)
        let item = try vendingMachine.vend(row: 1, column: 1)
        XCTAssertEqual(item, "Candy Bar")
    } catch {
        XCTFail("Unexpected failure: \(error)")
    }
}
If the implementation of VendingMachine.vend(row:column:) changes during development such that it throws an error in this situation, the test will fail as it should.

One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it's not one that's anticipated in the execution of the test.

In addition, tests do not currently support throwing an error from within an assertion, requiring any code that throws an error to be invoked outside the assertion itself using the same techniques described above.

Finally, since Swift error handling is a general mechanism that developers should be implementing in their own applications and frameworks, we need to make it straightforward to write tests that ensure code that implements error handling does so correctly.

Proposed solution
I propose several related solutions to this issue:

Allow test methods to throw errors.
Allow test assertion expressions to throw errors.
Add an assertion for checking errors.
These solutions combine to make writing tests that involve thrown errors much more succinct.

Allowing Test Methods to Throw Errors

First, we can allow test methods to throw errors if desired, thus allowing the do..catch construct to be omitted when the test isn't directly checking error handling. This makes the code a developer writes when they're not explicitly trying to test error handling much cleaner.

Moving the handling of errors thrown by tests into XCTest itself also ensures they can be treated as unexpected failures, since the mechanism to do so is currently private to the framework.

With this, the test from the previous section can become:

func testVendingOneItem() throws {
    vendingMachine.deposit(5)
    let item = try vendingMachine.vend(row: 1, column: 1)
    XCTAssertEqual(item, "Candy Bar")
}
This shows much more directly that the test is intended to check a specific non-error case, and that the developer is relying on the framework to handle unexpected errors.

Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}
This can eliminate otherwise-dangerous uses of try! and streamline code that needs to make multiple assertions in a row.

Adding a "Throws Error" Assertion

In order to test code that throws an error, it would be useful to have an assertion that expects an error to be thrown in a particular case. Right now a developer writing code to test that an error is thrown has to test that error themselves:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        var vendingFailed = false
        do {
            _ = try vendingMachine.vend(row: 1, column: 1))
        } catch {
            vendingFailed = true
        }
        XCTAssert(vendingFailed)
    }
If we add an assertion that specifically checks whether an error was thrown, this code will be significantly streamlined:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
    }
Of course, some code may want to just detect that an error was thrown, but other code may need to check that the details of the thrown error are correct. We can take advantage of Swift's trailing closure syntax to enable this, by passing the thrown error (if any) to a closure that can itself contain assertions:

    XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
        guard let vendingError = error as? VendingMachineError else {
            XCTFail("Unexpected type of error thrown: \(error)")
            return
        }
        
        XCTAssertEquals(vendingError.item, "Candy Bar")
        XCTAssertEquals(vendingError.price, 5)
        XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
    }
This lets a developer very concisely describe an error condition for their code, in whatever level of detail they desire.

Detailed design
The design of each of the above components is slightly different, based on the functionality provided.

Tests That Throw

In order to enable test methods to throw an error, we will need to update XCTest to support test methods with a () throws -> Void signature in addition to test methods with a () -> Voidsignature as it already supports.

We will need to ensure tests that do throw an error have that error caught, and that it registers an unexpected failure.

Assertions That Throw

In order to allow assertions to throw an exception, we will need to enhance our existing assertions' @autoclosure expression parameters to add throws to their signature.

Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not, this will not result in a combinatorial explosion of assertion overrides, and will let developers naturally write code that may throw an error within an assertion.

We will treat any error thrown from within an assertion expression as an unexpected failure because while all assertions represent a test for some form of failure, they're not specifically checking for a thrown error.

The "Throws Error" Assertion

To write tests for code that throws error, we will add a new assertion function to XCTest with the following prototype:

public func XCTAssertThrowsError(
    @autoclosure expression: () throws -> Void,
                  _ message: String = "",
                       file: StaticString = __FILE__,
                       line: UInt = __LINE__,
             _ errorHandler: (error: ErrorType) -> Void = { _ in })
Rather than treat an error thrown from its expression as a failure, this will treat the lack of an error thrown from its expression as an expected failure.

Furthermore, so long as an error is thrown, the error will be passed to the errorHandler block passed as a trailing closure, where the developer may make further assertions against it.

In both cases, the new assertion function is generic on an ErrorType in order to ensure that little to no casting will be required in the trailing closure.

Impact on existing code
There should be little impact on existing test code because we are only adding features and API, not changing existing features or API.

All existing tests should continue to work as implemented, and can easily adopt the new conventions we're making available to become more concise and intention-revealing with respect to their error handling as shown above.

Alternatives considered
We considered asking developers continue using XCTest as-is, and encouraging them to use Swift's native error handling to both suppress and check the validity of errors. We also considered adding additional ways of registering failures when doing this, so that developers could register unexpected failures themselves.

While this would result in developers using the language the same way in their tests as in their functional code, this would also result in much more verbose tests. We rejected this approach because such verbosity can be a significant obstacle to testing.

Making it quick and clean to write tests for error handling could also encourage developers to implement error handling in their code as they need it, rather than to try to work around the feature because of any perceived difficulty in testing.

We considered adding the ability to check that a specific error was thrown in XCTAssertThrowsError, but this would require the ErrorType passed to also conform to Equatable, which is also unnecessary given that this can easily be checked in a trailing closure if desired. (In some cases a developer may just want to ensure an error is thrown rather than a specific error is thrown.)

We explicitly chose not to offer a comprehensive suite of DoesNotThrowError assertions for XCTest in Swift, though we do offer such DoesNotThrow assertions for XCTest in Objective-C. We feel these are of limited utility given that our plan is for all assertions (except XCTAssertThrowsError) to treat any thrown error as a failure.

We explicitly chose not to offer any additional support for Objective-C exceptions beyond what we already provide: In the Xcode implementation of XCTest, an Objective-C exception that occurs within one of our existing assertions or tests will result in a test failure; doing more than this is not practical given that it's possible to neither catch and handle nor generate an Objective-C exception in Swift.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160109/8e036da8/attachment.html>