[swift-evolution] Proposal: XCTest Support for Swift Error Handling
Chris Hanson
chanson at apple.com
Sun Jan 10 18:34:23 CST 2016
The degree of change to which XCTest is open via the Swift process would be best discussed as a separate thread on either swift-corelibs-xctest or swift-evolution. Please keep this thread to discussion of the error handling proposal. Thanks.
-- Chris
Sent from my iPad
> On Jan 10, 2016, at 4:34 AM, James Campbell <james at supmenow.com> wrote:
>
> I would love it if we could do a full review of XCtest in general. As there are other things it could help with I.r mocking or allowing us to express tests in a BDD way
>
> Sent from my iPhone
>
>> On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <swift-evolution at swift.org> wrote:
>>
>> I have on the order of ~700 tests in XCTest in Swift-language projects. I'm considering migrating away from XCTest, although not over this issue.
>>
>> This proposal IMO addresses an important problem, but I am not convinced it actually solves it. #2 & #3 are basically sound API designs. It is a mystery to me why #3 "generated some debate" as this is a feature I already implement manually, but I can't address unknown concerns. I can tell you I implement this, and nothing terrible has happened to me so far.
>>
>> #1 I would not use. The rest of this comment explains why.
>>
>> Currently I write tests about like this:
>>
>> try! hopefullyNothingBad()
>>
>> Now this is "bad" because it "crashes" but that's (big sigh) actually "good" because the debugger stops and/or I get a crash report that identifies at least "some" line where something bad definitely happened.
>>
>> Now that is not everything I want to know–I want someone to tell me a story of how this error was created deep down in the bowels of my application, how it spent its kindergarten years in the models layer before being passed into a controller where it was rethrown onto a different dispatch queue and finally ended up in my unit test–but we can't have everything. So I settle for collecting a line number from the test case and then going hunting by hand.
>>
>> When the test function throws we no longer even find out a line number in the test case anymore, because the error is passed into XCTest and the information is lost. We have just the name of the test case (I assume; the proposal is silent on this issue, but that's the only way I can think of to implement it), and some of my tests are pretty long. So, that makes it even harder to track down.
>>
>> This sounds like a small thing but my test coverage is so thorough on mature projects that mostly what I turn up are heisenbugs that reproduce with 2% probability. So getting the report from the CI that has the most possible detail is critical, because if the report is not thorough enough for you to guess the bug, too bad, because that's all the information you get and the bug is not reproducible.
>>
>> For that reason, I won't use #1. I hesitate about whether to call it bad idea altogether, or whether it's just not to my taste. My sense is it's probably somewhere in the middle of those two poles.
>>
>> I would use #2 and #3, assuming that I don't first migrate out to a non-XCTest framework.
>>
>>
>>
>>> On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <swift-evolution at swift.org> wrote:
>>>
>>> We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.
>>>
>>> We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.
>>>
>>> -- Chris Hanson (chanson at apple.com)
>>>
>>>
>>> XCTest Support for Swift Error Handling
>>> Proposal: SE-NNNN
>>> Author(s): Chris Hanson
>>> Status: Review
>>> Review manager: TBD
>>> Introduction
>>> Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.
>>>
>>> Motivation
>>> Currently, if a developer wants to use a call that may throw an error in a test, they need to use Swift's do..catch construct in their test because tests are not themselves allowed to throw errors.
>>>
>>> As an example, a vending machine object that has had insufficient funds deposited may throw an error if asked to vend an item. A test for that situation could reasonably use the do..catchconstruct to check that this occurs as expected. However, that means all other tests also need to use either a do..catch or try! construct — and the failure of a try! is catastrophic, so do..catch would be preferred simply for better reporting within tests.
>>>
>>> func testVendingOneItem() {
>>> do {
>>> vendingMachine.deposit(5)
>>> let item = try vendingMachine.vend(row: 1, column: 1)
>>> XCTAssertEqual(item, "Candy Bar")
>>> } catch {
>>> XCTFail("Unexpected failure: \(error)")
>>> }
>>> }
>>> If the implementation of VendingMachine.vend(row:column:) changes during development such that it throws an error in this situation, the test will fail as it should.
>>>
>>> One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it's not one that's anticipated in the execution of the test.
>>>
>>> In addition, tests do not currently support throwing an error from within an assertion, requiring any code that throws an error to be invoked outside the assertion itself using the same techniques described above.
>>>
>>> Finally, since Swift error handling is a general mechanism that developers should be implementing in their own applications and frameworks, we need to make it straightforward to write tests that ensure code that implements error handling does so correctly.
>>>
>>> Proposed solution
>>> I propose several related solutions to this issue:
>>>
>>> Allow test methods to throw errors.
>>> Allow test assertion expressions to throw errors.
>>> Add an assertion for checking errors.
>>> These solutions combine to make writing tests that involve thrown errors much more succinct.
>>>
>>> Allowing Test Methods to Throw Errors
>>>
>>> First, we can allow test methods to throw errors if desired, thus allowing the do..catch construct to be omitted when the test isn't directly checking error handling. This makes the code a developer writes when they're not explicitly trying to test error handling much cleaner.
>>>
>>> Moving the handling of errors thrown by tests into XCTest itself also ensures they can be treated as unexpected failures, since the mechanism to do so is currently private to the framework.
>>>
>>> With this, the test from the previous section can become:
>>>
>>> func testVendingOneItem() throws {
>>> vendingMachine.deposit(5)
>>> let item = try vendingMachine.vend(row: 1, column: 1)
>>> XCTAssertEqual(item, "Candy Bar")
>>> }
>>> This shows much more directly that the test is intended to check a specific non-error case, and that the developer is relying on the framework to handle unexpected errors.
>>>
>>> Allowing Test Assertions to Throw Errors
>>>
>>> We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:
>>>
>>> func testVendingMultipleItemsWithSufficientFunds() {
>>> vendingMachine.deposit(10)
>>> XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
>>> XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
>>> }
>>> This can eliminate otherwise-dangerous uses of try! and streamline code that needs to make multiple assertions in a row.
>>>
>>> Adding a "Throws Error" Assertion
>>>
>>> In order to test code that throws an error, it would be useful to have an assertion that expects an error to be thrown in a particular case. Right now a developer writing code to test that an error is thrown has to test that error themselves:
>>>
>>> func testVendingFailsWithInsufficientFunds() {
>>> vendingMachine.deposit(1)
>>> var vendingFailed = false
>>> do {
>>> _ = try vendingMachine.vend(row: 1, column: 1))
>>> } catch {
>>> vendingFailed = true
>>> }
>>> XCTAssert(vendingFailed)
>>> }
>>> If we add an assertion that specifically checks whether an error was thrown, this code will be significantly streamlined:
>>>
>>> func testVendingFailsWithInsufficientFunds() {
>>> vendingMachine.deposit(1)
>>> XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
>>> }
>>> Of course, some code may want to just detect that an error was thrown, but other code may need to check that the details of the thrown error are correct. We can take advantage of Swift's trailing closure syntax to enable this, by passing the thrown error (if any) to a closure that can itself contain assertions:
>>>
>>> XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
>>> guard let vendingError = error as? VendingMachineError else {
>>> XCTFail("Unexpected type of error thrown: \(error)")
>>> return
>>> }
>>>
>>> XCTAssertEquals(vendingError.item, "Candy Bar")
>>> XCTAssertEquals(vendingError.price, 5)
>>> XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
>>> }
>>> This lets a developer very concisely describe an error condition for their code, in whatever level of detail they desire.
>>>
>>> Detailed design
>>> The design of each of the above components is slightly different, based on the functionality provided.
>>>
>>> Tests That Throw
>>>
>>> In order to enable test methods to throw an error, we will need to update XCTest to support test methods with a () throws -> Void signature in addition to test methods with a () -> Voidsignature as it already supports.
>>>
>>> We will need to ensure tests that do throw an error have that error caught, and that it registers an unexpected failure.
>>>
>>> Assertions That Throw
>>>
>>> In order to allow assertions to throw an exception, we will need to enhance our existing assertions' @autoclosure expression parameters to add throws to their signature.
>>>
>>> Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not, this will not result in a combinatorial explosion of assertion overrides, and will let developers naturally write code that may throw an error within an assertion.
>>>
>>> We will treat any error thrown from within an assertion expression as an unexpected failure because while all assertions represent a test for some form of failure, they're not specifically checking for a thrown error.
>>>
>>> The "Throws Error" Assertion
>>>
>>> To write tests for code that throws error, we will add a new assertion function to XCTest with the following prototype:
>>>
>>> public func XCTAssertThrowsError(
>>> @autoclosure expression: () throws -> Void,
>>> _ message: String = "",
>>> file: StaticString = __FILE__,
>>> line: UInt = __LINE__,
>>> _ errorHandler: (error: ErrorType) -> Void = { _ in })
>>> Rather than treat an error thrown from its expression as a failure, this will treat the lack of an error thrown from its expression as an expected failure.
>>>
>>> Furthermore, so long as an error is thrown, the error will be passed to the errorHandler block passed as a trailing closure, where the developer may make further assertions against it.
>>>
>>> In both cases, the new assertion function is generic on an ErrorType in order to ensure that little to no casting will be required in the trailing closure.
>>>
>>> Impact on existing code
>>> There should be little impact on existing test code because we are only adding features and API, not changing existing features or API.
>>>
>>> All existing tests should continue to work as implemented, and can easily adopt the new conventions we're making available to become more concise and intention-revealing with respect to their error handling as shown above.
>>>
>>> Alternatives considered
>>> We considered asking developers continue using XCTest as-is, and encouraging them to use Swift's native error handling to both suppress and check the validity of errors. We also considered adding additional ways of registering failures when doing this, so that developers could register unexpected failures themselves.
>>>
>>> While this would result in developers using the language the same way in their tests as in their functional code, this would also result in much more verbose tests. We rejected this approach because such verbosity can be a significant obstacle to testing.
>>>
>>> Making it quick and clean to write tests for error handling could also encourage developers to implement error handling in their code as they need it, rather than to try to work around the feature because of any perceived difficulty in testing.
>>>
>>> We considered adding the ability to check that a specific error was thrown in XCTAssertThrowsError, but this would require the ErrorType passed to also conform to Equatable, which is also unnecessary given that this can easily be checked in a trailing closure if desired. (In some cases a developer may just want to ensure an error is thrown rather than a specific error is thrown.)
>>>
>>> We explicitly chose not to offer a comprehensive suite of DoesNotThrowError assertions for XCTest in Swift, though we do offer such DoesNotThrow assertions for XCTest in Objective-C. We feel these are of limited utility given that our plan is for all assertions (except XCTAssertThrowsError) to treat any thrown error as a failure.
>>>
>>> We explicitly chose not to offer any additional support for Objective-C exceptions beyond what we already provide: In the Xcode implementation of XCTest, an Objective-C exception that occurs within one of our existing assertions or tests will result in a test failure; doing more than this is not practical given that it's possible to neither catch and handle nor generate an Objective-C exception in Swift.
>>>
>>>
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160110/c7c94b22/attachment.html>
More information about the swift-evolution
mailing list