[swift-evolution] Proposal: XCTest Support for Swift Error Handling

James Campbell james at supmenow.com
Sun Jan 10 10:05:26 CST 2016

I wouldn't mind if we could have something closer to this.

On Sun, Jan 10, 2016 at 3:01 PM, Ross O'Brien <narrativium+swift at gmail.com>

> I've been wondering for a while, while writing unit tests, why we still
> start with "func test[behaviourOfThing]() {" and not "test behaviourOfThing
> {". It's not just about less typing: parsing a function for the 'test'
> prefix is an Objective C holdover, and doesn't feel Swift to me, and it has
> tests behaving like functions when - as illustrated in this proposal - they
> should have different behaviours. It should be clearer when reading code
> whether a function is a test or a helper function to make tests easier to
> write.
> On Sun, Jan 10, 2016 at 12:34 PM, James Campbell via swift-evolution <
> swift-evolution at swift.org> wrote:
>> I would love it if we could do a full review of XCtest in general. As
>> there are other things it could help with I.r mocking or allowing us to
>> express tests in a BDD way
>> Sent from my iPhone
>> On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <
>> swift-evolution at swift.org> wrote:
>> I have on the order of ~700 tests in XCTest in Swift-language projects.
>> I'm considering migrating away from XCTest, although not over this issue.
>> This proposal IMO addresses an important problem, but I am not convinced
>> it actually solves it.  #2 & #3 are basically sound API designs.  It is a
>> mystery to me why #3 "generated some debate" as this is a feature I already
>> implement manually, but I can't address unknown concerns.  I can tell you I
>> implement this, and nothing terrible has happened to me so far.
>> #1 I would not use.  The rest of this comment explains why.
>> Currently I write tests about like this:
>> try! hopefullyNothingBad()
>> Now this is "bad" because it "crashes" but that's (big sigh) actually
>> "good" because the debugger stops and/or I get a crash report that
>> identifies at least "some" line where something bad definitely happened.
>> Now that is not everything I *want* to know–I *want* someone to tell me
>> a story of how this error was created deep down in the bowels of my
>> application, how it spent its kindergarten years in the models layer before
>> being passed into a controller where it was rethrown onto a different
>> dispatch queue and finally ended up in my unit test–but we can't have
>> everything.  So I settle for collecting a line number from the test case
>> and then going hunting by hand.
>> When the test function throws we no longer even find out a line number in
>> the test case anymore, because the error is passed into XCTest and the
>> information is lost.  We have just the name of the test case (I assume; the
>> proposal is silent on this issue, but that's the only way I can think of to
>> implement it), and some of my tests are pretty long.  So, that makes it
>> even harder to track down.
>> This sounds like a small thing but my test coverage is so thorough on
>> mature projects that mostly what I turn up are heisenbugs that reproduce
>> with 2% probability.  So getting the report from the CI that has the most
>> possible detail is critical, because if the report is not thorough enough
>> for you to guess the bug, too bad, because that's all the information you
>> get and the bug is not reproducible.
>> For that reason, I won't use #1.  I hesitate about whether to call it bad
>> idea altogether, or whether it's just not to my taste.  My sense is it's
>> probably somewhere in the middle of those two poles.
>> I would use #2 and #3, assuming that I don't first migrate out to a
>> non-XCTest framework.
>> On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <
>> swift-evolution at swift.org> wrote:
>> We’d like feedback on a proposed design for adding support for Swift
>> error handling to XCTest, attached below. I’ll mostly let the proposal
>> speak for itself, but there are three components to it: Allowing test
>> methods to throw errors, allowing the expressions evaluated by assertions
>> to throw errors, and adding an assertion for checking error handling.
>> We’d love to hear your feedback. We’re particularly interested in some
>> feedback on the idea of allowing the expressions evaluated by assertions to
>> throw errors; it’s generated some debate because it results in writing test
>> code slightly differently than other code that makes use of Swift error
>> handling, so any thoughts on it would be particularly appreciated.
>>   -- Chris Hanson (chanson at apple.com)
>> XCTest Support for Swift Error Handling
>>    - Proposal: SE-NNNN
>>    <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
>>    - Author(s): Chris Hanson <https://github.com/eschaton>
>>    - Status: *Review*
>>    - Review manager: TBD
>> Introduction
>> Swift 2 introduced a new error handling mechanism that, for completeness,
>> needs to be accommodated by our testing frameworks. Right now, to write
>> tests that involve methods that may throw an error, a developer needs to
>> incorporate significant boilerplate into their test. We should move this
>> into the framework in several ways, so tests of code that interacts with
>> Swift error handling is concise and intention-revealing.
>> Motivation
>> Currently, if a developer wants to use a call that may throw an error in
>> a test, they need to use Swift's do..catch construct in their test
>> because tests are not themselves allowed to throw errors.
>> As an example, a vending machine object that has had insufficient funds
>> deposited may throw an error if asked to vend an item. A test for that
>> situation could reasonably use the do..catchconstruct to check that this
>> occurs as expected. However, that means all other tests *also* need to
>> use either a do..catch or try! construct — and the failure of a try! is
>> catastrophic, so do..catch would be preferred simply for better
>> reporting within tests.
>> func testVendingOneItem() {
>>     do {
>>         vendingMachine.deposit(5)
>>         let item = try vendingMachine.vend(row: 1, column: 1)
>>         XCTAssertEqual(item, "Candy Bar")
>>     } catch {
>>         XCTFail("Unexpected failure: \(error)")
>>     }}
>> If the implementation of VendingMachine.vend(row:column:) changes during
>> development such that it throws an error in this situation, the test will
>> fail as it should.
>> One other downside of the above is that a failure caught this way will be
>> reported as an *expected failure*, which would normally be a failure for
>> which XCTest is explicitly testing via an assertion. This failure should
>> ideally be treated as an *unexpected failure*, as it's not one that's
>> anticipated in the execution of the test.
>> In addition, tests do not currently support throwing an error from within
>> an assertion, requiring any code that throws an error to be invoked outside
>> the assertion itself using the same techniques described above.
>> Finally, since Swift error handling is a general mechanism that
>> developers should be implementing in their own applications and frameworks,
>> we need to make it straightforward to write tests that ensure code that
>> implements error handling does so correctly.
>> Proposed solution
>> I propose several related solutions to this issue:
>>    1. Allow test methods to throw errors.
>>    2. Allow test assertion expressions to throw errors.
>>    3. Add an assertion for checking errors.
>> These solutions combine to make writing tests that involve thrown errors
>> much more succinct.
>> Allowing Test Methods to Throw Errors
>> First, we can allow test methods to throw errors if desired, thus
>> allowing the do..catch construct to be omitted when the test isn't
>> directly checking error handling. This makes the code a developer writes
>> when they're not explicitly trying to test error handling much cleaner.
>> Moving the handling of errors thrown by tests into XCTest itself also
>> ensures they can be treated as unexpected failures, since the mechanism to
>> do so is currently private to the framework.
>> With this, the test from the previous section can become:
>> func testVendingOneItem() throws {
>>     vendingMachine.deposit(5)
>>     let item = try vendingMachine.vend(row: 1, column: 1)
>>     XCTAssertEqual(item, "Candy Bar")}
>> This shows much more directly that the test is intended to check a
>> specific non-error case, and that the developer is relying on the framework
>> to handle unexpected errors.
>> Allowing Test Assertions to Throw Errors
>> We can also allow the @autoclosure expression that is passed into an
>> assertion to throw an error, and treat that error as an unexpected failure
>> (since the code is being invoked in an assertion that isn't directly
>> related to error handling). For example:
>> func testVendingMultipleItemsWithSufficientFunds() {
>>     vendingMachine.deposit(10)
>>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
>>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}
>> This can eliminate otherwise-dangerous uses of try! and streamline code
>> that needs to make multiple assertions in a row.
>> Adding a "Throws Error" Assertion
>> In order to test code that throws an error, it would be useful to have an
>> assertion that expects an error to be thrown in a particular case. Right
>> now a developer writing code to test that an error is thrown has to test
>> that error themselves:
>>     func testVendingFailsWithInsufficientFunds() {
>>         vendingMachine.deposit(1)
>>         var vendingFailed = false
>>         do {
>>             _ = try vendingMachine.vend(row: 1, column: 1))
>>         } catch {
>>             vendingFailed = true
>>         }
>>         XCTAssert(vendingFailed)
>>     }
>> If we add an assertion that specifically checks whether an error was
>> thrown, this code will be significantly streamlined:
>>     func testVendingFailsWithInsufficientFunds() {
>>         vendingMachine.deposit(1)
>>         XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
>>     }
>> Of course, some code may want to just detect that an error was thrown,
>> but other code may need to check that the details of the thrown error are
>> correct. We can take advantage of Swift's trailing closure syntax to enable
>> this, by passing the thrown error (if any) to a closure that can itself
>> contain assertions:
>>     XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
>>         guard let vendingError = error as? VendingMachineError else {
>>             XCTFail("Unexpected type of error thrown: \(error)")
>>             return
>>         }
>>         XCTAssertEquals(vendingError.item, "Candy Bar")
>>         XCTAssertEquals(vendingError.price, 5)
>>         XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
>>     }
>> This lets a developer very concisely describe an error condition for
>> their code, in whatever level of detail they desire.
>> Detailed design
>> The design of each of the above components is slightly different, based
>> on the functionality provided.
>> Tests That Throw
>> In order to enable test methods to throw an error, we will need to update
>> XCTest to support test methods with a () throws -> Void signature in
>> addition to test methods with a () -> Voidsignature as it already
>> supports.
>> We will need to ensure tests that do throw an error have that error
>> caught, and that it registers an unexpected failure.
>> Assertions That Throw
>> In order to allow assertions to throw an exception, we will need to
>> enhance our existing assertions' @autoclosure expression parameters to
>> add throws to their signature.
>> Because Swift defines a closure that can throw an error to be a proper
>> supertype of a closure that does not, this *will not* result in a
>> combinatorial explosion of assertion overrides, and will let developers
>> naturally write code that may throw an error within an assertion.
>> We will treat any error thrown from within an assertion expression as an
>> *unexpected* failure because while all assertions represent a test for
>> some form of failure, they're not specifically checking for a thrown error.
>> The "Throws Error" Assertion
>> To write tests for code that throws error, we will add a new assertion
>> function to XCTest with the following prototype:
>> public func XCTAssertThrowsError(
>>     @autoclosure expression: () throws -> Void,
>>                   _ message: String = "",
>>                        file: StaticString = __FILE__,
>>                        line: UInt = __LINE__,
>>              _ errorHandler: (error: ErrorType) -> Void = { _ in })
>> Rather than treat an error thrown from its expression as a failure, this
>> will treat *the lack of* an error thrown from its expression as an
>> expected failure.
>> Furthermore, so long as an error is thrown, the error will be passed to
>> the errorHandler block passed as a trailing closure, where the developer
>> may make further assertions against it.
>> In both cases, the new assertion function is generic on an ErrorType in
>> order to ensure that little to no casting will be required in the trailing
>> closure.
>> Impact on existing code
>> There should be little impact on existing test code because we are only
>> adding features and API, not changing existing features or API.
>> All existing tests should continue to work as implemented, and can easily
>> adopt the new conventions we're making available to become more concise and
>> intention-revealing with respect to their error handling as shown above.
>> Alternatives considered
>> We considered asking developers continue using XCTest as-is, and
>> encouraging them to use Swift's native error handling to both suppress and
>> check the validity of errors. We also considered adding additional ways of
>> registering failures when doing this, so that developers could register
>> unexpected failures themselves.
>> While this would result in developers using the language the same way in
>> their tests as in their functional code, this would also result in much
>> more verbose tests. We rejected this approach because such verbosity can be
>> a significant obstacle to testing.
>> Making it quick and clean to write tests for error handling could also
>> encourage developers to implement error handling in their code as they need
>> it, rather than to try to work around the feature because of any perceived
>> difficulty in testing.
>> We considered adding the ability to check that a specific error was
>> thrown in XCTAssertThrowsError, but this would require the ErrorType passed
>> to also conform to Equatable, which is also unnecessary given that this
>> can easily be checked in a trailing closure if desired. (In some cases a
>> developer may just want to ensure *an error* is thrown rather than *a
>> specific error* is thrown.)
>> We explicitly chose *not* to offer a comprehensive suite of
>> DoesNotThrowError assertions for XCTest in Swift, though we do offer
>> such DoesNotThrow assertions for XCTest in Objective-C. We feel these
>> are of limited utility given that our plan is for all assertions (except
>> XCTAssertThrowsError) to treat any thrown error as a failure.
>> We explicitly chose not to offer any additional support for Objective-C
>> exceptions beyond what we already provide: In the Xcode implementation of
>> XCTest, an Objective-C exception that occurs within one of our existing
>> assertions or tests will result in a test failure; doing more than this is
>> not practical given that it's possible to neither catch and handle nor
>> generate an Objective-C exception in Swift.
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution

 Wizard
james at supmenow.com
+44 7523 279 698
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160110/09d35327/attachment.html>

More information about the swift-evolution mailing list