[swift-evolution] Proposal: XCTest Support for Swift Error Handling

[Chris Hanson]

As Kevin Ballard pointed out, the examples give won’t currently compile and give the error “Call can throw, but it is executed in a non-throwing autoclosure.” I also don’t think they match up to what most developers would write. If it did compile, I would expect a developer to either just tack “throws” onto the test method, or to write something more like this:

func testFoo() {
    var foo = Foo()
    do {
        XCTAssertEqual(try foo.doSomething(), 42)
    } catch(
        XCTFail("testFoo failed: threw error \(error)")
    }
}

Then if they turn out to need easy matching between unexpected failures and individual calls, they will wind up writing multiple do…try blocks in their tests, one per expression that can throw, until they have sufficient granularity that a test failure in a log can instantly pinpoint the code that failed.

I think that would wind up cluttering up tests that are mostly intended not to be about error handling. That’s why I think it’s better to just let all of the assertions catch thrown errors as well, regardless of the substitutability argument; the assertions are already “special” by doing delayed rather than immediate evaluation, so I don’t think we’re really changing them all that much by allowing the assertion expression to throw.

Ultimately, I think allowing assertion expressions to throw makes tests easier to write as well as to read & maintain over time by moving as much of the burden of error handling on the success path to the testing framework itself. In this model, the only error handling code a developer needs to write in their tests is actually testing their error handling.

  -- Chris


> On Jan 11, 2016, at 6:15 PM, Philippe Hausler <phausler at apple.com> wrote:
> 
> I think I am in full agreement with Dmitri on this: The XCTAssertThrowsError should be the only one that interacts with errors and that others should not try to hijack the try.
> 
> But super-big +1 on the XCTAssertThrowsError being in place to verify error cases; swift-corelibs-foundation is missing a lot of testing of error cases and it would be nice to have an effort to start verifying those (and their structures)
> 
>> On Jan 10, 2016, at 11:18 AM, Dmitri Gribenko via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>> 
>> On Sat, Jan 9, 2016 at 6:58 PM, Chris Hanson via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>> Allowing Test Assertions to Throw Errors
>> 
>> We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:
>> 
>> func testVendingMultipleItemsWithSufficientFunds() {
>>     vendingMachine.deposit(10)
>>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
>>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
>> }
>> 
>> I have significant concerns about doing this.  Consider the following code:
>> 
>> var foo = Foo()
>> do {
>>     XCTAssertEqual(try foo.doSomething(), 42)
>> } catch {
>>     XCTAssertEqual(foo.success, false)
>> }
>> 
>> Adding ‘throws’ to the autoclosures will:
>> 
>> (1) change meaning of existing tests (the example above used to proceed to the catch block in case of errors, and will start failing with this change),
>> 
>> (2) hijacks ‘try’ and defeats its purpose — being able to understand the control flow.  Developers know that if they see a ‘try’, it should match with either a do-catch, or be used in a throwing function.  Adding this API makes the code misleading.
>> 
>> Note that although (2) applies to the XCTAssertThrowsError() API, because the name of that API makes it clear it is doing something special with error handling, I’m not concerned about it.  But adding error handling to regular assertion functions has the potential to create misleading code.
>> 
>> Changing the way control flow works breaks one of the basic language features — substitutability:
>> 
>> let bar1 = try bar()
>> foo(bar1)
>> 
>> should be equivalent to:
>> 
>> foo(try bar())
>> 
>> Dmitri
>> 
>> -- 
>> main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
>> (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com <mailto:gribozavr at gmail.com>>*/
>>  _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160111/d2993a18/attachment.html>