[swift-evolution] Proposal: XCTest Support for Swift Error Handling

[Ross O'Brien]

I've been wondering for a while, while writing unit tests, why we still
start with "func test[behaviourOfThing]() {" and not "test behaviourOfThing
{". It's not just about less typing: parsing a function for the 'test'
prefix is an Objective C holdover, and doesn't feel Swift to me, and it has
tests behaving like functions when - as illustrated in this proposal - they
should have different behaviours. It should be clearer when reading code
whether a function is a test or a helper function to make tests easier to
write.



On Sun, Jan 10, 2016 at 12:34 PM, James Campbell via swift-evolution <
swift-evolution at swift.org> wrote:

> I would love it if we could do a full review of XCtest in general. As
> there are other things it could help with I.r mocking or allowing us to
> express tests in a BDD way
>
> Sent from my iPhone
>
> On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <
> swift-evolution at swift.org> wrote:
>
> I have on the order of ~700 tests in XCTest in Swift-language projects.
> I'm considering migrating away from XCTest, although not over this issue.
>
> This proposal IMO addresses an important problem, but I am not convinced
> it actually solves it.  #2 & #3 are basically sound API designs.  It is a
> mystery to me why #3 "generated some debate" as this is a feature I already
> implement manually, but I can't address unknown concerns.  I can tell you I
> implement this, and nothing terrible has happened to me so far.
>
> #1 I would not use.  The rest of this comment explains why.
>
> Currently I write tests about like this:
>
> try! hopefullyNothingBad()
>
>
> Now this is "bad" because it "crashes" but that's (big sigh) actually
> "good" because the debugger stops and/or I get a crash report that
> identifies at least "some" line where something bad definitely happened.
>
> Now that is not everything I *want* to know–I *want* someone to tell me a
> story of how this error was created deep down in the bowels of my
> application, how it spent its kindergarten years in the models layer before
> being passed into a controller where it was rethrown onto a different
> dispatch queue and finally ended up in my unit test–but we can't have
> everything.  So I settle for collecting a line number from the test case
> and then going hunting by hand.
>
> When the test function throws we no longer even find out a line number in
> the test case anymore, because the error is passed into XCTest and the
> information is lost.  We have just the name of the test case (I assume; the
> proposal is silent on this issue, but that's the only way I can think of to
> implement it), and some of my tests are pretty long.  So, that makes it
> even harder to track down.
>
> This sounds like a small thing but my test coverage is so thorough on
> mature projects that mostly what I turn up are heisenbugs that reproduce
> with 2% probability.  So getting the report from the CI that has the most
> possible detail is critical, because if the report is not thorough enough
> for you to guess the bug, too bad, because that's all the information you
> get and the bug is not reproducible.
>
> For that reason, I won't use #1.  I hesitate about whether to call it bad
> idea altogether, or whether it's just not to my taste.  My sense is it's
> probably somewhere in the middle of those two poles.
>
> I would use #2 and #3, assuming that I don't first migrate out to a
> non-XCTest framework.
>
>
>
> On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <
> swift-evolution at swift.org> wrote:
>
> We’d like feedback on a proposed design for adding support for Swift error
> handling to XCTest, attached below. I’ll mostly let the proposal speak for
> itself, but there are three components to it: Allowing test methods to
> throw errors, allowing the expressions evaluated by assertions to throw
> errors, and adding an assertion for checking error handling.
>
> We’d love to hear your feedback. We’re particularly interested in some
> feedback on the idea of allowing the expressions evaluated by assertions to
> throw errors; it’s generated some debate because it results in writing test
> code slightly differently than other code that makes use of Swift error
> handling, so any thoughts on it would be particularly appreciated.
>
>   -- Chris Hanson (chanson at apple.com)
>
>
> XCTest Support for Swift Error Handling
>
>    - Proposal: SE-NNNN
>    <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
>    - Author(s): Chris Hanson <https://github.com/eschaton>
>    - Status: *Review*
>    - Review manager: TBD
>
> Introduction
>
> Swift 2 introduced a new error handling mechanism that, for completeness,
> needs to be accommodated by our testing frameworks. Right now, to write
> tests that involve methods that may throw an error, a developer needs to
> incorporate significant boilerplate into their test. We should move this
> into the framework in several ways, so tests of code that interacts with
> Swift error handling is concise and intention-revealing.
> Motivation
>
> Currently, if a developer wants to use a call that may throw an error in a
> test, they need to use Swift's do..catch construct in their test because
> tests are not themselves allowed to throw errors.
>
> As an example, a vending machine object that has had insufficient funds
> deposited may throw an error if asked to vend an item. A test for that
> situation could reasonably use the do..catchconstruct to check that this
> occurs as expected. However, that means all other tests *also* need to
> use either a do..catch or try! construct — and the failure of a try! is
> catastrophic, so do..catch would be preferred simply for better reporting
> within tests.
>
> func testVendingOneItem() {
>     do {
>         vendingMachine.deposit(5)
>         let item = try vendingMachine.vend(row: 1, column: 1)
>         XCTAssertEqual(item, "Candy Bar")
>     } catch {
>         XCTFail("Unexpected failure: \(error)")
>     }}
>
> If the implementation of VendingMachine.vend(row:column:) changes during
> development such that it throws an error in this situation, the test will
> fail as it should.
>
> One other downside of the above is that a failure caught this way will be
> reported as an *expected failure*, which would normally be a failure for
> which XCTest is explicitly testing via an assertion. This failure should
> ideally be treated as an *unexpected failure*, as it's not one that's
> anticipated in the execution of the test.
>
> In addition, tests do not currently support throwing an error from within
> an assertion, requiring any code that throws an error to be invoked outside
> the assertion itself using the same techniques described above.
>
> Finally, since Swift error handling is a general mechanism that developers
> should be implementing in their own applications and frameworks, we need to
> make it straightforward to write tests that ensure code that implements
> error handling does so correctly.
> Proposed solution
>
> I propose several related solutions to this issue:
>
>    1. Allow test methods to throw errors.
>    2. Allow test assertion expressions to throw errors.
>    3. Add an assertion for checking errors.
>
> These solutions combine to make writing tests that involve thrown errors
> much more succinct.
> Allowing Test Methods to Throw Errors
>
> First, we can allow test methods to throw errors if desired, thus allowing
> the do..catch construct to be omitted when the test isn't directly
> checking error handling. This makes the code a developer writes when
> they're not explicitly trying to test error handling much cleaner.
>
> Moving the handling of errors thrown by tests into XCTest itself also
> ensures they can be treated as unexpected failures, since the mechanism to
> do so is currently private to the framework.
>
> With this, the test from the previous section can become:
>
> func testVendingOneItem() throws {
>     vendingMachine.deposit(5)
>     let item = try vendingMachine.vend(row: 1, column: 1)
>     XCTAssertEqual(item, "Candy Bar")}
>
> This shows much more directly that the test is intended to check a
> specific non-error case, and that the developer is relying on the framework
> to handle unexpected errors.
> Allowing Test Assertions to Throw Errors
>
> We can also allow the @autoclosure expression that is passed into an
> assertion to throw an error, and treat that error as an unexpected failure
> (since the code is being invoked in an assertion that isn't directly
> related to error handling). For example:
>
> func testVendingMultipleItemsWithSufficientFunds() {
>     vendingMachine.deposit(10)
>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}
>
> This can eliminate otherwise-dangerous uses of try! and streamline code
> that needs to make multiple assertions in a row.
> Adding a "Throws Error" Assertion
>
> In order to test code that throws an error, it would be useful to have an
> assertion that expects an error to be thrown in a particular case. Right
> now a developer writing code to test that an error is thrown has to test
> that error themselves:
>
>     func testVendingFailsWithInsufficientFunds() {
>         vendingMachine.deposit(1)
>         var vendingFailed = false
>         do {
>             _ = try vendingMachine.vend(row: 1, column: 1))
>         } catch {
>             vendingFailed = true
>         }
>         XCTAssert(vendingFailed)
>     }
>
> If we add an assertion that specifically checks whether an error was
> thrown, this code will be significantly streamlined:
>
>     func testVendingFailsWithInsufficientFunds() {
>         vendingMachine.deposit(1)
>         XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
>     }
>
> Of course, some code may want to just detect that an error was thrown, but
> other code may need to check that the details of the thrown error are
> correct. We can take advantage of Swift's trailing closure syntax to enable
> this, by passing the thrown error (if any) to a closure that can itself
> contain assertions:
>
>     XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
>         guard let vendingError = error as? VendingMachineError else {
>             XCTFail("Unexpected type of error thrown: \(error)")
>             return
>         }
>
>         XCTAssertEquals(vendingError.item, "Candy Bar")
>         XCTAssertEquals(vendingError.price, 5)
>         XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
>     }
>
> This lets a developer very concisely describe an error condition for their
> code, in whatever level of detail they desire.
> Detailed design
>
> The design of each of the above components is slightly different, based on
> the functionality provided.
> Tests That Throw
>
> In order to enable test methods to throw an error, we will need to update
> XCTest to support test methods with a () throws -> Void signature in
> addition to test methods with a () -> Voidsignature as it already
> supports.
>
> We will need to ensure tests that do throw an error have that error
> caught, and that it registers an unexpected failure.
> Assertions That Throw
>
> In order to allow assertions to throw an exception, we will need to
> enhance our existing assertions' @autoclosure expression parameters to
> add throws to their signature.
>
> Because Swift defines a closure that can throw an error to be a proper
> supertype of a closure that does not, this *will not* result in a
> combinatorial explosion of assertion overrides, and will let developers
> naturally write code that may throw an error within an assertion.
>
> We will treat any error thrown from within an assertion expression as an
> *unexpected* failure because while all assertions represent a test for
> some form of failure, they're not specifically checking for a thrown error.
> The "Throws Error" Assertion
>
> To write tests for code that throws error, we will add a new assertion
> function to XCTest with the following prototype:
>
> public func XCTAssertThrowsError(
>     @autoclosure expression: () throws -> Void,
>                   _ message: String = "",
>                        file: StaticString = __FILE__,
>                        line: UInt = __LINE__,
>              _ errorHandler: (error: ErrorType) -> Void = { _ in })
>
> Rather than treat an error thrown from its expression as a failure, this
> will treat *the lack of* an error thrown from its expression as an
> expected failure.
>
> Furthermore, so long as an error is thrown, the error will be passed to
> the errorHandler block passed as a trailing closure, where the developer
> may make further assertions against it.
>
> In both cases, the new assertion function is generic on an ErrorType in
> order to ensure that little to no casting will be required in the trailing
> closure.
> Impact on existing code
>
> There should be little impact on existing test code because we are only
> adding features and API, not changing existing features or API.
>
> All existing tests should continue to work as implemented, and can easily
> adopt the new conventions we're making available to become more concise and
> intention-revealing with respect to their error handling as shown above.
> Alternatives considered
>
> We considered asking developers continue using XCTest as-is, and
> encouraging them to use Swift's native error handling to both suppress and
> check the validity of errors. We also considered adding additional ways of
> registering failures when doing this, so that developers could register
> unexpected failures themselves.
>
> While this would result in developers using the language the same way in
> their tests as in their functional code, this would also result in much
> more verbose tests. We rejected this approach because such verbosity can be
> a significant obstacle to testing.
>
> Making it quick and clean to write tests for error handling could also
> encourage developers to implement error handling in their code as they need
> it, rather than to try to work around the feature because of any perceived
> difficulty in testing.
>
> We considered adding the ability to check that a specific error was thrown
> in XCTAssertThrowsError, but this would require the ErrorType passed to
> also conform to Equatable, which is also unnecessary given that this can
> easily be checked in a trailing closure if desired. (In some cases a
> developer may just want to ensure *an error* is thrown rather than *a
> specific error* is thrown.)
>
> We explicitly chose *not* to offer a comprehensive suite of
> DoesNotThrowError assertions for XCTest in Swift, though we do offer such
> DoesNotThrow assertions for XCTest in Objective-C. We feel these are of
> limited utility given that our plan is for all assertions (except
> XCTAssertThrowsError) to treat any thrown error as a failure.
>
> We explicitly chose not to offer any additional support for Objective-C
> exceptions beyond what we already provide: In the Xcode implementation of
> XCTest, an Objective-C exception that occurs within one of our existing
> assertions or tests will result in a test failure; doing more than this is
> not practical given that it's possible to neither catch and handle nor
> generate an Objective-C exception in Swift.
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160110/2b67a5cb/attachment.html>