[swift-evolution] Proposal: XCTest Support for Swift Error Handling

Ross O'Brien narrativium+swift at gmail.com
Sun Jan 10 13:29:36 CST 2016

RSpec, Kiwi and similar have hierarchical tests which are good because they
reduce duplication and take into account the dependencies of tests - tests
on the performance of objects being conditional on the verified successful
creation of those objects, or on duplicated circumstances - but they also
nest tests into a 'pyramid of doom' as we had prior to "if let" allowing
multiple bindings.

It'd be nice if we could refactor huge nested structures like that into
flatter subtests. Ideally I'd like there to be a way to reduce repetition
of duplicated code in tests (e.g. for a series of identical tests
determining if a given data structure is valid, a way to call those tests
with one command). It's possible to call XCTest functions in helper
functions now, but hard to match any failure messages with the test which
called the helper function. Perhaps such sub-test methods could throw a
series of test failure messages so this can be determined more easily.

On Sun, Jan 10, 2016 at 4:05 PM, James Campbell <james at supmenow.com> wrote:

> I wouldn't mind if we could have something closer to this.
> http://rspec.info/
> On Sun, Jan 10, 2016 at 3:01 PM, Ross O'Brien <narrativium+swift at gmail.com
> > wrote:
>> I've been wondering for a while, while writing unit tests, why we still
>> start with "func test[behaviourOfThing]() {" and not "test behaviourOfThing
>> {". It's not just about less typing: parsing a function for the 'test'
>> prefix is an Objective C holdover, and doesn't feel Swift to me, and it has
>> tests behaving like functions when - as illustrated in this proposal - they
>> should have different behaviours. It should be clearer when reading code
>> whether a function is a test or a helper function to make tests easier to
>> write.
>> On Sun, Jan 10, 2016 at 12:34 PM, James Campbell via swift-evolution <
>> swift-evolution at swift.org> wrote:
>>> I would love it if we could do a full review of XCtest in general. As
>>> there are other things it could help with I.r mocking or allowing us to
>>> express tests in a BDD way
>>> Sent from my iPhone
>>> On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <
>>> swift-evolution at swift.org> wrote:
>>> I have on the order of ~700 tests in XCTest in Swift-language projects.
>>> I'm considering migrating away from XCTest, although not over this issue.
>>> This proposal IMO addresses an important problem, but I am not convinced
>>> it actually solves it.  #2 & #3 are basically sound API designs.  It is a
>>> mystery to me why #3 "generated some debate" as this is a feature I already
>>> implement manually, but I can't address unknown concerns.  I can tell you I
>>> implement this, and nothing terrible has happened to me so far.
>>> #1 I would not use.  The rest of this comment explains why.
>>> Currently I write tests about like this:
>>> try! hopefullyNothingBad()
>>> Now this is "bad" because it "crashes" but that's (big sigh) actually
>>> "good" because the debugger stops and/or I get a crash report that
>>> identifies at least "some" line where something bad definitely happened.
>>> Now that is not everything I *want* to know–I *want* someone to tell me
>>> a story of how this error was created deep down in the bowels of my
>>> application, how it spent its kindergarten years in the models layer before
>>> being passed into a controller where it was rethrown onto a different
>>> dispatch queue and finally ended up in my unit test–but we can't have
>>> everything.  So I settle for collecting a line number from the test case
>>> and then going hunting by hand.
>>> When the test function throws we no longer even find out a line number
>>> in the test case anymore, because the error is passed into XCTest and the
>>> information is lost.  We have just the name of the test case (I assume; the
>>> proposal is silent on this issue, but that's the only way I can think of to
>>> implement it), and some of my tests are pretty long.  So, that makes it
>>> even harder to track down.
>>> This sounds like a small thing but my test coverage is so thorough on
>>> mature projects that mostly what I turn up are heisenbugs that reproduce
>>> with 2% probability.  So getting the report from the CI that has the most
>>> possible detail is critical, because if the report is not thorough enough
>>> for you to guess the bug, too bad, because that's all the information you
>>> get and the bug is not reproducible.
>>> For that reason, I won't use #1.  I hesitate about whether to call it
>>> bad idea altogether, or whether it's just not to my taste.  My sense is
>>> it's probably somewhere in the middle of those two poles.
>>> I would use #2 and #3, assuming that I don't first migrate out to a
>>> non-XCTest framework.
>>> On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <
>>> swift-evolution at swift.org> wrote:
>>> We’d like feedback on a proposed design for adding support for Swift
>>> error handling to XCTest, attached below. I’ll mostly let the proposal
>>> speak for itself, but there are three components to it: Allowing test
>>> methods to throw errors, allowing the expressions evaluated by assertions
>>> to throw errors, and adding an assertion for checking error handling.
>>> We’d love to hear your feedback. We’re particularly interested in some
>>> feedback on the idea of allowing the expressions evaluated by assertions to
>>> throw errors; it’s generated some debate because it results in writing test
>>> code slightly differently than other code that makes use of Swift error
>>> handling, so any thoughts on it would be particularly appreciated.
>>>   -- Chris Hanson (chanson at apple.com)
>>> XCTest Support for Swift Error Handling
>>>    - Proposal: SE-NNNN
>>>    <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
>>>    - Author(s): Chris Hanson <https://github.com/eschaton>
>>>    - Status: *Review*
>>>    - Review manager: TBD
>>> Introduction
>>> Swift 2 introduced a new error handling mechanism that, for
>>> completeness, needs to be accommodated by our testing frameworks. Right
>>> now, to write tests that involve methods that may throw an error, a
>>> developer needs to incorporate significant boilerplate into their test. We
>>> should move this into the framework in several ways, so tests of code that
>>> interacts with Swift error handling is concise and intention-revealing.
>>> Motivation
>>> Currently, if a developer wants to use a call that may throw an error in
>>> a test, they need to use Swift's do..catch construct in their test
>>> because tests are not themselves allowed to throw errors.
>>> As an example, a vending machine object that has had insufficient funds
>>> deposited may throw an error if asked to vend an item. A test for that
>>> situation could reasonably use the do..catchconstruct to check that
>>> this occurs as expected. However, that means all other tests *also* need
>>> to use either a do..catch or try! construct — and the failure of a try! is
>>> catastrophic, so do..catch would be preferred simply for better
>>> reporting within tests.
>>> func testVendingOneItem() {
>>>     do {
>>>         vendingMachine.deposit(5)
>>>         let item = try vendingMachine.vend(row: 1, column: 1)
>>>         XCTAssertEqual(item, "Candy Bar")
>>>     } catch {
>>>         XCTFail("Unexpected failure: \(error)")
>>>     }}
>>> If the implementation of VendingMachine.vend(row:column:) changes
>>> during development such that it throws an error in this situation, the test
>>> will fail as it should.
>>> One other downside of the above is that a failure caught this way will
>>> be reported as an *expected failure*, which would normally be a failure
>>> for which XCTest is explicitly testing via an assertion. This failure
>>> should ideally be treated as an *unexpected failure*, as it's not one
>>> that's anticipated in the execution of the test.
>>> In addition, tests do not currently support throwing an error from
>>> within an assertion, requiring any code that throws an error to be invoked
>>> outside the assertion itself using the same techniques described above.
>>> Finally, since Swift error handling is a general mechanism that
>>> developers should be implementing in their own applications and frameworks,
>>> we need to make it straightforward to write tests that ensure code that
>>> implements error handling does so correctly.
>>> Proposed solution
>>> I propose several related solutions to this issue:
>>>    1. Allow test methods to throw errors.
>>>    2. Allow test assertion expressions to throw errors.
>>>    3. Add an assertion for checking errors.
>>> These solutions combine to make writing tests that involve thrown errors
>>> much more succinct.
>>> Allowing Test Methods to Throw Errors
>>> First, we can allow test methods to throw errors if desired, thus
>>> allowing the do..catch construct to be omitted when the test isn't
>>> directly checking error handling. This makes the code a developer writes
>>> when they're not explicitly trying to test error handling much cleaner.
>>> Moving the handling of errors thrown by tests into XCTest itself also
>>> ensures they can be treated as unexpected failures, since the mechanism to
>>> do so is currently private to the framework.
>>> With this, the test from the previous section can become:
>>> func testVendingOneItem() throws {
>>>     vendingMachine.deposit(5)
>>>     let item = try vendingMachine.vend(row: 1, column: 1)
>>>     XCTAssertEqual(item, "Candy Bar")}
>>> This shows much more directly that the test is intended to check a
>>> specific non-error case, and that the developer is relying on the framework
>>> to handle unexpected errors.
>>> Allowing Test Assertions to Throw Errors
>>> We can also allow the @autoclosure expression that is passed into an
>>> assertion to throw an error, and treat that error as an unexpected failure
>>> (since the code is being invoked in an assertion that isn't directly
>>> related to error handling). For example:
>>> func testVendingMultipleItemsWithSufficientFunds() {
>>>     vendingMachine.deposit(10)
>>>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
>>>     XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}
>>> This can eliminate otherwise-dangerous uses of try! and streamline code
>>> that needs to make multiple assertions in a row.
>>> Adding a "Throws Error" Assertion
>>> In order to test code that throws an error, it would be useful to have
>>> an assertion that expects an error to be thrown in a particular case. Right
>>> now a developer writing code to test that an error is thrown has to test
>>> that error themselves:
>>>     func testVendingFailsWithInsufficientFunds() {
>>>         vendingMachine.deposit(1)
>>>         var vendingFailed = false
>>>         do {
>>>             _ = try vendingMachine.vend(row: 1, column: 1))
>>>         } catch {
>>>             vendingFailed = true
>>>         }
>>>         XCTAssert(vendingFailed)
>>>     }
>>> If we add an assertion that specifically checks whether an error was
>>> thrown, this code will be significantly streamlined:
>>>     func testVendingFailsWithInsufficientFunds() {
>>>         vendingMachine.deposit(1)
>>>         XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
>>>     }
>>> Of course, some code may want to just detect that an error was thrown,
>>> but other code may need to check that the details of the thrown error are
>>> correct. We can take advantage of Swift's trailing closure syntax to enable
>>> this, by passing the thrown error (if any) to a closure that can itself
>>> contain assertions:
>>>     XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
>>>         guard let vendingError = error as? VendingMachineError else {
>>>             XCTFail("Unexpected type of error thrown: \(error)")
>>>             return
>>>         }
>>>         XCTAssertEquals(vendingError.item, "Candy Bar")
>>>         XCTAssertEquals(vendingError.price, 5)
>>>         XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
>>>     }
>>> This lets a developer very concisely describe an error condition for
>>> their code, in whatever level of detail they desire.
>>> Detailed design
>>> The design of each of the above components is slightly different, based
>>> on the functionality provided.
>>> Tests That Throw
>>> In order to enable test methods to throw an error, we will need to
>>> update XCTest to support test methods with a () throws -> Void signature
>>> in addition to test methods with a () -> Voidsignature as it already
>>> supports.
>>> We will need to ensure tests that do throw an error have that error
>>> caught, and that it registers an unexpected failure.
>>> Assertions That Throw
>>> In order to allow assertions to throw an exception, we will need to
>>> enhance our existing assertions' @autoclosure expression parameters to
>>> add throws to their signature.
>>> Because Swift defines a closure that can throw an error to be a proper
>>> supertype of a closure that does not, this *will not* result in a
>>> combinatorial explosion of assertion overrides, and will let developers
>>> naturally write code that may throw an error within an assertion.
>>> We will treat any error thrown from within an assertion expression as an
>>> *unexpected* failure because while all assertions represent a test for
>>> some form of failure, they're not specifically checking for a thrown error.
>>> The "Throws Error" Assertion
>>> To write tests for code that throws error, we will add a new assertion
>>> function to XCTest with the following prototype:
>>> public func XCTAssertThrowsError(
>>>     @autoclosure expression: () throws -> Void,
>>>                   _ message: String = "",
>>>                        file: StaticString = __FILE__,
>>>                        line: UInt = __LINE__,
>>>              _ errorHandler: (error: ErrorType) -> Void = { _ in })
>>> Rather than treat an error thrown from its expression as a failure, this
>>> will treat *the lack of* an error thrown from its expression as an
>>> expected failure.
>>> Furthermore, so long as an error is thrown, the error will be passed to
>>> the errorHandler block passed as a trailing closure, where the
>>> developer may make further assertions against it.
>>> In both cases, the new assertion function is generic on an ErrorType in
>>> order to ensure that little to no casting will be required in the trailing
>>> closure.
>>> Impact on existing code
>>> There should be little impact on existing test code because we are only
>>> adding features and API, not changing existing features or API.
>>> All existing tests should continue to work as implemented, and can
>>> easily adopt the new conventions we're making available to become more
>>> concise and intention-revealing with respect to their error handling as
>>> shown above.
>>> Alternatives considered
>>> We considered asking developers continue using XCTest as-is, and
>>> encouraging them to use Swift's native error handling to both suppress and
>>> check the validity of errors. We also considered adding additional ways of
>>> registering failures when doing this, so that developers could register
>>> unexpected failures themselves.
>>> While this would result in developers using the language the same way in
>>> their tests as in their functional code, this would also result in much
>>> more verbose tests. We rejected this approach because such verbosity can be
>>> a significant obstacle to testing.
>>> Making it quick and clean to write tests for error handling could also
>>> encourage developers to implement error handling in their code as they need
>>> it, rather than to try to work around the feature because of any perceived
>>> difficulty in testing.
>>> We considered adding the ability to check that a specific error was
>>> thrown in XCTAssertThrowsError, but this would require the ErrorType passed
>>> to also conform to Equatable, which is also unnecessary given that this
>>> can easily be checked in a trailing closure if desired. (In some cases a
>>> developer may just want to ensure *an error* is thrown rather than *a
>>> specific error* is thrown.)
>>> We explicitly chose *not* to offer a comprehensive suite of
>>> DoesNotThrowError assertions for XCTest in Swift, though we do offer
>>> such DoesNotThrow assertions for XCTest in Objective-C. We feel these
>>> are of limited utility given that our plan is for all assertions (except
>>> XCTAssertThrowsError) to treat any thrown error as a failure.
>>> We explicitly chose not to offer any additional support for Objective-C
>>> exceptions beyond what we already provide: In the Xcode implementation of
>>> XCTest, an Objective-C exception that occurs within one of our existing
>>> assertions or tests will result in a test failure; doing more than this is
>>> not practical given that it's possible to neither catch and handle nor
>>> generate an Objective-C exception in Swift.
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
> --
>  Wizard
> james at supmenow.com
> +44 7523 279 698
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160110/a800acc5/attachment.html>

More information about the swift-evolution mailing list