[swift-evolution] Proposal: XCTest Support for Swift Error Handling
James Campbell
james at supmenow.com
Sun Jan 10 15:19:59 CST 2016
Unfortunately a lot of current libraries involve hacks on-top of XCTest. I
think this belongs here as we should start having 1st class support.
On Sun, Jan 10, 2016 at 7:07 PM, Daniel Steinberg <daniel at dimsumthinking.com
> wrote:
> +1 for RSpec-like BDD style support in tests - though that doesn’t feel
> like a swift-evolution type of discussion
>
>
> On Jan 10, 2016, at 11:05 AM, James Campbell via swift-evolution <
> swift-evolution at swift.org> wrote:
>
> I wouldn't mind if we could have something closer to this.
> http://rspec.info/
>
> On Sun, Jan 10, 2016 at 3:01 PM, Ross O'Brien <narrativium+swift at gmail.com
> > wrote:
>
>> I've been wondering for a while, while writing unit tests, why we still
>> start with "func test[behaviourOfThing]() {" and not "test behaviourOfThing
>> {". It's not just about less typing: parsing a function for the 'test'
>> prefix is an Objective C holdover, and doesn't feel Swift to me, and it has
>> tests behaving like functions when - as illustrated in this proposal - they
>> should have different behaviours. It should be clearer when reading code
>> whether a function is a test or a helper function to make tests easier to
>> write.
>>
>>
>>
>> On Sun, Jan 10, 2016 at 12:34 PM, James Campbell via swift-evolution <
>> swift-evolution at swift.org> wrote:
>>
>>> I would love it if we could do a full review of XCtest in general. As
>>> there are other things it could help with I.r mocking or allowing us to
>>> express tests in a BDD way
>>>
>>> Sent from my iPhone
>>>
>>> On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <
>>> swift-evolution at swift.org> wrote:
>>>
>>> I have on the order of ~700 tests in XCTest in Swift-language projects.
>>> I'm considering migrating away from XCTest, although not over this issue.
>>>
>>> This proposal IMO addresses an important problem, but I am not convinced
>>> it actually solves it. #2 & #3 are basically sound API designs. It is a
>>> mystery to me why #3 "generated some debate" as this is a feature I already
>>> implement manually, but I can't address unknown concerns. I can tell you I
>>> implement this, and nothing terrible has happened to me so far.
>>>
>>> #1 I would not use. The rest of this comment explains why.
>>>
>>> Currently I write tests about like this:
>>>
>>> try! hopefullyNothingBad()
>>>
>>>
>>> Now this is "bad" because it "crashes" but that's (big sigh) actually
>>> "good" because the debugger stops and/or I get a crash report that
>>> identifies at least "some" line where something bad definitely happened.
>>>
>>> Now that is not everything I *want* to know–I *want* someone to tell me
>>> a story of how this error was created deep down in the bowels of my
>>> application, how it spent its kindergarten years in the models layer before
>>> being passed into a controller where it was rethrown onto a different
>>> dispatch queue and finally ended up in my unit test–but we can't have
>>> everything. So I settle for collecting a line number from the test case
>>> and then going hunting by hand.
>>>
>>> When the test function throws we no longer even find out a line number
>>> in the test case anymore, because the error is passed into XCTest and the
>>> information is lost. We have just the name of the test case (I assume; the
>>> proposal is silent on this issue, but that's the only way I can think of to
>>> implement it), and some of my tests are pretty long. So, that makes it
>>> even harder to track down.
>>>
>>> This sounds like a small thing but my test coverage is so thorough on
>>> mature projects that mostly what I turn up are heisenbugs that reproduce
>>> with 2% probability. So getting the report from the CI that has the most
>>> possible detail is critical, because if the report is not thorough enough
>>> for you to guess the bug, too bad, because that's all the information you
>>> get and the bug is not reproducible.
>>>
>>> For that reason, I won't use #1. I hesitate about whether to call it
>>> bad idea altogether, or whether it's just not to my taste. My sense is
>>> it's probably somewhere in the middle of those two poles.
>>>
>>> I would use #2 and #3, assuming that I don't first migrate out to a
>>> non-XCTest framework.
>>>
>>>
>>>
>>> On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <
>>> swift-evolution at swift.org> wrote:
>>>
>>> We’d like feedback on a proposed design for adding support for Swift
>>> error handling to XCTest, attached below. I’ll mostly let the proposal
>>> speak for itself, but there are three components to it: Allowing test
>>> methods to throw errors, allowing the expressions evaluated by assertions
>>> to throw errors, and adding an assertion for checking error handling.
>>>
>>> We’d love to hear your feedback. We’re particularly interested in some
>>> feedback on the idea of allowing the expressions evaluated by assertions to
>>> throw errors; it’s generated some debate because it results in writing test
>>> code slightly differently than other code that makes use of Swift error
>>> handling, so any thoughts on it would be particularly appreciated.
>>>
>>> -- Chris Hanson (chanson at apple.com)
>>>
>>>
>>> XCTest Support for Swift Error Handling
>>>
>>> - Proposal: SE-NNNN
>>> <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
>>> - Author(s): Chris Hanson <https://github.com/eschaton>
>>> - Status: *Review*
>>> - Review manager: TBD
>>>
>>> Introduction
>>>
>>> Swift 2 introduced a new error handling mechanism that, for
>>> completeness, needs to be accommodated by our testing frameworks. Right
>>> now, to write tests that involve methods that may throw an error, a
>>> developer needs to incorporate significant boilerplate into their test. We
>>> should move this into the framework in several ways, so tests of code that
>>> interacts with Swift error handling is concise and intention-revealing.
>>> Motivation
>>>
>>> Currently, if a developer wants to use a call that may throw an error in
>>> a test, they need to use Swift's do..catch construct in their test
>>> because tests are not themselves allowed to throw errors.
>>>
>>> As an example, a vending machine object that has had insufficient funds
>>> deposited may throw an error if asked to vend an item. A test for that
>>> situation could reasonably use the do..catchconstruct to check that
>>> this occurs as expected. However, that means all other tests *also* need
>>> to use either a do..catch or try! construct — and the failure of a try! is
>>> catastrophic, so do..catch would be preferred simply for better
>>> reporting within tests.
>>>
>>> func testVendingOneItem() {
>>> do {
>>> vendingMachine.deposit(5)
>>> let item = try vendingMachine.vend(row: 1, column: 1)
>>> XCTAssertEqual(item, "Candy Bar")
>>> } catch {
>>> XCTFail("Unexpected failure: \(error)")
>>> }}
>>>
>>> If the implementation of VendingMachine.vend(row:column:) changes
>>> during development such that it throws an error in this situation, the test
>>> will fail as it should.
>>>
>>> One other downside of the above is that a failure caught this way will
>>> be reported as an *expected failure*, which would normally be a failure
>>> for which XCTest is explicitly testing via an assertion. This failure
>>> should ideally be treated as an *unexpected failure*, as it's not one
>>> that's anticipated in the execution of the test.
>>>
>>> In addition, tests do not currently support throwing an error from
>>> within an assertion, requiring any code that throws an error to be invoked
>>> outside the assertion itself using the same techniques described above.
>>>
>>> Finally, since Swift error handling is a general mechanism that
>>> developers should be implementing in their own applications and frameworks,
>>> we need to make it straightforward to write tests that ensure code that
>>> implements error handling does so correctly.
>>> Proposed solution
>>>
>>> I propose several related solutions to this issue:
>>>
>>> 1. Allow test methods to throw errors.
>>> 2. Allow test assertion expressions to throw errors.
>>> 3. Add an assertion for checking errors.
>>>
>>> These solutions combine to make writing tests that involve thrown errors
>>> much more succinct.
>>> Allowing Test Methods to Throw Errors
>>>
>>> First, we can allow test methods to throw errors if desired, thus
>>> allowing the do..catch construct to be omitted when the test isn't
>>> directly checking error handling. This makes the code a developer writes
>>> when they're not explicitly trying to test error handling much cleaner.
>>>
>>> Moving the handling of errors thrown by tests into XCTest itself also
>>> ensures they can be treated as unexpected failures, since the mechanism to
>>> do so is currently private to the framework.
>>>
>>> With this, the test from the previous section can become:
>>>
>>> func testVendingOneItem() throws {
>>> vendingMachine.deposit(5)
>>> let item = try vendingMachine.vend(row: 1, column: 1)
>>> XCTAssertEqual(item, "Candy Bar")}
>>>
>>> This shows much more directly that the test is intended to check a
>>> specific non-error case, and that the developer is relying on the framework
>>> to handle unexpected errors.
>>> Allowing Test Assertions to Throw Errors
>>>
>>> We can also allow the @autoclosure expression that is passed into an
>>> assertion to throw an error, and treat that error as an unexpected failure
>>> (since the code is being invoked in an assertion that isn't directly
>>> related to error handling). For example:
>>>
>>> func testVendingMultipleItemsWithSufficientFunds() {
>>> vendingMachine.deposit(10)
>>> XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
>>> XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}
>>>
>>> This can eliminate otherwise-dangerous uses of try! and streamline code
>>> that needs to make multiple assertions in a row.
>>> Adding a "Throws Error" Assertion
>>>
>>> In order to test code that throws an error, it would be useful to have
>>> an assertion that expects an error to be thrown in a particular case. Right
>>> now a developer writing code to test that an error is thrown has to test
>>> that error themselves:
>>>
>>> func testVendingFailsWithInsufficientFunds() {
>>> vendingMachine.deposit(1)
>>> var vendingFailed = false
>>> do {
>>> _ = try vendingMachine.vend(row: 1, column: 1))
>>> } catch {
>>> vendingFailed = true
>>> }
>>> XCTAssert(vendingFailed)
>>> }
>>>
>>> If we add an assertion that specifically checks whether an error was
>>> thrown, this code will be significantly streamlined:
>>>
>>> func testVendingFailsWithInsufficientFunds() {
>>> vendingMachine.deposit(1)
>>> XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
>>> }
>>>
>>> Of course, some code may want to just detect that an error was thrown,
>>> but other code may need to check that the details of the thrown error are
>>> correct. We can take advantage of Swift's trailing closure syntax to enable
>>> this, by passing the thrown error (if any) to a closure that can itself
>>> contain assertions:
>>>
>>> XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
>>> guard let vendingError = error as? VendingMachineError else {
>>> XCTFail("Unexpected type of error thrown: \(error)")
>>> return
>>> }
>>>
>>> XCTAssertEquals(vendingError.item, "Candy Bar")
>>> XCTAssertEquals(vendingError.price, 5)
>>> XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
>>> }
>>>
>>> This lets a developer very concisely describe an error condition for
>>> their code, in whatever level of detail they desire.
>>> Detailed design
>>>
>>> The design of each of the above components is slightly different, based
>>> on the functionality provided.
>>> Tests That Throw
>>>
>>> In order to enable test methods to throw an error, we will need to
>>> update XCTest to support test methods with a () throws -> Void signature
>>> in addition to test methods with a () -> Voidsignature as it already
>>> supports.
>>>
>>> We will need to ensure tests that do throw an error have that error
>>> caught, and that it registers an unexpected failure.
>>> Assertions That Throw
>>>
>>> In order to allow assertions to throw an exception, we will need to
>>> enhance our existing assertions' @autoclosure expression parameters to
>>> add throws to their signature.
>>>
>>> Because Swift defines a closure that can throw an error to be a proper
>>> supertype of a closure that does not, this *will not* result in a
>>> combinatorial explosion of assertion overrides, and will let developers
>>> naturally write code that may throw an error within an assertion.
>>>
>>> We will treat any error thrown from within an assertion expression as an
>>> *unexpected* failure because while all assertions represent a test for
>>> some form of failure, they're not specifically checking for a thrown error.
>>> The "Throws Error" Assertion
>>>
>>> To write tests for code that throws error, we will add a new assertion
>>> function to XCTest with the following prototype:
>>>
>>> public func XCTAssertThrowsError(
>>> @autoclosure expression: () throws -> Void,
>>> _ message: String = "",
>>> file: StaticString = __FILE__,
>>> line: UInt = __LINE__,
>>> _ errorHandler: (error: ErrorType) -> Void = { _ in })
>>>
>>> Rather than treat an error thrown from its expression as a failure, this
>>> will treat *the lack of* an error thrown from its expression as an
>>> expected failure.
>>>
>>> Furthermore, so long as an error is thrown, the error will be passed to
>>> the errorHandler block passed as a trailing closure, where the
>>> developer may make further assertions against it.
>>>
>>> In both cases, the new assertion function is generic on an ErrorType in
>>> order to ensure that little to no casting will be required in the trailing
>>> closure.
>>> Impact on existing code
>>>
>>> There should be little impact on existing test code because we are only
>>> adding features and API, not changing existing features or API.
>>>
>>> All existing tests should continue to work as implemented, and can
>>> easily adopt the new conventions we're making available to become more
>>> concise and intention-revealing with respect to their error handling as
>>> shown above.
>>> Alternatives considered
>>>
>>> We considered asking developers continue using XCTest as-is, and
>>> encouraging them to use Swift's native error handling to both suppress and
>>> check the validity of errors. We also considered adding additional ways of
>>> registering failures when doing this, so that developers could register
>>> unexpected failures themselves.
>>>
>>> While this would result in developers using the language the same way in
>>> their tests as in their functional code, this would also result in much
>>> more verbose tests. We rejected this approach because such verbosity can be
>>> a significant obstacle to testing.
>>>
>>> Making it quick and clean to write tests for error handling could also
>>> encourage developers to implement error handling in their code as they need
>>> it, rather than to try to work around the feature because of any perceived
>>> difficulty in testing.
>>>
>>> We considered adding the ability to check that a specific error was
>>> thrown in XCTAssertThrowsError, but this would require the ErrorType passed
>>> to also conform to Equatable, which is also unnecessary given that this
>>> can easily be checked in a trailing closure if desired. (In some cases a
>>> developer may just want to ensure *an error* is thrown rather than *a
>>> specific error* is thrown.)
>>>
>>> We explicitly chose *not* to offer a comprehensive suite of
>>> DoesNotThrowError assertions for XCTest in Swift, though we do offer
>>> such DoesNotThrow assertions for XCTest in Objective-C. We feel these
>>> are of limited utility given that our plan is for all assertions (except
>>> XCTAssertThrowsError) to treat any thrown error as a failure.
>>>
>>> We explicitly chose not to offer any additional support for Objective-C
>>> exceptions beyond what we already provide: In the Xcode implementation of
>>> XCTest, an Objective-C exception that occurs within one of our existing
>>> assertions or tests will result in a test failure; doing more than this is
>>> not practical given that it's possible to neither catch and handle nor
>>> generate an Objective-C exception in Swift.
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>>
>>>
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>>
>>>
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>>
>>>
>>
>
>
> --
> Wizard
> james at supmenow.com
> +44 7523 279 698
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
>
--
Wizard
james at supmenow.com
+44 7523 279 698
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160110/252bfb83/attachment.html>
More information about the swift-evolution
mailing list