[swift-evolution] Proposal: XCTest Support for Swift Error Handling
James Campbell
james at supmenow.com
Sun Jan 10 10:05:26 CST 2016
I wouldn't mind if we could have something closer to this.
http://rspec.info/
On Sun, Jan 10, 2016 at 3:01 PM, Ross O'Brien <narrativium+swift at gmail.com>
wrote:
> I've been wondering for a while, while writing unit tests, why we still
> start with "func test[behaviourOfThing]() {" and not "test behaviourOfThing
> {". It's not just about less typing: parsing a function for the 'test'
> prefix is an Objective C holdover, and doesn't feel Swift to me, and it has
> tests behaving like functions when - as illustrated in this proposal - they
> should have different behaviours. It should be clearer when reading code
> whether a function is a test or a helper function to make tests easier to
> write.
>
>
>
> On Sun, Jan 10, 2016 at 12:34 PM, James Campbell via swift-evolution <
> swift-evolution at swift.org> wrote:
>
>> I would love it if we could do a full review of XCtest in general. As
>> there are other things it could help with I.r mocking or allowing us to
>> express tests in a BDD way
>>
>> Sent from my iPhone
>>
>> On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <
>> swift-evolution at swift.org> wrote:
>>
>> I have on the order of ~700 tests in XCTest in Swift-language projects.
>> I'm considering migrating away from XCTest, although not over this issue.
>>
>> This proposal IMO addresses an important problem, but I am not convinced
>> it actually solves it. #2 & #3 are basically sound API designs. It is a
>> mystery to me why #3 "generated some debate" as this is a feature I already
>> implement manually, but I can't address unknown concerns. I can tell you I
>> implement this, and nothing terrible has happened to me so far.
>>
>> #1 I would not use. The rest of this comment explains why.
>>
>> Currently I write tests about like this:
>>
>> try! hopefullyNothingBad()
>>
>>
>> Now this is "bad" because it "crashes" but that's (big sigh) actually
>> "good" because the debugger stops and/or I get a crash report that
>> identifies at least "some" line where something bad definitely happened.
>>
>> Now that is not everything I *want* to know–I *want* someone to tell me
>> a story of how this error was created deep down in the bowels of my
>> application, how it spent its kindergarten years in the models layer before
>> being passed into a controller where it was rethrown onto a different
>> dispatch queue and finally ended up in my unit test–but we can't have
>> everything. So I settle for collecting a line number from the test case
>> and then going hunting by hand.
>>
>> When the test function throws we no longer even find out a line number in
>> the test case anymore, because the error is passed into XCTest and the
>> information is lost. We have just the name of the test case (I assume; the
>> proposal is silent on this issue, but that's the only way I can think of to
>> implement it), and some of my tests are pretty long. So, that makes it
>> even harder to track down.
>>
>> This sounds like a small thing but my test coverage is so thorough on
>> mature projects that mostly what I turn up are heisenbugs that reproduce
>> with 2% probability. So getting the report from the CI that has the most
>> possible detail is critical, because if the report is not thorough enough
>> for you to guess the bug, too bad, because that's all the information you
>> get and the bug is not reproducible.
>>
>> For that reason, I won't use #1. I hesitate about whether to call it bad
>> idea altogether, or whether it's just not to my taste. My sense is it's
>> probably somewhere in the middle of those two poles.
>>
>> I would use #2 and #3, assuming that I don't first migrate out to a
>> non-XCTest framework.
>>
>>
>>
>> On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <
>> swift-evolution at swift.org> wrote:
>>
>> We’d like feedback on a proposed design for adding support for Swift
>> error handling to XCTest, attached below. I’ll mostly let the proposal
>> speak for itself, but there are three components to it: Allowing test
>> methods to throw errors, allowing the expressions evaluated by assertions
>> to throw errors, and adding an assertion for checking error handling.
>>
>> We’d love to hear your feedback. We’re particularly interested in some
>> feedback on the idea of allowing the expressions evaluated by assertions to
>> throw errors; it’s generated some debate because it results in writing test
>> code slightly differently than other code that makes use of Swift error
>> handling, so any thoughts on it would be particularly appreciated.
>>
>> -- Chris Hanson (chanson at apple.com)
>>
>>
>> XCTest Support for Swift Error Handling
>>
>> - Proposal: SE-NNNN
>> <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
>> - Author(s): Chris Hanson <https://github.com/eschaton>
>> - Status: *Review*
>> - Review manager: TBD
>>
>> Introduction
>>
>> Swift 2 introduced a new error handling mechanism that, for completeness,
>> needs to be accommodated by our testing frameworks. Right now, to write
>> tests that involve methods that may throw an error, a developer needs to
>> incorporate significant boilerplate into their test. We should move this
>> into the framework in several ways, so tests of code that interacts with
>> Swift error handling is concise and intention-revealing.
>> Motivation
>>
>> Currently, if a developer wants to use a call that may throw an error in
>> a test, they need to use Swift's do..catch construct in their test
>> because tests are not themselves allowed to throw errors.
>>
>> As an example, a vending machine object that has had insufficient funds
>> deposited may throw an error if asked to vend an item. A test for that
>> situation could reasonably use the do..catchconstruct to check that this
>> occurs as expected. However, that means all other tests *also* need to
>> use either a do..catch or try! construct — and the failure of a try! is
>> catastrophic, so do..catch would be preferred simply for better
>> reporting within tests.
>>
>> func testVendingOneItem() {
>> do {
>> vendingMachine.deposit(5)
>> let item = try vendingMachine.vend(row: 1, column: 1)
>> XCTAssertEqual(item, "Candy Bar")
>> } catch {
>> XCTFail("Unexpected failure: \(error)")
>> }}
>>
>> If the implementation of VendingMachine.vend(row:column:) changes during
>> development such that it throws an error in this situation, the test will
>> fail as it should.
>>
>> One other downside of the above is that a failure caught this way will be
>> reported as an *expected failure*, which would normally be a failure for
>> which XCTest is explicitly testing via an assertion. This failure should
>> ideally be treated as an *unexpected failure*, as it's not one that's
>> anticipated in the execution of the test.
>>
>> In addition, tests do not currently support throwing an error from within
>> an assertion, requiring any code that throws an error to be invoked outside
>> the assertion itself using the same techniques described above.
>>
>> Finally, since Swift error handling is a general mechanism that
>> developers should be implementing in their own applications and frameworks,
>> we need to make it straightforward to write tests that ensure code that
>> implements error handling does so correctly.
>> Proposed solution
>>
>> I propose several related solutions to this issue:
>>
>> 1. Allow test methods to throw errors.
>> 2. Allow test assertion expressions to throw errors.
>> 3. Add an assertion for checking errors.
>>
>> These solutions combine to make writing tests that involve thrown errors
>> much more succinct.
>> Allowing Test Methods to Throw Errors
>>
>> First, we can allow test methods to throw errors if desired, thus
>> allowing the do..catch construct to be omitted when the test isn't
>> directly checking error handling. This makes the code a developer writes
>> when they're not explicitly trying to test error handling much cleaner.
>>
>> Moving the handling of errors thrown by tests into XCTest itself also
>> ensures they can be treated as unexpected failures, since the mechanism to
>> do so is currently private to the framework.
>>
>> With this, the test from the previous section can become:
>>
>> func testVendingOneItem() throws {
>> vendingMachine.deposit(5)
>> let item = try vendingMachine.vend(row: 1, column: 1)
>> XCTAssertEqual(item, "Candy Bar")}
>>
>> This shows much more directly that the test is intended to check a
>> specific non-error case, and that the developer is relying on the framework
>> to handle unexpected errors.
>> Allowing Test Assertions to Throw Errors
>>
>> We can also allow the @autoclosure expression that is passed into an
>> assertion to throw an error, and treat that error as an unexpected failure
>> (since the code is being invoked in an assertion that isn't directly
>> related to error handling). For example:
>>
>> func testVendingMultipleItemsWithSufficientFunds() {
>> vendingMachine.deposit(10)
>> XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
>> XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}
>>
>> This can eliminate otherwise-dangerous uses of try! and streamline code
>> that needs to make multiple assertions in a row.
>> Adding a "Throws Error" Assertion
>>
>> In order to test code that throws an error, it would be useful to have an
>> assertion that expects an error to be thrown in a particular case. Right
>> now a developer writing code to test that an error is thrown has to test
>> that error themselves:
>>
>> func testVendingFailsWithInsufficientFunds() {
>> vendingMachine.deposit(1)
>> var vendingFailed = false
>> do {
>> _ = try vendingMachine.vend(row: 1, column: 1))
>> } catch {
>> vendingFailed = true
>> }
>> XCTAssert(vendingFailed)
>> }
>>
>> If we add an assertion that specifically checks whether an error was
>> thrown, this code will be significantly streamlined:
>>
>> func testVendingFailsWithInsufficientFunds() {
>> vendingMachine.deposit(1)
>> XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
>> }
>>
>> Of course, some code may want to just detect that an error was thrown,
>> but other code may need to check that the details of the thrown error are
>> correct. We can take advantage of Swift's trailing closure syntax to enable
>> this, by passing the thrown error (if any) to a closure that can itself
>> contain assertions:
>>
>> XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
>> guard let vendingError = error as? VendingMachineError else {
>> XCTFail("Unexpected type of error thrown: \(error)")
>> return
>> }
>>
>> XCTAssertEquals(vendingError.item, "Candy Bar")
>> XCTAssertEquals(vendingError.price, 5)
>> XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
>> }
>>
>> This lets a developer very concisely describe an error condition for
>> their code, in whatever level of detail they desire.
>> Detailed design
>>
>> The design of each of the above components is slightly different, based
>> on the functionality provided.
>> Tests That Throw
>>
>> In order to enable test methods to throw an error, we will need to update
>> XCTest to support test methods with a () throws -> Void signature in
>> addition to test methods with a () -> Voidsignature as it already
>> supports.
>>
>> We will need to ensure tests that do throw an error have that error
>> caught, and that it registers an unexpected failure.
>> Assertions That Throw
>>
>> In order to allow assertions to throw an exception, we will need to
>> enhance our existing assertions' @autoclosure expression parameters to
>> add throws to their signature.
>>
>> Because Swift defines a closure that can throw an error to be a proper
>> supertype of a closure that does not, this *will not* result in a
>> combinatorial explosion of assertion overrides, and will let developers
>> naturally write code that may throw an error within an assertion.
>>
>> We will treat any error thrown from within an assertion expression as an
>> *unexpected* failure because while all assertions represent a test for
>> some form of failure, they're not specifically checking for a thrown error.
>> The "Throws Error" Assertion
>>
>> To write tests for code that throws error, we will add a new assertion
>> function to XCTest with the following prototype:
>>
>> public func XCTAssertThrowsError(
>> @autoclosure expression: () throws -> Void,
>> _ message: String = "",
>> file: StaticString = __FILE__,
>> line: UInt = __LINE__,
>> _ errorHandler: (error: ErrorType) -> Void = { _ in })
>>
>> Rather than treat an error thrown from its expression as a failure, this
>> will treat *the lack of* an error thrown from its expression as an
>> expected failure.
>>
>> Furthermore, so long as an error is thrown, the error will be passed to
>> the errorHandler block passed as a trailing closure, where the developer
>> may make further assertions against it.
>>
>> In both cases, the new assertion function is generic on an ErrorType in
>> order to ensure that little to no casting will be required in the trailing
>> closure.
>> Impact on existing code
>>
>> There should be little impact on existing test code because we are only
>> adding features and API, not changing existing features or API.
>>
>> All existing tests should continue to work as implemented, and can easily
>> adopt the new conventions we're making available to become more concise and
>> intention-revealing with respect to their error handling as shown above.
>> Alternatives considered
>>
>> We considered asking developers continue using XCTest as-is, and
>> encouraging them to use Swift's native error handling to both suppress and
>> check the validity of errors. We also considered adding additional ways of
>> registering failures when doing this, so that developers could register
>> unexpected failures themselves.
>>
>> While this would result in developers using the language the same way in
>> their tests as in their functional code, this would also result in much
>> more verbose tests. We rejected this approach because such verbosity can be
>> a significant obstacle to testing.
>>
>> Making it quick and clean to write tests for error handling could also
>> encourage developers to implement error handling in their code as they need
>> it, rather than to try to work around the feature because of any perceived
>> difficulty in testing.
>>
>> We considered adding the ability to check that a specific error was
>> thrown in XCTAssertThrowsError, but this would require the ErrorType passed
>> to also conform to Equatable, which is also unnecessary given that this
>> can easily be checked in a trailing closure if desired. (In some cases a
>> developer may just want to ensure *an error* is thrown rather than *a
>> specific error* is thrown.)
>>
>> We explicitly chose *not* to offer a comprehensive suite of
>> DoesNotThrowError assertions for XCTest in Swift, though we do offer
>> such DoesNotThrow assertions for XCTest in Objective-C. We feel these
>> are of limited utility given that our plan is for all assertions (except
>> XCTAssertThrowsError) to treat any thrown error as a failure.
>>
>> We explicitly chose not to offer any additional support for Objective-C
>> exceptions beyond what we already provide: In the Xcode implementation of
>> XCTest, an Objective-C exception that occurs within one of our existing
>> assertions or tests will result in a test failure; doing more than this is
>> not practical given that it's possible to neither catch and handle nor
>> generate an Objective-C exception in Swift.
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>>
>
--
Wizard
james at supmenow.com
+44 7523 279 698
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160110/09d35327/attachment.html>
More information about the swift-evolution
mailing list