[swift-evolution] Testing Assertions

[Jérôme Duquennoy]

It is true that testing an assert is not really possible, as it basically crashes the app.

But we have to take into account that the behaviour of the assert method is not strait-forward : it depends on what is the optimisation level. Here is an extract of the inline doc of the assert method :

* In playgrounds and -Onone builds (the default for Xcode's Debug
  configuration): if `condition` evaluates to false, stop program
  execution in a debuggable state after printing `message`.
* In -O builds (the default for Xcode's Release configuration),
  `condition` is not evaluated, and there are no effects.
* In -Ounchecked builds, `condition` is not evaluated, but the
  optimizer may assume that it *would* evaluate to `true`. Failure
  to satisfy that assumption in -Ounchecked builds is a serious
  programming error.

I have the feeling that assertions are not meant to be tested, as they are not meant to be executed in production code.
And I have to add that writing tests that would not behave the same depending on the optimisation level would make me feel rather uncomfortable.

Maybe throwing an error would be more adapted for a validation that must also occur on production builds ?
Then, testing it would be pretty easy, with an extension to XCTestCase like that :

extension XCTestCase {

  func XCTAssertThrows(file: String = __FILE__, line: UInt = __LINE__, _ closure:() throws -> Void) {
    do {
      try closure();
      XCTFail("Closure did not throw an error", file: file, line: line);
    } catch {
      // expected, nothing to do
    }
  }
  
  func XCTAssertNoThrow<T>(file: String = __FILE__, line: UInt = __LINE__, _ closure:() throws -> T) -> T? {
    do {
      return try closure();
    } catch let error {
      XCTFail("Closure throw unexpected error \(error)", file: file, line: line);
    }
    return nil;
  }
  
}

Jérôme


> On 05 Jan 2016, at 01:42, Tony Parker via swift-evolution <swift-evolution at swift.org> wrote:
> 
> Hi Mohamed,
> 
> I agree it’s very difficult to test assertions in XCTest today. This approach looks interesting, but I’m not sure how it’s possible to implement within XCTest’s current architecture. Do you have any idea how this would be implemented? 
> 
> - Tony
> 
>> On Dec 31, 2015, at 1:35 AM, Mohamed Ebrahim Afifi via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>> 
>> Right now, we cannot easily test failed assertions (assert, assertionFailure, precondition, preconditionFailure, fatalError) in our own code without some hacks in the code under test. Like this example https://github.com/mohamede1945/AssertionsTestingExample <https://github.com/mohamede1945/AssertionsTestingExample>
>> 
>> So, my suggestion is to add for XCTest something very similar to expectCrashLater that is defined here https://github.com/apple/swift/blob/9b15d03b73b9e8a6dbd3f71b5c78660a359e8e26/stdlib/private/StdlibUnittest/StdlibUnittest.swift.gyb <https://github.com/apple/swift/blob/9b15d03b73b9e8a6dbd3f71b5c78660a359e8e26/stdlib/private/StdlibUnittest/StdlibUnittest.swift.gyb>
>> and used in tests like this example https://github.com/apple/swift/blob/master/validation-test/stdlib/Assert.swift <https://github.com/apple/swift/blob/master/validation-test/stdlib/Assert.swift>
>> 
>> What do you think?
>> 
>> Best Regards,
>> Mohamed Afifi
>>  _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 
> 
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160105/b04bbfbe/attachment.html>