[swift-evolution] Solving the issue of unit-testing precondition with the Standard Library?

[David Hart]

I don't agree because as soon as you have complicated preconditions, you will have bugs in them which need unit testing. As recently as Friday, the approach on Stack Overflow allowed me to test and uncover several bugs/holes in my preconditions.

> On 13 Mar 2016, at 09:40, Haravikk via swift-evolution <swift-evolution at swift.org> wrote:
> 
> I’m unsure about this; a precondition should validate that a method has been called correctly, so if it fails then it may indicate a flaw with your test, so it makes sense for it to stop. You can use assert() to also test for edge-cases within your method that could have unexpected results (due to implementation details). The unit test meanwhile should just pass in valid input, and test for expected output. At least that’s how I try to do it, i.e- with a unit test doing black-box testing only, and assert() handling white-box testing (implementation details and edge cases that could affect them). This leaves the precondition to check input; if one fails then it indicates a very basic error that may invalidate your remaining tests anyway.
> 
> 
> If you do want to handle them however, then it seems like a compiler option to replace them with thrown errors might make most sense; this would mean duplicating code with preconditions (or that calls such methods) so that there’s one path with thrown preconditions and one without, the compiler can then select which path to use based upon the call-site in your unit test (e.g- if you have a catch block for PreconditionError, or an attribute like @preconditionError or something), and discard any paths that aren’t used. It’s a complex option, but it would definitely work, and would only be enabled when testing by default.
> 
>> On 13 Mar 2016, at 00:32, Dmitri Gribenko via swift-evolution <swift-evolution at swift.org> wrote:
>> 
>> On Sat, Mar 12, 2016 at 4:28 PM, Jonathan Tang via swift-evolution
>> <swift-evolution at swift.org> wrote:
>>> 
>>> 
>>> On Sat, Mar 12, 2016 at 1:41 PM, David Hart via swift-evolution
>>> <swift-evolution at swift.org> wrote:
>>>> 
>>>> I am deeply interested in finding solutions for allowing unit-tests of
>>>> preconditions. Without them, I believe we are leaving many holes in our
>>>> tests and coverage. The solution found in the Swift project of forking the
>>>> process seems fairly complicated to implement in XCTest.
>>>> 
>>>> I found a solution online that works by overriding the precondition
>>>> function with a function that calls a configurable closure which defaults to
>>>> the original precondition function. It would be great if the Standard
>>>> Library allowed this by default so that XCTest could use it to offer full
>>>> support for precondition unit tests.
>>>> 
>>>> http://stackoverflow.com/a/31349339
>>>> 
>>>> Is this imaginable?
>>> 
>>> 
>>> +1 to this being an important problem to solve, but I'm not sure about the
>>> specific solution.  Are there performance or security impacts to production
>>> code by having this?  Could the forked-process solution be implemented once
>>> in a framework and then everybody just uses it transparently?  The
>>> forked-process approach also has the advantage of catching crashes for any
>>> other unexpected reason (eg. division by zero), and of ensuring that test
>>> executions are hermetically sealed without data leaking between test
>>> instances.
>> 
>> I completely agree that the approach that uses process isolation is
>> the right way to go, not just for testing preconditions, but for all
>> other reasons you mention.
>> 
>> Dmitri
>> 
>> -- 
>> main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
>> (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160313/699ea135/attachment.html>