[swift-corelibs-dev] [xctest] Removing outliers from performance tests

[Tony Parker]

Hi Drew,

Thanks for the detailed info on your issue. I see you filed a radar, and that is indeed the best way to make sure an issue on Darwin platforms is addressed. Unfortunately our corelibs implementation of XCTest isn’t ready yet for performance testing.

- Tony

> On Dec 10, 2015, at 3:41 AM, Drew Crawford via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:
> 
> Hello folks,
> 
> I’m one of the heavy users of XCTest.measureBlock as it exists in Xcode 7.2.  To give some hard numbers, I have ~50 performance tests in an OSX framework project, occupying about 20m wall clock time total.  This occurs on a per-commit basis.
> 
> The current implementation of measureBlock as it currently exists in closed-source Xcode is something like this:
> 
> 1.  Run 10 trials
> 2.  Compare the average across those 10 trials to some baseline
> 3.  Compare the stdev across those 10 trials to some standard value (10% by default)
> 
> There are really a lot of problems with this algorithm, but maybe the biggest one is how it handles outliers.  If you have a test suite running for 20m, chances are “something” is going to happen on the build server in that time.  System background task, software update, gremlins etc.
> 
> So what happens lately is exactly *one* of the 10 * 50 = 500 total measureBlocks takes a really long time, and it is a different failure each time (e.g., it’s not my code, I swear).  A result like this for some test is typical:
> 
> <Screen Shot 2015-12-10 at 5.12.13 AM.png>
> 
> 
> The probability of this kind of error grows exponentially with the test suite size.  If we assume for an individual measureBlock that it only fails due to “chance” .01% of the time, then the overall test suite at N = 500 will only pass 60% of the time.  This is very vaguely consistent with what I experience at my scale—e.g. a test suite that does not really tell me if my code is broken or not.
> 
> IMO the problem here is one of experiment design.  From the data in the screenshot, this very well might be a real performance regression that should be properly investigated.  It is only when I tell you a lot of extra information—e.g. that this test will pass fine the next 100 executions and it’s part of an enormous test suite where something is bound to fail—that a failure due to random chance seems likely.  In other words, running 10 iterations and pretending that will find performance regressions is a poor approach.
> 
> I’ve done some prototyping on algorithms that use a dynamically sized number of trials to find performance regressions.  Apple employees, see rdar://21315474 <rdar://21315474> for an algorithm for a sliding window for performance tests (that also has other benefits, like measuring nanosecond-scale performance).  I am certainly willing to contrib that work in the open if there’s consensus it’s a good direction.
> 
> However, now that this is happening in the open, I’m interested in getting others’ thoughts on this problem.  Surely I am not the only serious user of performance tests, and maybe people with better statistics backgrounds than I have can suggest an appropriate solution.
> 
> Drew
> 
> _______________________________________________
> swift-corelibs-dev mailing list
> swift-corelibs-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-corelibs-dev/attachments/20151212/c4646f54/attachment.html>