[swift-corelibs-dev] [xctest] Removing outliers from performance tests

[Mike Ferris]

I think that there’s a lot of room for improvement in how we measure and analyze perf test results. And I like some of your ideas of adaptive numbers of runs and what to do with outlier results, etc… that you presented below.

We would very much like to not diverge the API of the CoreLibs XCTest and Xcode’s.

There may be a bit more room for some variation in the behavior. So talking about doing something where the number of runs varies dynamically not via API but through smarter execution mechanisms, or doing better statistical analysis such as removing outlier results, this would introduce some change in the results that might be reported between the two implementations, but it would not give rise to people writing tests that would not work cross-platform.

Further, there’s always room for discussion of taking such ideas, and even API additions and supporting them in Xcode’s XCTest as well. But that sort of discussion will have to include the practicalities of schedule and resource on the Xcode side (and, to some unavoidable extent, there will be aspects to that discussion that will not be as transparent since we’ll be weighing that work against other work for the testing team that cannot be discussed as freely.)

This year, our primary goal for the core libraries is to broaden the implementation of the existing APIs in the OS X versions of the frameworks. For XCTest, we would also love to come up with some better answer for test discovery and potentially elicit help from the community in achieving that. Beyond that, the Xcode team’s bandwidth for incorporating other things that would necessaitate change to the Xcode XCTest is going to be limited.

Mike


> On Dec 12, 2015, at 6:10 PM, Drew Crawford via swift-corelibs-dev <swift-corelibs-dev at swift.org> wrote:
> 
>> Unfortunately our corelibs implementation of XCTest isn’t ready yet for performance testing.
> 
> That's why I'm here; I'm taking the temperature on implementing it.  I'm at the pain level where I need a solution in the next several months, even if the solution is to code it up myself.  My tests have failed 10x over this so far today.
> 
> I think the real question is, if I did implement basic performance testing, and I did implement a variable-sized window of runs, would that departure from the Old XCTest behavior (which uses 10 runs) disqualify the PR?  It's a basic compatibility question about how close we need to follow the Old XCTest behavior.
> 
> e.g. if XCS wanted to migrate to corelibs-xctest and it used variable #s of runs, presumably that would be an undertaking for the XCS team.  But I don't know whether those concerns (if they exist) play a role in what this project decides to do.
> 
> I'm going to do something on this problem eventually, unless someone else solves it first.  I'm just trying to work out whether I can do something upstream or whether this is a better candidate for an independent effort.
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> swift-corelibs-dev mailing list
> swift-corelibs-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev