[swift-corelibs-dev] [xctest] Removing outliers from performance tests
drew at sealedabstract.com
Sat Dec 12 20:10:26 CST 2015
> Unfortunately our corelibs implementation of XCTest isn’t ready yet for performance testing.
That's why I'm here; I'm taking the temperature on implementing it. I'm at the pain level where I need a solution in the next several months, even if the solution is to code it up myself. My tests have failed 10x over this so far today.
I think the real question is, if I did implement basic performance testing, and I did implement a variable-sized window of runs, would that departure from the Old XCTest behavior (which uses 10 runs) disqualify the PR? It's a basic compatibility question about how close we need to follow the Old XCTest behavior.
e.g. if XCS wanted to migrate to corelibs-xctest and it used variable #s of runs, presumably that would be an undertaking for the XCS team. But I don't know whether those concerns (if they exist) play a role in what this project decides to do.
I'm going to do something on this problem eventually, unless someone else solves it first. I'm just trying to work out whether I can do something upstream or whether this is a better candidate for an independent effort.
More information about the swift-corelibs-dev