[swift-dev] Measuring MEAN Performance (was: Questions about Swift-CI)
atrick at apple.com
Mon Jun 12 16:56:53 CDT 2017
> On Jun 12, 2017, at 2:55 PM, Michael Gottesman <mgottesman at apple.com> wrote:
>> Current approach to detecting performance changes is fragile for tests that have very low absolute runtime, as they are easily over the 5% improvement/regression threshold when the test machine gets a little bit noisy. For example in benchmark on PR #9806 <https://github.com/apple/swift/pull/9806#issuecomment-303370149>:
>> BitCount 12 14 +16.7% 0.86x
>> SuffixCountableRange 10 11 +10.0% 0.91x
>> MapReduce 303 331 +9.2% 0.92x
>> These are all false changes (and there are quite a few more there).
> The current design assumes that in such cases, the workload will be increased so that is not an issue.
That is also a valid fix for the problem, which I forgot to mention.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the swift-dev