[swift-dev] Measuring MEAN Performance (was: Questions about Swift-CI)

[Andrew Trick]

> On Jun 12, 2017, at 2:55 PM, Michael Gottesman <mgottesman at apple.com> wrote:
> 
>> Current approach to detecting performance changes is fragile for tests that have very low absolute runtime, as they are easily over the 5% improvement/regression threshold when the test machine gets a little bit noisy. For example in benchmark on PR #9806 <https://github.com/apple/swift/pull/9806#issuecomment-303370149>:
>> 
>> BitCount	12	14	+16.7%	0.86x
>> SuffixCountableRange	10	11	+10.0%	0.91x
>> MapReduce	303	331	+9.2%	0.92x
>> These are all false changes (and there are quite a few more there).
> 
> The current design assumes that in such cases, the workload will be increased so that is not an issue.

That is also a valid fix for the problem, which I forgot to mention.
-Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20170612/da5dec1c/attachment.html>