[swift-dev] Measuring MEAN Performance (was: Questions about Swift-CI)
atrick at apple.com
Mon Jun 12 20:27:15 CDT 2017
> On Jun 12, 2017, at 5:55 PM, Pavol Vaskovic <pali at pali.sk> wrote:
> On Tue, Jun 13, 2017 at 2:31 AM, Michael Gottesman <mgottesman at apple.com <mailto:mgottesman at apple.com>> wrote:
>> I don't think we can get more consistent test results just from re-running tests that were detected as changes in the first pass, as described in SR-4669 <https://bugs.swift.org/browse/SR-4669>, because that improves accuracy only for one side of the comparison - the branch. When the measurement error is with the baseline from the master, re-running the branch would not help.
> When we are benchmarking, we can always have access to the baseline compiler by stashing the build directory. So we can always take more samples (in fact when I was talking about re-running I always assumed we would).
> Well, if I understand correctly how the swift-CI builds perf-PR, then switching between master and branch from Benchmark_Driver is not possible...
> Or are you thinking about manual benchmarking scenario?
I was thinking (hoping) Benchmark_Driver would support this and we could ask for support from CI to call the driver that way.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the swift-dev