[swift-dev] Measuring MEAN Performance (was: Questions about Swift-CI)

[Pavol Vaskovic]

On Tue, Jun 13, 2017 at 2:31 AM, Michael Gottesman <mgottesman at apple.com>
wrote:
>
> I don't think we can get more consistent test results just from re-running
> tests that were detected as changes in the first pass, as described in
> SR-4669 <https://bugs.swift.org/browse/SR-4669>, because that improves
> accuracy only for one side of the comparison - the branch. When the
> measurement error is with the baseline from the master, re-running the
> branch would not help.
>
>
> When we are benchmarking, we can always have access to the baseline
> compiler by stashing the build directory. So we can always take more
> samples (in fact when I was talking about re-running I always assumed we
> would).
>

Well, if I understand correctly how the swift-CI builds perf-PR, then
switching between master and branch from Benchmark_Driver is not possible...

Or are you thinking about manual benchmarking scenario?

--Pavol
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20170613/72632994/attachment.html>