[swift-dev] Measuring MEAN Performance (was: Questions about Swift-CI)
pali at pali.sk
Tue Jun 13 09:11:27 CDT 2017
On Tue, Jun 13, 2017 at 8:51 AM, Andrew Trick <atrick at apple.com> wrote:
> I’m confused though because I thought we agreed that all samples need to
> run with exactly the same number of iterations. So, there would be one
> short run to find the desired `num_iters` for each benchmark, then each
> subsequent invocation of the benchmark harness would be handed `num_iters`
> as input.
That was agreed on in the discussion about measuring memory consumption (PR
MAX_RSS was variable between runs, due to dynamic `num_iters` adjustment
inside `DriverUtils` to fit the ~1s budget.
This could work for keeping the num_iters same during comparison between
the [master] and [branch], give we logged the num_iters from [master] and
used them to drive [branch] MAX_RSS memory. I don't know how to extend this
to make memory consumption comparable between different measurement runs
(over time...), tough.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the swift-dev