[swift-dev] Measuring MEAN Performance (was: Questions about Swift-CI)

[Pavol Vaskovic]

On Tue, Jun 13, 2017 at 8:51 AM, Andrew Trick <atrick at apple.com> wrote:

> I’m confused though because I thought we agreed that all samples need to
> run with exactly the same number of iterations. So, there would be one
> short run to find the desired `num_iters` for each benchmark, then each
> subsequent invocation of the benchmark harness would be handed `num_iters`
> as input.
>

That was agreed on in the discussion about measuring memory consumption (PR
8793) <https://github.com/apple/swift/pull/8793#issuecomment-297834790>.
MAX_RSS was variable between runs, due to dynamic `num_iters` adjustment
inside `DriverUtils` to fit the ~1s budget.

This could work for keeping the num_iters same during comparison between
the [master] and [branch], give we logged the num_iters from [master] and
used them to drive [branch] MAX_RSS memory. I don't know how to extend this
to make memory consumption comparable between different measurement runs
(over time...), tough.

--Pavol
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20170613/19233b47/attachment.html>