<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">So I did a bit more research. Check out how LNT does this:<br class=""><div class=""><br class=""></div><div class=""><a href="https://github.com/llvm-mirror/lnt/search?utf8=%E2%9C%93&q=mann-whitney&type=" class="">https://github.com/llvm-mirror/lnt</a></div><div class=""><br class=""></div><div class="">I talked with Chris Matthews (+CC) about how LNT uses Mann-Whitney. In the following let n be the number of samples taken. From what he told me this is what LNT does:</div><div class=""><br class=""></div><div class="">1. If n is < 5, then some sort of computation around confidence intervals is used.</div><div class="">2. If the number of samples is > 5, then Mann-Whitney U is done.</div><div class=""><br class=""></div><div class="">I am not 100% sure what 1 is, but I think it has to do with some sort of quartile measurements. I.e. Find the median of the new data and make sure it is within +- median absolute deviation (basically mean + std-dev but more robust to errors). I believe the code is in LNT so we can find it for sure.</div><div class=""><br class=""></div><div class="">Thus in my mind the natural experiment here in terms of Mann-Whitney U.</div><div class=""><br class=""></div><div class="">1. This seems to suggest that for small numbers we do some sort of simple comparison that we do today and if a regression is "identified", we grab more samples of before/after and run mann-whitney u.</div><div class="">2. Try out different versions of n. I am not 100% sure if 5 is the right or wrong answer or if it should be dependent on the test.</div><div class=""><br class=""></div><div class="">Chris, did I get it right?</div><div class=""><br class=""></div><div class="">Michael</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Jun 13, 2017, at 7:11 AM, Pavol Vaskovic via swift-dev <<a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">On Tue, Jun 13, 2017 at 8:51 AM, Andrew Trick <span dir="ltr" class=""><<a href="mailto:atrick@apple.com" target="_blank" class="">atrick@apple.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><span class=""><div class="">I’m confused though because I thought we agreed that all samples need to run with exactly the same number of iterations. So, there would be one short run to find the desired `num_iters` for each benchmark, then each subsequent invocation of the benchmark harness would be handed `num_iters` as input.<br class=""></div></span></div></div></blockquote></div><br class=""></div><div class="gmail_extra">That was agreed on in the <a href="https://github.com/apple/swift/pull/8793#issuecomment-297834790" class="">discussion about measuring memory consumption (PR 8793)</a>. MAX_RSS was variable between runs, due to dynamic `num_iters` adjustment inside `DriverUtils` to fit the ~1s budget.</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">This could work for keeping the num_iters same during comparison between the [master] and [branch], give we logged the num_iters from [master] and used them to drive [branch] MAX_RSS memory. I don't know how to extend this to make memory consumption comparable between different measurement runs (over time...), tough.</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">--Pavol</div></div>
_______________________________________________<br class="">swift-dev mailing list<br class=""><a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-dev<br class=""></div></blockquote></div><br class=""></div></body></html>