<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jun 12, 2017, at 5:55 PM, Pavol Vaskovic <<a href="mailto:pali@pali.sk" class="">pali@pali.sk</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Jun 13, 2017 at 2:31 AM, Michael Gottesman <span dir="ltr" class=""><<a href="mailto:mgottesman@apple.com" target="_blank" class="">mgottesman@apple.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><span class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><div style="font-size:12.8px" class="">I don't think we can get more consistent test results just from re-running tests that were detected as changes in the first pass, as described in <a class="m_-7747474691262636825gmail-m_2966171682885730286gmail-issue-link" href="https://bugs.swift.org/browse/SR-4669" style="color:rgb(59,115,175);font-family:Arial,sans-serif;font-size:14px;white-space:nowrap;background-color:rgb(245,245,245)" target="_blank">SR-4669</a>, because that improves accuracy only for one side of the comparison - the branch. When the measurement error is with the baseline from the master, re-running the branch would not help.<br class=""></div></div></div></div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">When we are benchmarking, we can always have access to the baseline compiler by stashing the build directory. So we can always take more samples (in fact when I was talking about re-running I always assumed we would).</div></div></div></blockquote></div></div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Well, if I understand correctly how the swift-CI builds perf-PR, then switching between master and branch from Benchmark_Driver is not possible...</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">Or are you thinking about manual benchmarking scenario?</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">--Pavol</div></div>
</div></blockquote></div><br class=""><div class="">I was thinking (hoping) Benchmark_Driver would support this and we could ask for support from CI to call the driver that way.</div><div class=""><br class=""></div><div class="">-Andy</div></body></html>