<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jun 12, 2017, at 4:45 PM, Pavol Vaskovic <<a href="mailto:pali@pali.sk" class="">pali@pali.sk</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">We really have two problems:</div><div class="">1. spurious results </div><div class="">2. the turnaround time for the entire benchmark suite</div><div class=""><br class=""></div></div></div></blockquote><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div style="font-size: 12.8px;" class="">I don't think we can get more consistent test results just from re-running tests that were detected as changes in the first pass, as described in <a class="gmail-m_2966171682885730286gmail-issue-link" href="https://bugs.swift.org/browse/SR-4669" target="_blank" style="color: rgb(59, 115, 175); font-family: Arial, sans-serif; font-size: 14px; white-space: nowrap; background-color: rgb(245, 245, 245);">SR-4669</a>, because that improves accuracy only for one side of the comparison - the branch. When the measurement error is with the baseline from the master, re-running the branch would not help.</div></div></div></blockquote></div><br class=""><div class="">My understanding of this feature is that it would rerun both branches (or possibly whichever is slower or more jittery, but that’s probably over complicating it).</div><div class=""><br class=""></div><div class="">-Andy</div></body></html>