<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jun 12, 2017, at 2:55 PM, Michael Gottesman <<a href="mailto:mgottesman@apple.com" class="">mgottesman@apple.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Current approach to detecting performance changes is fragile for tests that have very low absolute runtime, as they are easily over the 5% improvement/regression threshold when the test machine gets a little bit noisy. For example in<span class="Apple-converted-space"> </span><a href="https://github.com/apple/swift/pull/9806#issuecomment-303370149" target="_blank" class="">benchmark on PR #9806</a>:</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><table class="" style="box-sizing: border-box; border-collapse: collapse; margin-top: 0px; margin-bottom: 16px; display: block; width: 668px; overflow: auto; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 14px;"><tbody class="" style="box-sizing: border-box;"><tr class="" style="box-sizing: border-box; border-top-width: 1px; border-top-style: solid; border-top-color: rgb(198, 203, 209);"><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">BitCount</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">12</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">14</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">+16.7%</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);"><span class="" style="box-sizing: border-box; font-weight: 600;">0.86x</span></td></tr><tr class="" style="box-sizing: border-box; background-color: rgb(246, 248, 250); border-top-width: 1px; border-top-style: solid; border-top-color: rgb(198, 203, 209);"><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">SuffixCountableRange</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">10</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">11</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">+10.0%</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);"><span class="" style="box-sizing: border-box; font-weight: 600;">0.91x</span></td></tr><tr class="" style="box-sizing: border-box; border-top-width: 1px; border-top-style: solid; border-top-color: rgb(198, 203, 209);"><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">MapReduce</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">303</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">331</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">+9.2%</td><td class="" style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);"><span class="" style="box-sizing: border-box; font-weight: 600;">0.92x</span></td></tr></tbody></table></blockquote><div class="">These are all false changes (and there are quite a few more there).</div></div></div></div></div></blockquote><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">The current design assumes that in such cases, the workload will be increased so that is not an issue.</div></div></div></div></blockquote></div><br class=""><div class="">That is also a valid fix for the problem, which I forgot to mention.</div><div class="">-Andy</div></body></html>