<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">IMO, we should remove/disable this test unless it can be “really” fixed. It is unacceptable to have a test that we know injects noise into our CI systems. Having to second guess whether failures are “real” or not undermines their value, and we should continue to stomp out any nondeterminism from the build and test cycle.<div class=""><br class=""></div><div class="">-Chris</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 16, 2016, at 11:34 AM, Ben Langmuir via swift-dev <<a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I was tired of seeing and writing these emails:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(198, 153, 28); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">commit 24f685c1d29cd926b1674e099bfcca6ed8673491</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Author: Ben Langmuir <<a href="mailto:blangmuir@apple.com" class="">blangmuir@apple.com</a>></span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Date: Wed Mar 16 10:36:02 2016</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227); min-height: 17px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Decrease the probability that stdlib/Hashing.swift will fail</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227); min-height: 17px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> </span><br class="webkit-block-placeholder"></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Bump the number of trials from 10 to 14, which makes the test orders of</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> magnitude less likely to fail. For a range of size 10, doing 10 trials</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> meant that a single call to checkRange would fail ~0.03% of the time;</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> with 14 trials the spurious failure rate is ~0.0003%. We have 10 calls</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> to checkRange in this test with ranges of size <= 10.</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227); min-height: 17px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> </span><br class="webkit-block-placeholder"></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> While this test didn't fail that often before, we have a very large</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> number of automated builds and it has been a constant low-level source</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> of friction that this test fails and we have to look at and then ignore</span></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; color: rgb(120, 142, 149); background-color: rgb(253, 246, 227);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> the results.</span></div></div><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Mar 16, 2016, at 10:01 AM, Jordan Rose via swift-dev <<a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Unrelated and harmless: this test is probabilistic.</div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Mar 16, 2016, at 10:00 , <a href="mailto:no-reply@swift.org" class="">no-reply@swift.org</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><h2 style="font-family: 'Helvetica Neue', sans-serif; font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; width: 890px; background-color: red; color: white; text-align: center;" class=""></h2><h1 style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">[<font class="FAILURE" style="color: red;">FAILURE</font>] oss-swift-incremental-RA-osx [#2761]</h1><table style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><tbody class=""><tr style="text-align: left;" class=""><th class="">Build URL:</th><td class=""><a href="https://ci.swift.org/job/oss-swift-incremental-RA-osx/2761/" style="color: rgb(0, 122, 255); text-decoration: none;" class="">https://ci.swift.org/job/oss-swift-incremental-RA-osx/2761/</a></td></tr><tr style="text-align: left;" class=""><th class="">Project:</th><td class="">oss-swift-incremental-RA-osx</td></tr><tr style="text-align: left;" class=""><th class="">Date of build:</th><td class="">Wed, 16 Mar 2016 09:31:51 -0700</td></tr><tr style="text-align: left;" class=""><th class="">Build duration:</th><td class="">29 min</td></tr></tbody></table><h2 style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Identified problems:</h2><ul style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><li class="">Regression test failed: This build failed because a regression test in the test suite FAILed. Below is a list of all errors:<ul class=""><li class=""><a href="https://ci.swift.org//job/oss-swift-incremental-RA-osx/2761/consoleFull#-1004795036fca400bf-2f4a-462e-b517-e058d770b2d7" style="color: rgb(0, 122, 255); text-decoration: none;" class="">Indication 1</a></li></ul></li></ul><h2 style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Tests:<span class="Apple-converted-space"> </span></h2><table width="100%" style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><tbody class=""><tr style="text-align: left;" class=""><td class="bg1" colspan="2"></td></tr><tr style="text-align: left;" class=""><td class="bg2" colspan="2">Name:<span class="Apple-converted-space"> </span><b class="">Swift</b><br class="">Failed: 1 test(s), Passed: 7772 test(s), Total: 7773 test(s)</td></tr><tr bgcolor="white" style="text-align: left;" class=""><td class="test_failed" colspan="2"><b class=""><ul class=""><li class="">Failed:<span class="Apple-converted-space"> </span><a href="https://ci.swift.org/job/oss-swift-incremental-RA-osx/2761/testReport/junit/Swift/stdlib/Hashing_swift" style="color: rgb(0, 122, 255); text-decoration: none;" class="">Swift.stdlib.Hashing.swift</a></li></ul></b></td></tr><tr style="text-align: left;" class=""><td class=""></td></tr><tr style="text-align: left;" class=""><td class="bg2" colspan="2">Name:<span class="Apple-converted-space"> </span><b class="">Swift-Unit</b><br class="">Failed: 0 test(s), Passed: 278 test(s), Total: 278 test(s)</td></tr><tr style="text-align: left;" class=""><td class=""></td></tr></tbody></table><br style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><h2 style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Changes</h2><ul style="color: rgb(57, 57, 57); font-family: 'Helvetica Neue', sans-serif; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><li class="">Commit<span class="Apple-converted-space"> </span><b class="">c43943d7a884f7b43a6ac06877c29656d5214653</b><span class="Apple-converted-space"> </span>by<span class="Apple-converted-space"> </span><b class="">jordan_rose:</b><p class="commit-message" style="font-family: Menlo, fixed; white-space: pre-wrap;">[build-script] Remove -X option for "edit-only" Xcode projects.</p><ul class=""><li class="change-edit"><b class="">edit</b>: utils/build-script-impl</li><li class="change-edit"><b class="">edit</b>: README.md</li><li class="change-edit"><b class="">edit</b>: CMakeLists.txt</li><li class="change-edit"><b class="">edit</b>: cmake/modules/SwiftAddCustomCommandTarget.cmake</li><li class="change-edit"><b class="">edit</b>: utils/build-script</li></ul></li></ul></div></blockquote></div><br class=""></div>_______________________________________________<br class="">swift-dev mailing list<br class=""><a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a><br class=""><a href="https://lists.swift.org/mailman/listinfo/swift-dev" class="">https://lists.swift.org/mailman/listinfo/swift-dev</a><br class=""></div></blockquote></div><br class=""></div>_______________________________________________<br class="">swift-dev mailing list<br class=""><a href="mailto:swift-dev@swift.org" class="">swift-dev@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-dev<br class=""></div></blockquote></div><br class=""></div></body></html>