<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place.<div class=""><br class=""></div><div class="">We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <<a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><blockquote type="cite" class=""><div class=""><span class="" style="font-family: HelveticaNeue; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?</span></div></blockquote></div><br class=""><div class="">Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.</div></div></div></blockquote><div class=""><br class=""></div><div class="">That’s our current thinking.</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.</div><div class=""><br class=""></div><div class="">There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.</div><div class=""><br class=""></div><div class="">I'm simply pushing that decision out to individual test frameworks to study. XCTest thinks you should have test suites that are human curated. That's one way to skin the cat. Let's give another test framework the flexibility to think differently on this problem.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Sounds super interesting. We should make sure the protocol we design can do these things.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><blockquote type="cite" class=""><span class="" style="font-family: HelveticaNeue;">I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.</span></blockquote><br class=""></div><div class="">Thanks for picking this up. Getting basic testing support in place is really important, and IMO the current proposal is really strong. I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :-)</div></div></div></blockquote></div><br class="" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><img src="https://u2002410.ct.sendgrid.net/wf/open?upn=cbMbdH1LnH6O78Q-2BHw3jtU8ikibH470Fh9meAJKpwSqpJeTKnMxtBKMcwgO6eNd7RiBAIn9BZYPOs2eHgzutG3iFkECUBpSh2YJRbpFs5gYIFdbbC8ItMNA7gp-2BqGK4cWwBss4mY6Uv8sMAwKkzb1-2B1rswus8P2lPVMLer7xFPfSEQS6c96lmiGJyiZhqf-2F-2BJOnpV-2BhXXERxFHP94bzeH6uMWF-2B662RYrOYdPl-2FmMCE-3D" alt="" width="1" height="1" border="0" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; height: 1px !important; width: 1px !important; border-width: 0px !important; margin: 0px !important; padding: 0px !important;" class=""><span style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><span class="Apple-converted-space"> </span>_______________________________________________</span><br style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">swift-build-dev mailing list</span><br style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:swift-build-dev@swift.org" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">swift-build-dev@swift.org</a><br style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">https://lists.swift.org/mailman/listinfo/swift-build-dev</a></div></blockquote></div><br class=""></div></body></html>