[swift-dev] smoke test not very smoky?

[Jordan Rose]

> On Oct 19, 2016, at 10:17, Dave Abrahams via swift-dev <swift-dev at swift.org> wrote:
> 
> 
> on Wed Oct 19 2016, Jordan Rose <swift-dev-AT-swift.org> wrote:
> 
>>> On Oct 19, 2016, at 9:44, Dave Abrahams via swift-dev <swift-dev at swift.org> wrote:
>>> 
>>> 
>>> on Wed Oct 19 2016, Dave Abrahams <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:
>>> 
>>>> It still seems like, for a smoke test, we're doing way too much work.
>>>> This appears to be much more than what I get from build-script -t when
>>>> I run tests locally.  Maybe I'm misunderstanding the intended role of
>>>> our smoke tests, but since nobody is correcting me, I'm betting not.
>> 
>> Even smoke tests should run the validation tests…
> 
> What's the point of distinguishing validation from other tests if even
> the smoke tests run them?

I think the validation tests are tests that compiler developers don’t bother running locally before they move to a PR. I’d be very concerned about landing changes in master without having run the validation tests—that would get us back to the days of consistent failures because someone forgot to update compiler_crashers.

> 
>>> Someone wrote to me privately:
>>> 
>>> "buildbot_linux_1404" preset used in Linux smoke test contains
>>> "--long-test".
>>> 
>>> This seems wrong to me.  Can we fix it?
>> 
>> …but I could see "long tests" going either way.
> 
> Well, what is “smoke” supposed to mean?  If it requires running
> validation tests and maybe even long tests, what is the “smoke test”
> distinction *for*?

It only builds and runs one platform per builder (no iOS/watchOS/tvOS), and doesn’t necessarily build and test all the downstream projects.

I agree that if we are to have something called “smoke test”, 40 minutes is too long.

Jordan