[swift-evolution] [Review] SE-0019 Swift Testing (Package Manager)

[Max Howell]

>> Yes, and indeed, this isn’t really acceptable, but I think any changes to how this work would involve a discussion on the swift-build-dev mailing list. Really how targets depend on each other and external dependencies in the Package.swift manifest needs some work in general.
> 
> Given that you’ve already recognized the future need for more flexible test-module-specific build config, and now there’s also this concern about per-test-module dependencies, is it worth considering giving each test module its own (possibly implicit) target in Package.swift? That might kill several birds with one stone.

I’m not sure it’s worth it yet. You can’t really do anything with targets in Package.swift yet (just specify internal inter-module dependencies). I’d rather wait until the manifest format had some more functionality and had settled more, rather than slap things in without more understanding about how the manifest is going to actually be used.

>>>>> 3. You propose that “building a module also builds that module's corresponding tests.” Does this apply only to the top-level package, or to all of its dependencies? For example, if my FooApp depends on BarLib, and BarLib’s tests depend on HugeFancyTestFramework, do I have to download and compile HugeFancyTestFramework in order to build FooApp? (Hopefully not!)
>>>> 
>>>> HugeFancyTestFramework will not be downloaded or built.
>>> 
>>> That’s a relief!
>> 
>> Probably people would want an option to do this though. You should be running and checking the tests of your dependencies somewhat regularly.
>> 
>> But when developing your own software, this would be a waste of engineering time.
> 
> Indeed, clearly that should not be the default. In general, assuming unnecessary building to be “almost free” is a mistake. Carthage’s authors hit this: they built for all platforms by default, stubbornly resisted fine-tuning of the build process at the command line, and then were deluged with complaints as soon as popular libs like Alamofire started building for iOS, OS X, and watchOS even for apps that only needed one of the platforms.
> 

> However, having the option to run dependency tests seems very nice. Something I tentatively like about this proposal is that it nudges library authors to make their tests work immediately on checkout with no further futzing. Running dependency tests would push further in that direction.
> 
> A particular advantage of running all the dependencies’ tests would be to test them against the specific versions of all indirect dependencies used in production. In other words, if MyApp uses FooLib, FooLib depends on BarLib, and FooLib’s author tested against BarLib 1.0 but 1.1 inadvertently introduced a change that breaks FooLib, then running all of MyApp’s dependency tests might catch the breakage.

I agree and I wish we could force all tests to be run. But this is just idealism. Practically all of us would just start building with the “don’t do that flag”. 

I think at the least when we have a “publish” command we will run all the tests. As well as lint. In day to day development we cannot expect developers to wait, but we have a duty to ensure the packaging ecosystem we help create is as high quality as possible.

Max