[swift-evolution] [Review] SE-0019 Swift Testing (Package Manager)

[Paul Cantrell]

> On Jan 5, 2016, at 2:32 PM, Max Howell <max.howell at apple.com> wrote:
> 
>> On Jan 5, 2016, at 12:21 PM, Paul Cantrell <cantrell at pobox.com> wrote:
>> 
>>> On Jan 5, 2016, at 2:14 PM, Max Howell <max.howell at apple.com> wrote:
>>> 
>>>> 1. The proposal talks in several places about “test modules” (for example, “test-module is created per subdirectory of Tests”). How do these test modules interact with Package.swift? Does each of them have a separate target? If so, how is the test module name specified in the Target(…) entry?
>>> 
>>> There is no support for referring to test targets in the Package.swift as part of this proposal.
>>> 
>>>> 2. Perhaps answered by #1, how does Package.swift specify test-only dependencies (e.g. Quick) that are necessary to build tests, but should not be exported to downstream projects?
>>> 
>>> This is already a part of the package manager: https://github.com/apple/swift-package-manager/pull/74
>> 
>> Cool. Hadn’t seen the private dependencies yet.
>> 
>> Taking 1 and 2 together, does that mean that all dependencies necessary for _all_ the test modules must be downloaded & compiled in order to run _any_ or them? Not ideal, but not a deal-killer either
> 
> Yes, and indeed, this isn’t really acceptable, but I think any changes to how this work would involve a discussion on the swift-build-dev mailing list. Really how targets depend on each other and external dependencies in the Package.swift manifest needs some work in general.

Given that you’ve already recognized the future need for more flexible test-module-specific build config, and now there’s also this concern about per-test-module dependencies, is it worth considering giving each test module its own (possibly implicit) target in Package.swift? That might kill several birds with one stone.

(Poor sweet little birdies. Always getting the short end of that idiom.)

>>>> 3. You propose that “building a module also builds that module's corresponding tests.” Does this apply only to the top-level package, or to all of its dependencies? For example, if my FooApp depends on BarLib, and BarLib’s tests depend on HugeFancyTestFramework, do I have to download and compile HugeFancyTestFramework in order to build FooApp? (Hopefully not!)
>>> 
>>> HugeFancyTestFramework will not be downloaded or built.
>> 
>> That’s a relief!
> 
> Probably people would want an option to do this though. You should be running and checking the tests of your dependencies somewhat regularly.
> 
> But when developing your own software, this would be a waste of engineering time.

Indeed, clearly that should not be the default. In general, assuming unnecessary building to be “almost free” is a mistake. Carthage’s authors hit this: they built for all platforms by default, stubbornly resisted fine-tuning of the build process at the command line, and then were deluged with complaints as soon as popular libs like Alamofire started building for iOS, OS X, and watchOS even for apps that only needed one of the platforms.

However, having the option to run dependency tests seems very nice. Something I tentatively like about this proposal is that it nudges library authors to make their tests work immediately on checkout with no further futzing. Running dependency tests would push further in that direction.

A particular advantage of running all the dependencies’ tests would be to test them against the specific versions of all indirect dependencies used in production. In other words, if MyApp uses FooLib, FooLib depends on BarLib, and FooLib’s author tested against BarLib 1.0 but 1.1 inadvertently introduced a change that breaks FooLib, then running all of MyApp’s dependency tests might catch the breakage.

I realize I’ve wandered from the task at hand. I’ll get a proper review together after digesting. Thanks for the clarifications.

Cheers,

Paul