[swift-build-dev] Test-only dependencies

[Paul Cantrell]

I’ve been working on getting Siesta up and running with SwiftPM, and have hit snags running the tests. At Ankit Aggarwal’s encouragement, I’m opening a discussion thread here.

By “test-only dependencies,” I mean external packages required by a package’s tests, but not by the package's exported targets. For example, Siesta’s tests depend on Quick, Nimble, and Nocilla, but apps that use Siesta should not have to build those dependencies and would very much not want them baked into the resulting binary.

Proper support for test-only dependencies would have three essential features:

They are not downloaded or built by “swift build,” but are downloaded and built by “swift test.”
Any modules they define do not end up in the compiled binary, and are not available for the package’s non-test targets to import.
A package ignores the test-only dependencies of its dependencies.

Those are the essentials. Also nice to have:

The ability to limit test-only dependencies to specific test modules (e.g. an integration test module might pull in some heavyweight context that isn’t necessary or desirable for basic regression tests).
The ability to import / run tests defined in an external module. (Probably a separate discussion.)

To open the discussion, I see three general approaches:

Approach 1: Special testDependencies attribute

It looks like SwiftPM flirted with this, then abandoned it?

Pros: Clear.

Cons: Creates (another) special exception for tests. Doesn’t obviously generalize to handle the “specific test modules” nice-to-have above.

Approach 2: Per-target external dependencies

SwiftPM could allow individual targets to specify external dependencies, something like this:

    targets: [
      Target(
        name: "SiestaTests",
        dependencies: [
          .Package(url: "https://github.com/Quick/Quick", majorVersion: 0)
        ])
    ],

Pros: Covers the “essential features” above in a way that generalizes nicely to other interesting use cases beside testing.

Cons: Requires redundant dependency declarations across test targets.

Approach 3: Target groups

Bundler provides arbitrary logical gem groupings, which Rails uses to isolate dependencies to development, test, and/or production environments. For example:

    group :development, :test do
      gem 'byebug'
      gem 'spring'
    end

    group :test do
      gem 'minitest-spec-rails'
      gem 'factory_girl_rails'
    end

    group :development
      gem 'letter_opener'
    end

    group :production
      gem 'puma'
    end

Apps can then decide which groups to include depending on their runtime environment.

Pros: Gives the clarity & conciseness of #1 with the generality benefits of #2.

Cons: Lots of additional structure here, both explicit and implicit. Not clear how useful this would be in SwiftPM: because Ruby is so dynamic, all dependencies are runtime dependencies, so code can decide whether it’s “in test mode” or “in production” before looking for modules. In SwiftPM, all dependencies are compile-time dependencies, so arbitrary logical groups lose the benefits that come with runtime resolution.

________


I’m throwing all of this out there in a state of general ignorance about SwiftPM’s direction and status. Hopefully it spurs some useful discussion.

Cheers,

Paul

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160828/e3aaf46e/attachment.html>