[swift-evolution] [Draft] Test-Only Package Dependencies and Targets

[Ankit Aggarwal]

> On 25-Jan-2017, at 4:02 AM, Robert Widmann via swift-evolution <swift-evolution at swift.org> wrote:
> 
> Hello Swift Community,
> 
> Harlan Haskins and I have been working on libraries <https://github.com/trill-lang> to make interacting with LLVM and Clang’s APIs more elegant with native Swift interfaces.  While writing up the packages we realized the package manager wouldn’t allow us to specify testing targets and test-only dependencies.  To rectify that, I have attached a draft proposal for adding test-only targets and dependency fields to the Swift Package manager.  This proposal can also be read in gist form <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38>.
> 
> Cheers,
> 
> ~Robert Widmann
> 

Thanks for driving this! It is a very desirable feature which needs proposal work. Comments inline.



> Test-Only Package Dependencies and Targets
> 
> Proposal: SE-NNNN <https://gist.github.com/CodaFi/NNNN-filename.md>
> Authors: Harlan Haskins <https://github.com/harlanhaskins>, Robert Widmann <https://github.com/codafi>
> Review Manager: TBD
> Status: Awaiting review
>  <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38#introduction>Introduction
> 
> This proposal reinstates Swift package manager’s ability to fetch dependencies and build targets scoped exclusively to the testing module(s) of a given package.
> 
> Swift-evolution thread: Discussion thread topic for that proposal <https://lists.swift.org/pipermail/swift-evolution/>
>  <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38#motivation>Motivation
> 
> Soon after SE-0019 <https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md#test-only-dependencies> identified the need for richer test-only dependencies and targets, a decision was made to remove the package manager’s fledgling ability to treat certain dependencies as test-only. This has led to a myriad of clever-but-needlessly-complex workarounds ([1] <https://github.com/ReactiveCocoa/ReactiveSwift/blob/master/.travis.yml#L85>, [2] <https://github.com/ReactiveX/RxSwift/blob/master/Package.swift#L3>, [3] <https://github.com/Quick/Quick/blob/master/.Package.test.swift>) on the part of 3rd parties to recover the feature themselves. In addition, the Swift community has come up with a number of their own frameworks to augment functionality in XCTest but depending on these external testing frameworks is brittle and difficult to get right.
> 
>  <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38#proposed-solution>Proposed solution
> 
> We propose the re-introduction of the testDependencies parameter in Package Manifests to support external test-only dependencies. To support local test-only targets we also propose the introduction of the testTargets parameter and an extension of the existing swift test command to support individual invocation of these targets.
> 
>  <https://gist.github.com/CodaFi/6bd83e5315c7d30aeaf4154ed3b03a38#detailed-design>Detailed design
> 
> The behavior of the new testDependencies parameter mirrors that of the existing dependencies parameter with one important difference: fetched dependencies are only built to support package-defined test targets as part of an invocation of swift test.
> 
> import PackageDescription
> 
> let package = Package(
>     name: "Foo",
>     targets: [
>         Target(name: "Foo")
>     ],
>     dependencies: [
>         .Package(url: "https://github.com/org/ana.git <https://github.com/org/ana.git>", versions: Version(1,0,0)...Version(1,9,9)),
>     ],
>     testDependencies: [
>         .Package(url: "https://github.com/org/anism.git <https://github.com/org/anism.git>", versions: Version(1,0,0)...Version(1,9,9)),
>     ]
> )

I think this feature should be called local dependencies (or maybe dev dependencies) because it can be used for tests as well as regular targets. 
As an example say you have a networking library package and you want to create an example CLI target which uses a JSON mapper package. You wouldn't want to vend the CLI tool when you act as a dependency to other packages, or include the JSON mapper in your dependencies. Test dependency doesn't sound right in that context.

After the product proposal <https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md> is implemented, you will be able to control what you vend to your clients, and we thought about adding ability to define dependencies in-line with target dependencies but left it out of the proposal to keep it simpler. Maybe this proposal can add that instead of a separate `testDependencies` property. Consider this manifest:

let package = Package(
    name: "FooNetworking",
    targets: [
        Target(
            name: "FooNetworking"),
        Target(
            name: "FooNetworkingExample",
            dependencies: [
                "FooNetworking", 
                .package(url: "/path/to/BarJSON.git", version: "1.0.0", product: "BarJSON"), // Note: This doesn't actually exists right now.
            ]),
        Target(
            name: "FooNetworkingTests",
            dependencies: [
                "FooNetworking", 
                .package(url: "/path/to/Quick.git", version: "1.0.0", product: "Quick"), // Note: This doesn't actually exists right now.
            ]),
   ],
   products: [
       .Library(name: "FooNetworking", targets: ["FooNetworking"]),
   ]
)

This manifest defines three targets and one product.

FooNetworking: The base target and the actual library.
FooNetworkingExample: The example cli tool. It depends on FooNetworking target and BarJSON product from an external package.
FooNetworkingTests: The test target depends on an external package Quick.

Both BarJSON and Quick are local dependencies to this package. If FooNetworkingExample was also vended as a product, BarJSON would automatically become a regular external dependency.

> Similarly, the behavior of the testTargets field mirrors that of the existing targets field but defines a set of targets that are only built during an invocation of swift test. Importantly, a target defined in testTargets may reference a target defined in targets but not vice-versa. Should that behavior be needed, the test target should be promoted to a “full” target.
> 
> import PackageDescription
> 
> let package = Package(
>     name: "SwiftPM",
>     targets: [
>         Target(
>             name: "PackageDescription",
>             dependencies: []),
> 
>         // MARK: Support libraries
> 
>         Target(
>             /** Cross-platform access to bare `libc` functionality. */
>             name: "libc",
>             dependencies: []),
>         Target(
>             /** “Swifty” POSIX functions from libc */
>             name: "POSIX",
>             dependencies: ["libc"]),
>         Target(
>             /** Basic support library */
>             name: "Basic",
>             dependencies: ["libc", "POSIX"]),
> 
>         /* Omitted for Brevity */
>      ],
>      testTargets: [
>         Target(
>             name: "BasicPerformanceTests",
>             dependencies: ["Basic"]),
>        /* Omitted for Brevity */
>      ]
>  )
> Finally, with well-defined test targets in hand, we propose swift test be amended to support individual test execution.
> 
> 
> We propose the following syntax to execute all tests of all known test targets.
> 
> $ swift test
> To run a set of specific test cases, reference the module-defining test target and the specific name of a subclass of XCTestCase:
> 
> $ swift test TestModule.TestCase
> To run an individual test case, reference the module-defining test target, the name of the test case subclass, and the name of the test:
> 
> $ swift test TestModule.TestCase.exampleTest

We already have all these features. A target is a test target if it has a suffix Tests and is placed under Tests/ directory — we might loosen these restrictions with custom target conventions feature. This proposal <https://github.com/apple/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md> could clarify further.

Here is a summary of what `swift test` currently supports:
* Test targets are only built (and run) when you run `swift test`.
* `swift test --skip-build` skips building of test targets and runs whatever was last built.
* `swift test -s <test-module>.<test-case>` will run a test case class.
* `swift test -s <test-module>.<test-case>/<test>` will run an individual test.
* `swift test --list-tests` will list all tests in above format.
* `swift test --parallel` will run tests in parallel.

You can view these options and their help text using `swift test --help`.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20170125/5dc2700c/attachment.html>