[swift-build-dev] Draft proposal: "Test Executables"

[George King]

Daniel, thanks for the feedback.

First, a process question: in general, do package manager proposals end up in the swift-evolution repo, or is that only for the language proper? If so, then I'll put it in a github fork of swift-evolution so that we can track changes. If not, then what is the final destination?

More comments inline...

> On Jul 8, 2016, at 6:43 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> 
> Hi George!
> 
> Generally speaking, I like the direction for SwiftPM of making unit tests consistent with the other kinds of targets, and so the "is a test" becomes just a flag for each target rather than a somewhat different kind of target.
> 
> Your proposal calls out a couple things that there aren't concrete details on: why exactly do we need the output file with a list of test executables? Where and when would it be written, and how would other targets find it? I suspect this is something we should consider independently, and I would recommend exploring whether it is something we should expose as an API targets could use instead of a build artifact.

My basic assumption in all of this is that XCTest is not currently equipped to do process IO testing, and that as a first step, executable tests would be run by a third party test harness. The nature of the harness is left unspecified, apart from being a command line tool.

The output list is, I admit, not so well thought out. I was thinking that it was a simple way for swiftpm to communicate to a harness script the executables that had been produced.

Take this example harness.sh (totally made up):
```
set -e
swift test # if the XCTest unit tests fail, or any of the test executables do not build, this exits nonzero and the harness aborts.
built_test_executables_path='.build/built_test_executables.txt' # the output list - this name would be a swiftpm convention.
for test in $(cat "$built_test_executables_path"); do
  echo "running executable test: $test"
  $test # if the test fails, harness aborts.
  echo
done
```

A real testing framework would specify test expectations for each of these executables, so the list might not be so useful as I had initially thought.

The real problem to address is that swiftpm must communicate the availability of up-to-date test executables to the third party harness. If it doesn't, then when a build fails, the harness is liable to test a stale executable. Perhaps the simple solution is for swiftpm to delete the previous product when a build starts, so that consumers cannot possibly run tests on stale products.

My other concern is that we don't want harnesses trying to parse the output messages of swiftpm, and the output list concept was an attempt to mitigate that. At the moment, I think that as long as stale products are removed and their locations are sufficiently stable and obvious, a 3rd party harness should not need any further information from swiftpm because it ought to have test expectations written down anyway.


> The big part of this proposal is a change to the conventions for test targets, and I'm not sure exactly what convention you are suggesting -- that `main.swift` inside a module within Tests just become an executable target?

Yes. I'm proposing that it parallel the existing convention for package targets; the presence of main.swift distinguishes a library from an executable in `Sources`, and it could do the exact same thing in `Tests`.


> I am tempted towards a slight different direction:
> 
>  - Rename test targets to `<NAME>Tests`, which matches the Xcode convention and solves an existing problem where the module name of the test module isn't apparent in the file system (despite being required to be named in the source code sometimes).

I'm not sure what you mean; at first I interpreted this as suggesting that for package `Foo` there would be a top level directory called `FooTests` instead of `Tests`. But from your example below, now I don't think so. Do you mean that a `Tests/Foo` compiles to `FooTests`?

Regardless, I think we should strive to enhance the parallels between how `Sources` works, and how `Tests` works.
* Swiftpm looks for sources in just two canonical places: `Sources` or `src`. This is a nice, easy convention. Looking for sources in `Tests` similar.
* Swiftpm can either interprets the contents of Sources as "single target" (sources contains source files), or "multiple target" (sources contains subdirectories of source files). `Tests` should be the same.
* In the "single target" interpretation of the `Tests` directory contents, swiftpm could infer a name for the test module, just like it does for a single target in sources. This could be `<PACKAGE-NAME>Tests`.
* In the "multiple target" interpretation of `Tests`, the test modules are named according to the subdirectories (I think this is how it works now). Maybe you just mean appending the "Tests" suffix to these, which seems fine.

>  - Accept all target forms under Tests/, so `Tests/foo/main.swift` would declare a new executable `foo`, just like it would in Sources. The only special behavior of these targets would be that they wouldn't automatically be built by downstream package dependencies.

Yes, this is what I had in mind. I assume that downstream packages don't build the unit tests either; again, I advocate for parallel behavior.

> Are there other reasons why we need a separate "test executable" concept other than just putting another executable into `Sources/`, if SwiftPM itself isn't going to tie custom behaviors to them? Currently I can think of two reasons: organizational clarity and the inability to specify dependencies of test modules.

I agree about organizational clarity.

I do not know what you mean about inability to specify dependencies of test modules. Is this because you cannot write them down in Package.swift, like you would for a regular target? Is there any reason why we cannot just support the same sorts of specifications of dependencies (and anything else) for tests as we do for regular targets?

More important than not building the executables downstream, I don't want to expose the test executables as public, whatever that means. As far as I can tell, swiftpm does not yet do anything to make package executables available to the end user, but it might in the future. For example, Pypi can generate "entry points" that end up in the python bin directory. If swiftpm started making packages executables available to the user on the PATH or any way, keeping tests out of that process would be important.

Put another way, it does not seem far fetched that swiftpm could gain an `swift package install` command or similar, and it would be unfortunate if published packages had in the meantime been publishing test executables as public targets.


> Bear in mind that the perspective I am coming from is that even if you want to do integration tests of test executables, the "driver" of those executables would still be an XCTest module adapting them, at least until we had third-party testing support.

This is the basic source of confusion I think; you mentioned in our initial slack chat that 3rd party testing frameworks was a concern, so I leapt to the conclusion that XCTest would not be the driver.

My perspective is that I already have my own harness written (github.com/gwk/iotest <http://github.com/gwk/iotest> - it's for personal use and depends on a fast-changing utility library so it might appear broken on a given day). I use it to test executable targets that I'm producing via swiftpm now.

I'm curious to hear a bit more about what you consider to be "3rd party testing support", because apart from this proposal, I can't think of anything else that iotest needs from the package manager!


Finally, I should add a few motivating examples for why test executables would be useful. The obvious case is for libraries that offer IO functionality; imagine a command line argument parser library, or an error logging framework. I have found that it is easiest to test these kinds of things with a harness that can specify environment, process arguments, stdin, and current working directory structure, and then capture exit status, stdout and stderr from a test process. iotest does this, and has a few nice tricks like printing diffs of the expected and actual output. Offering these sorts of features through XCTest would be a much larger proposal, so I think it is worth trying to carve out the test executable part as a first step.

> 
>  - Daniel
> 
> On Thursday, July 7, 2016, George King via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
> Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George
> 
> Test Executables
> 
> Proposal: SE-NNNN <http://nnnn-filename.md/>
> Author: George King <https://github.com/gwk>
> Status: Awaiting review
> Review manager: TBD
> Introduction
> 
> The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.
> 
> Motivation
> 
> Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.
> 
> XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.
> 
> Consider the existing features of swiftpm as two orthogonal axes:
> 
> Build, Test
> Library, Executable
> Of the four combinations, only "Test Executable" is missing.
> 
> One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.
> 
> Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:
> 
> Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
> For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
> Test executables would only be built by swift test, not swift build.
> Proposed solution
> 
> swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.
> 
> The test command should, upon encountering a test executable directory, build the executable.
> 
> If the test executable build fails, test should report a test failure and return a nonzero exit code.
> 
> If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.
> 
> Future improvements could include invoking a test command specified in Package.swift upon successful build.
> 
> Detailed design
> 
> I am unclear on the exact semantics of @testable import (I see no difference when I remove the at testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.
> 
> swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.
> 
> Impact on existing code
> 
> None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.
> 
> Alternatives considered
> 
> No action
> 
> See the list of problems with treating test executables as regular targets in the "Motivation" section above.
> 
> Complete integration testing with XCTest
> 
> This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160709/8e954a95/attachment.html>