[swift-build-dev] Draft proposal: "Test Executables"

[Daniel Dunbar]

Hi George,

Sorry for the delayed reply, I lost track of this while on vacation.

> On Jul 9, 2016, at 2:31 PM, George King via swift-build-dev <swift-build-dev at swift.org> wrote:
> 
> Daniel, thanks for the feedback.
> 
> First, a process question: in general, do package manager proposals end up in the swift-evolution repo, or is that only for the language proper? If so, then I'll put it in a github fork of swift-evolution so that we can track changes. If not, then what is the final destination?

They do end up in the swift-evolution repo, usually once the proposal is ready for official review and accept/reject stamp then a PR is created for the swift-evolution repo. Once review starts it is assigned a number and a feedback period.

> More comments inline...
> 
>> On Jul 8, 2016, at 6:43 PM, Daniel Dunbar <daniel at zuster.org <mailto:daniel at zuster.org>> wrote:
>> 
>> Hi George!
>> 
>> Generally speaking, I like the direction for SwiftPM of making unit tests consistent with the other kinds of targets, and so the "is a test" becomes just a flag for each target rather than a somewhat different kind of target.
>> 
>> Your proposal calls out a couple things that there aren't concrete details on: why exactly do we need the output file with a list of test executables? Where and when would it be written, and how would other targets find it? I suspect this is something we should consider independently, and I would recommend exploring whether it is something we should expose as an API targets could use instead of a build artifact.
> 
> My basic assumption in all of this is that XCTest is not currently equipped to do process IO testing, and that as a first step, executable tests would be run by a third party test harness.

It is possible to use XCTest to drive subprocess for IO testing, and you can even use dynamic test reporting to expose the results as distinct tests (although I am not sure if this can be done on Linux). So it should be possible, it just might not be convenient (e.g., if your "XCTest" ends up being call this subprocess and expect it to report 0).

In the interest of ensuring a focused & tractable proposal I would recommend decoupling this proposal itself from the 3rd-party test harness problem, which also deserves a proposal. That way each is easy to design and discuss.

> The nature of the harness is left unspecified, apart from being a command line tool.
> 
> The output list is, I admit, not so well thought out. I was thinking that it was a simple way for swiftpm to communicate to a harness script the executables that had been produced.

I propose just dropping this requirement to start with... individual projects will already know the names of the executables they expected to be produced. If we do want this to be automated, then it probably makes the most sense to be a general purpose feature for SwiftPM to report all of its artifacts, which could be used for a wide variety of purposes.

> Take this example harness.sh (totally made up):
> ```
> set -e
> swift test # if the XCTest unit tests fail, or any of the test executables do not build, this exits nonzero and the harness aborts.
> built_test_executables_path='.build/built_test_executables.txt' # the output list - this name would be a swiftpm convention.
> for test in $(cat "$built_test_executables_path"); do
>   echo "running executable test: $test"
>   $test # if the test fails, harness aborts.
>   echo
> done
> ```
> 
> A real testing framework would specify test expectations for each of these executables, so the list might not be so useful as I had initially thought.
> 
> The real problem to address is that swiftpm must communicate the availability of up-to-date test executables to the third party harness. If it doesn't, then when a build fails, the harness is liable to test a stale executable. Perhaps the simple solution is for swiftpm to delete the previous product when a build starts, so that consumers cannot possibly run tests on stale products.
> 
> My other concern is that we don't want harnesses trying to parse the output messages of swiftpm, and the output list concept was an attempt to mitigate that. At the moment, I think that as long as stale products are removed and their locations are sufficiently stable and obvious, a 3rd party harness should not need any further information from swiftpm because it ought to have test expectations written down anyway.

We don't currently ensure stale products are always removed, but I think we should treat that as a bug not something we need a feature for.

>> The big part of this proposal is a change to the conventions for test targets, and I'm not sure exactly what convention you are suggesting -- that `main.swift` inside a module within Tests just become an executable target?
> 
> Yes. I'm proposing that it parallel the existing convention for package targets; the presence of main.swift distinguishes a library from an executable in `Sources`, and it could do the exact same thing in `Tests`.
> 
> 
>> I am tempted towards a slight different direction:
>> 
>>  - Rename test targets to `<NAME>Tests`, which matches the Xcode convention and solves an existing problem where the module name of the test module isn't apparent in the file system (despite being required to be named in the source code sometimes).
> 
> I'm not sure what you mean; at first I interpreted this as suggesting that for package `Foo` there would be a top level directory called `FooTests` instead of `Tests`. But from your example below, now I don't think so. Do you mean that a `Tests/Foo` compiles to `FooTests`?

This ended up being codified in Anders' proposal here:
  https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160718/000555.html <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160718/000555.html>

> Regardless, I think we should strive to enhance the parallels between how `Sources` works, and how `Tests` works.

I agree.

> * Swiftpm looks for sources in just two canonical places: `Sources` or `src`. This is a nice, easy convention. Looking for sources in `Tests` similar.
> * Swiftpm can either interprets the contents of Sources as "single target" (sources contains source files), or "multiple target" (sources contains subdirectories of source files). `Tests` should be the same.

This seems reasonable to me.

> * In the "single target" interpretation of the `Tests` directory contents, swiftpm could infer a name for the test module, just like it does for a single target in sources. This could be `<PACKAGE-NAME>Tests`.

Agreed.

> * In the "multiple target" interpretation of `Tests`, the test modules are named according to the subdirectories (I think this is how it works now). Maybe you just mean appending the "Tests" suffix to these, which seems fine.

I meant making Tests explicit in the file organization.


>>  - Accept all target forms under Tests/, so `Tests/foo/main.swift` would declare a new executable `foo`, just like it would in Sources. The only special behavior of these targets would be that they wouldn't automatically be built by downstream package dependencies.
> 
> Yes, this is what I had in mind. I assume that downstream packages don't build the unit tests either; again, I advocate for parallel behavior.
> 
>> Are there other reasons why we need a separate "test executable" concept other than just putting another executable into `Sources/`, if SwiftPM itself isn't going to tie custom behaviors to them? Currently I can think of two reasons: organizational clarity and the inability to specify dependencies of test modules.
> 
> I agree about organizational clarity.
> 
> I do not know what you mean about inability to specify dependencies of test modules. Is this because you cannot write them down in Package.swift, like you would for a regular target?

Yup. Anders' proposal will fix this.

> Is there any reason why we cannot just support the same sorts of specifications of dependencies (and anything else) for tests as we do for regular targets?

In the current system, the reason is that users have no idea how to "spell" the name of the test module.

> More important than not building the executables downstream, I don't want to expose the test executables as public, whatever that means. As far as I can tell, swiftpm does not yet do anything to make package executables available to the end user, but it might in the future. For example, Pypi can generate "entry points" that end up in the python bin directory. If swiftpm started making packages executables available to the user on the PATH or any way, keeping tests out of that process would be important.
> 
> Put another way, it does not seem far fetched that swiftpm could gain an `swift package install` command or similar, and it would be unfortunate if published packages had in the meantime been publishing test executables as public targets.

This makes sense to me.

The only problem I see with this line of reasoning is that it only extends to one category ("Tests" in this case). If we were to support multiple categories it would become unwieldy if each new category was always mirrored in the file organization, and we would probably want to move to using attributes in the manifest. At that point, one might question why we didn't use an attribute to say "this executable is private" in the first place. So I think it comes down to motivating whether or not "test executables" are common/special enough to motivate having a special case for. Obviously we already decided Test modules are...

For what it is worth, my personal opinion is that the convention does make sense, mostly because it seems very intuitive. I don't expect test executables to be *that* common, but test support libraries (code shared among test modules, but not needed by clients) are, and having a nice convention for those seems valuable.

>> Bear in mind that the perspective I am coming from is that even if you want to do integration tests of test executables, the "driver" of those executables would still be an XCTest module adapting them, at least until we had third-party testing support.
> 
> This is the basic source of confusion I think; you mentioned in our initial slack chat that 3rd party testing frameworks was a concern, so I leapt to the conclusion that XCTest would not be the driver.
> 
> My perspective is that I already have my own harness written (github.com/gwk/iotest <http://github.com/gwk/iotest> - it's for personal use and depends on a fast-changing utility library so it might appear broken on a given day). I use it to test executable targets that I'm producing via swiftpm now.
> 
> I'm curious to hear a bit more about what you consider to be "3rd party testing support", because apart from this proposal, I can't think of anything else that iotest needs from the package manager!

My opinion is that what we want is for `swift test` to behave as a relatively framework neutral driver for each packages tests, while still being able to offer sophisticated features (parallel testing, performance tests, test filtering). I want test framework offers to be able to integrate with SwiftPM in a way that the "UI" for accessing the tests remains the same. I want "3rd-party" testing frameworks to feel just as native as XCTest (given sufficiently motivated framework providers) while also offering a uniform UI to SwiftPM users independent of the specific framework.

What this means is that we need a couple things:
1. We need some "protocol" (either an actual Swift protocol, or something else) by which we get all the information we need about tests. This will need to evolve over time, but the initial proposal would need to define where the protocol is, how it is implemented, as well as the actual APIs etc. I have ideas about how this can work... :)
2. We need some way to find the right protocol for a given package (i.e., what framework is in use). 

I don't think something as simple as assuming "each test executable is a test harness and exits with 0 or 1", because that limits what you can do with test executables (not all of them may be harnesses) but also just because the protocol isn't very rich. I would like for features like "list all tests" or "run this one test" to always work and behave uniformly -- that will require more work for us and more work for 3rd party framework providers (although hopefully not that much), but in the grand scheme of things will be much better for users IMHO.

HTH,
 - Daniel

> 
> Finally, I should add a few motivating examples for why test executables would be useful. The obvious case is for libraries that offer IO functionality; imagine a command line argument parser library, or an error logging framework. I have found that it is easiest to test these kinds of things with a harness that can specify environment, process arguments, stdin, and current working directory structure, and then capture exit status, stdout and stderr from a test process. iotest does this, and has a few nice tricks like printing diffs of the expected and actual output. Offering these sorts of features through XCTest would be a much larger proposal, so I think it is worth trying to carve out the test executable part as a first step.
> 
>> 
>>  - Daniel
>> 
>> On Thursday, July 7, 2016, George King via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org>> wrote:
>> Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George
>> 
>> Test Executables
>> 
>> Proposal: SE-NNNN <http://nnnn-filename.md/>
>> Author: George King <https://github.com/gwk>
>> Status: Awaiting review
>> Review manager: TBD
>> Introduction
>> 
>> The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.
>> 
>> Motivation
>> 
>> Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.
>> 
>> XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.
>> 
>> Consider the existing features of swiftpm as two orthogonal axes:
>> 
>> Build, Test
>> Library, Executable
>> Of the four combinations, only "Test Executable" is missing.
>> 
>> One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.
>> 
>> Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:
>> 
>> Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
>> For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
>> Test executables would only be built by swift test, not swift build.
>> Proposed solution
>> 
>> swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.
>> 
>> The test command should, upon encountering a test executable directory, build the executable.
>> 
>> If the test executable build fails, test should report a test failure and return a nonzero exit code.
>> 
>> If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.
>> 
>> Future improvements could include invoking a test command specified in Package.swift upon successful build.
>> 
>> Detailed design
>> 
>> I am unclear on the exact semantics of @testable import (I see no difference when I remove the at testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.
>> 
>> swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.
>> 
>> Impact on existing code
>> 
>> None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.
>> 
>> Alternatives considered
>> 
>> No action
>> 
>> See the list of problems with treating test executables as regular targets in the "Motivation" section above.
>> 
>> Complete integration testing with XCTest
>> 
>> This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.
>> 
> 
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160720/73113818/attachment.html>