[swift-build-dev] Draft proposal: "Test Executables"

[Daniel Dunbar]

Hi George!

Generally speaking, I like the direction for SwiftPM of making unit tests
consistent with the other kinds of targets, and so the "is a test" becomes
just a flag for each target rather than a somewhat different kind of target.

Your proposal calls out a couple things that there aren't concrete details
on: why exactly do we need the output file with a list of test executables?
Where and when would it be written, and how would other targets find it? I
suspect this is something we should consider independently, and I would
recommend exploring whether it is something we should expose as an API
targets could use instead of a build artifact.

The big part of this proposal is a change to the conventions for test
targets, and I'm not sure exactly what convention you are suggesting --
that `main.swift` inside a module within Tests just become an executable
target? I am tempted towards a slight different direction:

 - Rename test targets to `<NAME>Tests`, which matches the Xcode convention
and solves an existing problem where the module name of the test module
isn't apparent in the file system (despite being required to be named in
the source code sometimes).
 - Accept all target forms under Tests/, so `Tests/foo/main.swift` would
declare a new executable `foo`, just like it would in Sources. The only
special behavior of these targets would be that they wouldn't automatically
be built by downstream package dependencies.

Are there other reasons why we need a separate "test executable" concept
other than just putting another executable into `Sources/`, if SwiftPM
itself isn't going to tie custom behaviors to them? Currently I can think
of two reasons: organizational clarity and the inability to specify
dependencies of test modules.

Bear in mind that the perspective I am coming from is that even if you want
to do integration tests of test executables, the "driver" of those
executables would still be an XCTest module adapting them, at least until
we had third-party testing support.

 - Daniel

On Thursday, July 7, 2016, George King via swift-build-dev <
swift-build-dev at swift.org> wrote:

> Hello, here is a rough draft proposal, inspired by a conversation with
> Daniel Dunbar last week on the Slack channel. I realize it is light on
> details, but hopefully this will be a starting point for figuring out how
> to better support building dedicated test executable targets. Let me know
> what you think, and if there is anything else I can do to move this
> forward. Thanks, -GeorgeTest Executables
>
>    - Proposal: SE-NNNN <http://NNNN-filename.md>
>    - Author: George King <https://github.com/gwk>
>    - Status: *Awaiting review*
>    - Review manager: TBD
>
> Introduction
>
> The package manager currently supports testing via XCTest, but does not
> provide support for other testing methodologies. In particular, facilities
> for building test executables directly in the package manager would make
> command line testing of Swift packages easier and more robust.
> Motivation
>
> Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is,
> broadly speaking, the practice of testing multiple software components such
> as functions and classes (which may be "unit tested" individually) in
> aggregate. Integration tests can reveal bugs arising from interactions
> between components, can validate the intended usage patterns for individual
> components, and serve as references for that intended usage.
>
> XCTest offers facilities
> <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for
> unit testing, performance testing,and GUI testing, but not for testing the
> basic IO behavior of a process. Since the swift package manager supports
> building command-line executables, it should support testing them as well.
>
> Consider the existing features of swiftpm as two orthogonal axes:
>
>    - Build, Test
>    - Library, Executable
>
> Of the four combinations, only "Test Executable" is missing.
>
> One preliminary step towards process-based testing would be to add
> conventions for building executables that are meant strictly for testing,
> rather than for public usage. These would be exercised by some external
> testing harness; adding such capabilities to XCTest is beyond the scope of
> this proposal.
>
> Currently, test executables can simply be placed in the Sources, where
> they are compiled as regular executable targets. Distinguishing them as
> test executables would be helpful for several reasons:
>
>    - Putting all test code in the Tests directory will prevent test
>    executables from being exposed to library consumers, and clarifies
>    developer intent.
>    - For simple projects that produce a single library module, adding a
>    test executable no longer necessitates moving the library code from
>    Sources/ to Sources/[Module] to allow for the second target.
>    - Test executables would only be built by swift test, not swift build.
>
> Proposed solution
>
> swift test should distinguish between unit test directories and
> executable directories, just as swift build distinguishes between libary
> and executable directories.
>
> The test command should, upon encountering a test executable directory,
> build the executable.
>
> If the test executable build fails, test should report a test failure and
> return a nonzero exit code.
>
> If the build succeeds, test should report a test success and clearly
> output the path to the resulting test executable, for consumption by
> external test harness scripts.
>
> Future improvements could include invoking a test command specified in
> Package.swift upon successful build.
> Detailed design
>
> I am unclear on the exact semantics of @testable import (I see no
> difference when I remove the at testable modifier from a test case), but I
> assume it has to do with linking a unit test against the library under
> test. Superficially, it makes sense that a test executable would have the
> same requirements and use the same syntax.
>
> swift test should output a list of successfully built executable test
> paths, for consumption by an external test harness. This should be a text
> file listing one path per line, so that simple shell scripts can run the
> tests without having to parse formats like json or yaml. The list should
> contain only executables that built successfully (or where deemed unchanged
> by dependency calculation), so that nonexistant and/or stale tests do not
> get executed. If a use case emerges that requires a list of all tests, then
> perhaps a complete listing could be produced separately.
> Impact on existing code
>
> None; swift test currently appears to ignore subdirectories in Tests that
> do not contain XCTest-based unit tests.
> Alternatives consideredNo action
>
> See the list of problems with treating test executables as regular targets
> in the "Motivation" section above.
> Complete integration testing with XCTest
>
> This would be a much larger undertaking, and it seems likely that such a
> solution would necessitate the essentials of this proposal anyway.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160708/05ca46e2/attachment.html>