[swift-build-dev] Draft proposal: "Test Executables"

[George King]

Hi list,
Here is an updated draft of my "Test Executables" proposal. This draft drops the less well defined ideas, and tightens up some of the motivation. Let me know what you think, and what more needs to be done.
Thanks,
George



Test Executables

Proposal: SE-NNNN <applewebdata://CA2DA431-9CC7-418E-8DC8-FA9AB46534B8/NNNN-filename.md>
Author: George King <https://github.com/gwk>
Status: Awaiting review / DRAFT
Review manager: TBD
Introduction

The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables (that is, executables meant strictly for testing and not as an end product) in the Swift Package Manager would make command line testing of Swift packages easier and more robust. We propose to add to the package manager the ability to specify and build test-only executable targets that are distinct from regular executables.

Motivation

Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.

XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing, and GUI testing, but not for testing the basic IO behavior of a process. To be clear, this proposal does not address XCTest at all; while adding whole process, IO oriented APIs to XCTest sounds desirable, it is a larger project and out of scope for this proposal.

Instead, this proposal simply asks for the ability to build executables specifically for testing, rather than for public usage. The intended use case is execution via an external test harness, or from within an XCTest case via NSTask/Process.

Consider the existing features of swiftpm as two orthogonal axes:

Build, Test
Library, Executable
Of the four combinations, only "Test Executable" is missing.

Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:

Test executables would only be built by swift test, not swift build.
Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
For simple projects that produce a single library module, adding a test executable will no longer necessitate moving the library code from Sources/ to Sources/[Module] to allow for the second target.
Proposed solution

swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.

The swift test command should, upon encountering a test executable directory, build the executable.

If the test executable build fails, swift test should report a test failure and return a nonzero exit code. An external test harness can then run the test executables as it sees fit. Note that for an XCTest case to invoke the test executable, test executables would need to be built prior to running test cases.

Future improvements could include invoking an external test command specified in Package.swift upon successful build.

Detailed design

I am unclear on the exact semantics of @testable import (I see no difference when I remove the @testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.

Impact on existing code

None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.

Alternatives considered

No action

See the list of problems with treating test executables as regular targets in the "Motivation" section above.

Complete integration testing with XCTest

This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.



> On Jul 7, 2016, at 7:07 PM, George King <george.w.king at gmail.com> wrote:
> 
> Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George
> 
> Test Executables
> 
> Proposal: SE-NNNN <x-msg://24/NNNN-filename.md>
> Author: George King <https://github.com/gwk>
> Status: Awaiting review
> Review manager: TBD
> Introduction
> 
> The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.
> 
> Motivation
> 
> Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.
> 
> XCTest offers facilities <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.
> 
> Consider the existing features of swiftpm as two orthogonal axes:
> 
> Build, Test
> Library, Executable
> Of the four combinations, only "Test Executable" is missing.
> 
> One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.
> 
> Currently, test executables can simply be placed in the Sources, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:
> 
> Putting all test code in the Tests directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.
> For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from Sources/ to Sources/[Module] to allow for the second target.
> Test executables would only be built by swift test, not swift build.
> Proposed solution
> 
> swift test should distinguish between unit test directories and executable directories, just as swift build distinguishes between libary and executable directories.
> 
> The test command should, upon encountering a test executable directory, build the executable.
> 
> If the test executable build fails, test should report a test failure and return a nonzero exit code.
> 
> If the build succeeds, test should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.
> 
> Future improvements could include invoking a test command specified in Package.swift upon successful build.
> 
> Detailed design
> 
> I am unclear on the exact semantics of @testable import (I see no difference when I remove the at testable modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.
> 
> swift test should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.
> 
> Impact on existing code
> 
> None; swift test currently appears to ignore subdirectories in Tests that do not contain XCTest-based unit tests.
> 
> Alternatives considered
> 
> No action
> 
> See the list of problems with treating test executables as regular targets in the "Motivation" section above.
> 
> Complete integration testing with XCTest
> 
> This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of this proposal anyway.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20161110/fe4feb0e/attachment.html>