[swift-build-dev] Draft proposal: "Test Executables"

[Ankit Agarwal]

This proposal sounds similar to allowing regular modules (lib+exec) inside
Tests/ and IMO that is better than just allowing executables and is more
general. Right now we error out when some directory doesn't have Tests
prefix in Tests/ folder, We can just allow that and use same rule of
main.swift to distinguish between lib and exec.

George, would doing that solve your problem?

On Sat, Nov 12, 2016 at 6:39 AM, Daniel Dunbar via swift-build-dev <
swift-build-dev at swift.org> wrote:

> This change makes sense to me, and the proposal looks good.
>
> This change makes sense on its own, but it should also mitigate the
> current pain point where we don't have test-only dependencies, so people
> can't easily have "example" main executables which don't get build by
> downstream clients.
>
> A couple minor comments inline...
>
> On Jul 7, 2016, at 4:07 PM, George King via swift-build-dev <
> swift-build-dev at swift.org> wrote:
>
> Hello, here is a rough draft proposal, inspired by a conversation with
> Daniel Dunbar last week on the Slack channel. I realize it is light on
> details, but hopefully this will be a starting point for figuring out how
> to better support building dedicated test executable targets. Let me know
> what you think, and if there is anything else I can do to move this
> forward. Thanks, -GeorgeTest Executables
>
>    - Proposal: SE-NNNN
>    - Author: George King <https://github.com/gwk>
>    - Status: *Awaiting review*
>    - Review manager: TBD
>
> Introduction
>
> The package manager currently supports testing via XCTest, but does not
> provide support for other testing methodologies. In particular, facilities
> for building test executables directly in the package manager would make
> command line testing of Swift packages easier and more robust.
> Motivation
>
> Integration testing <https://en.wikipedia.org/wiki/Integration_testing> is,
> broadly speaking, the practice of testing multiple software components such
> as functions and classes (which may be "unit tested" individually) in
> aggregate. Integration tests can reveal bugs arising from interactions
> between components, can validate the intended usage patterns for individual
> components, and serve as references for that intended usage.
>
> XCTest offers facilities
> <https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8> for
> unit testing, performance testing,and GUI testing, but not for testing the
> basic IO behavior of a process. Since the swift package manager supports
> building command-line executables, it should support testing them as well.
>
> Consider the existing features of swiftpm as two orthogonal axes:
>
>    - Build, Test
>    - Library, Executable
>
> Of the four combinations, only "Test Executable" is missing.
>
> One preliminary step towards process-based testing would be to add
> conventions for building executables that are meant strictly for testing,
> rather than for public usage. These would be exercised by some external
> testing harness; adding such capabilities to XCTest is beyond the scope of
> this proposal.
>
> Currently, test executables can simply be placed in the Sources, where
> they are compiled as regular executable targets. Distinguishing them as
> test executables would be helpful for several reasons:
>
>    - Putting all test code in the Tests directory will prevent test
>    executables from being exposed to library consumers, and clarifies
>    developer intent.
>    - For simple projects that produce a single library module, adding a
>    test executable no longer necessitates moving the library code from
>    Sources/ to Sources/[Module] to allow for the second target.
>    - Test executables would only be built by swift test, not swift build.
>
> Proposed solution
>
> swift test should distinguish between unit test directories and
> executable directories, just as swift build distinguishes between libary
> and executable directories.
>
> The test command should, upon encountering a test executable directory,
> build the executable.
>
> If the test executable build fails, test should report a test failure and
> return a nonzero exit code.
>
> If the build succeeds, test should report a test success and clearly
> output the path to the resulting test executable, for consumption by
> external test harness scripts.
>
> Future improvements could include invoking a test command specified in
> Package.swift upon successful build.
> Detailed design
>
> I am unclear on the exact semantics of @testable import (I see no
> difference when I remove the at testable modifier from a test case), but I
> assume it has to do with linking a unit test against the library under
> test. Superficially, it makes sense that a test executable would have the
> same requirements and use the same syntax.
>
>
> @testable just gives you access to code which is `internal`, *if* the
> module was built with testability enabled. I don't think this proposal
> needs any change to manage it.
>
> swift test should output a list of successfully built executable test
> paths, for consumption by an external test harness. This should be a text
> file listing one path per line, so that simple shell scripts can run the
> tests without having to parse formats like json or yaml. The list should
> contain only executables that built successfully (or where deemed unchanged
> by dependency calculation), so that nonexistant and/or stale tests do not
> get executed. If a use case emerges that requires a list of all tests, then
> perhaps a complete listing could be produced separately.
>
>
> I don't think we should prescribe `swift test` output for these
> executables. Is what you want a mechanism so the tests know where to find
> them? If so, I would probably something else where SwiftPM just writes a
> compact representation of what it built into the build directory -- those
> kinds of small tool behaviors (versus package convention / manifest) don't
> need to be at the level of a Swift evolution proposal though.
>
> As for doing it in a way that tests won't need to rerun if it has changed,
> I think that is out of scope for this proposal. There are too many other
> complicated factors (like what if the executable picks up some data file
> which *has* changed) for it to seem like something to role into what is an
> otherwise straightforward proposal.
>
>  - Daniel
>
> Impact on existing code
>
> None; swift test currently appears to ignore subdirectories in Tests that
> do not contain XCTest-based unit tests.
> Alternatives consideredNo action
>
> See the list of problems with treating test executables as regular targets
> in the "Motivation" section above.
> Complete integration testing with XCTest
>
> This would be a much larger undertaking, and it seems likely that such a
> solution would necessitate the essentials of this proposal anyway.
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev
>
>
>
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev
>
>


-- 
Ankit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20161114/d0855317/attachment.html>