<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">This change makes sense to me, and the proposal looks good.<div class=""><br class=""></div><div class="">This change makes sense on its own, but it should also mitigate the current pain point where we don't have test-only dependencies, so people can't easily have "example" main executables which don't get build by downstream clients.</div><div class=""><br class=""></div><div class="">A couple minor comments inline...</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Jul 7, 2016, at 4:07 PM, George King via swift-build-dev <<a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><h1 id="test-executables-by-convention" style="padding-bottom: 0.3em; line-height: 1.2; border-bottom-width: 1px; border-bottom-style: solid; font-weight: normal; border-color: rgba(255, 255, 255, 0.180392); font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; widows: 1;" class=""><span style="font-size: 12px;" class="">Hello, here is a rough draft proposal, inspired by a conversation with Daniel Dunbar last week on the Slack channel. I realize it is light on details, but hopefully this will be a starting point for figuring out how to better support building dedicated test executable targets. Let me know what you think, and if there is anything else I can do to move this forward. Thanks, -George</span></h1><h1 id="test-executables-by-convention" style="padding-bottom: 0.3em; line-height: 1.2; border-bottom-width: 1px; border-bottom-style: solid; font-weight: normal; border-color: rgba(255, 255, 255, 0.180392); font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; widows: 1;" class="">Test Executables</h1><ul style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><li class="">Proposal: <a href="x-msg://152/NNNN-filename.md" style="text-decoration: none;" class="">SE-NNNN</a></li><li class="">Author: <a href="https://github.com/gwk" style="text-decoration: none;" class="">George King</a></li><li class="">Status: <strong class="">Awaiting review</strong></li><li class="">Review manager: TBD</li></ul><h2 id="introduction" style="font-weight: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Introduction</h2><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">The package manager currently supports testing via XCTest, but does not provide support for other testing methodologies. In particular, facilities for building test executables directly in the package manager would make command line testing of Swift packages easier and more robust.</p><h2 id="motivation" style="font-weight: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Motivation</h2><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><font class=""><a href="https://en.wikipedia.org/wiki/Integration_testing" style="text-decoration: none;" class="">Integration testing</a> is, broadly speaking, the practice of testing multiple software components such as functions and classes (which may be "unit tested" individually) in aggregate. Integration tests can reveal bugs arising from interactions between components, can validate the intended usage patterns for individual components, and serve as references for that intended usage.</font></p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">XCTest offers <a href="https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/03-testing_basics.html#//apple_ref/doc/uid/TP40014132-CH3-SW8" style="text-decoration: none;" class="">facilities</a> for unit testing, performance testing,and GUI testing, but not for testing the basic IO behavior of a process. Since the swift package manager supports building command-line executables, it should support testing them as well.</p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Consider the existing features of swiftpm as two orthogonal axes:</p><ul style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><li class="">Build, Test</li><li class="">Library, Executable</li></ul><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Of the four combinations, only "Test Executable" is missing.</p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">One preliminary step towards process-based testing would be to add conventions for building executables that are meant strictly for testing, rather than for public usage. These would be exercised by some external testing harness; adding such capabilities to XCTest is beyond the scope of this proposal.</p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Currently, test executables can simply be placed in the <code style="line-height: 19px;" class="">Sources</code>, where they are compiled as regular executable targets. Distinguishing them as test executables would be helpful for several reasons:</p><ul style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><li class="">Putting all test code in the <code style="line-height: 19px;" class="">Tests</code> directory will prevent test executables from being exposed to library consumers, and clarifies developer intent.</li><li class="">For simple projects that produce a single library module, adding a test executable no longer necessitates moving the library code from <code style="line-height: 19px;" class="">Sources/</code> to <code style="line-height: 19px;" class="">Sources/[Module]</code> to allow for the second target.</li><li class="">Test executables would only be built by <code style="line-height: 19px;" class="">swift test</code>, not <code style="line-height: 19px;" class="">swift build</code>.</li></ul><h2 id="proposed-solution" style="font-weight: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Proposed solution</h2><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><font face="Helvetica" class=""><code style="line-height: 19px;" class="">swift test</code> should distinguish between unit test directories and executable directories, just as <code style="line-height: 19px;" class="">swift build</code> distinguishes between libary and executable directories.</font></p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">The <code style="line-height: 19px;" class="">test</code> command should, upon encountering a test executable directory, build the executable.</p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">If the test executable build fails, <code style="line-height: 19px;" class="">test</code> should report a test failure and return a nonzero exit code.</p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">If the build succeeds, <code style="line-height: 19px;" class="">test</code> should report a test success and clearly output the path to the resulting test executable, for consumption by external test harness scripts.</p><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Future improvements could include invoking a test command specified in <code style="line-height: 19px;" class="">Package.swift</code> upon successful build.</p><h2 id="detailed-design" style="font-weight: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Detailed design</h2><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">I am unclear on the exact semantics of <code style="line-height: 19px;" class="">@testable import</code> (I see no difference when I remove the<code style="line-height: 19px;" class="">@testable</code> modifier from a test case), but I assume it has to do with linking a unit test against the library under test. Superficially, it makes sense that a test executable would have the same requirements and use the same syntax.</p><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div>@testable just gives you access to code which is `internal`, *if* the module was built with testability enabled. I don't think this proposal needs any change to manage it.<br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><font face="Helvetica" class=""><code style="line-height: 19px;" class="">swift test</code> should output a list of successfully built executable test paths, for consumption by an external test harness. This should be a text file listing one path per line, so that simple shell scripts can run the tests without having to parse formats like json or yaml. The list should contain only executables that built successfully (or where deemed unchanged by dependency calculation), so that nonexistant and/or stale tests do not get executed. If a use case emerges that requires a list of all tests, then perhaps a complete listing could be produced separately.</font></p></div></div></blockquote><div><br class=""></div>I don't think we should prescribe `swift test` output for these executables. Is what you want a mechanism so the tests know where to find them? If so, I would probably something else where SwiftPM just writes a compact representation of what it built into the build directory -- those kinds of small tool behaviors (versus package convention / manifest) don't need to be at the level of a Swift evolution proposal though.</div><div><br class=""></div><div>As for doing it in a way that tests won't need to rerun if it has changed, I think that is out of scope for this proposal. There are too many other complicated factors (like what if the executable picks up some data file which *has* changed) for it to seem like something to role into what is an otherwise straightforward proposal.</div><div><br class=""></div><div> - Daniel<br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><h2 id="impact-on-existing-code" style="font-weight: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Impact on existing code</h2><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">None; <code style="line-height: 19px;" class="">swift test</code> currently appears to ignore subdirectories in <code style="line-height: 19px;" class="">Tests</code> that do not contain XCTest-based unit tests.</p><h2 id="alternatives-considered" style="font-weight: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">Alternatives considered</h2><h3 id="no-action" style="font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><span style="font-size: 15px; font-weight: normal;" class="">No action</span></h3><p style="font-size: 14px; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class="">See the list of problems with treating test executables as regular targets in the "Motivation" section above.</p><h3 id="complete-integration-testing-with-xctest" style="font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><span style="font-size: 15px; font-weight: normal;" class="">Complete integration testing with XCTest</span></h3><p style="font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: 22px; widows: 1;" class=""><span style="font-size: 14px;" class="">This would be a much larger undertaking, and it seems likely that such a solution would necessitate the essentials of </span><span style="font-size: 14px;" class="">this proposal anyway.</span></p></div>_______________________________________________<br class="">swift-build-dev mailing list<br class=""><a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-build-dev<br class=""></div></blockquote></div><br class=""></div></body></html>