<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">FWIW, and this isn't really an answer, but the convention I have personally adopted (in Xcode, not SwiftPM) to deal with this problem is that performance tests always go into their own module, so it can have a different set of build dependencies, flags, workflows, etc.<div class=""><br class=""></div><div class="">This is a workable model, but it feels unfortunate to me because it is partly in response to both (a) how testability is currently defined, and (b) how XCTest defines performance tests. I have dreams we will eventually improve (a), which makes me wonder if adding a convention specifically for performance tests is the right direction.</div><div class=""><br class=""></div><div class="">On the other hand, I also believe performance tests end up having very different workflow requirements than regular tests. That may mean they deserve a special case independent of the @testability problem.</div><div class=""><br class=""></div><div class="">I am very strongly opposed to any workflow which only works by building with @testability enabled in release mode. Those workflows don't work well for very performance sensitive code which depend critically on compiler optimization, and which need that behavior to be tracked by performance tests to defend against regressions.</div><div class=""><br class=""></div><div class="">BTW, this is somewhat related to the topic Brian raised recently:</div><div class=""> <a href="https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160725/000567.html" class="">https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160725/000567.html</a></div><div class=""><br class=""></div><div class="">Also, this is basically part of:</div><div class=""> <a href="https://bugs.swift.org/browse/SR-1354" class="">https://bugs.swift.org/browse/SR-1354</a></div><div class=""><br class=""></div><div class="">I think it would be worth brainstorming what a performance test "convention" would look like, and then see if the proposal has sufficient interest and can justify its own existence versus alternatives.</div><div class=""><br class=""></div><div class=""> - Daniel</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Aug 1, 2016, at 11:33 AM, Ankit Agarwal via swift-build-dev <<a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi,<div class=""><br class=""></div><div class="">Currently SwiftPM builds tests in debug configuration with testability enabled i.e. @testable import Module which gives tests internal level access to that module. This works ok for normal unit tests but there is no way to run performance tests in release mode right now.</div><div class=""><br class=""></div><div class="">SwiftPM can enable a release config with testability for building and running tests but enabling testability will remove some compiler optimisations and will not be the final release code which will be shipping and users would almost always want to run perf tests on the final release code. Not enabling testability in release mode tests will make the code that uses @testable import fail to compile.</div><div class=""><br class=""></div><div class="">There are two potential solutions which I can think of:</div><div class="">1. let testability be enabled in release mode tests and not care about the difference.</div><div class=""><br class=""></div><div class="">2. Have another convention especially for performance tests. A PerfTests directory besides Tests directory which will always test in release mode with testability off and user is expected not to use the @testable imports there.</div><div class=""><br class=""></div><div class="">Thoughts?</div><div class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature">Ankit<br class=""><br class=""></div>
<br class=""><br class="">
</div></div>
_______________________________________________<br class="">swift-build-dev mailing list<br class=""><a href="mailto:swift-build-dev@swift.org" class="">swift-build-dev@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-build-dev<br class=""></div></blockquote></div><br class=""></div></body></html>