A workable convention could be to detect classes which end with "*PerformanceTests" and when building, create their own test module that doesn't have testability enabled. So we'd end up with building two underlying test modules and running them both under the hood. All that users would need to do is 1) name some of their test classes with PerformanceTests at the end and 2) not use @testable import in those test classes. <br><br>Might be too much magic, but I'm just building on top of *Tests naming conventions that we're already using. <br><br>Honza<br><br><div class="gmail_quote"><div dir="ltr">On Mon, Aug 1, 2016 at 9:33 PM Daniel Dunbar via swift-build-dev <<a href="mailto:swift-build-dev@swift.org">swift-build-dev@swift.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">FWIW, and this isn't really an answer, but the convention I have personally adopted (in Xcode, not SwiftPM) to deal with this problem is that performance tests always go into their own module, so it can have a different set of build dependencies, flags, workflows, etc.<div><br></div><div>This is a workable model, but it feels unfortunate to me because it is partly in response to both (a) how testability is currently defined, and (b) how XCTest defines performance tests. I have dreams we will eventually improve (a), which makes me wonder if adding a convention specifically for performance tests is the right direction.</div><div><br></div><div>On the other hand, I also believe performance tests end up having very different workflow requirements than regular tests. That may mean they deserve a special case independent of the @testability problem.</div><div><br></div><div>I am very strongly opposed to any workflow which only works by building with @testability enabled in release mode. Those workflows don't work well for very performance sensitive code which depend critically on compiler optimization, and which need that behavior to be tracked by performance tests to defend against regressions.</div><div><br></div><div>BTW, this is somewhat related to the topic Brian raised recently:</div><div> <a href="https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160725/000567.html" target="_blank">https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160725/000567.html</a></div><div><br></div><div>Also, this is basically part of:</div><div> <a href="https://bugs.swift.org/browse/SR-1354" target="_blank">https://bugs.swift.org/browse/SR-1354</a></div><div><br></div><div>I think it would be worth brainstorming what a performance test "convention" would look like, and then see if the proposal has sufficient interest and can justify its own existence versus alternatives.</div></div><div style="word-wrap:break-word"><div><br></div><div> - Daniel</div></div><div style="word-wrap:break-word"><div><br><div><blockquote type="cite"><div>On Aug 1, 2016, at 11:33 AM, Ankit Agarwal via swift-build-dev <<a href="mailto:swift-build-dev@swift.org" target="_blank">swift-build-dev@swift.org</a>> wrote:</div><br><div><div dir="ltr">Hi,<div><br></div><div>Currently SwiftPM builds tests in debug configuration with testability enabled i.e. @testable import Module which gives tests internal level access to that module. This works ok for normal unit tests but there is no way to run performance tests in release mode right now.</div><div><br></div><div>SwiftPM can enable a release config with testability for building and running tests but enabling testability will remove some compiler optimisations and will not be the final release code which will be shipping and users would almost always want to run perf tests on the final release code. Not enabling testability in release mode tests will make the code that uses @testable import fail to compile.</div><div><br></div><div>There are two potential solutions which I can think of:</div><div>1. let testability be enabled in release mode tests and not care about the difference.</div><div><br></div><div>2. Have another convention especially for performance tests. A PerfTests directory besides Tests directory which will always test in release mode with testability off and user is expected not to use the @testable imports there.</div><div><br></div><div>Thoughts?</div><div><div><br></div>-- <br><div>Ankit<br><br></div>
<br><br>
</div></div>
_______________________________________________<br>swift-build-dev mailing list<br><a href="mailto:swift-build-dev@swift.org" target="_blank">swift-build-dev@swift.org</a><br><a href="https://lists.swift.org/mailman/listinfo/swift-build-dev" target="_blank">https://lists.swift.org/mailman/listinfo/swift-build-dev</a><br></div></blockquote></div><br></div></div>_______________________________________________<br>
swift-build-dev mailing list<br>
<a href="mailto:swift-build-dev@swift.org" target="_blank">swift-build-dev@swift.org</a><br>
<a href="https://lists.swift.org/mailman/listinfo/swift-build-dev" rel="noreferrer" target="_blank">https://lists.swift.org/mailman/listinfo/swift-build-dev</a><br>
</blockquote></div>