[swift-build-dev] Release build performance tests in SwiftPM

[Daniel Dunbar]

FWIW, and this isn't really an answer, but the convention I have personally adopted (in Xcode, not SwiftPM) to deal with this problem is that performance tests always go into their own module, so it can have a different set of build dependencies, flags, workflows, etc.

This is a workable model, but it feels unfortunate to me because it is partly in response to both (a) how testability is currently defined, and (b) how XCTest defines performance tests. I have dreams we will eventually improve (a), which makes me wonder if adding a convention specifically for performance tests is the right direction.

On the other hand, I also believe performance tests end up having very different workflow requirements than regular tests. That may mean they deserve a special case independent of the @testability problem.

I am very strongly opposed to any workflow which only works by building with @testability enabled in release mode. Those workflows don't work well for very performance sensitive code which depend critically on compiler optimization, and which need that behavior to be tracked by performance tests to defend against regressions.

BTW, this is somewhat related to the topic Brian raised recently:
  https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160725/000567.html

Also, this is basically part of:
  https://bugs.swift.org/browse/SR-1354 <https://bugs.swift.org/browse/SR-1354>

I think it would be worth brainstorming what a performance test "convention" would look like, and then see if the proposal has sufficient interest and can justify its own existence versus alternatives.

 - Daniel

> On Aug 1, 2016, at 11:33 AM, Ankit Agarwal via swift-build-dev <swift-build-dev at swift.org> wrote:
> 
> Hi,
> 
> Currently SwiftPM builds tests in debug configuration with testability enabled i.e. @testable import Module which gives tests internal level access to that module. This works ok for normal unit tests but there is no way to run performance tests in release mode right now.
> 
> SwiftPM can enable a release config with testability for building and running tests but enabling testability will remove some compiler optimisations and will not be the final release code which will be shipping and users would almost always want to run perf tests on the final release code. Not enabling testability in release mode tests will make the code that uses @testable import fail to compile.
> 
> There are two potential solutions which I can think of:
> 1. let testability be enabled in release mode tests and not care about the difference.
> 
> 2. Have another convention especially for performance tests. A PerfTests directory besides Tests directory which will always test in release mode with testability off and user is expected not to use the @testable imports there.
> 
> Thoughts?
> 
> -- 
> Ankit
> 
> 
> 
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160801/ad7f2b59/attachment.html>