[swift-build-dev] Release build performance tests in SwiftPM

[Honza Dvorsky]

A workable convention could be to detect classes which end with
"*PerformanceTests" and when building, create their own test module that
doesn't have testability enabled. So we'd end up with building two
underlying test modules and running them both under the hood. All that
users would need to do is 1) name some of their test classes with
PerformanceTests at the end and 2) not use @testable import in those test
classes.

Might be too much magic, but I'm just building on top of *Tests naming
conventions that we're already using.

Honza

On Mon, Aug 1, 2016 at 9:33 PM Daniel Dunbar via swift-build-dev <
swift-build-dev at swift.org> wrote:

> FWIW, and this isn't really an answer, but the convention I have
> personally adopted (in Xcode, not SwiftPM) to deal with this problem is
> that performance tests always go into their own module, so it can have a
> different set of build dependencies, flags, workflows, etc.
>
> This is a workable model, but it feels unfortunate to me because it is
> partly in response to both (a) how testability is currently defined, and
> (b) how XCTest defines performance tests. I have dreams we will eventually
> improve (a), which makes me wonder if adding a convention specifically for
> performance tests is the right direction.
>
> On the other hand, I also believe performance tests end up having very
> different workflow requirements than regular tests. That may mean they
> deserve a special case independent of the @testability problem.
>
> I am very strongly opposed to any workflow which only works by building
> with @testability enabled in release mode. Those workflows don't work well
> for very performance sensitive code which depend critically on compiler
> optimization, and which need that behavior to be tracked by performance
> tests to defend against regressions.
>
> BTW, this is somewhat related to the topic Brian raised recently:
>
> https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160725/000567.html
>
> Also, this is basically part of:
>   https://bugs.swift.org/browse/SR-1354
>
> I think it would be worth brainstorming what a performance test
> "convention" would look like, and then see if the proposal has sufficient
> interest and can justify its own existence versus alternatives.
>
>  - Daniel
>
> On Aug 1, 2016, at 11:33 AM, Ankit Agarwal via swift-build-dev <
> swift-build-dev at swift.org> wrote:
>
> Hi,
>
> Currently SwiftPM builds tests in debug configuration with testability
> enabled i.e. @testable import Module which gives tests internal level
> access to that module. This works ok for normal unit tests but there is no
> way to run performance tests in release mode right now.
>
> SwiftPM can enable a release config with testability for building and
> running tests but enabling testability will remove some
> compiler optimisations and will not be the final release code which will be
> shipping and users would almost always want to run perf tests on the final
> release code. Not enabling testability in release mode tests will make the
> code that uses @testable import fail to compile.
>
> There are two potential solutions which I can think of:
> 1. let testability be enabled in release mode tests and not care about the
> difference.
>
> 2. Have another convention especially for performance tests. A PerfTests
> directory besides Tests directory which will always test in release mode
> with testability off and user is expected not to use the @testable imports
> there.
>
> Thoughts?
>
> --
> Ankit
>
>
>
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev
>
>
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160802/cd6b17b0/attachment.html>