[swift-build-dev] Performance testing via SwiftPM and XCTest
Brian Gesiak
modocache at gmail.com
Sun Jul 24 12:01:53 CDT 2016
Hello corelibs-dev and build-dev,
Back in May, Brian Croom implemented performance testing in
swift-corelibs-xctest:
https://github.com/apple/swift-corelibs-xctest/pull/109
I’d love to see Swift developers use this feature to measure the
performance of their code. I think we’ll need to add functionality to
swift-corelibs-xctest and SwiftPM in order to do so.
The problem: recording performance test baselines
In order for performance tests to be useful, Apple’s Xcode provides a way
to record “baseline” metrics. Baseline metrics allow a developer to
indicate “this performance test should never be slower than 1.2 seconds on
average, with 10% standard deviation as ‘wiggle room’”. When Apple XCTest
tests are run, they are informed of the baseline metrics that have been set
in Xcode. Apple XCTest performance tests that have a baseline registered
will fail if performance becomes slower than the acceptable amount.
If we could provide swift-corelibs-xctest with a mapping from each
performance test to its baseline metric, it would be easy to write the code
to fail a test if it didn’t perform well enough. That mapping, however, is
the tricky part. Here’s why:
- The mapping needs to group metrics based on the host machine running
the test. Performance will of course vary based on the hardware, so it’s
important to make sure performance baselines set on a Raspberry Pi aren’t
used when testing on a Mac Pro.
- The mapping also needs to group metrics based on the target machine.
Using Apple XCTest, a developer can start a test suite run from their
MacBook Pro (macOS 64-bit), and see the results of the performance tests
when run on their iPhone 6s (iOS armv7s). I don’t think this is relevant to
swift-corelibs-xctest just yet — as far as I know, SwiftPM is not capable
of cross-compilation, so the host machine will always be identical to the
target machine. Still, we should design something flexible enough for this
scenario.
Xcode’s solution: plist files
Xcode’s solves this problem using two kinds of .plist files. I tried
creating a sample project, named Perforate.xcodeproj, which contained a
single performance test. Here’s what Xcode created:
<!-- Perforate.xcodeproj/xcshareddata/xcbaselines/DA77262F1D447DB300735C93.xcbaseline/Info.plist
-->
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC
"-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist
version="1.0"><dict>
<!-- runDestinationsByUUID: These are the host/target machine
groups. -->
<key>runDestinationsByUUID</key>
<dict>
<!--
It appears each group is given a UUID, but to
be honest, I'm not sure why.
It seems like these should be "keyed" on
aspects of the host/target machines.
As-is, I imagine Xcode and Apple XCTest need
to traverse each group's
`localComputer`, `targetArchitecture`, and
`targetDevice`'s values in order to find a match.
-->
<key>8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5</key>
<dict>
<!-- Information about the host machine:
number of CPUs, cores, etc. -->
<key>localComputer</key>
<dict>
<key>busSpeedInMHz</key>
<integer>100</integer>
<key>cpuCount</key>
<integer>1</integer>
<key>cpuKind</key>
<string>Intel Core i7</string>
<key>cpuSpeedInMHz</key>
<integer>2800</integer>
<key>logicalCPUCoresPerPackage</key>
<integer>8</integer>
<key>modelCode</key>
<string>MacBookPro11,3</string>
<key>physicalCPUCoresPerPackage</key>
<integer>4</integer>
<key>platformIdentifier</key>
<string>com.apple.platform.macosx</string>
</dict>
<!-- The target architecture and device are
stored as separate keys. -->
<key>targetArchitecture</key>
<string>x86_64</string>
<key>targetDevice</key>
<dict>
<key>modelCode</key>
<string>iPhone8,2</string>
<key>platformIdentifier</key>
<string>com.apple.platform.iphonesimulator</string>
</dict>
</dict>
</dict></dict></plist>
<!-- Perforate.xcodeproj/xcshareddata/xcbaselines/DA77262F1D447DB300735C93.xcbaseline/8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5.plist
-->
<!-- Notice that this file is named after the `runDestinationsByUUID`
key from the first file: 8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5. -->
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC
"-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist
version="1.0"><dict>
<key>classNames</key>
<dict>
<key>PerforateTests</key>
<dict>
<!-- The metrics are mapped by class name and
test method name to performance metrics. -->
<key>test_uniqueOrdered_performance</key>
<dict>
<!-- There are several categories of
performance metrics. The only one publicly available in Apple XCTest
so far is wall clock time. -->
<key>com.apple.XCTPerformanceMetric_WallClockTime</key>
<dict>
<key>baselineAverage</key>
<real>0.5</real>
<key>baselineIntegrationDisplayName</key>
<string>Local Baseline</string>
</dict>
</dict>
</dict>
</dict></dict></plist>
Proposed solution for SwiftPM/swift-corelibs-xctest: JSON files
I think we can mimic Xcode’s approach here. Here’s what I’m proposing:
- swift-corelibs-xctest’s test runner should take a --performance-metrics
<PATH> argument, where <PATH> is the location of a file containing JSON
that looks pretty much exactly like the
8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5.plist from above:
{
"classNames": {
"PerforateTests": {
"test_uniqueOrdered_performance": {
"baselineAverage": "0.5",
"baselineIntegrationDisplayName": "Local Baseline"
}
}
}}
- SwiftPM’s swift test command should also take a --performance-metrics
<PATH> argument, where <PATH> is the location of a file containing JSON
that looks pretty much exactly like the
xcbaselines/DA77262F1D447DB300735C93.xcbaseline/Info.plist from above
(by default, --performance-metrics could be set to the same path as
the swift
test --build-path directory):
{
"runDestinationsByUUID": {
"8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5": {
"localComputer": {
"busSpeedInMHz": "100",
# ...
},
"targetArchitecture": "x86_64",
"targetDevice": {
# We might need to change these keys, since "modelCode" seems
very Apple-specific.
"modelCode": "linux",
"platformIdentifier": "Ubuntu 15.04",
}
}
}
}
Personally, I think the format of the plist files Xcode and Apple XCTest
generate could be improved. Still, I think it’d be nice to stick to the
same format (as much as possible) for swift-corelibs-xctest, just to keep
things simple.
Thoughts?
I admit that I don’t have much experience using Apple XCTest’s performance
testing functionality, so I might be missing something here. Does anyone
have any feedback on this idea? I’d like to incorporate your feedback, and
perhaps submit a Swift Evolution proposal for this feature.
- Brian Gesiak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160724/b6a80b05/attachment.html>
More information about the swift-build-dev
mailing list