<div dir="ltr"><div class="markdown-here-wrapper" style=""><p style="margin:0px 0px 1.2em!important">I received some feedback on this proposal from Ankit Aggarwal, which centered on how developers would edit and update their baseline metrics. Here’s what I’m envisioning specifically:</p>
<h1 id="two-new-command-line-options-for-swift-test-" style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">Two new command-line options for <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test</code></h1>
<ol style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test --performance-metrics <path></code>. This is a path to a directory where JSON files containing the baseline metrics for the tests will be stored. By default, this path will be <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">MyPackage/Tests/PerformanceMetrics</code>. In my previous email, I suggested the default <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics</code> path could be set to the same path as the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test --build-path</code> directory, but I have reconsidered. This is because I think developers would want to check their baseline metrics JSON files into source control, so that they can share metrics with one another, and with their continuous integration servers.</li>
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test --performance-metrics-update <mode></code>, where <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)"><mode></code> is one of <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">{all|new|better|worse|none}</code>. This specifies the behavior SwiftPM should take when writing baseline metrics data into the JSON files at the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics</code> path.<ul style="margin:1.2em 0px;padding-left:2em;margin:0px;padding-left:1em">
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">all</code>: Write baseline metrics data for all performance test cases. If metrics for those test cases already exist in the JSON, they are overwritten.</li>
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">new</code>: Only write baseline metrics for performance test cases that did not already exist in the baseline metrics JSON. This is the default.</li>
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">better</code>: Only write baseline metrics for performance test cases whose performance has improved compared to the last time they were run. If baseline metrics for those test cases already exist in the JSON, they are overwritten. If baseline metrics for those test cases does not exist in the JSON, they are written to the JSON.</li>
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">worse</code>: Only write baseline metrics for performance test cases whose performance has worsened compared to the last time they were run. If baseline metrics for those test cases already exist in the JSON, they are overwritten. If baseline metrics for those test cases does not exist in the JSON, they are written to the JSON.</li>
</ul>
</li>
</ol>
<h1 id="two-new-command-line-options-for-swift-corelibs-xctest-executables" style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">Two new command-line options for swift-corelibs-xctest executables</h1>
<ol style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics <path></code>. This is a path to a JSON file containing a mapping from test cases to baseline metrics.<ul style="margin:1.2em 0px;padding-left:2em;margin:0px;padding-left:1em">
<li style="margin:0.5em 0px">If not specified, performance tests are not run against any baseline metrics, and so will never fail.</li>
<li style="margin:0.5em 0px">If specified, performance test cases will be run against these metrics. Based on the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics-update</code> mode (see below), performance test cases may fail if their performance does not meet the baseline.</li>
</ul>
</li>
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics-update <mode></code>. Same as the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test --performance-metrics-update</code> parameter.</li>
</ol>
<h1 id="-performancemetrics-directory" style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">PerformanceMetrics</code> directory</h1>
<p style="margin:0px 0px 1.2em!important">If a package’s tests contain any performance tests (i.e.: tests that call <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">XCTestCase.measure()</code>, <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">XCTestCase.measureMetrics()</code>, etc.), running <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test</code> will result in the following directories and files being generated:</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248);white-space:pre;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important">MyPackage/
.build/
Sources/
Tests/
LinuxMain.swift
MyPackage/
MyPackageTests.swift
PerformanceMetrics/ # Generated if any performance tests are run. This is the path specified by --performance-metrics.
MyPackage/
Destinations.json # Contains a mapping of "runDestinationsByUUID".
8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5.json # An individual run destination's baseline metrics.
</code></pre><p style="margin:0px 0px 1.2em!important">In order to avoid name collisions, developers will no longer be able to name their test modules “PerformanceMetrics”.</p>
<h1 id="what-happens-when-swift-test-is-run" style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">What happens when <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test</code> is run</h1>
<ol style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test</code>, using the default arguments, would be the equivalent of <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test --performance-metrics ./Tests/PerformanceMetrics --performance-metrics-update new</code>.</li>
<li style="margin:0.5em 0px">SwiftPM determines which of the destinations defined in <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">Destinations.json</code> to pass to XCTest. For example, if testing on a macOS 64-bit system with one processor, SwiftPM attempts to find a run destination UUID in <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">Destinations.json</code> that matches those criteria. If no <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">Destinations.json</code> file exists, SwiftPM creates a mapping in memory.</li>
<li style="margin:0.5em 0px">SwiftPM invokes <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">LinuxMain.swift</code>, passing swift-corelibs-xctest the path to a run destination’s baseline metrics file (in this case, <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics 8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5.json</code>), as well as the update behavior (<code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">all</code>, <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">new</code>, <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">better</code>, or <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">worse</code>). This file may not already exist, such as in the case that a <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">Destinations.json</code> file did not exist, or that a JSON file for this particular run destination did not exist.</li>
<li style="margin:0.5em 0px">swift-corelibs-xctest parses the JSON in the baseline metrics JSON file it is given, and stores in memory the mappings from test cases to their baseline metrics. If the file is empty or does not exist, swift-corelibs-xctest stores an empty mapping.</li>
<li style="margin:0.5em 0px">swift-corelibs-xctest runs the tests. If a test exists in the mapping from step 4, it compares its performance to the baseline metric. If the performance is worse, and the update behavior is <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">new</code> or <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">better</code>, the test case is failed.</li>
<li style="margin:0.5em 0px">swift-corelibs-xctest writes to the baseline metrics JSON file, based on the specified update behavior. If the file does not already exist, swift-corelibs-xctest creates the file, then writes to it.</li>
<li style="margin:0.5em 0px">After running the tests, SwiftPM determines whether the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5.json</code> file contains any data. If it does, and this run destination did not exist in <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">Destinations.json</code> in step (2), then SwiftPM writes the new destination to the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">Destinations.json</code> file.</li>
</ol>
<h1 id="how-will-this-work-on-darwin-" style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">How will this work on Darwin?</h1>
<p style="margin:0px 0px 1.2em!important">I realized while writing this email that I have no clue how to get this working on Darwin. Is it even possible to specify the paths to performance baseline plist files to Apple XCTest on the command line? This seems like a prerequisite to supporting performance testing via SwiftPM on Darwin.</p>
<p style="margin:0px 0px 1.2em!important">It would be great to hear from someone on the developer tools team on this topic (+cc Daniel Dunbar, Mike Ferris). I’ll try and figure out how this works in Apple XCTest, and will send an update when I do.</p>
<h1 id="thoughts-" style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">Thoughts?</h1>
<p style="margin:0px 0px 1.2em!important">As before, I’d love to hear any feedback you all may have on this proposal.<br><br>- Brian Gesiak</p>
<div title="MDH:SSByZWNlaXZlZCBzb21lIGZlZWRiYWNrIG9uIHRoaXMgcHJvcG9zYWwgZnJvbSBBbmtpdCBBZ2dh
cndhbCwgd2hpY2ggY2VudGVyZWQgb24gaG93IGRldmVsb3BlcnMgd291bGQgZWRpdCBhbmQgdXBk
YXRlIHRoZWlyIGJhc2VsaW5lIG1ldHJpY3MuIEhlcmUncyB3aGF0IEknbSBlbnZpc2lvbmluZyBz
cGVjaWZpY2FsbHk6PGJyPjxicj4jIFR3byBuZXcgY29tbWFuZC1saW5lIG9wdGlvbnMgZm9yIGBz
d2lmdCB0ZXN0YDxicj48YnI+MS4gYHN3aWZ0IHRlc3QgLS1wZXJmb3JtYW5jZS1tZXRyaWNzICZs
dDtwYXRoJmd0O2AuIFRoaXMgaXMgYSBwYXRoIHRvIGEgZGlyZWN0b3J5IHdoZXJlIEpTT04gZmls
ZXMgY29udGFpbmluZyB0aGUgYmFzZWxpbmUgbWV0cmljcyBmb3IgdGhlIHRlc3RzIHdpbGwgYmUg
c3RvcmVkLiBCeSBkZWZhdWx0LCB0aGlzIHBhdGggd2lsbCBiZSBgTXlQYWNrYWdlL1Rlc3RzLzx3
YnI+UGVyZm9ybWFuY2VNZXRyaWNzYC4gSW4gbXkgcHJldmlvdXMgZW1haWwsIEkgc3VnZ2VzdGVk
IHRoZSBkZWZhdWx0IGAtLXBlcmZvcm1hbmNlLW1ldHJpY3NgIHBhdGjCoDxzcGFuIHN0eWxlPSJm
b250LXNpemU6IDEyLjhweDsiPmNvdWxkIGJlIHNldCB0byB0aGUgc2FtZSBwYXRoIGFzIHRoZSBg
c3dpZnQgdGVzdCAtLWJ1aWxkLXBhdGhgPC9zcGFuPjxmb250IGZhY2U9ImNvbnNvbGFzLCBpbmNv
bnNvbGF0YSwgY291cmllciwgbW9ub3NwYWNlIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMS4w
NXB4OyB3aGl0ZS1zcGFjZTogcHJlLXdyYXA7IGJhY2tncm91bmQtY29sb3I6IHJnYigyNDgsIDI0
OCwgMjQ4KTsiPiZuYnNwOzwvc3Bhbj48L2ZvbnQ+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTIu
OHB4OyI+ZGlyZWN0b3J5LCBidXQgSSBoYXZlIHJlY29uc2lkZXJlZC4gVGhpcyBpcyBiZWNhdXNl
IEkgdGhpbmsgZGV2ZWxvcGVycyB3b3VsZCB3YW50IHRvIGNoZWNrIHRoZWlyIGJhc2VsaW5lIG1l
dHJpY3MgSlNPTiBmaWxlcyBpbnRvIHNvdXJjZSBjb250cm9sLCBzbyB0aGF0IHRoZXkgY2FuIHNo
YXJlIG1ldHJpY3Mgd2l0aCBvbmUgYW5vdGhlciwgYW5kIHdpdGggdGhlaXIgY29udGludW91cyBp
bnRlZ3JhdGlvbiBzZXJ2ZXJzLjwvc3Bhbj48ZGl2PjIuIGBzd2lmdCB0ZXN0IC0tcGVyZm9ybWFu
Y2UtbWV0cmljcy11cGRhdGUgJmx0O21vZGUmZ3Q7YCwgd2hlcmUgYCZsdDttb2RlJmd0O2AgaXMg
b25lIG9mIGB7YWxsfG5ld3xiZXR0ZXJ8d29yc2V8bm9uZX1gLiBUaGlzIHNwZWNpZmllcyB0aGUg
YmVoYXZpb3IgU3dpZnRQTSBzaG91bGQgdGFrZSB3aGVuIHdyaXRpbmcgYmFzZWxpbmUgbWV0cmlj
cyBkYXRhIGludG8gdGhlIEpTT04gZmlsZXMgYXQgdGhlIGAtLXBlcmZvcm1hbmNlLW1ldHJpY3Ng
IHBhdGguPGJyPiZuYnNwOyAtIGBhbGxgOiBXcml0ZSBiYXNlbGluZSBtZXRyaWNzIGRhdGEgZm9y
IGFsbCBwZXJmb3JtYW5jZSB0ZXN0IGNhc2VzLiBJZiBtZXRyaWNzIGZvciB0aG9zZSB0ZXN0IGNh
c2VzIGFscmVhZHkgZXhpc3QgaW4gdGhlIEpTT04sIHRoZXkgYXJlIG92ZXJ3cml0dGVuLjxicj4m
bmJzcDsgLSBgbmV3YDogT25seSB3cml0ZSBiYXNlbGluZSBtZXRyaWNzIGZvciBwZXJmb3JtYW5j
ZSB0ZXN0IGNhc2VzIHRoYXQgZGlkIG5vdCBhbHJlYWR5IGV4aXN0IGluIHRoZSBiYXNlbGluZSBt
ZXRyaWNzIEpTT04uIFRoaXMgaXMgdGhlIGRlZmF1bHQuPGJyPiZuYnNwOyAtIGBiZXR0ZXJgOiBP
bmx5IHdyaXRlIGJhc2VsaW5lIG1ldHJpY3MgZm9yIHBlcmZvcm1hbmNlIHRlc3QgY2FzZXMgd2hv
c2UgcGVyZm9ybWFuY2UgaGFzIGltcHJvdmVkIGNvbXBhcmVkIHRvIHRoZSBsYXN0IHRpbWUgdGhl
eSB3ZXJlIHJ1bi4gSWYgYmFzZWxpbmUgbWV0cmljcyBmb3IgdGhvc2UgdGVzdCBjYXNlcyBhbHJl
YWR5IGV4aXN0IGluIHRoZSBKU09OLCB0aGV5IGFyZSBvdmVyd3JpdHRlbi4gSWYgYmFzZWxpbmUg
bWV0cmljcyBmb3IgdGhvc2UgdGVzdCBjYXNlcyBkb2VzIG5vdCBleGlzdCBpbiB0aGUgSlNPTiwg
dGhleSBhcmUgd3JpdHRlbiB0byB0aGUgSlNPTi48YnI+Jm5ic3A7IC0gYHdvcnNlYDogT25seSB3
cml0ZSBiYXNlbGluZSBtZXRyaWNzIGZvciBwZXJmb3JtYW5jZSB0ZXN0IGNhc2VzIHdob3NlIHBl
cmZvcm1hbmNlIGhhcyB3b3JzZW5lZCBjb21wYXJlZCB0byB0aGUgbGFzdCB0aW1lIHRoZXkgd2Vy
ZSBydW4uJm5ic3A7SWYgYmFzZWxpbmUgbWV0cmljcyBmb3IgdGhvc2UgdGVzdCBjYXNlcyBhbHJl
YWR5IGV4aXN0IGluIHRoZSBKU09OLCB0aGV5IGFyZSBvdmVyd3JpdHRlbi4gSWYgYmFzZWxpbmUg
bWV0cmljcyBmb3IgdGhvc2UgdGVzdCBjYXNlcyBkb2VzIG5vdCBleGlzdCBpbiB0aGUgSlNPTiwg
dGhleSBhcmUgd3JpdHRlbiB0byB0aGUgSlNPTi48YnI+PGJyPiMgVHdvIG5ldyBjb21tYW5kLWxp
bmUgb3B0aW9ucyBmb3Igc3dpZnQtY29yZWxpYnMteGN0ZXN0IGV4ZWN1dGFibGVzPGJyPjxicj4x
LiBgLS1wZXJmb3JtYW5jZS1tZXRyaWNzICZsdDtwYXRoJmd0O2AuIFRoaXMgaXMgYSBwYXRoIHRv
IGEgSlNPTiBmaWxlIGNvbnRhaW5pbmcgYSBtYXBwaW5nIGZyb20gdGVzdCBjYXNlcyB0byBiYXNl
bGluZSBtZXRyaWNzLjxicj4mbmJzcDsgLSBJZiBub3Qgc3BlY2lmaWVkLCBwZXJmb3JtYW5jZSB0
ZXN0cyBhcmUgbm90IHJ1biBhZ2FpbnN0IGFueSBiYXNlbGluZSBtZXRyaWNzLCBhbmQgc28gd2ls
bCBuZXZlciBmYWlsLjxicj4mbmJzcDsgLSBJZiBzcGVjaWZpZWQsIHBlcmZvcm1hbmNlIHRlc3Qg
Y2FzZXMgd2lsbCBiZSBydW4gYWdhaW5zdCB0aGVzZSBtZXRyaWNzLiBCYXNlZCBvbiB0aGUgYC0t
cGVyZm9ybWFuY2UtbWV0cmljcy11cGRhdGVgIG1vZGUgKHNlZSBiZWxvdyksIHBlcmZvcm1hbmNl
IHRlc3QgY2FzZXMgbWF5IGZhaWwgaWYgdGhlaXIgcGVyZm9ybWFuY2UgZG9lcyBub3QgbWVldCB0
aGUgYmFzZWxpbmUuPGJyPjIuIGAtLXBlcmZvcm1hbmNlLW1ldHJpY3MtdXBkYXRlICZsdDttb2Rl
Jmd0O2AuIFNhbWUgYXMgdGhlIGBzd2lmdCB0ZXN0IC0tcGVyZm9ybWFuY2UtbWV0cmljcy11cGRh
dGVgIHBhcmFtZXRlci48YnI+PGJyPiMgYFBlcmZvcm1hbmNlTWV0cmljc2AgZGlyZWN0b3J5PC9k
aXY+PGRpdj48YnI+PC9kaXY+PGRpdj5JZiBhIHBhY2thZ2UncyB0ZXN0cyBjb250YWluIGFueSBw
ZXJmb3JtYW5jZSB0ZXN0cyAoaS5lLjogdGVzdHMgdGhhdCBjYWxsIGBYQ1Rlc3RDYXNlLm1lYXN1
cmUoKWAsIGBYQ1Rlc3RDYXNlLm1lYXN1cmVNZXRyaWNzKClgLCBldGMuKSwgcnVubmluZyBgc3dp
ZnQgdGVzdGAgd2lsbCByZXN1bHQgaW4gdGhlIGZvbGxvd2luZyBkaXJlY3RvcmllcyBhbmQgZmls
ZXMgYmVpbmcgZ2VuZXJhdGVkOjxicj48YnI+YGBgPGJyPk15UGFja2FnZS88YnI+Jm5ic3A7ICZu
YnNwOyAuYnVpbGQvPGJyPiZuYnNwOyAmbmJzcDsgU291cmNlcy88YnI+Jm5ic3A7ICZuYnNwOyBU
ZXN0cy88YnI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IExpbnV4TWFpbi5zd2lmdDxicj4m
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgTXlQYWNrYWdlLzxicj4mbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyBNeVBhY2thZ2VUZXN0cy5zd2lmdDxicj4mbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgUGVyZm9ybWFuY2VNZXRyaWNzLyAmbmJzcDsjIEdlbmVyYXRl
ZCBpZiBhbnkgcGVyZm9ybWFuY2UgdGVzdHMgYXJlIHJ1bi4gVGhpcyBpcyB0aGUgcGF0aCBzcGVj
aWZpZWQgYnkgLS1wZXJmb3JtYW5jZS1tZXRyaWNzLjxicj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyBNeVBhY2thZ2UvPGJyPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgRGVzdGluYXRpb25zLmpzb24gJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IyBDb250YWlucyBhIG1hcHBpbmcgb2YgInJ1bkRl
c3RpbmF0aW9uc0J5VVVJRCIuPGJyPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgOENFOUUwNTEtOUFCNi00NEFGLThCODAtRjJERUZENDA5Q0I1
Lmpzb24gJm5ic3A7IyBBbiBpbmRpdmlkdWFsIHJ1biBkZXN0aW5hdGlvbidzIGJhc2VsaW5lIG1l
dHJpY3MuPGJyPmBgYDxicj48YnI+SW4gb3JkZXIgdG8gYXZvaWQgbmFtZSBjb2xsaXNpb25zLCBk
ZXZlbG9wZXJzIHdpbGwgbm8gbG9uZ2VyIGJlIGFibGUgdG8gbmFtZSB0aGVpciB0ZXN0IG1vZHVs
ZXMgIlBlcmZvcm1hbmNlTWV0cmljcyIuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj4jIFdoYXQg
aGFwcGVucyB3aGVuIGBzd2lmdCB0ZXN0YCBpcyBydW48YnI+PGJyPjEuIGBzd2lmdCB0ZXN0YCwg
dXNpbmcgdGhlIGRlZmF1bHQgYXJndW1lbnRzLCB3b3VsZCBiZSB0aGUgZXF1aXZhbGVudCBvZiBg
c3dpZnQgdGVzdCAtLXBlcmZvcm1hbmNlLW1ldHJpY3MgLi9UZXN0cy9QZXJmb3JtYW5jZU1ldHJp
Y3MgLS1wZXJmb3JtYW5jZS1tZXRyaWNzLXVwZGF0ZSBuZXdgLjwvZGl2PjxkaXY+Mi4gU3dpZnRQ
TSBkZXRlcm1pbmVzIHdoaWNoIG9mIHRoZSBkZXN0aW5hdGlvbnMgZGVmaW5lZCBpbiBgRGVzdGlu
YXRpb25zLmpzb25gIHRvIHBhc3MgdG8gWENUZXN0LiBGb3IgZXhhbXBsZSwgaWYgdGVzdGluZyBv
biBhIG1hY09TIDY0LWJpdCBzeXN0ZW0gd2l0aCBvbmUgcHJvY2Vzc29yLCBTd2lmdFBNIGF0dGVt
cHRzIHRvIGZpbmQgYSBydW4gZGVzdGluYXRpb24gVVVJRCBpbiBgRGVzdGluYXRpb25zLmpzb25g
IHRoYXQgbWF0Y2hlcyB0aG9zZSBjcml0ZXJpYS4gSWYgbm8gYERlc3RpbmF0aW9ucy5qc29uYCBm
aWxlIGV4aXN0cywgU3dpZnRQTSBjcmVhdGVzIGEgbWFwcGluZyBpbiBtZW1vcnkuPGJyPjMuIFN3
aWZ0UE0gaW52b2tlcyBgTGludXhNYWluLnN3aWZ0YCwgcGFzc2luZyBzd2lmdC1jb3JlbGlicy14
Y3Rlc3QgdGhlIHBhdGggdG8gYSBydW4gZGVzdGluYXRpb24ncyBiYXNlbGluZSBtZXRyaWNzIGZp
bGUgKGluIHRoaXMgY2FzZSwgYC0tcGVyZm9ybWFuY2UtbWV0cmljcyA4Q0U5RTA1MS05QUI2LTQ0
QUYtOEI4MC1GMkRFRkQ0MDlDQjUuanNvbmApLCBhcyB3ZWxsIGFzIHRoZSB1cGRhdGUgYmVoYXZp
b3IgKGBhbGxgLCBgbmV3YCwgYGJldHRlcmAsIG9yIGB3b3JzZWApLiBUaGlzIGZpbGUgbWF5IG5v
dCBhbHJlYWR5IGV4aXN0LCBzdWNoIGFzIGluIHRoZSBjYXNlIHRoYXQgYSBgRGVzdGluYXRpb25z
Lmpzb25gIGZpbGUgZGlkIG5vdCBleGlzdCwgb3IgdGhhdCBhIEpTT04gZmlsZSBmb3IgdGhpcyBw
YXJ0aWN1bGFyIHJ1biBkZXN0aW5hdGlvbiBkaWQgbm90IGV4aXN0Ljxicj40LiBzd2lmdC1jb3Jl
bGlicy14Y3Rlc3QgcGFyc2VzIHRoZSBKU09OIGluIHRoZSBiYXNlbGluZSBtZXRyaWNzIEpTT04g
ZmlsZSBpdCBpcyBnaXZlbiwgYW5kIHN0b3JlcyBpbiBtZW1vcnkgdGhlIG1hcHBpbmdzIGZyb20g
dGVzdCBjYXNlcyB0byB0aGVpciBiYXNlbGluZSBtZXRyaWNzLiBJZiB0aGUgZmlsZSBpcyBlbXB0
eSBvciBkb2VzIG5vdCBleGlzdCwgc3dpZnQtY29yZWxpYnMteGN0ZXN0IHN0b3JlcyBhbiBlbXB0
eSBtYXBwaW5nLjxicj41LiBzd2lmdC1jb3JlbGlicy14Y3Rlc3QgcnVucyB0aGUgdGVzdHMuIElm
IGEgdGVzdCBleGlzdHMgaW4gdGhlIG1hcHBpbmcgZnJvbSBzdGVwIDQsIGl0IGNvbXBhcmVzIGl0
cyBwZXJmb3JtYW5jZSB0byB0aGUgYmFzZWxpbmUgbWV0cmljLiBJZiB0aGUgcGVyZm9ybWFuY2Ug
aXMgd29yc2UsIGFuZCB0aGUgdXBkYXRlIGJlaGF2aW9yIGlzIGBuZXdgIG9yIGBiZXR0ZXJgLCB0
aGUgdGVzdCBjYXNlIGlzIGZhaWxlZC48YnI+Ni4gc3dpZnQtY29yZWxpYnMteGN0ZXN0IHdyaXRl
cyB0byB0aGUgYmFzZWxpbmUgbWV0cmljcyBKU09OIGZpbGUsIGJhc2VkIG9uIHRoZSBzcGVjaWZp
ZWQgdXBkYXRlIGJlaGF2aW9yLiBJZiB0aGUgZmlsZSBkb2VzIG5vdCBhbHJlYWR5IGV4aXN0LCBz
d2lmdC1jb3JlbGlicy14Y3Rlc3QgY3JlYXRlcyB0aGUgZmlsZSwgdGhlbiB3cml0ZXMgdG8gaXQu
PC9kaXY+PGRpdj43LiBBZnRlciBydW5uaW5nIHRoZSB0ZXN0cywgU3dpZnRQTSBkZXRlcm1pbmVz
IHdoZXRoZXIgdGhlIGA4Q0U5RTA1MS05QUI2LTQ0QUYtOEI4MC1GMkRFRkQ0MDlDQjUuanNvbmAg
ZmlsZSBjb250YWlucyBhbnkgZGF0YS4gSWYgaXQgZG9lcywgYW5kIHRoaXMgcnVuIGRlc3RpbmF0
aW9uIGRpZCBub3QgZXhpc3QgaW4gYERlc3RpbmF0aW9ucy5qc29uYCBpbiBzdGVwICgyKSwgdGhl
biBTd2lmdFBNIHdyaXRlcyB0aGUgbmV3IGRlc3RpbmF0aW9uIHRvIHRoZSBgRGVzdGluYXRpb25z
Lmpzb25gIGZpbGUuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj4jIEhvdyB3aWxsIHRoaXMgd29y
ayBvbiBEYXJ3aW4/PGJyPjxicj5JIHJlYWxpemVkIHdoaWxlIHdyaXRpbmcgdGhpcyBlbWFpbCB0
aGF0IEkgaGF2ZSBubyBjbHVlIGhvdyB0byBnZXQgdGhpcyB3b3JraW5nIG9uIERhcndpbi4gSXMg
aXQgZXZlbiBwb3NzaWJsZSB0byBzcGVjaWZ5IHRoZSBwYXRocyB0byBwZXJmb3JtYW5jZSBiYXNl
bGluZSBwbGlzdCBmaWxlcyB0byBBcHBsZSBYQ1Rlc3Qgb24gdGhlIGNvbW1hbmQgbGluZT8gVGhp
cyBzZWVtcyBsaWtlIGEgcHJlcmVxdWlzaXRlIHRvIHN1cHBvcnRpbmcgcGVyZm9ybWFuY2UgdGVz
dGluZyB2aWEgU3dpZnRQTSBvbiBEYXJ3aW4uPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5JdCB3
b3VsZCBiZSBncmVhdCB0byBoZWFyIGZyb20gc29tZW9uZSBvbiB0aGUgZGV2ZWxvcGVyIHRvb2xz
IHRlYW0gb24gdGhpcyB0b3BpYyAoK2NjIERhbmllbCBEdW5iYXIsIE1pa2UgRmVycmlzKS4gSSds
bCB0cnkgYW5kIGZpZ3VyZSBvdXQgaG93IHRoaXMgd29ya3MgaW4gQXBwbGUgWENUZXN0LCBhbmQg
d2lsbCBzZW5kIGFuIHVwZGF0ZSB3aGVuIEkgZG8uPGJyPjxicj4jIFRob3VnaHRzPzxicj48YnI+
QXMgYmVmb3JlLCBJJ2QgbG92ZSB0byBoZWFyIGFueSBmZWVkYmFjayB5b3UgYWxsIG1heSBoYXZl
IG9uIHRoaXMgcHJvcG9zYWwuPGJyPjxicj4tIEJyaWFuIEdlc2lhazxicj48YnI+PGJyPjwvZGl2
PjxkaXY+PGJyPjxicj48YnI+PC9kaXY+" style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jul 24, 2016 at 1:01 PM, Brian Gesiak <span dir="ltr"><<a href="mailto:modocache@gmail.com" target="_blank">modocache@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><p style="margin:0px 0px 1.2em!important">Hello corelibs-dev and build-dev,</p>
<p style="margin:0px 0px 1.2em!important">Back in May, Brian Croom implemented performance testing in swift-corelibs-xctest: <a href="https://github.com/apple/swift-corelibs-xctest/pull/109" target="_blank">https://github.com/apple/swift-corelibs-xctest/pull/109</a></p>
<p style="margin:0px 0px 1.2em!important">I’d love to see Swift developers use this feature to measure the performance of their code. I think we’ll need to add functionality to swift-corelibs-xctest and SwiftPM in order to do so.</p>
<h1 style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">The problem: recording performance test baselines</h1>
<p style="margin:0px 0px 1.2em!important">In order for performance tests to be useful, Apple’s Xcode provides a way to record “baseline” metrics. Baseline metrics allow a developer to indicate “this performance test should never be slower than 1.2 seconds on average, with 10% standard deviation as ‘wiggle room’”. When Apple XCTest tests are run, they are informed of the baseline metrics that have been set in Xcode. Apple XCTest performance tests that have a baseline registered will fail if performance becomes slower than the acceptable amount.</p>
<p style="margin:0px 0px 1.2em!important">If we could provide swift-corelibs-xctest with a mapping from each performance test to its baseline metric, it would be easy to write the code to fail a test if it didn’t perform well enough. That mapping, however, is the tricky part. Here’s why:</p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">The mapping needs to group metrics based on the host machine running the test. Performance will of course vary based on the hardware, so it’s important to make sure performance baselines set on a Raspberry Pi aren’t used when testing on a Mac Pro.</li>
<li style="margin:0.5em 0px">The mapping also needs to group metrics based on the target machine. Using Apple XCTest, a developer can start a test suite run from their MacBook Pro (macOS 64-bit), and see the results of the performance tests when run on their iPhone 6s (iOS armv7s). I don’t think this is relevant to swift-corelibs-xctest just yet — as far as I know, SwiftPM is not capable of cross-compilation, so the host machine will always be identical to the target machine. Still, we should design something flexible enough for this scenario.</li>
</ul>
<h1 style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">Xcode’s solution: plist files</h1>
<p style="margin:0px 0px 1.2em!important">Xcode’s solves this problem using two kinds of .plist files. I tried creating a sample project, named <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">Perforate.xcodeproj</code>, which contained a single performance test. Here’s what Xcode created:</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248);white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important;display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248)"><span style="color:rgb(153,153,136);font-style:italic"><!-- Perforate.xcodeproj/xcshareddata/xcbaselines/DA77262F1D447DB300735C93.xcbaseline/Info.plist --></span>
<span style="color:rgb(153,153,153);font-weight:bold"><?xml version="1.0" encoding="UTF-8"?></span>
<span style="color:rgb(153,153,153);font-weight:bold"><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "<a href="http://www.apple.com/DTDs/PropertyList-1.0.dtd" target="_blank">http://www.apple.com/DTDs/PropertyList-1.0.dtd</a>"></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">plist</span> <span style="color:rgb(0,128,128)">version</span>=<span style="color:rgb(221,17,68)">"1.0"</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(153,153,136);font-style:italic"><!-- runDestinationsByUUID: These are the host/target machine groups. --></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>runDestinationsByUUID<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(153,153,136);font-style:italic"><!--
It appears each group is given a UUID, but to be honest, I'm not sure why.
It seems like these should be "keyed" on aspects of the host/target machines.
As-is, I imagine Xcode and Apple XCTest need to traverse each group's
`localComputer`, `targetArchitecture`, and `targetDevice`'s values in order to find a match.
--></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(153,153,136);font-style:italic"><!-- Information about the host machine: number of CPUs, cores, etc. --></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>localComputer<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>busSpeedInMHz<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>100<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>cpuCount<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>1<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>cpuKind<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>Intel Core i7<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>cpuSpeedInMHz<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>2800<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>logicalCPUCoresPerPackage<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>8<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>modelCode<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>MacBookPro11,3<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>physicalCPUCoresPerPackage<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>4<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">integer</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>platformIdentifier<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>com.apple.platform.macosx<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(153,153,136);font-style:italic"><!-- The target architecture and device are stored as separate keys. --></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>targetArchitecture<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>x86_64<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>targetDevice<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>modelCode<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>iPhone8,2<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>platformIdentifier<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>com.apple.platform.iphonesimulator<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">plist</span>></span>
</code></pre>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248);white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important;display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248)"><span style="color:rgb(153,153,136);font-style:italic"><!-- Perforate.xcodeproj/xcshareddata/xcbaselines/DA77262F1D447DB300735C93.xcbaseline/8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5.plist --></span>
<span style="color:rgb(153,153,136);font-style:italic"><!-- Notice that this file is named after the `runDestinationsByUUID` key from the first file: 8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5. --></span>
<span style="color:rgb(153,153,153);font-weight:bold"><?xml version="1.0" encoding="UTF-8"?></span>
<span style="color:rgb(153,153,153);font-weight:bold"><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "<a href="http://www.apple.com/DTDs/PropertyList-1.0.dtd" target="_blank">http://www.apple.com/DTDs/PropertyList-1.0.dtd</a>"></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">plist</span> <span style="color:rgb(0,128,128)">version</span>=<span style="color:rgb(221,17,68)">"1.0"</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>classNames<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>PerforateTests<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(153,153,136);font-style:italic"><!-- The metrics are mapped by class name and test method name to performance metrics. --></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>test_uniqueOrdered_performance<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(153,153,136);font-style:italic"><!-- There are several categories of performance metrics. The only one publicly available in Apple XCTest so far is wall clock time. --></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>com.apple.XCTPerformanceMetric_WallClockTime<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>baselineAverage<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">real</span>></span>0.5<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">real</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>baselineIntegrationDisplayName<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">key</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"><<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>Local Baseline<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">string</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">dict</span>></span>
<span style="color:rgb(0,0,128);font-weight:normal"></<span style="color:rgb(153,0,0);font-weight:bold;color:rgb(0,0,128);font-weight:normal">plist</span>></span>
</code></pre>
<h1 style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">Proposed solution for SwiftPM/swift-corelibs-xctest: JSON files</h1>
<p style="margin:0px 0px 1.2em!important">I think we can mimic Xcode’s approach here. Here’s what I’m proposing:</p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">swift-corelibs-xctest’s test runner should take a <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics <PATH></code> argument, where <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)"><PATH></code> is the location of a file containing JSON that looks pretty much exactly like the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5.plist</code> from above:</li>
</ul>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248);white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important;display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248)">{
"<span style="color:rgb(0,128,128)">classNames</span>": <span>{
"<span style="color:rgb(0,128,128)">PerforateTests</span>": <span>{
"<span style="color:rgb(0,128,128)">test_uniqueOrdered_performance</span>": <span>{
"<span style="color:rgb(0,128,128)">baselineAverage</span>": <span><span style="color:rgb(221,17,68)">"0.5"</span></span>,
"<span style="color:rgb(0,128,128)">baselineIntegrationDisplayName</span>": <span><span style="color:rgb(221,17,68)">"Local Baseline"</span>
</span>}
</span>}
</span>}
</span>}
</code></pre>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">SwiftPM’s <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test</code> command should also take a <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics <PATH></code> argument, where <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)"><PATH></code> is the location of a file containing JSON that looks pretty much exactly like the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">xcbaselines/DA77262F1D447DB300735C93.xcbaseline/Info.plist</code> from above (by default, <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">--performance-metrics</code> could be set to the same path as the <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swift test --build-path</code> directory): </li>
</ul>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248);white-space:pre-wrap;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important;display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248)">{
"runDestinationsByUUID": {
"8CE9E051-9AB6-44AF-8B80-F2DEFD409CB5": {
"localComputer": {
"busSpeedInMHz": "100",
# ...
},
"targetArchitecture": "x86_64",
"targetDevice": {
# We might need to change these keys, since "modelCode" seems very Apple-specific.
"modelCode": "linux",
"platformIdentifier": "Ubuntu 15.04",
}
}
}
}
</code></pre>
<p style="margin:0px 0px 1.2em!important">Personally, I think the format of the plist files Xcode and Apple XCTest generate could be improved. Still, I think it’d be nice to stick to the same format (as much as possible) for swift-corelibs-xctest, just to keep things simple.</p>
<h1 style="margin:1.3em 0px 1em;padding:0px;font-weight:bold;font-size:1.6em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221)">Thoughts?</h1>
<p style="margin:0px 0px 1.2em!important">I admit that I don’t have much experience using Apple XCTest’s performance testing functionality, so I might be missing something here. Does anyone have any feedback on this idea? I’d like to incorporate your feedback, and perhaps submit a Swift Evolution proposal for this feature.</p><span class="HOEnZb"><font color="#888888"><p style="margin:0px 0px 1.2em!important">- Brian Gesiak</p>
<div title="MDH:SGVsbG8gY29yZWxpYnMtZGV2IGFuZCBidWlsZC1kZXYsPGRpdj48YnI+PC9kaXY+PGRpdj5CYWNr
IGluIE1heSwgQnJpYW4gQ3Jvb20gaW1wbGVtZW50ZWQgcGVyZm9ybWFuY2UgdGVzdGluZyBpbiBz
d2lmdC1jb3JlbGlicy14Y3Rlc3Q6Jm5ic3A7PGEgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL2Fw
cGxlL3N3aWZ0LWNvcmVsaWJzLXhjdGVzdC9wdWxsLzEwOSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBz
Oi8vZ2l0aHViLmNvbS88d2JyPmFwcGxlL3N3aWZ0LWNvcmVsaWJzLXhjdGVzdC88d2JyPnB1bGwv
MTA5PC9hPjxicj48YnI+SSdkIGxvdmUgdG8gc2VlIFN3aWZ0IGRldmVsb3BlcnMgdXNlIHRoaXMg
ZmVhdHVyZSB0byBtZWFzdXJlIHRoZSBwZXJmb3JtYW5jZSBvZiB0aGVpciBjb2RlLiBJIHRoaW5r
IHdlJ2xsIG5lZWQgdG8gYWRkIGZ1bmN0aW9uYWxpdHkgdG8gc3dpZnQtY29yZWxpYnMteGN0ZXN0
IGFuZCBTd2lmdFBNIGluIG9yZGVyIHRvIGRvIHNvLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+
IyBUaGUgcHJvYmxlbTogcmVjb3JkaW5nIHBlcmZvcm1hbmNlIHRlc3QgYmFzZWxpbmVzPC9kaXY+
PGRpdj48YnI+PC9kaXY+PGRpdj5JbiBvcmRlciBmb3IgcGVyZm9ybWFuY2UgdGVzdHMgdG8gYmUg
dXNlZnVsLCBBcHBsZSdzIFhjb2RlIHByb3ZpZGVzIGEgd2F5IHRvIHJlY29yZCAiYmFzZWxpbmUi
IG1ldHJpY3MuIEJhc2VsaW5lIG1ldHJpY3MgYWxsb3cgYSBkZXZlbG9wZXIgdG8gaW5kaWNhdGUg
InRoaXMgcGVyZm9ybWFuY2UgdGVzdCBzaG91bGQgbmV2ZXIgYmUgc2xvd2VyIHRoYW4gMS4yIHNl
Y29uZHMgb24gYXZlcmFnZSwgd2l0aCAxMCUgc3RhbmRhcmQgZGV2aWF0aW9uIGFzICd3aWdnbGUg
cm9vbSciLiBXaGVuIEFwcGxlIFhDVGVzdCB0ZXN0cyBhcmUgcnVuLCB0aGV5IGFyZSBpbmZvcm1l
ZCBvZiB0aGUgYmFzZWxpbmUgbWV0cmljcyB0aGF0IGhhdmUgYmVlbiBzZXQgaW4gWGNvZGUuIEFw
cGxlIFhDVGVzdCBwZXJmb3JtYW5jZSB0ZXN0cyB0aGF0IGhhdmUgYSBiYXNlbGluZSByZWdpc3Rl
cmVkIHdpbGwgZmFpbCBpZiBwZXJmb3JtYW5jZSBiZWNvbWVzIHNsb3dlciB0aGFuIHRoZSBhY2Nl
cHRhYmxlIGFtb3VudC48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PklmIHdlIGNvdWxkIHByb3Zp
ZGUgc3dpZnQtY29yZWxpYnMteGN0ZXN0IHdpdGggYSBtYXBwaW5nIGZyb20gZWFjaCBwZXJmb3Jt
YW5jZSB0ZXN0IHRvIGl0cyBiYXNlbGluZSBtZXRyaWMsIGl0IHdvdWxkIGJlIGVhc3kgdG8gd3Jp
dGUgdGhlIGNvZGUgdG8gZmFpbCBhIHRlc3QgaWYgaXQgZGlkbid0IHBlcmZvcm0gd2VsbCBlbm91
Z2guIFRoYXQgbWFwcGluZywgaG93ZXZlciwgaXMgdGhlIHRyaWNreSBwYXJ0LiBIZXJlJ3Mgd2h5
Ojxicj48YnI+LSBUaGUgbWFwcGluZyBuZWVkcyB0byBncm91cCBtZXRyaWNzIGJhc2VkIG9uIHRo
ZSBob3N0IG1hY2hpbmUgcnVubmluZyB0aGUgdGVzdC4gUGVyZm9ybWFuY2Ugd2lsbCBvZiBjb3Vy
c2UgdmFyeSBiYXNlZCBvbiB0aGUgaGFyZHdhcmUsIHNvIGl0J3MgaW1wb3J0YW50IHRvIG1ha2Ug
c3VyZSBwZXJmb3JtYW5jZSBiYXNlbGluZXMgc2V0IG9uIGEgUmFzcGJlcnJ5IFBpIGFyZW4ndCB1
c2VkIHdoZW4gdGVzdGluZyBvbiBhIE1hYyBQcm8uPGJyPi0gVGhlIG1hcHBpbmcgYWxzbyBuZWVk
cyB0byBncm91cCBtZXRyaWNzIGJhc2VkIG9uIHRoZSB0YXJnZXQgbWFjaGluZS4gVXNpbmcgQXBw
bGUgWENUZXN0LCBhIGRldmVsb3BlciBjYW4gc3RhcnQgYSB0ZXN0IHN1aXRlIHJ1biBmcm9tIHRo
ZWlyIE1hY0Jvb2sgUHJvIChtYWNPUyA2NC1iaXQpLCBhbmQgc2VlIHRoZSByZXN1bHRzIG9mIHRo
ZSBwZXJmb3JtYW5jZSB0ZXN0cyB3aGVuIHJ1biBvbiB0aGVpciBpUGhvbmUgNnMgKGlPUyBhcm12
N3MpLiBJIGRvbid0IHRoaW5rIHRoaXMgaXMgcmVsZXZhbnQgdG8gc3dpZnQtY29yZWxpYnMteGN0
ZXN0IGp1c3QgeWV0IC0tIGFzIGZhciBhcyBJIGtub3csIFN3aWZ0UE0gaXMgbm90IGNhcGFibGUg
b2YgY3Jvc3MtY29tcGlsYXRpb24sIHNvIHRoZSBob3N0IG1hY2hpbmUgd2lsbCBhbHdheXMgYmUg
aWRlbnRpY2FsIHRvIHRoZSB0YXJnZXQgbWFjaGluZS4gU3RpbGwsIHdlIHNob3VsZCBkZXNpZ24g
c29tZXRoaW5nIGZsZXhpYmxlIGVub3VnaCBmb3IgdGhpcyBzY2VuYXJpby48YnI+PGJyPiMgWGNv
ZGUncyBzb2x1dGlvbjogcGxpc3QgZmlsZXM8YnI+PGJyPlhjb2RlJ3Mgc29sdmVzIHRoaXMgcHJv
YmxlbSB1c2luZyB0d28ga2luZHMgb2YgLnBsaXN0IGZpbGVzLiBJIHRyaWVkIGNyZWF0aW5nIGEg
c2FtcGxlIHByb2plY3QsIG5hbWVkIGBQZXJmb3JhdGUueGNvZGVwcm9qYCwgd2hpY2ggY29udGFp
bmVkIGEgc2luZ2xlIHBlcmZvcm1hbmNlIHRlc3QuIEhlcmUncyB3aGF0IFhjb2RlIGNyZWF0ZWQ6
PC9kaXY+PGRpdj48YnI+YGBgeG1sPGJyPiZsdDshLS0mbmJzcDtQZXJmb3JhdGUueGNvZGVwcm9q
L3hjc2hhcmVkZGF0YS94Y2Jhc2VsaW5lcy9EQTc3MjYyRjFENDQ3REIzMDA3MzVDOTMueGNiYXNl
bGluZS9JbmZvLnBsaXN0IC0tJmd0Ozxicj48YnI+PGRpdj4mbHQ7P3htbCB2ZXJzaW9uPSIxLjAi
IGVuY29kaW5nPSJVVEYtOCI/Jmd0OzwvZGl2PjxkaXY+Jmx0OyFET0NUWVBFIHBsaXN0IFBVQkxJ
QyAiLS8vQXBwbGUvL0RURCBQTElTVCAxLjAvL0VOIiAiaHR0cDovL3d3dy5hcHBsZS5jb20vRFRE
cy9Qcm9wZXJ0eUxpc3QtMS4wLmR0ZCImZ3Q7PC9kaXY+PGRpdj4mbHQ7cGxpc3QgdmVyc2lvbj0i
MS4wIiZndDs8L2Rpdj48ZGl2PiZsdDtkaWN0Jmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZsdDshLS0gcnVuRGVzdGluYXRpb25zQnlVVUlEOiBUaGVzZSBhcmUgdGhl
IGhvc3QvdGFyZ2V0IG1hY2hpbmUgZ3JvdXBzLiZuYnNwOy0tJmd0OzwvZGl2PjxkaXY+Jm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtrZXkmZ3Q7cnVuRGVzdGluYXRpb25zQnlVVUlEJmx0
Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2RpY3Qm
Z3Q7PGJyPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJmx0OyEtLTxicj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyBJdCBhcHBlYXJzIGVh
Y2ggZ3JvdXAgaXMgZ2l2ZW4gYSBVVUlELCBidXQgdG8gYmUgaG9uZXN0LCBJJ20gbm90IHN1cmUg
d2h5LjwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgSXQgc2VlbXMgbGlrZSB0
aGVzZSBzaG91bGQgYmUgImtleWVkIiBvbiBhc3BlY3RzIG9mIHRoZSBob3N0L3RhcmdldCBtYWNo
aW5lcy48L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IEFzLWlzLCBJIGltYWdp
bmUgWGNvZGUgYW5kIEFwcGxlIFhDVGVzdCBuZWVkIHRvIHRyYXZlcnNlIGVhY2ggZ3JvdXAnczwv
ZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgYGxvY2FsQ29tcHV0ZXJgLCBgdGFy
Z2V0QXJjaGl0ZWN0dXJlYCwgYW5kIGB0YXJnZXREZXZpY2VgJ3MgdmFsdWVzIGluIG9yZGVyIHRv
IGZpbmQgYSBtYXRjaC48YnI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAtLSZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2tleSZndDs4Q0U5RTA1MS05QUI2
LTQ0QUYtOEI4MC1GMkRFRkQ0MDlDQjUmbHQ7L2tleSZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2RpY3QmZ3Q7
PGJyPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDshLS0gSW5mb3JtYXRpb24gYWJvdXQg
dGhlIGhvc3QgbWFjaGluZTogbnVtYmVyIG9mIENQVXMsIGNvcmVzLCBldGMuIC0tJmd0OzwvZGl2
PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2tleSZndDtsb2NhbENvbXB1dGVy
Jmx0Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7ZGlj
dCZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbHQ7a2V5Jmd0O2J1c1NwZWVkSW5NSHombHQ7L2tleSZndDs8L2Rpdj48
ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbHQ7aW50ZWdlciZndDsxMDAmbHQ7L2ludGVnZXImZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2tleSZndDtj
cHVDb3VudCZsdDsva2V5Jmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtpbnRlZ2VyJmd0OzEmbHQ7L2ludGVnZXIm
Z3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJmx0O2tleSZndDtjcHVLaW5kJmx0Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O3N0
cmluZyZndDtJbnRlbCBDb3JlIGk3Jmx0Oy9zdHJpbmcmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2tleSZndDtj
cHVTcGVlZEluTUh6Jmx0Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2ludGVnZXImZ3Q7MjgwMCZsdDsv
aW50ZWdlciZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7a2V5Jmd0O2xvZ2ljYWxDUFVDb3Jlc1BlclBhY2thZ2Um
bHQ7L2tleSZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7aW50ZWdlciZndDs4Jmx0Oy9pbnRlZ2VyJmd0OzwvZGl2
PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZsdDtrZXkmZ3Q7bW9kZWxDb2RlJmx0Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O3N0cmluZyZn
dDtNYWNCb29rUHJvMTEsMyZsdDsvc3RyaW5nJmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtrZXkmZ3Q7cGh5c2lj
YWxDUFVDb3Jlc1BlclBhY2thZ2UmbHQ7L2tleSZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7aW50ZWdlciZndDs0
Jmx0Oy9pbnRlZ2VyJmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtrZXkmZ3Q7cGxhdGZvcm1JZGVudGlmaWVyJmx0
Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJmx0O3N0cmluZyZndDtjb20uYXBwbGUucGxhdGZvcm0ubWFjb3N4
Jmx0Oy9zdHJpbmcmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7
L2RpY3QmZ3Q7PGJyPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDshLS0gVGhlIHRhcmdl
dCBhcmNoaXRlY3R1cmUgYW5kIGRldmljZSBhcmUgc3RvcmVkIGFzIHNlcGFyYXRlIGtleXMuIC0t
Jmd0Ozxicj48L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtrZXkmZ3Q7
dGFyZ2V0QXJjaGl0ZWN0dXJlJmx0Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbHQ7c3RyaW5nJmd0O3g4Nl82NCZsdDsvc3RyaW5nJmd0OzwvZGl2PjxkaXY+
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2tleSZndDt0YXJnZXREZXZpY2UmbHQ7L2tl
eSZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtkaWN0Jmd0Ozwv
ZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZsdDtrZXkmZ3Q7bW9kZWxDb2RlJmx0Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O3N0cmlu
ZyZndDtpUGhvbmU4LDImbHQ7L3N0cmluZyZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7a2V5Jmd0O3BsYXRmb3Jt
SWRlbnRpZmllciZsdDsva2V5Jmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtzdHJpbmcmZ3Q7Y29tLmFwcGxlLnBs
YXRmb3JtLmlwaG9uZXNpbXVsYXRvciZsdDsvc3RyaW5nJmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJmx0Oy9kaWN0Jmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7L2RpY3QmZ3Q7PC9k
aXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0Oy9kaWN0Jmd0OzwvZGl2Pjxk
aXY+Jmx0Oy9kaWN0Jmd0OzwvZGl2PjxkaXY+Jmx0Oy9wbGlzdCZndDs8YnI+YGBgPC9kaXY+PC9k
aXY+PGRpdj48YnI+PC9kaXY+PGRpdj5gYGB4bWw8YnI+Jmx0OyEtLSZuYnNwO1BlcmZvcmF0ZS54
Y29kZXByb2oveGNzaGFyZWRkYXRhL3hjYmFzZWxpbmVzL0RBNzcyNjJGMUQ0NDdEQjMwMDczNUM5
My54Y2Jhc2VsaW5lLzhDRTlFMDUxLTlBQjYtNDRBRi04QjgwLUYyREVGRDQwOUNCNS5wbGlzdCAt
LSZndDs8YnI+PGJyPiZsdDshLS0gTm90aWNlIHRoYXQgdGhpcyBmaWxlIGlzIG5hbWVkIGFmdGVy
IHRoZSBgcnVuRGVzdGluYXRpb25zQnlVVUlEYCBrZXkgZnJvbSB0aGUgZmlyc3QgZmlsZTombmJz
cDs4Q0U5RTA1MS05QUI2LTQ0QUYtOEI4MC1GMkRFRkQ0MDlDQjUuIC0tJmd0Ozxicj48YnI+PGRp
dj4mbHQ7P3htbCB2ZXJzaW9uPSIxLjAiIGVuY29kaW5nPSJVVEYtOCI/Jmd0OzwvZGl2PjxkaXY+
Jmx0OyFET0NUWVBFIHBsaXN0IFBVQkxJQyAiLS8vQXBwbGUvL0RURCBQTElTVCAxLjAvL0VOIiAi
aHR0cDovL3d3dy5hcHBsZS5jb20vRFREcy9Qcm9wZXJ0eUxpc3QtMS4wLmR0ZCImZ3Q7PC9kaXY+
PGRpdj4mbHQ7cGxpc3QgdmVyc2lvbj0iMS4wIiZndDs8L2Rpdj48ZGl2PiZsdDtkaWN0Jmd0Ozxi
cj48L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7a2V5Jmd0O2NsYXNz
TmFtZXMmbHQ7L2tleSZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bHQ7ZGljdCZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2tleSZndDtQZXJmb3JhdGVUZXN0cyZsdDsva2V5Jmd0
OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbHQ7ZGljdCZndDs8YnI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0
OyEtLSBUaGUgbWV0cmljcyBhcmUgbWFwcGVkIGJ5IGNsYXNzIG5hbWUgYW5kIHRlc3QgbWV0aG9k
IG5hbWUgdG8gcGVyZm9ybWFuY2UgbWV0cmljcy4gLS0mZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbHQ7a2V5Jmd0O3Rlc3RfdW5pcXVlT3JkZXJlZF9wZXJmb3JtYW5j
ZSZsdDsva2V5Jmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2Rp
Y3QmZ3Q7PGJyPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbHQ7IS0tIFRoZXJlIGFyZSBzZXZlcmFsIGNhdGVnb3JpZXMgb2YgcGVyZm9ybWFu
Y2UgbWV0cmljcy4gVGhlIG9ubHkgb25lIHB1YmxpY2x5IGF2YWlsYWJsZSBpbiBBcHBsZSBYQ1Rl
c3Qgc28gZmFyIGlzIHdhbGwgY2xvY2sgdGltZS4gLS0mZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O2tleSZndDtj
b20uYXBwbGUuWENUUGVyZm9ybWFuY2VNZXRyaWNfV2FsbENsb2NrVGltZSZsdDsva2V5Jmd0Ozwv
ZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZsdDtkaWN0Jmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bHQ7a2V5Jmd0O2Jhc2VsaW5lQXZlcmFnZSZsdDsva2V5Jmd0OzwvZGl2PjxkaXY+Jm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbHQ7cmVhbCZndDswLjUmbHQ7L3JlYWwmZ3Q7PC9kaXY+PGRpdj4m
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZsdDtrZXkmZ3Q7YmFzZWxpbmVJbnRlZ3JhdGlvbkRp
c3BsYXlOYW1lJmx0Oy9rZXkmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZsdDtzdHJpbmcmZ3Q7TG9jYWwgQmFzZWxpbmUmbHQ7L3N0cmluZyZndDs8L2Rpdj48ZGl2PiZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7
L2RpY3QmZ3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7L2RpY3Qm
Z3Q7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZsdDsvZGljdCZndDs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbHQ7L2RpY3QmZ3Q7PC9kaXY+PGRpdj4mbHQ7L2RpY3QmZ3Q7PC9kaXY+PGRpdj4m
bHQ7L3BsaXN0Jmd0OzwvZGl2PmBgYDxicj48YnI+IyBQcm9wb3NlZCBzb2x1dGlvbiBmb3IgU3dp
ZnRQTS9zd2lmdC1jb3JlbGlicy14Y3Rlc3Q6IEpTT04gZmlsZXM8YnI+PGJyPkkgdGhpbmsgd2Ug
Y2FuIG1pbWljIFhjb2RlJ3MgYXBwcm9hY2ggaGVyZS4gSGVyZSdzIHdoYXQgSSdtIHByb3Bvc2lu
Zzo8YnI+PGJyPi0gc3dpZnQtY29yZWxpYnMteGN0ZXN0J3MgdGVzdCBydW5uZXIgc2hvdWxkIHRh
a2UgYSBgLS1wZXJmb3JtYW5jZS1tZXRyaWNzICZsdDtQQVRIJmd0O2AgYXJndW1lbnQsIHdoZXJl
IGAmbHQ7UEFUSCZndDtgIGlzIHRoZSBsb2NhdGlvbiBvZiBhIGZpbGUgY29udGFpbmluZyBKU09O
IHRoYXQgbG9va3MgcHJldHR5IG11Y2ggZXhhY3RseSBsaWtlIHRoZSBgOENFOUUwNTEtOUFCNi00
NEFGLThCODAtRjJERUZENDA5Q0I1LnBsaXN0YCBmcm9tIGFib3ZlOjwvZGl2PjxkaXY+PGJyPmBg
YGpzb248YnI+PGRpdj57PC9kaXY+PGRpdj4mbmJzcDsgImNsYXNzTmFtZXMiOiB7PC9kaXY+PGRp
dj4mbmJzcDsgJm5ic3A7ICJQZXJmb3JhdGVUZXN0cyI6IHs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJz
cDsgJm5ic3A7ICJ0ZXN0X3VuaXF1ZU9yZGVyZWRfcGVyZm9ybWFuY2UiOiB7PC9kaXY+PGRpdj4m
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgImJhc2VsaW5lQXZlcmFnZSI6ICIwLjUiLDwvZGl2
PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICJiYXNlbGluZUludGVncmF0aW9uRGlz
cGxheU5hbWUiOiAiTG9jYWwgQmFzZWxpbmUiPC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNw
OyB9PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7IH08L2Rpdj48ZGl2PiZuYnNwOyB9PC9kaXY+PGRp
dj59PC9kaXY+YGBgPGJyPjxicj4tIFN3aWZ0UE0ncyBgc3dpZnQgdGVzdGAgY29tbWFuZCBzaG91
bGQgYWxzbyB0YWtlIGEgYC0tcGVyZm9ybWFuY2UtbWV0cmljcyAmbHQ7UEFUSCZndDtgIGFyZ3Vt
ZW50LCB3aGVyZSBgJmx0O1BBVEgmZ3Q7YCBpcyB0aGUgbG9jYXRpb24gb2YgYSBmaWxlIGNvbnRh
aW5pbmcgSlNPTiB0aGF0IGxvb2tzIHByZXR0eSBtdWNoIGV4YWN0bHkgbGlrZSB0aGUgYHhjYmFz
ZWxpbmVzL0RBNzcyNjJGMUQ0NDdEQjMwMDczNUM5My54Y2Jhc2VsaW5lL0luZm8ucGxpc3RgIGZy
b20gYWJvdmUgKGJ5Jm5ic3A7ZGVmYXVsdCwgYC0tcGVyZm9ybWFuY2UtbWV0cmljc2AgY291bGQg
YmUgc2V0IHRvIHRoZSBzYW1lIHBhdGggYXMgdGhlIGBzd2lmdCB0ZXN0IC0tYnVpbGQtcGF0aGAg
ZGlyZWN0b3J5KTombmJzcDs8L2Rpdj48ZGl2Pjxicj5gYGBqc29uPGJyPjxkaXY+ezwvZGl2Pjxk
aXY+Jm5ic3A7ICJydW5EZXN0aW5hdGlvbnNCeVVVSUQiOiB7PC9kaXY+PGRpdj4mbmJzcDsgJm5i
c3A7ICI4Q0U5RTA1MS05QUI2LTQ0QUYtOEI4MC1GMkRFRkQ0MDlDQjUiOiB7PC9kaXY+PGRpdj4m
bmJzcDsgJm5ic3A7ICZuYnNwOyAibG9jYWxDb21wdXRlciI6IHs8L2Rpdj48ZGl2PiZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAiYnVzU3BlZWRJbk1IeiI6ICIxMDAiLDwvZGl2PjxkaXY+Jm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICMgLi4uPC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZu
YnNwOyB9LDwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgInRhcmdldEFyY2hpdGVjdHVy
ZSI6ICJ4ODZfNjQiLDwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgInRhcmdldERldmlj
ZSI6IHs8YnI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICMgV2UgbWlnaHQgbmVlZCB0byBj
aGFuZ2UgdGhlc2Uga2V5cywgc2luY2UgIm1vZGVsQ29kZSIgc2VlbXMgdmVyeSBBcHBsZS1zcGVj
aWZpYy48L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAibW9kZWxDb2RlIjog
ImxpbnV4Iiw8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAicGxhdGZvcm1J
ZGVudGlmaWVyIjogIlVidW50dSAxNS4wNCIsPC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNw
OyB9PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7IH08L2Rpdj48ZGl2PiZuYnNwOyB9PC9kaXY+PGRp
dj59PC9kaXY+YGBgPGJyPjxicj5QZXJzb25hbGx5LCBJIHRoaW5rIHRoZSBmb3JtYXQgb2YgdGhl
IHBsaXN0IGZpbGVzIFhjb2RlIGFuZCBBcHBsZSBYQ1Rlc3QgZ2VuZXJhdGUgY291bGQgYmUgaW1w
cm92ZWQuIFN0aWxsLCBJIHRoaW5rIGl0J2QgYmUgbmljZSB0byBzdGljayB0byB0aGUgc2FtZSBm
b3JtYXQgKGFzIG11Y2ggYXMgcG9zc2libGUpIGZvciBzd2lmdC1jb3JlbGlicy14Y3Rlc3QsIGp1
c3QgdG8ga2VlcCB0aGluZ3Mgc2ltcGxlLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+IyBUaG91
Z2h0cz88YnI+PGJyPkkgYWRtaXQgdGhhdCBJIGRvbid0IGhhdmUgbXVjaCBleHBlcmllbmNlIHVz
aW5nIEFwcGxlIFhDVGVzdCdzIHBlcmZvcm1hbmNlIHRlc3RpbmcgZnVuY3Rpb25hbGl0eSwgc28g
SSBtaWdodCBiZSBtaXNzaW5nIHNvbWV0aGluZyBoZXJlLiBEb2VzIGFueW9uZSBoYXZlIGFueSBm
ZWVkYmFjayBvbiB0aGlzIGlkZWE/IEknZCBsaWtlIHRvIGluY29ycG9yYXRlIHlvdXIgZmVlZGJh
Y2ssIGFuZCBwZXJoYXBzIHN1Ym1pdCBhIFN3aWZ0IEV2b2x1dGlvbiBwcm9wb3NhbCBmb3IgdGhp
cyBmZWF0dXJlLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+LSBCcmlhbiBHZXNpYWs8L2Rpdj48
ZGl2Pjxicj48L2Rpdj4=" style="min-height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></font></span></div></div>
</blockquote></div><br></div>