<div dir="ltr"><div class="markdown-here-wrapper" style=""><p style="margin:0px 0px 1.2em!important">Hello all!</p>
<p style="margin:0px 0px 1.2em!important"><a href="https://bugs.swift.org/browse/SR-710">SR-710</a> tracks a major goal for Swift 3: having SwiftPM/corelibs-xctest automatically generate a list of test methods to execute. The implementations we’re considering span various parts of the codebase: libIDE, SourceKit, the reflection APIs, etc. We need input from people familiar with these components.</p>
<p style="margin:0px 0px 1.2em!important">Here’s the issue: currently, users of corelibs-xctest must manually list each test they wish to execute:</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code class="hljs language-swift" style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248);white-space:pre;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important;display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248)"><span class="hljs-class"><span class="hljs-keyword" style="color:rgb(51,51,51);font-weight:bold">class</span> <span class="hljs-title" style="color:rgb(153,0,0);font-weight:bold;color:rgb(68,85,136);font-weight:bold">MyTestCase</span>: <span class="hljs-title" style="color:rgb(153,0,0);font-weight:bold;color:rgb(68,85,136);font-weight:bold">XCTestCase</span> </span>{
<span class="hljs-keyword" style="color:rgb(51,51,51);font-weight:bold">static</span> <span class="hljs-keyword" style="color:rgb(51,51,51);font-weight:bold">var</span> allTests: [(<span class="hljs-type" style="color:rgb(68,85,136);font-weight:bold">String</span>, <span class="hljs-type" style="color:rgb(68,85,136);font-weight:bold">MyTestCase</span> -> () throws -> <span class="hljs-type" style="color:rgb(68,85,136);font-weight:bold">Void</span>)] {
<span class="hljs-keyword" style="color:rgb(51,51,51);font-weight:bold">return</span> [
(<span class="hljs-string" style="color:rgb(221,17,68)">"testFoo"</span>, testFoo)
]
}
<span class="hljs-func"><span class="hljs-keyword" style="color:rgb(51,51,51);font-weight:bold">func</span> <span class="hljs-title" style="color:rgb(153,0,0);font-weight:bold">testFoo</span><span class="hljs-params">()</span> </span>{
<span class="hljs-type" style="color:rgb(68,85,136);font-weight:bold">XCTAssert</span>(<span class="hljs-built_in" style="color:rgb(0,134,179)">true</span>)
}
<span class="hljs-comment" style="color:rgb(153,153,136);font-style:italic">// The user forgot to list this method in `allTests`, so it is never run.</span>
<span class="hljs-func"><span class="hljs-keyword" style="color:rgb(51,51,51);font-weight:bold">func</span> <span class="hljs-title" style="color:rgb(153,0,0);font-weight:bold">testBar</span><span class="hljs-params">()</span> </span>{
<span class="hljs-type" style="color:rgb(68,85,136);font-weight:bold">XCTAssert</span>(<span class="hljs-built_in" style="color:rgb(0,134,179)">true</span>)
}
}
</code></pre>
<p style="margin:0px 0px 1.2em!important">This is tedious and error-prone. We need to do better by Swift 3!</p>
<p style="margin:0px 0px 1.2em!important">Apple XCTest uses Objective-C reflection at runtime to compile a list of <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">NSInvocation</code> to execute as tests. We can’t use the same approach in Swift: as far as I know, there’s no reflection API that allows us to find instance methods defined on a class, and adding such an API is considered a <a href="https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001682.html">stretch goal for Swift 3</a>.</p>
<p style="margin:0px 0px 1.2em!important">Several commenters in <a href="https://bugs.swift.org/browse/SR-710">SR-710</a> have suggested using libIDE or SourceKit to determine the list of tests. I believe this is the most feasible approach that could be implemented in time for Swift 3. A few caveats:</p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">SourceKit is coupled to XPC, which only works on OS X.</li>
<li style="margin:0.5em 0px">Logic like <a href="https://github.com/apple/swift/blob/12593eff135ab2fa99529e9fa8ecc61ce268cd45/tools/SourceKit/include/SourceKit/Core/LangSupport.h#L54"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">SourceKit::FuncDeclEntityInfo.IsTestCandidate</code></a> exists in SourceKit, but not libIDE.</li>
<li style="margin:0.5em 0px">libIDE defines a C++ interface, so we could not use it from Swift (should we choose to use Swift to generate the list of test methods).</li>
</ul>
<p style="margin:0px 0px 1.2em!important">I think we have three concrete options for implementing <a href="https://bugs.swift.org/browse/SR-710">SR-710</a>:</p>
<ol style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">Port SourceKit to Linux, using a different form of IPC since XPC is not available.</li>
<li style="margin:0.5em 0px">Move business logic like <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">IsTestCandidate</code> to libIDE. If the tool to generate the list of tests will be written in Swift, we can add a C header (like libclang or sourcekitd). The tool would link against libIDE and use the C header.</li>
<li style="margin:0.5em 0px">Move business logic like <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">IsTestCandidate</code> to libIDE and add a <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swiftc</code> option to interface with its functionality. The tool that generates the list of tests would invoke <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">swiftc</code> to get the list.</li>
</ol>
<p style="margin:0px 0px 1.2em!important">I think #2 is the best option. It’s less work than both #1 and #3. I believe logic like <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">IsTestCandidate</code> belongs in libIDE anyway—SourceKit should stick to XPC and asynchronous communication with libIDE.</p>
<p style="margin:0px 0px 1.2em!important">Not being an expert in many of these components, I have several questions:</p>
<ul style="margin:1.2em 0px;padding-left:2em">
<li style="margin:0.5em 0px">I’m assuming the reflection API to return a list of instance methods on a <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">XCTestCase</code> subclass is not ready yet, and won’t be for some time. Is this accurate?</li>
<li style="margin:0.5em 0px">I’m assuming that SourceKit is intended to be an asynchronous wrapper over libIDE, and that logic like <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)">IsTestCandidate</code> should be moved to libIDE. Is this accurate?</li>
<li style="margin:0.5em 0px">I’m assuming that SourceKit is coupled with XPC, and that it would be more work to port it to Linux than it would be to move its logic to libIDE. Is this accurate?</li>
</ul>
<p style="margin:0px 0px 1.2em!important">If you have thoughts/feedback, please reply to this email or comment on <a href="https://bugs.swift.org/browse/SR-710">SR-710</a>. Your input would be greatly appreciated!!</p>
<p style="margin:0px 0px 1.2em!important">Brian Gesiak</p>
<div title="MDH:SGVsbG8gYWxsITxkaXY+PGJyPltTUi03MTBdKDxhIGhyZWY9Imh0dHBzOi8vYnVncy5zd2lmdC5v
cmcvYnJvd3NlL1NSLTcxMCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vYnVncy5zd2lmdC5vcmcv
YnJvd3NlLzx3YnI+U1ItNzEwPC9hPikgdHJhY2tzIGEgbWFqb3IgZ29hbCBmb3IgU3dpZnQgMzog
aGF2aW5nIFN3aWZ0UE0vY29yZWxpYnMtPHdicj54Y3Rlc3QgYXV0b21hdGljYWxseSBnZW5lcmF0
ZSBhIGxpc3Qgb2YgdGVzdCBtZXRob2RzIHRvIGV4ZWN1dGUuIFRoZSBpbXBsZW1lbnRhdGlvbnMg
d2UncmUgY29uc2lkZXJpbmcgc3BhbiB2YXJpb3VzIHBhcnRzIG9mIHRoZSBjb2RlYmFzZTogbGli
SURFLCBTb3VyY2VLaXQsIHRoZSByZWZsZWN0aW9uIEFQSXMsIGV0Yy4gV2UgbmVlZCBpbnB1dCBm
cm9tIHBlb3BsZSBmYW1pbGlhciB3aXRoIHRoZXNlIGNvbXBvbmVudHMuPGJyPjxicj5IZXJlJ3Mg
dGhlIGlzc3VlOiBjdXJyZW50bHksIHVzZXJzIG9mIGNvcmVsaWJzLXhjdGVzdCBtdXN0IG1hbnVh
bGx5IGxpc3QgZWFjaCB0ZXN0IHRoZXkgd2lzaCB0byBleGVjdXRlOjxicj48YnI+YGBgc3dpZnQ8
YnI+PGRpdj5jbGFzcyBNeVRlc3RDYXNlOiBYQ1Rlc3RDYXNlIHs8L2Rpdj48ZGl2PiZuYnNwOyAm
bmJzcDsgc3RhdGljIHZhciBhbGxUZXN0czogWyhTdHJpbmcsIE15VGVzdENhc2UgLSZndDsgKCkg
dGhyb3dzIC0mZ3Q7IFZvaWQpXSB7PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgcmV0dXJuIFs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICgidGVzdEZvbyIsIHRlc3RGb28pPC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDsgXTwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyB9PC9kaXY+PGRpdj48YnI+PC9kaXY+
PGRpdj4mbmJzcDsgJm5ic3A7IGZ1bmMgdGVzdEZvbygpIHs8L2Rpdj48ZGl2PiZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyBYQ1RBc3NlcnQodHJ1ZSk8L2Rpdj48ZGl2PiZuYnNwOyAmbmJzcDsg
fTwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+Jm5ic3A7ICZuYnNwOyAvLyBUaGUgdXNlciBmb3Jn
b3QgdG8gbGlzdCB0aGlzIG1ldGhvZCBpbiBgYWxsVGVzdHNgLCBzbyBpdCBpcyBuZXZlciBydW4u
PC9kaXY+PGRpdj4mbmJzcDsgJm5ic3A7IGZ1bmMgdGVzdEJhcigpIHs8YnI+Jm5ic3A7ICZuYnNw
OyAmbmJzcDsgJm5ic3A7IFhDVEFzc2VydCh0cnVlKTxicj4mbmJzcDsgJm5ic3A7IH08L2Rpdj48
ZGl2Pn08L2Rpdj5gYGA8ZGl2Pjxicj48L2Rpdj48ZGl2PlRoaXMgaXMgdGVkaW91cyBhbmQgZXJy
b3ItcHJvbmUuIFdlIG5lZWQgdG8gZG8gYmV0dGVyIGJ5IFN3aWZ0IDMhPGJyPjxicj5BcHBsZSBY
Q1Rlc3QgdXNlcyBPYmplY3RpdmUtQyByZWZsZWN0aW9uIGF0IHJ1bnRpbWUgdG8gY29tcGlsZSBh
IGxpc3Qgb2YgYE5TSW52b2NhdGlvbmAgdG8gZXhlY3V0ZSBhcyB0ZXN0cy4gV2UgY2FuJ3QgdXNl
IHRoZSBzYW1lIGFwcHJvYWNoIGluIFN3aWZ0OiBhcyBmYXIgYXMgSSBrbm93LCB0aGVyZSdzIG5v
IHJlZmxlY3Rpb24gQVBJIHRoYXQgYWxsb3dzIHVzIHRvIGZpbmQgaW5zdGFuY2UgbWV0aG9kcyBk
ZWZpbmVkIG9uIGEgY2xhc3MsIGFuZCBhZGRpbmcgc3VjaCBhbiBBUEkgaXMgY29uc2lkZXJlZCBh
IFtzdHJldGNoIGdvYWwgZm9yIFN3aWZ0IDNdKDxhIGhyZWY9Imh0dHBzOi8vbGlzdHMuc3dpZnQu
b3JnL3BpcGVybWFpbC9zd2lmdC1ldm9sdXRpb24vV2Vlay1vZi1Nb24tMjAxNTEyMDcvMDAxNjgy
Lmh0bWwiIHRhcmdldD0iX2JsYW5rIj5odHRwczovL2xpc3RzLnN3aWZ0Lm9yZy88d2JyPnBpcGVy
bWFpbC9zd2lmdC1ldm9sdXRpb24vPHdicj5XZWVrLW9mLU1vbi0yMDE1MTIwNy8wMDE2ODIuPHdi
cj5odG1sPC9hPikuPGJyPjxicj5TZXZlcmFsIGNvbW1lbnRlcnMgaW4gW1NSLTcxMF0oPGEgaHJl
Zj0iaHR0cHM6Ly9idWdzLnN3aWZ0Lm9yZy9icm93c2UvU1ItNzEwIiB0YXJnZXQ9Il9ibGFuayI+
aHR0cHM6Ly9idWdzLnN3aWZ0Lm9yZy9icm93c2UvPHdicj5TUi03MTA8L2E+KSBoYXZlIHN1Z2dl
c3RlZCB1c2luZyBsaWJJREUgb3IgU291cmNlS2l0IHRvIGRldGVybWluZSB0aGUgbGlzdCBvZiB0
ZXN0cy4gSSBiZWxpZXZlIHRoaXMgaXMgdGhlIG1vc3QgZmVhc2libGUgYXBwcm9hY2ggdGhhdCBj
b3VsZCBiZSBpbXBsZW1lbnRlZCBpbiB0aW1lIGZvciBTd2lmdCAzLiBBIGZldyBjYXZlYXRzOjxi
cj48YnI+LSBTb3VyY2VLaXQgaXMgY291cGxlZCB0byBYUEMsIHdoaWNoIG9ubHkgd29ya3Mgb24g
T1MgWC48YnI+LSBMb2dpYyBsaWtlIFtgU291cmNlS2l0OjpGdW5jRGVjbEVudGl0eUluZm8uSXNU
ZXN0Q2FuZGlkYXRlYF0oaHR0cHM6Ly9naXRodWIuY29tL2FwcGxlL3N3aWZ0L2Jsb2IvMTI1OTNl
ZmYxMzVhYjJmYTk5NTI5ZTlmYThlY2M2MWNlMjY4Y2Q0NS90b29scy9Tb3VyY2VLaXQvaW5jbHVk
ZS9Tb3VyY2VLaXQvQ29yZS9MYW5nU3VwcG9ydC5oI0w1NCkgZXhpc3RzIGluIFNvdXJjZUtpdCwg
YnV0IG5vdCBsaWJJREUuPGJyPi0gbGliSURFIGRlZmluZXMgYSBDKysgaW50ZXJmYWNlLCBzbyB3
ZSBjb3VsZCBub3QgdXNlIGl0IGZyb20gU3dpZnQgKHNob3VsZCB3ZSBjaG9vc2UgdG8gdXNlIFN3
aWZ0IHRvIGdlbmVyYXRlIHRoZSBsaXN0IG9mIHRlc3QgbWV0aG9kcykuPGJyPjxicj5JIHRoaW5r
IHdlIGhhdmUgdGhyZWUgY29uY3JldGUgb3B0aW9ucyBmb3IgaW1wbGVtZW50aW5nJm5ic3A7W1NS
LTcxMF0oPGEgaHJlZj0iaHR0cHM6Ly9idWdzLnN3aWZ0Lm9yZy9icm93c2UvU1ItNzEwIiB0YXJn
ZXQ9Il9ibGFuayI+aHR0cHM6Ly9idWdzLnN3aWZ0Lm9yZy9icm93c2UvPHdicj5TUi03MTA8L2E+
KTo8YnI+PGJyPjxkaXY+MS4gUG9ydCBTb3VyY2VLaXQgdG8gTGludXgsIHVzaW5nIGEgZGlmZmVy
ZW50IGZvcm0gb2YgSVBDIHNpbmNlIFhQQyBpcyBub3QgYXZhaWxhYmxlLjwvZGl2PjxkaXY+Mi4g
TW92ZSBidXNpbmVzcyBsb2dpYyBsaWtlIGBJc1Rlc3RDYW5kaWRhdGVgIHRvIGxpYklERS4gSWYg
dGhlIHRvb2wgdG8gZ2VuZXJhdGUgdGhlIGxpc3Qgb2YgdGVzdHMgd2lsbCBiZSB3cml0dGVuIGlu
IFN3aWZ0LCB3ZSBjYW4gYWRkIGEgQyBoZWFkZXIgKGxpa2UgbGliY2xhbmcgb3Igc291cmNla2l0
ZCkuIFRoZSB0b29sIHdvdWxkIGxpbmsgYWdhaW5zdCBsaWJJREUgYW5kIHVzZSB0aGUgQyBoZWFk
ZXIuPC9kaXY+PGRpdj4zLiBNb3ZlIGJ1c2luZXNzIGxvZ2ljIGxpa2UgYElzVGVzdENhbmRpZGF0
ZWAgdG8gbGliSURFIGFuZCBhZGQgYSBgc3dpZnRjYCBvcHRpb24gdG8gaW50ZXJmYWNlIHdpdGgg
aXRzIGZ1bmN0aW9uYWxpdHkuIFRoZSB0b29sIHRoYXQgZ2VuZXJhdGVzIHRoZSBsaXN0IG9mIHRl
c3RzIHdvdWxkIGludm9rZSBgc3dpZnRjYCB0byBnZXQgdGhlIGxpc3QuPC9kaXY+PGRpdj48YnI+
PC9kaXY+PGRpdj5JIHRoaW5rICMyIGlzIHRoZSBiZXN0IG9wdGlvbi4gSXQncyBsZXNzIHdvcmsg
dGhhbiBib3RoICMxIGFuZCAjMy4gSSBiZWxpZXZlIGxvZ2ljIGxpa2UgYElzVGVzdENhbmRpZGF0
ZWAgYmVsb25ncyBpbiBsaWJJREUgYW55d2F5LS1Tb3VyY2VLaXQgc2hvdWxkIHN0aWNrIHRvIFhQ
QyBhbmQgYXN5bmNocm9ub3VzIGNvbW11bmljYXRpb24gd2l0aCBsaWJJREUuPGJyPjxicj48L2Rp
dj48ZGl2Pk5vdCBiZWluZyBhbiBleHBlcnQgaW4gbWFueSBvZiB0aGVzZSBjb21wb25lbnRzLCBJ
IGhhdmUgc2V2ZXJhbCBxdWVzdGlvbnM6PGJyPjxicj4tIEknbSBhc3N1bWluZyB0aGUgcmVmbGVj
dGlvbiBBUEkgdG8gcmV0dXJuIGEgbGlzdCBvZiBpbnN0YW5jZSBtZXRob2RzIG9uIGEgYFhDVGVz
dENhc2VgIHN1YmNsYXNzIGlzIG5vdCByZWFkeSB5ZXQsIGFuZCB3b24ndCBiZSBmb3Igc29tZSB0
aW1lLiBJcyB0aGlzIGFjY3VyYXRlPzxicj4tIEknbSBhc3N1bWluZyB0aGF0IFNvdXJjZUtpdCBp
cyBpbnRlbmRlZCB0byBiZSBhbiBhc3luY2hyb25vdXMgd3JhcHBlciBvdmVyIGxpYklERSwgYW5k
IHRoYXQgbG9naWMgbGlrZSBgSXNUZXN0Q2FuZGlkYXRlYCBzaG91bGQgYmUgbW92ZWQgdG8gbGli
SURFLiBJcyB0aGlzIGFjY3VyYXRlPzxicj4tIEknbSBhc3N1bWluZyB0aGF0IFNvdXJjZUtpdCBp
cyBjb3VwbGVkIHdpdGggWFBDLCBhbmQgdGhhdCBpdCB3b3VsZCBiZSBtb3JlIHdvcmsgdG8gcG9y
dCBpdCB0byBMaW51eCB0aGFuIGl0IHdvdWxkIGJlIHRvIG1vdmUgaXRzIGxvZ2ljIHRvIGxpYklE
RS4gSXMgdGhpcyBhY2N1cmF0ZT88L2Rpdj48L2Rpdj48L2Rpdj48ZGl2Pjxicj5JZiB5b3UgaGF2
ZSB0aG91Z2h0cy9mZWVkYmFjaywgcGxlYXNlIHJlcGx5IHRvIHRoaXMgZW1haWwgb3IgY29tbWVu
dCBvbiZuYnNwO1tTUi03MTBdKDxhIGhyZWY9Imh0dHBzOi8vYnVncy5zd2lmdC5vcmcvYnJvd3Nl
L1NSLTcxMCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vYnVncy5zd2lmdC5vcmcvYnJvd3NlLzx3
YnI+U1ItNzEwPC9hPikuIFlvdXIgaW5wdXQgd291bGQgYmUgZ3JlYXRseSBhcHByZWNpYXRlZCEh
PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5CcmlhbiBHZXNpYWs8YnI+PGJyPjwvZGl2Pg==" style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div></div>