<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Sorry for the delay in following up.</div><div class=""><br class=""></div><div class="">I have had several long discussions on this topic with Dmitri and others, which</div><div class="">I will try to summarize here:</div><div class=""><br class=""></div><div class="">*TL;DR*: We think the right long-term path forward is to pursue porting</div><div class="">SourceKit to Linux, and would like to explore that direction first before trying</div><div class="">to develop a compromise which either (a) uses internal and likely-to-break APIs</div><div class="">(e.g., dump-ast) or (b) devotes significant engineering resources to a "right</div><div class="">solution" for this problem, but which will only solve this problem and not other</div><div class="">issues where having an API for use with the Swift language.</div><div class=""><br class=""></div><div class="">## Problem Statement</div><div class=""><br class=""></div><div class="">The XCTest API has historically been defined in terms of methods with a</div><div class="">particular naming convention `test...` which were in subclasses of XCTestCase.</div><div class=""><br class=""></div><div class="">On OS X, these methods can be found via the Objective-C runtime but that does</div><div class="">not work on Linux. Our current solution on Linux requires manual specification</div><div class="">of all of the test methods, and is a huge maintenance burden for people trying</div><div class="">to use XCTest on Linux or maintain cross-platform projects.</div><div class=""><br class=""></div><div class="">## Background</div><div class=""><br class=""></div><div class="">This feature works on OS X in two ways:</div><div class=""><br class=""></div><div class="">1. As mentioned, tests are run by dynamic discovery through the Objective-C</div><div class=""> runtime when the bundle containing the tests is executed.</div><div class=""> </div><div class="">2. Within the Xcode IDE, tests are "discovered" (for use in UI) through the use</div><div class=""> of the Xcode indexer.</div><div class=""> </div><div class=""> The mechanism at work here, for Swift, is a combination of functionality in</div><div class=""> the indexing engine (to aggregate and query results) and the raw underlying</div><div class=""> data provided by SourceKit.</div><div class=""> </div><div class=""> We would like any implementation of this feature to share as much code as</div><div class=""> possible with SourceKit and Xcode's implementation in order for</div><div class=""> cross-platform projects to behave predictably.</div><div class=""> </div><div class="">It also happens that SwiftPM has several other needs for API-based interactions</div><div class="">with the functionality in the Swift compiler. Several examples:</div><div class=""> </div><div class="">* We would like to be able to enforce the strictness of the `Package.swift`</div><div class=""> manifest file format. This requires APIs to interact with the Swift</div><div class=""> AST. <a href="https://bugs.swift.org/browse/SR-1433" class="">https://bugs.swift.org/browse/SR-1433</a></div><div class=""> </div><div class="">* We would like to support advanced features for dealing with automatic</div><div class=""> inter-module dependencies. For example, one idea which has been proposed is</div><div class=""> that if a module attempted to import a module upon which it did not have a</div><div class=""> dependency, that we would prompt the developer if this dependency should</div><div class=""> automatically be added to the manifest. Doing this feature well requires</div><div class=""> APIs to interact with the Swift compiler as it parses source code and</div><div class=""> reports diagnostics.</div><div class=""> </div><div class="">* We would like to understand the current level of parallelism being used by</div><div class=""> the Swift compiler, so that llbuild can accurately schedule work. This</div><div class=""> requires APIs for interacting with an in-flight compilation process.</div><div class=""><br class=""></div><div class="">## Discussion</div><div class=""><br class=""></div><div class="">We discussed several avenues of attack on this problem. I will go through them</div><div class="">one by one. I am just attempting to summarize the conversation, obviously there</div><div class="">is a ton of nuance to each point, and hopefully other people will chime in if I</div><div class="">missed something important.</div><div class=""><br class=""></div><div class="">### Language Features</div><div class=""><br class=""></div><div class="">One way to view this problem is that XCTest's API (i.e., depending on test</div><div class="">naming and dynamic discovery) is a poor fit for Swift today, and that this</div><div class="">problem should be tackled in that direction.</div><div class=""><br class=""></div><div class="">For example, the Swift compiler itself has a test framework that does not depend</div><div class="">on dynamic discovery. One could also imagine language-level features which would</div><div class="">solve the arbitrary problem of wanting to find discoverable things; that could</div><div class="">take the form of an `@XCTest` attribute, or "generalized attribute" support.</div><div class=""><br class=""></div><div class="">The current mission for Swift 3, however, is API parity between Linux and OS X,</div><div class="">and so this direction does not lead to any short term solution. For that reason,</div><div class="">while many of us ultimately think this is the right long term direction, we need</div><div class="">to find another one as well.</div><div class=""><br class=""></div><div class="">### Custom "Supported" XCTest Feature</div><div class=""><br class=""></div><div class="">The next option is to pursue a custom, but "supported" feature intended to</div><div class="">tackle this problem head on. Some proposals that have come up are, e.g., a new</div><div class="">compiler flag which emits the list of test methods.</div><div class=""><br class=""></div><div class="">This approach has a couple unfortunate properties:</div><div class=""><br class=""></div><div class="">1. It is non-trivial. We can either design this as an incredibly XCTest specific</div><div class=""> feature requiring understanding of the backend (compiler directly emits</div><div class=""> metadata into .o file), or a midway feature (compiler tells us list of test</div><div class=""> methods, but then we have to manage incremental compilation and the desire to</div><div class=""> not compile things multiple times more than necessary ourselves).</div><div class=""> </div><div class="">2. It is not-reusable. The work we do here doesn't help with any of the other</div><div class=""> ways we want to use the Swift compiler as an API.</div><div class=""> </div><div class="">### Implement an API-based Interface</div><div class=""><br class=""></div><div class="">This approach means porting SourceKit to Linux, and then building this feature</div><div class="">on top of that (that itself will be non-trivial, which is something not to be</div><div class="">glossed over).</div><div class=""><br class=""></div><div class="">Everyone generally agrees we should have some kind of API-based access to the</div><div class="">compiler, so this is in line with our long-term direction, but in a way which is</div><div class="">actionable now (it does not require design).</div><div class=""><br class=""></div><div class="">This approach has its own issues:</div><div class=""><br class=""></div><div class="">1. Porting SourceKit may require a fair amount of work. The current code base is</div><div class=""> very OS X specific in particular areas (generally around the use of</div><div class=""> `libxpc`). This will require adding abstractions, developing alternatives,</div><div class=""> and writing the code and build system changes to get this to happen.</div><div class=""> </div><div class="">2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in</div><div class=""> a way that it can be used from SourceKit. If we do port and use SourceKit, it</div><div class=""> will mean the toolchains increase in size because we effectively will have</div><div class=""> two copies of many parts of Swift/Clang/LLVM installed.</div><div class=""> </div><div class="">3. By itself, this does not fully solve the issue. As mentioned in the</div><div class=""> background, SourceKit itself does not directly manage all parts of XCTest</div><div class=""> discovery today on OS X, it is a collaboration between SourceKit and the</div><div class=""> Xcode indexer to do discovery without execution. The expectation, however, is</div><div class=""> that we can, however, use SourceKit's APIs to implement this feature in a way</div><div class=""> that is "supported".</div><div class=""><br class=""></div><div class="">## Conclusion</div><div class=""><br class=""></div><div class="">The conclusion was that after weighing all of the tradeoffs, it made the most</div><div class="">sense to encourage porting of SourceKit to Linux and then using it to build out</div><div class="">the Linux test discovery feature. This was most in line with a desirable</div><div class="">long-term direction without being blocked on language design.</div><div class=""><br class=""></div><div class="">I don't think we are particularly firm in this conclusion. If you feel that the work</div><div class="">you have already invested gets us so close that it is worth prioritizing that</div><div class="">approach, then please push back.</div><div class=""><br class=""></div><div class=""> - Daniel</div><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On May 1, 2016, at 8:19 PM, Brian Gesiak <<a href="mailto:modocache@gmail.com" class="">modocache@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">I made an attempt at adding `swiftc -frontend -dump-xctest-methods`: <a href="https://github.com/apple/swift/pull/2364" class="">https://github.com/apple/swift/pull/2364</a><br class=""><br class="">Feedback appreciated!!<div class=""><br class=""></div><div class="">- Brian Gesiak</div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Sun, Apr 24, 2016 at 8:59 PM, Brian Gesiak <span dir="ltr" class=""><<a href="mailto:modocache@gmail.com" target="_blank" class="">modocache@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class=""><p style="margin:0px 0px 1.2em!important" class="">Thanks to <a href="https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6" target="_blank" class="">this commit</a> it looks as if <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)" class="">IsTestCandidate</code> has been moved out of SourceKit and into libIndex:</p>
<ul style="margin:1.2em 0px;padding-left:2em" class="">
<li style="margin:0.5em 0px" class=""><a href="https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754" target="_blank" class=""><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)" class="">isTestCandidate(swift::ValueDecl)</code></a></li>
<li style="margin:0.5em 0px" class=""><a href="https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102" target="_blank" class=""><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)" class="">swift::index::FuncDeclIndexSymbol.IsTestCandidate</code></a> and <a href="https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786" target="_blank" class="">where it’s set</a>.</li>
</ul><p style="margin:0px 0px 1.2em!important" class="">I’m looking into adding an option to swiftc to emit XCTest candidate methods. How does <code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);border-radius:3px;display:inline;background-color:rgb(248,248,248)" class="">swiftc -frontend -dump-xctest-methods</code> sound?</p><span class="HOEnZb"><font color="#888888" class=""><p style="margin:0px 0px 1.2em!important" class="">- Brian Gesiak</p>
<div title="MDH:PGRpdj5UaGFua3MgdG8gW3RoaXMgY29tbWl0XShodHRwczovL2dpdGh1Yi5jb20vYXBwbGUvc3dp
ZnQvY29tbWl0L2FkMjY5YjBlMWZiYzEyMDM3YWUyYzE2NjM0YjViNDUxMDYxNjU3YzYpIGl0IGxv
b2tzIGFzIGlmIGBJc1Rlc3RDYW5kaWRhdGVgIGhhcyBiZWVuIG1vdmVkIG91dCBvZiBTb3VyY2VL
aXQgYW5kIGludG8gbGliSW5kZXg6PGJyPjxicj4tIFtgaXNUZXN0Q2FuZGlkYXRlKHN3aWZ0OjpW
YWx1ZURlY2wpYF0oaHR0cHM6Ly9naXRodWIuY29tL2FwcGxlL3N3aWZ0L2Jsb2IvOGRhZDdmNzgw
MzQ3Nzg4ZjYwMzJhZDllMjVjZTUzNDBhZWNmNDA3My9saWIvSW5kZXgvSW5kZXguY3BwI0w3NTQp
PGJyPi0gW2Bzd2lmdDo6aW5kZXg6OkZ1bmNEZWNsSW5kZXhTeW1ib2wuSXNUZXN0Q2FuZGlkYXRl
YF0oaHR0cHM6Ly9naXRodWIuY29tL2FwcGxlL3N3aWZ0L2Jsb2IvNDFlNGU5YjZlZmM3NDVmMDRk
ZjIzYmQ2YTgwM2E0NjdjNTdhNjZiOC9pbmNsdWRlL3N3aWZ0L0luZGV4L0luZGV4U3ltYm9sLmgj
TDEwMikgYW5kIFt3aGVyZSBpdCdzIHNldF0oaHR0cHM6Ly9naXRodWIuY29tL2FwcGxlL3N3aWZ0
L2Jsb2IvOGRhZDdmNzgwMzQ3Nzg4ZjYwMzJhZDllMjVjZTUzNDBhZWNmNDA3My9saWIvSW5kZXgv
SW5kZXguY3BwI0w3ODYpLjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkknbSBsb29raW5n
IGludG8gYWRkaW5nIGFuIG9wdGlvbiB0byBzd2lmdGMgdG8gZW1pdCBYQ1Rlc3QgY2FuZGlkYXRl
IG1ldGhvZHMuIEhvdyBkb2VzIGBzd2lmdGMgLWZyb250ZW5kIC1kdW1wLXhjdGVzdC1tZXRob2Rz
YCBzb3VuZD88L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pi0gQnJpYW4gR2VzaWFrPC9kaXY+PGRp
dj48YnI+PC9kaXY+" style="min-height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0" class=""></div></font></span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br class=""><div class="gmail_quote">On Sun, Apr 17, 2016 at 5:50 PM, Drew Crawford <span dir="ltr" class=""><<a href="mailto:drew@sealedabstract.com" target="_blank" class="">drew@sealedabstract.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><span class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko <<a href="mailto:gribozavr@gmail.com" target="_blank" class="">gribozavr@gmail.com</a>> wrote:</div><br class=""><div class=""><div class=""><blockquote type="cite" style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">Hmm... but then wouldn't that more tightly couple the test discovery tool<br class="">and the Swift compiler? In an earlier email you said you "like #3 better<br class="">[...] because that would decouple the test discovery tool from the Swift<br class="">compiler." I think that part is confusing me.<br class=""></blockquote><br style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><span style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="">Sorry -- what I meant is that the compiler remains the point of truth</span><br style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><span style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="">about the language and can find the tests. The tools that actually</span><br style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><span style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="">generate glue code won't need to parse code, and would be decoupled</span><br style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><span style="font-family:HelveticaNeue;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important" class="">from the compiler.</span></div></div></blockquote></div><br class=""></span><div class="">I am internally shipping a test framework that discovers tests via an out-of-tree parser. Teaching swiftc about XCTest syntax is not sufficient to deprecate my parser, and therefore is not sufficient for the compiler to be the source of truth for my tests.</div></div></blockquote></div><br class=""></div>
</div></div></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></body></html>