[swift-dev] [SR-710][RFC] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

[Daniel Dunbar]

> On Apr 3, 2016, at 3:36 PM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
> 
> On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache at gmail.com> wrote:
>> I think #2 is the best option. It’s less work than both #1 and #3. I believe
>> logic like IsTestCandidate belongs in libIDE anyway—SourceKit should stick
>> to XPC and asynchronous communication with libIDE.
> 
> I like #3 better (an option to swiftc), because that would decouple
> the test discovery tool from the Swift compiler.  That would allow you
> to use the discovery tool with different compilers.  And, because we
> would avoid statically linking libIDE, it would mean one less copy of
> LLVM, Clang and Swift in the toolchain.

Ultimately my opinion is that it is likely that the package manager will want an API interface to Swift in any case. I personally would rather we simply plan on that.

I also would like to avoid duplicating anything in the toolchain, but think that should be done by moving the driver to sitting on top of a shared library.

>> Not being an expert in many of these components, I have several questions:
>> 
>> I’m assuming the reflection API to return a list of instance methods on a
>> XCTestCase subclass is not ready yet, and won’t be for some time. Is this
>> accurate?
> 
> I think so.
> 
>> I’m assuming that SourceKit is intended to be an asynchronous wrapper over
>> libIDE, and that logic like IsTestCandidate should be moved to libIDE. Is
>> this accurate?
> 
> SourceKit has a lot of functionality of its own, but moving this
> particular piece of logic to libIDE sounds reasonable.
> 
>> I’m assuming that SourceKit is coupled with XPC, and that it would be more
>> work to port it to Linux than it would be to move its logic to libIDE. Is
>> this accurate?
> 
> It is not tightly coupled with XPC, there is a portability layer that
> you could implement for Linux.  You would need to decide on an IPC
> mechanism and serialization format though.

For our purposes, I don't think we need IPC. I think a direct (C) library interface would be fine. Clients can implement the IPC/XPC if they need it.

>> If you have thoughts/feedback, please reply to this email or comment on
>> SR-710. Your input would be greatly appreciated!!
> 
> I'm wondering how feasible is it to change the XCTest API to
> accommodate better the Swift language that we have, rather than trying
> to add custom tooling to make the existing API work.  Adding magic
> tooling that adds behavior not present in the language seems unnatural
> to me.

I agree with you that it is unnatural, but I think this ship has sailed for XCTest, we have a need to support the existing API in a cross platform manner. 

My personal preference is that eventually we would build features like this on top of general support for attributes (a la Java/Python/C#).

> Compare with StdlibUnittest -- by using an API to build tests we get
> the following advantages:
> 
> - We completely avoid having the issue of test discovery, executing
> the code discovers the tests.  No reflection needed!

While I think StdlibUnittest is neat, I also believe that there are very good reasons for supporting test discovery in a test suite. I have used these features in other suites to great avail to create (bidirectional, sometimes) lit bridge adaptors between various test frameworks (Python unittest, googletest, XCTest).

In an IDE context, it is also very useful to be able to perform test discovery independent of test execution.

As one other example, I've used lit with suites with hundreds of thousands of tests... it would be unfortunate to have to dynamically discover all of those tests when the user is just trying to run a single one.

> - We can add attributes to tests (for example, skip, xfail).  In the
> current XCTest API this would require adding some kind of user-defined
> attributes, which is another language which is a long way from being
> designed and implemented.

This isn't necessarily the case, XCTest could in theory provide explicit APIs to do these things as part of test execution. I agree attributes are a better fit in the current model.

> - We can define data-parameterized tests.

> - Tests can be dynamically synthesized by control flow.  In the
> current XCTest API, dynamically generating tests would mean
> dynamically generating methods, which is even more far off than
> read-only method reflection.

FWIW, XCTest has some support for doing these kinds of things, they just don't take the form of the pattern-matched methods.

 - Daniel

> 
> Dmitri
> 
> -- 
> main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
> (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/