[swift-dev] [SR-710][RFC] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

[Dmitri Gribenko]

On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache at gmail.com> wrote:
> I think #2 is the best option. It’s less work than both #1 and #3. I believe
> logic like IsTestCandidate belongs in libIDE anyway—SourceKit should stick
> to XPC and asynchronous communication with libIDE.

I like #3 better (an option to swiftc), because that would decouple
the test discovery tool from the Swift compiler.  That would allow you
to use the discovery tool with different compilers.  And, because we
would avoid statically linking libIDE, it would mean one less copy of
LLVM, Clang and Swift in the toolchain.

> Not being an expert in many of these components, I have several questions:
>
> I’m assuming the reflection API to return a list of instance methods on a
> XCTestCase subclass is not ready yet, and won’t be for some time. Is this
> accurate?

I think so.

> I’m assuming that SourceKit is intended to be an asynchronous wrapper over
> libIDE, and that logic like IsTestCandidate should be moved to libIDE. Is
> this accurate?

SourceKit has a lot of functionality of its own, but moving this
particular piece of logic to libIDE sounds reasonable.

> I’m assuming that SourceKit is coupled with XPC, and that it would be more
> work to port it to Linux than it would be to move its logic to libIDE. Is
> this accurate?

It is not tightly coupled with XPC, there is a portability layer that
you could implement for Linux.  You would need to decide on an IPC
mechanism and serialization format though.

> If you have thoughts/feedback, please reply to this email or comment on
> SR-710. Your input would be greatly appreciated!!

I'm wondering how feasible is it to change the XCTest API to
accommodate better the Swift language that we have, rather than trying
to add custom tooling to make the existing API work.  Adding magic
tooling that adds behavior not present in the language seems unnatural
to me.

Compare with StdlibUnittest -- by using an API to build tests we get
the following advantages:

- We completely avoid having the issue of test discovery, executing
the code discovers the tests.  No reflection needed!

- We can add attributes to tests (for example, skip, xfail).  In the
current XCTest API this would require adding some kind of user-defined
attributes, which is another language which is a long way from being
designed and implemented.

- We can define data-parameterized tests.

- Tests can be dynamically synthesized by control flow.  In the
current XCTest API, dynamically generating tests would mean
dynamically generating methods, which is even more far off than
read-only method reflection.

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/