[swift-dev] [SR-710][RFC] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

Sun Apr 17 13:36:08 CDT 2016

In case anyone interested missed it, John Holdsworth mentions another
approach in a comment on https://bugs.swift.org/browse/SR-710: using
runtime metadata to find a list of tests. The linked project contains an
example. It:

1. Specifies a test method regex:
https://github.com/johnno1962/TestRunner/blob/201bd04af14fdebf1a7fa2ae0a419fc59d6266ed/TestRunner/AppDelegate.swift#L69
2. In Swift, the project exposes metadata information for Swift objects:
https://github.com/johnno1962/TestRunner/blob/201bd04af14fdebf1a7fa2ae0a419fc59d6266ed/TestRunner/TestRunner.swift#L52
3. Uses an UnsafeMutablePointer<ClassMetadataSwift> to get a reference to
the methods defined on an instance:
https://github.com/johnno1962/TestRunner/blob/201bd04af14fdebf1a7fa2ae0a419fc59d6266ed/TestRunner/TestRunner.swift#L97

I assume this approach won't be acceptable to some because it relies on
private information about Swift objects, but I don't know for sure. Please
chime in if this approach seems reasonable to you!

Based on the assumption that we can't use metadata, however, I'm going to
try to begin working on approaches #2 or #3, which I mentioned in my
original email. I think the preliminary work is the same (like
moving SourceKit::FuncDeclEntityInfo.IsTestCandidate to libIDE), so I'll
hold off on making a hard decision between #2 and #3 until necessary.

- Brian Gesiak

On Mon, Apr 4, 2016 at 12:23 AM, Daniel Dunbar <daniel_dunbar at apple.com>
wrote:

>
> > On Apr 3, 2016, at 3:36 PM, Dmitri Gribenko <gribozavr at gmail.com> wrote:
> >
> > On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache at gmail.com>
> wrote:
> >> I think #2 is the best option. It’s less work than both #1 and #3. I
> believe
> >> logic like IsTestCandidate belongs in libIDE anyway—SourceKit should
> stick
> >> to XPC and asynchronous communication with libIDE.
> >
> > I like #3 better (an option to swiftc), because that would decouple
> > the test discovery tool from the Swift compiler.  That would allow you
> > to use the discovery tool with different compilers.  And, because we
> > would avoid statically linking libIDE, it would mean one less copy of
> > LLVM, Clang and Swift in the toolchain.
>
> Ultimately my opinion is that it is likely that the package manager will
> want an API interface to Swift in any case. I personally would rather we
> simply plan on that.
>
> I also would like to avoid duplicating anything in the toolchain, but
> think that should be done by moving the driver to sitting on top of a
> shared library.
>
> >> Not being an expert in many of these components, I have several
> questions:
> >>
> >> I’m assuming the reflection API to return a list of instance methods on
> a
> >> XCTestCase subclass is not ready yet, and won’t be for some time. Is
> this
> >> accurate?
> >
> > I think so.
> >
> >> I’m assuming that SourceKit is intended to be an asynchronous wrapper
> over
> >> libIDE, and that logic like IsTestCandidate should be moved to libIDE.
> Is
> >> this accurate?
> >
> > SourceKit has a lot of functionality of its own, but moving this
> > particular piece of logic to libIDE sounds reasonable.
> >
> >> I’m assuming that SourceKit is coupled with XPC, and that it would be
> more
> >> work to port it to Linux than it would be to move its logic to libIDE.
> Is
> >> this accurate?
> >
> > It is not tightly coupled with XPC, there is a portability layer that
> > you could implement for Linux.  You would need to decide on an IPC
> > mechanism and serialization format though.
>
> For our purposes, I don't think we need IPC. I think a direct (C) library
> interface would be fine. Clients can implement the IPC/XPC if they need it.
>
> >> If you have thoughts/feedback, please reply to this email or comment on
> >> SR-710. Your input would be greatly appreciated!!
> >
> > I'm wondering how feasible is it to change the XCTest API to
> > accommodate better the Swift language that we have, rather than trying
> > to add custom tooling to make the existing API work.  Adding magic
> > tooling that adds behavior not present in the language seems unnatural
> > to me.
>
> I agree with you that it is unnatural, but I think this ship has sailed
> for XCTest, we have a need to support the existing API in a cross platform
> manner.
>
> My personal preference is that eventually we would build features like
> this on top of general support for attributes (a la Java/Python/C#).
>
> > Compare with StdlibUnittest -- by using an API to build tests we get
> > the following advantages:
> >
> > - We completely avoid having the issue of test discovery, executing
> > the code discovers the tests.  No reflection needed!
>
> While I think StdlibUnittest is neat, I also believe that there are very
> good reasons for supporting test discovery in a test suite. I have used
> these features in other suites to great avail to create (bidirectional,
> sometimes) lit bridge adaptors between various test frameworks (Python
> unittest, googletest, XCTest).
>
> In an IDE context, it is also very useful to be able to perform test
> discovery independent of test execution.
>
> As one other example, I've used lit with suites with hundreds of thousands
> of tests... it would be unfortunate to have to dynamically discover all of
> those tests when the user is just trying to run a single one.
>
> > - We can add attributes to tests (for example, skip, xfail).  In the
> > current XCTest API this would require adding some kind of user-defined
> > attributes, which is another language which is a long way from being
> > designed and implemented.
>
> This isn't necessarily the case, XCTest could in theory provide explicit
> APIs to do these things as part of test execution. I agree attributes are a
> better fit in the current model.
>
> > - We can define data-parameterized tests.
>
> > - Tests can be dynamically synthesized by control flow.  In the
> > current XCTest API, dynamically generating tests would mean
> > dynamically generating methods, which is even more far off than
> > read-only method reflection.
>
> FWIW, XCTest has some support for doing these kinds of things, they just
> don't take the form of the pattern-matched methods.
>
>  - Daniel
>
> >
> > Dmitri
> >
> > --
> > main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
> > (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160417/1346eb08/attachment.html>