[swift-dev] [SR-710][RFC] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

[Brian Gesiak]

Thanks for the feedback, Dmitri!

I think I misstated options #2 and #3: where I typed "move business logic
like IsTestCandidate to libIDE", I meant "move a lot of the
non-XCTest-specific logic from SourceKit to libIDE, such that implementing
IsTestCandidate would be trivial in SourceKit and our test generation
tool". The last thing I'd want to do is couple libIDE and XCTest!

On Sun, Apr 3, 2016 at 6:36 PM, Dmitri Gribenko <gribozavr at gmail.com> wrote:

> On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache at gmail.com> wrote:
> > I think #2 is the best option. It’s less work than both #1 and #3. I
> believe
> > logic like IsTestCandidate belongs in libIDE anyway—SourceKit should
> stick
> > to XPC and asynchronous communication with libIDE.
>
> I like #3 better (an option to swiftc), because that would decouple
> the test discovery tool from the Swift compiler.  That would allow you
> to use the discovery tool with different compilers.  And, because we
> would avoid statically linking libIDE, it would mean one less copy of
> LLVM, Clang and Swift in the toolchain.
>
>
My concern with adding a swiftc option would be deciding upon an interface
to it. Would we want something like clang-query--a generic method of
finding methods that match a set of criteria? That seems like it would be a
lot of work to implement.

Perhaps I'm misunderstanding what you have in mind for swiftc. Could you
elaborate?


> > Not being an expert in many of these components, I have several
> questions:
> >
> > I’m assuming the reflection API to return a list of instance methods on a
> > XCTestCase subclass is not ready yet, and won’t be for some time. Is this
> > accurate?
>
> I think so.
>
> > I’m assuming that SourceKit is intended to be an asynchronous wrapper
> over
> > libIDE, and that logic like IsTestCandidate should be moved to libIDE. Is
> > this accurate?
>
> SourceKit has a lot of functionality of its own, but moving this
> particular piece of logic to libIDE sounds reasonable.
>
> > I’m assuming that SourceKit is coupled with XPC, and that it would be
> more
> > work to port it to Linux than it would be to move its logic to libIDE. Is
> > this accurate?
>
> It is not tightly coupled with XPC, there is a portability layer that
> you could implement for Linux.  You would need to decide on an IPC
> mechanism and serialization format though.
>
> > If you have thoughts/feedback, please reply to this email or comment on
> > SR-710. Your input would be greatly appreciated!!
>
> I'm wondering how feasible is it to change the XCTest API to
> accommodate better the Swift language that we have, rather than trying
> to add custom tooling to make the existing API work.  Adding magic
> tooling that adds behavior not present in the language seems unnatural
> to me.
>
> Compare with StdlibUnittest -- by using an API to build tests we get
> the following advantages:
>
> - We completely avoid having the issue of test discovery, executing
> the code discovers the tests.  No reflection needed!
>
> - We can add attributes to tests (for example, skip, xfail).  In the
> current XCTest API this would require adding some kind of user-defined
> attributes, which is another language which is a long way from being
> designed and implemented.
>
> - We can define data-parameterized tests.
>
> - Tests can be dynamically synthesized by control flow.  In the
> current XCTest API, dynamically generating tests would mean
> dynamically generating methods, which is even more far off than
> read-only method reflection.
>

I definitely agree that all of these are fantastic features. However, the
corelibs-xctest README spells out the following goal for the project: "This
version of XCTest uses the same API as the XCTest you are familiar with
from Xcode. Our goal is to enable your project's tests to run on all Swift
platforms without having to rewrite them."

This goal often conflicts with what would be the best API for a Swift
testing framework--we are forced to use mutable reference types, extensive
subclassing, include a ton of optional properties--all in order to mirror
exactly the Apple XCTest API.

I suppose we *could* modify the SDK overlay for XCTest to provide an API
more suitable for Swift, but I think that would involve a lot of
coordination with Apple XCTest and Xcode teams, some swift-evolution
proposals, and a potentially harsh migration path for developers writing
their tests in Apple XCTest. So while I think corelibs-xctest and Apple
XCTest can (and should) evolve to better take advantage of Swift features
over time, I don't think we can migrate both before Swift 3's release.

(+CC Mike Ferris, who works on Apple XCTest and corelibs-xctest.)

- Brian Gesiak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160403/eb32fc4a/attachment.html>