[swift-dev] [SR-710][RFC] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

[Brian Gesiak]

Hello all!

SR-710 <https://bugs.swift.org/browse/SR-710> tracks a major goal for Swift
3: having SwiftPM/corelibs-xctest automatically generate a list of test
methods to execute. The implementations we’re considering span various
parts of the codebase: libIDE, SourceKit, the reflection APIs, etc. We need
input from people familiar with these components.

Here’s the issue: currently, users of corelibs-xctest must manually list
each test they wish to execute:

class MyTestCase: XCTestCase {
    static var allTests: [(String, MyTestCase -> () throws -> Void)] {
        return [
            ("testFoo", testFoo)
        ]
    }

    func testFoo() {
        XCTAssert(true)
    }

    // The user forgot to list this method in `allTests`, so it is never run.
    func testBar() {
        XCTAssert(true)
    }
}

This is tedious and error-prone. We need to do better by Swift 3!

Apple XCTest uses Objective-C reflection at runtime to compile a list of
NSInvocation to execute as tests. We can’t use the same approach in Swift:
as far as I know, there’s no reflection API that allows us to find instance
methods defined on a class, and adding such an API is considered a stretch
goal for Swift 3
<https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001682.html>
.

Several commenters in SR-710 <https://bugs.swift.org/browse/SR-710> have
suggested using libIDE or SourceKit to determine the list of tests. I
believe this is the most feasible approach that could be implemented in
time for Swift 3. A few caveats:

   - SourceKit is coupled to XPC, which only works on OS X.
   - Logic like SourceKit::FuncDeclEntityInfo.IsTestCandidate
   <https://github.com/apple/swift/blob/12593eff135ab2fa99529e9fa8ecc61ce268cd45/tools/SourceKit/include/SourceKit/Core/LangSupport.h#L54>
   exists in SourceKit, but not libIDE.
   - libIDE defines a C++ interface, so we could not use it from Swift
   (should we choose to use Swift to generate the list of test methods).

I think we have three concrete options for implementing SR-710
<https://bugs.swift.org/browse/SR-710>:

   1. Port SourceKit to Linux, using a different form of IPC since XPC is
   not available.
   2. Move business logic like IsTestCandidate to libIDE. If the tool to
   generate the list of tests will be written in Swift, we can add a C header
   (like libclang or sourcekitd). The tool would link against libIDE and use
   the C header.
   3. Move business logic like IsTestCandidate to libIDE and add a swiftc
   option to interface with its functionality. The tool that generates the
   list of tests would invoke swiftc to get the list.

I think #2 is the best option. It’s less work than both #1 and #3. I
believe logic like IsTestCandidate belongs in libIDE anyway—SourceKit
should stick to XPC and asynchronous communication with libIDE.

Not being an expert in many of these components, I have several questions:

   - I’m assuming the reflection API to return a list of instance methods
   on a XCTestCase subclass is not ready yet, and won’t be for some time.
   Is this accurate?
   - I’m assuming that SourceKit is intended to be an asynchronous wrapper
   over libIDE, and that logic like IsTestCandidate should be moved to
   libIDE. Is this accurate?
   - I’m assuming that SourceKit is coupled with XPC, and that it would be
   more work to port it to Linux than it would be to move its logic to libIDE.
   Is this accurate?

If you have thoughts/feedback, please reply to this email or comment on
SR-710 <https://bugs.swift.org/browse/SR-710>. Your input would be greatly
appreciated!!

Brian Gesiak
​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160403/fafbaebe/attachment.html>