[swift-dev] [SR-710][RFC] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

Fri May 6 15:04:19 CDT 2016

Sorry for the delay in following up.

I have had several long discussions on this topic with Dmitri and others, which
I will try to summarize here:

*TL;DR*: We think the right long-term path forward is to pursue porting
SourceKit to Linux, and would like to explore that direction first before trying
to develop a compromise which either (a) uses internal and likely-to-break APIs
(e.g., dump-ast) or (b) devotes significant engineering resources to a "right
solution" for this problem, but which will only solve this problem and not other
issues where having an API for use with the Swift language.

## Problem Statement

The XCTest API has historically been defined in terms of methods with a
particular naming convention `test...` which were in subclasses of XCTestCase.

On OS X, these methods can be found via the Objective-C runtime but that does
not work on Linux. Our current solution on Linux requires manual specification
of all of the test methods, and is a huge maintenance burden for people trying
to use XCTest on Linux or maintain cross-platform projects.

## Background

This feature works on OS X in two ways:

1. As mentioned, tests are run by dynamic discovery through the Objective-C
   runtime when the bundle containing the tests is executed.

2. Within the Xcode IDE, tests are "discovered" (for use in UI) through the use
   of the Xcode indexer.

   The mechanism at work here, for Swift, is a combination of functionality in
   the indexing engine (to aggregate and query results) and the raw underlying
   data provided by SourceKit.

   We would like any implementation of this feature to share as much code as
   possible with SourceKit and Xcode's implementation in order for
   cross-platform projects to behave predictably.

It also happens that SwiftPM has several other needs for API-based interactions
with the functionality in the Swift compiler. Several examples:

* We would like to be able to enforce the strictness of the `Package.swift`
  manifest file format. This requires APIs to interact with the Swift
  AST. https://bugs.swift.org/browse/SR-1433

* We would like to support advanced features for dealing with automatic
  inter-module dependencies. For example, one idea which has been proposed is
  that if a module attempted to import a module upon which it did not have a
  dependency, that we would prompt the developer if this dependency should
  automatically be added to the manifest. Doing this feature well requires
  APIs to interact with the Swift compiler as it parses source code and
  reports diagnostics.

* We would like to understand the current level of parallelism being used by
  the Swift compiler, so that llbuild can accurately schedule work. This
  requires APIs for interacting with an in-flight compilation process.

## Discussion

We discussed several avenues of attack on this problem. I will go through them
one by one. I am just attempting to summarize the conversation, obviously there
is a ton of nuance to each point, and hopefully other people will chime in if I
missed something important.

### Language Features

One way to view this problem is that XCTest's API (i.e., depending on test
naming and dynamic discovery) is a poor fit for Swift today, and that this
problem should be tackled in that direction.

For example, the Swift compiler itself has a test framework that does not depend
on dynamic discovery. One could also imagine language-level features which would
solve the arbitrary problem of wanting to find discoverable things; that could
take the form of an `@XCTest` attribute, or "generalized attribute" support.

The current mission for Swift 3, however, is API parity between Linux and OS X,
and so this direction does not lead to any short term solution. For that reason,
while many of us ultimately think this is the right long term direction, we need
to find another one as well.

### Custom "Supported" XCTest Feature

The next option is to pursue a custom, but "supported" feature intended to
tackle this problem head on. Some proposals that have come up are, e.g., a new
compiler flag which emits the list of test methods.

This approach has a couple unfortunate properties:

1. It is non-trivial. We can either design this as an incredibly XCTest specific
   feature requiring understanding of the backend (compiler directly emits
   metadata into .o file), or a midway feature (compiler tells us list of test
   methods, but then we have to manage incremental compilation and the desire to
   not compile things multiple times more than necessary ourselves).

2. It is not-reusable. The work we do here doesn't help with any of the other
   ways we want to use the Swift compiler as an API.

### Implement an API-based Interface

This approach means porting SourceKit to Linux, and then building this feature
on top of that (that itself will be non-trivial, which is something not to be
glossed over).

Everyone generally agrees we should have some kind of API-based access to the
compiler, so this is in line with our long-term direction, but in a way which is
actionable now (it does not require design).

This approach has its own issues:

1. Porting SourceKit may require a fair amount of work. The current code base is
   very OS X specific in particular areas (generally around the use of
   `libxpc`). This will require adding abstractions, developing alternatives,
   and writing the code and build system changes to get this to happen.

2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in
   a way that it can be used from SourceKit. If we do port and use SourceKit, it
   will mean the toolchains increase in size because we effectively will have
   two copies of many parts of Swift/Clang/LLVM installed.

3. By itself, this does not fully solve the issue. As mentioned in the
   background, SourceKit itself does not directly manage all parts of XCTest
   discovery today on OS X, it is a collaboration between SourceKit and the
   Xcode indexer to do discovery without execution. The expectation, however, is
   that we can, however, use SourceKit's APIs to implement this feature in a way
   that is "supported".

## Conclusion

The conclusion was that after weighing all of the tradeoffs, it made the most
sense to encourage porting of SourceKit to Linux and then using it to build out
the Linux test discovery feature. This was most in line with a desirable
long-term direction without being blocked on language design.

I don't think we are particularly firm in this conclusion. If you feel that the work
you have already invested gets us so close that it is worth prioritizing that
approach, then please push back.

 - Daniel

> On May 1, 2016, at 8:19 PM, Brian Gesiak <modocache at gmail.com> wrote:
> 
> I made an attempt at adding `swiftc -frontend -dump-xctest-methods`: https://github.com/apple/swift/pull/2364 <https://github.com/apple/swift/pull/2364>
> 
> Feedback appreciated!!
> 
> - Brian Gesiak
> 
> 
> On Sun, Apr 24, 2016 at 8:59 PM, Brian Gesiak <modocache at gmail.com <mailto:modocache at gmail.com>> wrote:
> Thanks to this commit <https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6> it looks as if IsTestCandidate has been moved out of SourceKit and into libIndex:
> 
> isTestCandidate(swift::ValueDecl) <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754>
> swift::index::FuncDeclIndexSymbol.IsTestCandidate <https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102> and where it’s set <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786>.
> I’m looking into adding an option to swiftc to emit XCTest candidate methods. How does swiftc -frontend -dump-xctest-methods sound?
> 
> - Brian Gesiak
> 
> 
> On Sun, Apr 17, 2016 at 5:50 PM, Drew Crawford <drew at sealedabstract.com <mailto:drew at sealedabstract.com>> wrote:
> 
>> On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko <gribozavr at gmail.com <mailto:gribozavr at gmail.com>> wrote:
>> 
>>> Hmm... but then wouldn't that more tightly couple the test discovery tool
>>> and the Swift compiler? In an earlier email you said you "like #3 better
>>> [...] because that would decouple the test discovery tool from the Swift
>>> compiler." I think that part is confusing me.
>> 
>> Sorry -- what I meant is that the compiler remains the point of truth
>> about the language and can find the tests.  The tools that actually
>> generate glue code won't need to parse code, and would be decoupled
>> from the compiler.
> 
> I am internally shipping a test framework that discovers tests via an out-of-tree parser.  Teaching swiftc about XCTest syntax is not sufficient to deprecate my parser, and therefore is not sufficient for the compiler to be the source of truth for my tests.
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160506/18a6bec8/attachment.html>