[swift-evolution] [swift-build-dev] [Review] SE-0019 Swift Testing (Package Manager)

[Drew Crawford]

If I may make an extended comment that may clear up some disconnects that are troubling other reviewers, that I have been grappling with very recently.  I am one of the more aggressive voices in favor of configurability on swift-build-dev–I actually maintain a small fork that has various configurability hacks so the project is usable for me–and I empathize with what David is saying, and what the other folks are saying about "mixed bags" and "first drafts" when a proposal does not entirely fit into my use case.

One thing I have finally learned about SwiftPM proposals is they are subtly different from Language proposals.  

Language proposals operate on a language that already exists, is already implemented, and is basically in a useful state, for various values of "basically".  Language proposals make small tweaks for small usability wins, and necessarily operate at a minute level of detail with all the problems ironed out to avoid conflicting with current code.  Language proposals tend to be largely adversarial: we have a problem, and there is a solution space; many will enter, few will win.  C-style for loops should either be in or out; there will only ever be one proposal on that topic; everybody has had plenty of time to think about the full implications of C-style for loops three years into the language.  If you think there is some deficiency in SE-0007 your only recourse is to raise objections on this list; the suggestion that you should instead write a different proposal to keep them sounds very much like a brush-off.

SwiftPM, on the other hand, is basically *unusable* (again for various values of "basically"), and there seems to be the prevailing presumption that before anything is done we must have a proposal.  As a consequence, there is no testing support in SwiftPM right now at all, and there is the general sense that before any testing can be added, some proposal must be accepted by this list.  I think this presumption may be so obvious to some people that nobody bothers to actually say it explicitly, whereas it was not at all obvious to me.  This result and the various follow-on consequences of it have only occurred to me recently.

It may be illustrative to think of how Swift might have evolved were it using the swift-evolution process from the beginning.  In that situation there would have first been a proposal to *introduce* C-style for loops (because obviously we need something) and then there would have been a proposal for *for-in loops* and finally there would be another proposal to *repeal the C-style for*.  We would have in fact had 3 proposals on the "for loop topic" (if not more, because I am probably forgetting key developments) and so a proposal for a very greenfield project is hardly a Final Design for All Time and more a permission to start solving the problem at all.

When that first C-style for-loop proposal was presented somebody very well could have said "I think we should have for-in loops", but it would not have been a brush-off to suggest that should be its own proposal that is separately discussed and implemented *in addition to* the C-style proposal, nor would it have been, in 2013, an objection to C-style-for.  It is easy for us in 2016 to smugly suppose we would have always had one for-loop model, but we had 2 models for years, and as the spirited discussion on SE-0007 has shown, it is not obvious to everyone that they should be combined, and the key arguments involve the details of optimizers and so on which are understood only recently.

Now, with the disclaimer that this is my intuition of the situation, rather than any written policy–and things like this should really be written–the present situation is like introducing the C-style for loops, because we need something.  I think the standard of evidence for this proposal (and in fact all SwiftPM proposals, and in fact all greenfield project proposals) should be much different than other proposals reviewed on this list.  The question we should ask ourselves is: "Is this proposal basically useful?  Should someone start implementing it tomorrow?" and not "Is this proposal exhaustively thorough?"  We are in month 1 of a project here, things are not going to be thorough.  If we are obviously going to box ourselves into a corner, that is one thing, and that class of objection is very important.

But SwiftPM is in a "bad" state (read: it's unimplemented, I don't suggest it's poorly-implemented).  You cannot run tests, you cannot link with libdispatch or Foundation, you cannot build a dynamic library, you cannot call an external build system, you cannot cross-compile, you cannot get an optimized build, you cannot define a DEBUG macro... *we have a lot of work to do*.  And as I understand it, most, if not all, of these problems are going to come across your desk as a proposal review.

Things like running the tests separately from builds are absolutely important.  Things like encouraging a diversity in the type of tests are absolutely important.  But I think it is more important to understand the semantics of this review: it is not, I believe, an absolute statement that we have considered and *rejected* all the things *not* included in the proposal.  It is, I believe, a statement that we have *permission* to implement all the things *in* the proposal.  So the absence of various things should not be worrying, but should instead be redirected into producing proposals that extend this one to support more usecases.  Particularly if these things are critical for motivated users (and trust me, I have testing problems not handled here that are critical for me), it seems like writing follow-on proposals is the productive path to accomplishing them, at least as far as I have understood the process so far.

To some extent these extensions are already happening, for example, I have a proposal still in swift-build-dev <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151228/000125.html> that describes in great detail the protocol given merely one sentence in the present proposal, which may be a first step toward diversity in test types.  I think somebody should draft a proposal for tests-without-builds.  But "we need more features than this" should be an argument for more proposals, not less.  Either that or we need to revisit the apparent presumption in the SwiftPM project about proposals-first, because we are never going to get anywhere by waiting for a Final Specification for Tests Forever before implementing any testing at all.  And I think there should be the feeling that we may even reverse things written here if they turn out to be wrong with more experience, like we did recently with C-style-for.

I really think this whole idea about proposals-first and evidentiary standards of review for greenfield projects needs to be written up and documented somewhere official, because I was quite confused about how SwiftPM is governed until recently (and maybe I'm still confused!  Who knows!) and people looked at me funny when I raised objections very much like David's on other topics in SwiftPM.  And those kinds of objections and an adversarial presumption are not unreasonable for people more familiar with the Language proposals as a background.  The lack of clarity about the semantics of a review creates misunderstandings that negatively impact both the people writing the proposals and the people trying in good faith to raise objections to them.  We can fix that.

> What is your evaluation of the proposal?

+1

I desperately need test support, I am currently doing something insane to be able to test my codebases on Linux.  The insane thing I am doing needs to die in a fire two weeks ago.

For that matter the insane thing that Foundation and SwiftPM themselves do to run their tests needs to die in a fire.  I think we are not quite there yet with Foundation, but this proposal gets us there (I think?) on SwiftPM.

I agree generally with Paul Cantrell's assessment that the actual details are a mixed bag.  Specifically:

Paul makes a convincing argument that --without-tests is the right default, and it agrees with my own intuition
I wonder if JUnit XML is really the format we want to encourage (as opposed to something JSON-based, etc.)  I certainly don't object to it as an option, and the current language is certainly vague enough to be agreeable to me, but I would want to discourage it as a default format.  We should have a native format in something easier to parse.
That build configuration for testing is lacking (e.g. -D TESTING or similar)
That dependencies for test modules are presently unspecified
That we need a test-without-build setting for David's case
That we need some overrideable behavior for "swift build -t" other than "run all test modules"
It probably goes without saying, but we should standardize on a particular return code for "tests failed" (as opposed to combining with the code for build failure or some other kind of error).

To be clear, I don't think any of these are important enough to derail this proposal, but they are things I think should be taken up at some point.

I do think there are a lot of strong points about the proposal:

That there is "one weird trick" (swift build -t) that can run tests for any Swift package, and ideally (e.g. through further extensions) we will extend this to support custom testing frameworks and so on so that there is no longer any reason for an ad-hoc way to run the tests
That tests are a first-class citizen and available anywhere that builds are available.  One of the early questions everyone has when they dip their toe into a new project are "how does it build" and "how do I run the test suite" and I really think greasing the wheels here is important to the overall future of the ecosystem.  Currently many of the github.com/apple <http://github.com/apple> projects do something "special snowflake" here, and this creates friction.  We have an opportunity to avoid this in the future.
I think there is an opportunity here to encourage "bug-reports-as-test-cases", which are the best kind of bug report.  Often when I submit bug reports I have a little sample project or something but it is not necessarily obvious how to turn that bug report into a test which would simplify upstream reproducing it and so on.  Some standardization here really has the opportunity to make it easy for me to submit bug reports as tests, which simplifies upstream's life considerably.

I think these strong points are a lot more important than some of the details in the first list that may be absent or wrong, and for those reasons I think we just need to do it and see how it goes.  The only thing that even makes me hesitate is the --without-tests default, but while I believe in that hill I don't want test support to die on it.

> 	* Is the problem being addressed significant enough to warrant a change to Swift?

Yes, tests are critical.

> 	* Does this proposal fit well with the feel and direction of Swift?

Basically.  I think we have a lot more work to do for use cases like David's (and also mine... and also others...) but we desperately need some basic way to run "some tests somewhere" and "swift build -t" is a fine way.

Organizing the tests into a "Tests" directory is logical for the common case and extends a current SPMism of using directory layout to infer intent.  For legacy interop with Xcode projects there may be other cases we need to take up separately.

> 	* If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I design and implement tools that wrap "xcodebuild test".  This proposal is light on a lot of details I care about, but the fact that there should be some way to build and run tests is not especially controversial.

I also work with Python3's unittest module quite often.  I think we can do a lot better than Python, but again this proposal is a first step that doesn't really touch on any issues I care about there.

> 	* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've followed this proposal from its first draft.  I've also written a follow-on proposal currently collecting feedback in swift-build-dev.


> On Jan 7, 2016, at 1:18 PM, David Owens II via swift-evolution <swift-evolution at swift.org> wrote:
> 
>> One thing I'll add here: Unless I misunderstand, I don't think that this is a design/architectural limitation, but rather just a limitation of what functionality is currently exposed. In your example with xcodebuild, architecturally we do have the ability to test without building, and that functionality is even exposed in the Xcode.app Product menu, we've just never implemented a flag to xcodebuild for it. But we haven't boxed ourselves into an architectural corner there; there's no improper coupling at a technical level. Likewise, while this initial proposal doesn't specify the ability to test without building, I think that would be easy to add later on when we want it. Does that address your concern here?
>> 
>> 	- Rick
> 
> No, it doesn’t. While on the Office for Mac team, we repeatedly asked for many of the items in Xcode and xcodebuild to be made less coupled with each of the steps in the process. We were told no in most cases and given hacky workarounds in others that were prone to break with each incremental update to Xcode. Most of these breaks were found out because we picked up the latest beta build and attempted to run our automation tools only to find that something subtle had changed in our workaround or the basis of the workaround is no longer there.
> 
> Sure, SPM is “open source”, however, it’s unrealistic for people/teams to maintain forks of the tools to support features they need. Maybe the changes get accepted into a pull request, maybe not.
> 
> Honestly, I’m a bit shocked that the separation of building and executing tests is such a controversial thing. There should be three high-level tasks:
> 
> 1. build
> 2. run-tests
> 3. build + run-tests
> 
> What I’m saying is that by not supporting #2, you’re creating an adoption blocker for any team that cannot or does not want to build every time it runs their tests. There are many reasons for this. I’ve already given several, however, another reason is instrumentation for things like code-coverage or product code markers, often happen as an intermediate step outside of the normal build operation. This is especially true in the cases of bug investigation. Saying that I cannot trigger the running of these tests without a build step baffling to me.
> 
> -David
> 
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20160108/8598ff11/attachment.html>