[swift-build-dev] protocol for third-party testing frameworks

Thu Mar 17 18:20:25 CDT 2016

Resending as Outlook apparently has trouble including the list instead of just the recipients…

---

Honestly, I’m still confused why this is the approach that is desired to be taken. Why not just provide a way to invoke different runners from within SwiftPM? Like Drew mentioned, this approach still isn’t going to solve the problems we had. We simply provided a mechanism to run various things. This made it extremely trivial to integrate the running of any type of test framework we wanted with no additional work required by anyone.

With this approach, it is trivial for us to support XCTest runners, shell script tests, tests that required files to be generated first, etc… 

I’ve also been doing some experiments with different ways to write tests, how would I go about doing that with SwiftPM? 

class Sum {
    func add(x: Int, _ y: Int) -> Int { return x + y }
}

func __test_sum_add_checkin() throws {
    let sum = Sum()
    assert(sum.add(4, 5) == 10, "Math is hard!")
    assert(sum.add(-3, 3) == 0)
}

Then with our build files:

(package

  :name "IntegratedUnitTests"

  :tasks {

    :build {

      :tool "atllbuild"

      :sources ["Sum.swift"]

      :name "math"

      :output-type "static-library"

      :publish-product true

      :compile-options ["-enable-testing"]

    }

    :test {

      :dependencies ["generate-test-file"]

      :tool "atllbuild"

      :sources ["sum_test.swift"]

      :name "sum_test"

      :output-type "executable"

      :publish-product true

      :link-with ["math.a"]

    }

    :generate-test-file {

      :dependencies ["build"]

      :tool "shell"

      :script "echo '@testable import math' > sum_test.swift && xcrun -sdk macosx swiftc -print-ast Sum.swift | grep __test | sed 's/internal func/try/g' | sed 's/throws//g' >> sum_test.swift"

    }

    :run {

      :dependencies ["test"]

      :tool "shell"

      :script "./bin/sum_test"

    }

  }

)

More details: http://owensd.io/blog/tooling-around---testing-in-swift/

It would seem the plan is that I’d then have to go write some additional wrapper if I simply wanted it to work in SwiftPM. That’s not a problem that scales very well.

-David

From: swift-build-dev-bounces at swift.org [mailto:swift-build-dev-bounces at swift.org] On Behalf Of Brian Pratt via swift-build-dev
Sent: Thursday, March 17, 2016 10:32 AM
To: Drew Crawford <drew at sealedabstract.com>; Max Howell <max.howell at apple.com>; Drew Crawford via swift-build-dev <swift-build-dev at swift.org>
Subject: Re: [swift-build-dev] protocol for third-party testing frameworks

To get a bit more concrete:

let package = Package(

  name: "MyCoolPackage",

  // etc

  targets: [

    Target(name: "MyCoolPackageTarget")

 ],

 testTargets: [

    Target(name: "MyCoolTestingTarget", dependencies: [.Dependency("one-of-my-test-dependencies")])

  ])

When someone else uses my package, MyCoolTestingTarget would not be built (although its sources would be present), but MyCoolPackageTarget would be -- if that makes sense. In this way, we'd get ad-hoc support for third-party testing frameworks and also improve the Package Manager to be able to be more useful as a build tool for a whole dev environment as opposed to just a build tool for creating and consuming packages.

The only drawback I can think of that falls out of this approach is that 3rd-party testing frameworks wouldn't support the `swift test` command-line invocation, but I'd personally rank that as a lower priority for me than just being able to get more granular control over what gets built in a local context vs what gets built when I'm a dependency of something else.

Hopefully that clears my idea up a bit more.

-- 

Brian Pratt

On March 17, 2016 at 10:05:23 AM, Brian Pratt (brian at pratt.io <mailto:brian at pratt.io> ) wrote:

Hey folks,

I think from my perspective SwiftPM could solve this problem by simply allowing me to define and build targets that aren't going to be included in the distributed package. Maybe a field like `testTargets` or something that allows the author downstream to link `testDependencies`?

SwiftPM is great as a package manager, but for a lot of my most common development workflows, it falls short because of this limitation -- I can't use it to build things that I don't mean to distribute as part of the package.

-- 
Brian Pratt

On March 16, 2016 at 8:39:02 PM, Drew Crawford via swift-build-dev (swift-build-dev at swift.org <mailto:swift-build-dev at swift.org> ) wrote:

Please take the lead on this. 

For background.  We required a solution around the time I was working on this originally, and given that upstream wasn't ready (which is understandable) we decided to go alone, and we're now committed to our solution.  So from our POV the problem has been solved.

On Mar 16, 2016, at 8:12 PM, Max Howell <max.howell at apple.com <mailto:max.howell at apple.com> > wrote:

It’s time to resurrect this proposal. 

By all means let’s talk a bit more here, but I hope Drew will submit a proposal, or if he doesn’t have the time I can write it up, co-authored and we’ll submit to evolution.

On Jan 12, 2016, at 5:02 PM, Max Howell <max.howell at apple.com <mailto:max.howell at apple.com> > wrote:

To follow up here, I’d like to put this particular proposal on the back burner for a couple weeks until we have the initial testing infrastructure in place. 

We should keep this proposal in mind as we build the testing infrastructure, as this protocol is in my mind at least, the more important of the two.

On Jan 6, 2016, at 11:58 AM, Max Howell via swift-build-dev <swift-build-dev at swift.org <mailto:swift-build-dev at swift.org> > wrote:

I envisaged command line arguments to `swift build`. What use cases are we talking about here for making which tests run more configurable?

Well I think some kind of CLI support (like perhaps passing the whole CLI to the testing framework?) makes sense.

That’s our current thinking.

Specifically one problem that motivates flexibility here is the per-commit continuous integration. If my test suite is 15 minutes, and I'm going to run that in a few different configurations  (32 vs 64-bit, or say we support iOS someday and I'm cross-compiling for N simulators or hardware devices, etc.) I'm in for a bad time trying to do all that every commit.

There are a lot of "solutions" here–parallelization, random sampling, running an abbreviated test suite ordinarily and the full test suite every 10th commit–a test suite that tries to maximize code coverage per unit time–there are all kinds of ways to look at this problem that may make sense to somebody.

I'm simply pushing that decision out to individual test frameworks to study.  XCTest thinks you should have test suites that are human curated.  That's one way to skin the cat.  Let's give another test framework the flexibility to think differently on this problem.

Sounds super interesting. We should make sure the protocol we design can do these things.

I’ll be working on the testing infrastructure as soon as the proposal has been reviewed. As I do so I’ll be making notes on this aspect.

Thanks for picking this up.  Getting basic testing support in place is really important, and IMO the current proposal is really strong.  I am focusing on some of these other areas not as a distraction but because I think the basic testing ideas are strong enough already that they don't need my help :-)

  <https://u2002410.ct.sendgrid.net/wf/open?upn=cbMbdH1LnH6O78Q-2BHw3jtU8ikibH470Fh9meAJKpwSqpJeTKnMxtBKMcwgO6eNd7RiBAIn9BZYPOs2eHgzutG3iFkECUBpSh2YJRbpFs5gYIFdbbC8ItMNA7gp-2BqGK4cWwBss4mY6Uv8sMAwKkzb1-2B1rswus8P2lPVMLer7xFPfSEQS6c96lmiGJyiZhqf-2F-2BJOnpV-2BhXXERxFHP94bzeH6uMWF-2B662RYrOYdPl-2FmMCE-3D>  _______________________________________________
swift-build-dev mailing list
 <mailto:swift-build-dev at swift.org> swift-build-dev at swift.org
 <https://lists.swift.org/mailman/listinfo/swift-build-dev> https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev at swift.org <mailto:swift-build-dev at swift.org> 
https://lists.swift.org/mailman/listinfo/swift-build-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20160317/b0ab3979/attachment.html>