<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Brian,</div><div class=""><br class=""></div><div class="">As Max noted, this part of the proposal really ought to be removed and pended for the next proposal, for third party testing framework support. There is a good discussion to be had about the merits of requiring a testing adaptor that conforms to a protocol for structured test results vs just expecting other test frameworks to report an overall success/fail, but that's not the focus of this proposal and shouldn't block implementing anything here. Please do raise your concerns here when we do discuss supporting other test frameworks, though!</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- Rick</div><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 19, 2016, at 1:02 PM, Brian Pratt via swift-evolution <<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class=""><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class=""><div class=""><div style="font-size:12.8px" class="">However what is sufficient? Is it mandatory for us to output JUnit-esque XML</div><div style="font-size:12.8px" class="">for all possible test frameworks? Would that restrict what innovations are</div><div style="font-size:12.8px" class="">possible by testing frameworks?</div></div><div style="font-size:12.8px" class=""><br class=""></div></blockquote><span style="font-size:12.8px" class="">"for all possible test frameworks"? I don't think you can do something like that unless the build-tool provides an entire output shell that testing frameworks can hook into for reporting at runtime... and in that case, yes, I personally think it would be overly restrictive on testing frameworks, at least from the perspective of a <i class="">build tool</i>. The build tool's responsibility is that it has to compile and run the tests, and then allow said tests to be configured to provide feedback in whatever way I can configure them to. The opinionated output proposed in this document would seem to overreach those responsibilities unless, as Max and Brian mentioned, it lived in XCTest or similar.</span></div><div class=""><br class=""></div><div class=""><span style="font-size:12.8px" class=""><br class=""></span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class=""><div class=""><div style="font-size:12.8px" class=""><div class="">Is it enough to simply report success or failure for the invocation? CI services</div><div class="">would then know if the tests ran to success or not and could simply provide</div><div class="">the output from the test framework as logs.</div></div></div></blockquote><span style="font-size:12.8px" class=""><br class=""></span><div class=""><span style="font-size:12.8px" class="">I'd say yes: Most CI tools I know of will just check the exit status of the commands they're provided, in the simplest cases... So, if the test framework provides a zero/non-zero status for pass/fail, that's enough for most CI services I've used (Travis, Wercker, Jenkins, to name a few). </span><span style="font-size:12.8px" class="">Yes, you can get a lot more detailed, but the implementations would probably differ for each service. Exit statuses are something they all understand, and something the build tool understands already too.</span></div><div class=""><span style="font-size:12.8px" class=""><br class=""></span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class=""><div class=""><div style="font-size:12.8px" class=""><br class="">If we are to provide XML/JSON output as part of this initial implementation</div><div style="font-size:12.8px" class="">then we must parse XCTest output or figure out some other way to integrate</div><div style="font-size:12.8px" class="">with XCTest so that the output format can be modified.</div></div><div style="font-size:12.8px" class=""><br class=""></div></blockquote><span style="font-size:12.8px" class="">Parsing the output feels like a fragile integration, and one that could handcuff what XCTest can change in the future. While it'd be possible to integrate with XCTest more directly (via a reporter registration system that I'm sure Brian can explain better than I can), I don't think the value of that has been defended very well from the perspective of the build tool: The win of getting uniform test output from a build tool is not worth the loss of simple testing integration, whether it be provided by a runtime library defining a set of output protocols, or by intercepting other tools' output and parsing them to transform it, or anything else I've seen mentioned.</span></div><div class=""><br class=""></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px" class=""><div class=""><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;line-height:25.6px" class="">We would like to get testing up to speed as soon as possible.</span></div><div class=""><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;line-height:25.6px" class=""><br class=""></span></div></blockquote><span style="font-size:12.8px" class="">If this is true, then we should make it clear our intention to push those details out to XCTest and allow it to innovate and report on its own terms. Anything else is going to slow things down. </span><span style="font-size:12.8px" class="">Right now, given that the output part of the proposal is getting pushback both from developers working on XCTest and developers who have worked on third-party testing frameworks, I'd say that's just too much scope for a build tool to be taking on.</span><div class=""><span style="font-size:12.8px" class=""><br class=""></span></div><div class=""><span style="font-size:12.8px" class="">Beyond that, there's a very low-effort way that's already included in the tool to get testing up-to-speed, and this would do that: let me define a target (or a list of targets) in my Package.swift that are for tests. Compile them, link them with my testDependencies, and run them, checking exit statuses for results. This achieves:</span><div class="">- the ability to build and run tests from the build-tool</div></div><div class="">- it works with XCTest. Today. Not in the future when they have registration for reporters or when the build-tool writes some test wrapper</div><div class="">- a simple integration point for third-party frameworks</div><div class="">- support for almost every single CI system in existence (if it can run Swift)</div><div class="">- a quick turnaround to put this in the hands of developers (who want it)</div><div class=""><br class=""></div><div class="">More notably, it leaves the concerns of actual testing output to the testing frameworks, where, as so far all parties who have commented on this review agree, those concerns actually belong. This would even fit in the confines of this proposal and could "kick the can down the road" for third-party testing protocols for quite some time as the build tool evolves. Anything else is going to:<br class=""></div><div class="">- limit the flexibility of test framework developers (including those who work on XCTest)</div><div class="">- require more development (in the case of creating an XCTest formatter/reporter, or a parser/transformer, or a protocol that XCTest itself will then have to implement)</div><div class="">- be more complex</div><div class=""><br class=""></div><div class="">All of those things are net losses, all for something that we can't even agree is <i class="">even philosophically a responsibility of the build tool</i>? Let's just drop the output idea, even if just for now, and start delivering first-class testability with the tools that already exist, please.</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Jan 19, 2016 at 1:50 PM, Max Howell via swift-evolution <span dir="ltr" class=""><<a href="mailto:swift-evolution@swift.org" target="_blank" class="">swift-evolution@swift.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><blockquote type="cite" class=""><div class=""><div class="">My only reservation is in regards to the following:<br class=""><br class=""><blockquote type="cite" class="">Executing a test from the terminal will produce user-readable output. This should incorporate colorization and other formatting similar to other testing tools to indicate the success and failure of different tests. [...] An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.<br class=""></blockquote><br class="">swift-corelibs-xctest is currently responsible for its own output--it<br class="">prints test results to stdout. How do the authors of the proposal plan<br class="">to implement colorization and formatting?<br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">This is a good point.</div><br class=""><blockquote type="cite" class=""><div class=""><div class="">Personally, I believe that the testing framework itself--in this case<br class="">swift-corelibs-xctest--should be responsible for colorization and<br class="">output formatting. I believe swift-corelibs-xctest will soon provide<br class="">an observation API similar to Apple XCTest (adding it has been<br class="">discussed several times; see:<br class=""><a href="https://github.com/apple/swift-corelibs-xctest/pull/40" target="_blank" class="">https://github.com/apple/swift-corelibs-xctest/pull/40</a>,<br class=""><a href="https://lists.swift.org/pipermail/swift-corelibs-dev/2015-December/000034.html" target="_blank" class="">https://lists.swift.org/pipermail/swift-corelibs-dev/2015-December/000034.html</a>).<br class="">Once it does, any user will be able to register whichever output<br class="">formatter they wish.<br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">The proposal is not specific about who is providing the output. I personally</div><div class="">expected the testing-framework to provide the default Terminal output</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div class="">I am concerned to find this proposal for swift-package-manager mention<br class="">output formatting, since I it doesn't seem like output formatters<br class="">belong in swift-package-manager itself, and the API for<br class="">swift-corelibs-xctest hasn't been implemented yet. I suggest deferring<br class="">the discussion on output formatting to a future proposal--it seems<br class="">like something that would be merely nice to have, whereas this<br class="">proposal states "we would like to get testing up to speed as soon as<br class="">possible.”</div></div></blockquote><div class=""><br class=""></div><div class="">I think the proposal is OK. We’re not clear about who provides the output,</div><div class="">so nothing is fixed.</div><div class=""><br class=""></div><div class="">When we work on the alternative testing framework proposal we will go into</div><div class="">more detail on this matter.</div><div class=""><br class=""></div><div class="">So TL;DR: for now XCTest will provide the output, colorization can be added</div><div class="">immediately with some output parsing but is probably better left for the next</div><div class="">proposal.</div><div class=""><br class=""></div><div class="">One part of the proposal cannot be met immediately:</div><div class=""><br class=""></div><div class=""></div><blockquote type="cite" class=""><div class=""><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;background-color:rgb(255,255,255)" class="">An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.</span></div></blockquote><div class=""><br class=""></div><div class=""><div class="">It is essential that `swift test` itself can provide some kind of standardized</div><div class="">output so other tools can use `swift test` as part of larger CI systems.</div><div class=""><br class=""></div><div class="">However what is sufficient? Is it mandatory for us to output JUnit-esque XML</div><div class="">for all possible test frameworks? Would that restrict what innovations are</div><div class="">possible by testing frameworks?</div><div class=""><br class=""></div><div class="">Is it enough to simply report success or failure for the invocation? CI services</div><div class="">would then know if the tests ran to success or not and could simply provide</div><div class="">the output from the test framework as logs.</div></div><div class=""><br class=""></div><div class="">If we are to provide XML/JSON output as part of this initial implementation</div><div class="">then we must parse XCTest output or figure out some other way to integrate</div><div class="">with XCTest so that the output format can be modified.</div><div class=""><br class=""></div><div class="">Really the proposal should be modified and this part removed pending the</div><div class="">next proposal, but it would be a shame to delay implementation by a further</div><div class="">week.</div><span class="HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div><div class="">Max</div></font></span></div></div><br class="">_______________________________________________<br class="">
swift-evolution mailing list<br class="">
<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a><br class="">
<a href="https://lists.swift.org/mailman/listinfo/swift-evolution" rel="noreferrer" target="_blank" class="">https://lists.swift.org/mailman/listinfo/swift-evolution</a><br class="">
<br class=""></blockquote></div><br class=""></div>
_______________________________________________<br class="">swift-evolution mailing list<br class=""><a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a><br class="">https://lists.swift.org/mailman/listinfo/swift-evolution<br class=""></div></blockquote></div><br class=""></body></html>