<div dir="ltr"><div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div style="font-size:12.8px">However what is sufficient? Is it mandatory for us to output JUnit-esque XML</div><div style="font-size:12.8px">for all possible test frameworks? Would that restrict what innovations are</div><div style="font-size:12.8px">possible by testing frameworks?</div></div><div style="font-size:12.8px"><br></div></blockquote><span style="font-size:12.8px">"for all possible test frameworks"? I don't think you can do something like that unless the build-tool provides an entire output shell that testing frameworks can hook into for reporting at runtime... and in that case, yes, I personally think it would be overly restrictive on testing frameworks, at least from the perspective of a <i>build tool</i>. The build tool's responsibility is that it has to compile and run the tests, and then allow said tests to be configured to provide feedback in whatever way I can configure them to. The opinionated output proposed in this document would seem to overreach those responsibilities unless, as Max and Brian mentioned, it lived in XCTest or similar.</span></div><div><br></div><div><span style="font-size:12.8px"><br></span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div style="font-size:12.8px"><div>Is it enough to simply report success or failure for the invocation? CI services</div><div>would then know if the tests ran to success or not and could simply provide</div><div>the output from the test framework as logs.</div></div></div></blockquote><span style="font-size:12.8px"><br></span><div><span style="font-size:12.8px">I'd say yes: Most CI tools I know of will just check the exit status of the commands they're provided, in the simplest cases... So, if the test framework provides a zero/non-zero status for pass/fail, that's enough for most CI services I've used (Travis, Wercker, Jenkins, to name a few). </span><span style="font-size:12.8px">Yes, you can get a lot more detailed, but the implementations would probably differ for each service. Exit statuses are something they all understand, and something the build tool understands already too.</span></div><div><span style="font-size:12.8px"><br></span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div style="font-size:12.8px"><br class="">If we are to provide XML/JSON output as part of this initial implementation</div><div style="font-size:12.8px">then we must parse XCTest output or figure out some other way to integrate</div><div style="font-size:12.8px">with XCTest so that the output format can be modified.</div></div><div style="font-size:12.8px"><br></div></blockquote><span style="font-size:12.8px">Parsing the output feels like a fragile integration, and one that could handcuff what XCTest can change in the future. While it'd be possible to integrate with XCTest more directly (via a reporter registration system that I'm sure Brian can explain better than I can), I don't think the value of that has been defended very well from the perspective of the build tool: The win of getting uniform test output from a build tool is not worth the loss of simple testing integration, whether it be provided by a runtime library defining a set of output protocols, or by intercepting other tools' output and parsing them to transform it, or anything else I've seen mentioned.</span></div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;line-height:25.6px">We would like to get testing up to speed as soon as possible.</span></div><div><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;line-height:25.6px"><br></span></div></blockquote><span style="font-size:12.8px">If this is true, then we should make it clear our intention to push those details out to XCTest and allow it to innovate and report on its own terms. Anything else is going to slow things down. </span><span style="font-size:12.8px">Right now, given that the output part of the proposal is getting pushback both from developers working on XCTest and developers who have worked on third-party testing frameworks, I'd say that's just too much scope for a build tool to be taking on.</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Beyond that, there's a very low-effort way that's already included in the tool to get testing up-to-speed, and this would do that: let me define a target (or a list of targets) in my Package.swift that are for tests. Compile them, link them with my testDependencies, and run them, checking exit statuses for results. This achieves:</span><div>- the ability to build and run tests from the build-tool</div></div><div>- it works with XCTest. Today. Not in the future when they have registration for reporters or when the build-tool writes some test wrapper</div><div>- a simple integration point for third-party frameworks</div><div>- support for almost every single CI system in existence (if it can run Swift)</div><div>- a quick turnaround to put this in the hands of developers (who want it)</div><div><br></div><div>More notably, it leaves the concerns of actual testing output to the testing frameworks, where, as so far all parties who have commented on this review agree, those concerns actually belong. This would even fit in the confines of this proposal and could "kick the can down the road" for third-party testing protocols for quite some time as the build tool evolves. Anything else is going to:<br></div><div>- limit the flexibility of test framework developers (including those who work on XCTest)</div><div>- require more development (in the case of creating an XCTest formatter/reporter, or a parser/transformer, or a protocol that XCTest itself will then have to implement)</div><div>- be more complex</div><div><br></div><div>All of those things are net losses, all for something that we can't even agree is <i>even philosophically a responsibility of the build tool</i>? Let's just drop the output idea, even if just for now, and start delivering first-class testability with the tools that already exist, please.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 19, 2016 at 1:50 PM, Max Howell via swift-evolution <span dir="ltr"><<a href="mailto:swift-evolution@swift.org" target="_blank">swift-evolution@swift.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><blockquote type="cite"><div><div>My only reservation is in regards to the following:<br><br><blockquote type="cite">Executing a test from the terminal will produce user-readable output. This should incorporate colorization and other formatting similar to other testing tools to indicate the success and failure of different tests. [...] An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.<br></blockquote><br>swift-corelibs-xctest is currently responsible for its own output--it<br>prints test results to stdout. How do the authors of the proposal plan<br>to implement colorization and formatting?<br></div></div></blockquote><div><br></div><div>This is a good point.</div><br><blockquote type="cite"><div><div>Personally, I believe that the testing framework itself--in this case<br>swift-corelibs-xctest--should be responsible for colorization and<br>output formatting. I believe swift-corelibs-xctest will soon provide<br>an observation API similar to Apple XCTest (adding it has been<br>discussed several times; see:<br><a href="https://github.com/apple/swift-corelibs-xctest/pull/40" target="_blank">https://github.com/apple/swift-corelibs-xctest/pull/40</a>,<br><a href="https://lists.swift.org/pipermail/swift-corelibs-dev/2015-December/000034.html" target="_blank">https://lists.swift.org/pipermail/swift-corelibs-dev/2015-December/000034.html</a>).<br>Once it does, any user will be able to register whichever output<br>formatter they wish.<br></div></div></blockquote><div><br></div><div>The proposal is not specific about who is providing the output. I personally</div><div>expected the testing-framework to provide the default Terminal output</div><div><br></div><blockquote type="cite"><div><div>I am concerned to find this proposal for swift-package-manager mention<br>output formatting, since I it doesn't seem like output formatters<br>belong in swift-package-manager itself, and the API for<br>swift-corelibs-xctest hasn't been implemented yet. I suggest deferring<br>the discussion on output formatting to a future proposal--it seems<br>like something that would be merely nice to have, whereas this<br>proposal states "we would like to get testing up to speed as soon as<br>possible.”</div></div></blockquote><div><br></div><div>I think the proposal is OK. We’re not clear about who provides the output,</div><div>so nothing is fixed.</div><div><br></div><div>When we work on the alternative testing framework proposal we will go into</div><div>more detail on this matter.</div><div><br></div><div>So TL;DR: for now XCTest will provide the output, colorization can be added</div><div>immediately with some output parsing but is probably better left for the next</div><div>proposal.</div><div><br></div><div>One part of the proposal cannot be met immediately:</div><div><br></div><div></div><blockquote type="cite"><div><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';font-size:16px;background-color:rgb(255,255,255)">An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.</span></div></blockquote><div><br></div><div><div>It is essential that `swift test` itself can provide some kind of standardized</div><div>output so other tools can use `swift test` as part of larger CI systems.</div><div><br></div><div>However what is sufficient? Is it mandatory for us to output JUnit-esque XML</div><div>for all possible test frameworks? Would that restrict what innovations are</div><div>possible by testing frameworks?</div><div><br></div><div>Is it enough to simply report success or failure for the invocation? CI services</div><div>would then know if the tests ran to success or not and could simply provide</div><div>the output from the test framework as logs.</div></div><div><br></div><div>If we are to provide XML/JSON output as part of this initial implementation</div><div>then we must parse XCTest output or figure out some other way to integrate</div><div>with XCTest so that the output format can be modified.</div><div><br></div><div>Really the proposal should be modified and this part removed pending the</div><div>next proposal, but it would be a shame to delay implementation by a further</div><div>week.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Max</div></font></span></div></div><br>_______________________________________________<br>
swift-evolution mailing list<br>
<a href="mailto:swift-evolution@swift.org">swift-evolution@swift.org</a><br>
<a href="https://lists.swift.org/mailman/listinfo/swift-evolution" rel="noreferrer" target="_blank">https://lists.swift.org/mailman/listinfo/swift-evolution</a><br>
<br></blockquote></div><br></div>