Clar JUnit XML Support - #4741

Personally, I'd rather vote for the Test Anything Protocol (https://testanything.org/) instead of the JUnit protocol. It's really simple, doesn't use XML, human readable and understood by quite a lot of tools. I think it's also used by git's test suite. I have no idea whether CTest understands it, though

Yeah, I would have loved to not have be anywhere near XML 😉. Note that my summary is actually invalid w.r.t to the XML schema.

My main gripe is that AFAIK we're "misusing" ctest when we do a monster-test-executable, as its expectation is that it is the test runner. IOW, it expects one ADD_TEST stanza per testcase. But this would enable its full range of functionality, like parallel (-j <jobs>), randomized (--schedule-random) runs. Test selection is available through -L/-R/-E <regex> so this wouldn't be missed, and there's a label system for things like online::push.

So (and this is pretty fuzzy because I'm randomly typing keywords), it seems ctest ought to be able to output NUnit XML, because that's one of the formats accepted by CDash (the CMake "dashboard").

And secondly the VSTS task only has support for XML formats, and most of them look somewhat VisualStudio-oriented (what are assemblies ?). Even though it's open-source, the cursory look I gave it made me think that it's only performing a "transfer" to some other part of the VSTS infrastructure, and I haven't been able to see if TAP support could be added to that.

The main question thus become : do we want to embrace ctest and change clar to be a dumber "test harness", only responsible of one specific test suite, maybe even a single test case ? I'd be willing to make a pass at this ; after more keyword-digging though, I've yet to find a definitive "you can report your tests using those formats" in the ctest documentation…

Note that I'm somewhat torn on the ctest issue, since it has obvious benefits that are missing from clar (not to be mean to clar, apart from the one-executable-issue, I'm actually impressed at how clean it looks and how well it works).

I should have fixed most of the issues with the XML, but I'm not sure : all Schema validators I could find seem to be confused by it (not expecting <testcase> in <testsuite> (!), and the only thing that didn't was xsdvi, which transforms schemas to SVG. At least my eyes and my brain-parser seem to think this should be at least be consumed, even if the reporting might not actually be helpful.

@ethomson If you don't mind and can afford it, I'd love to have the opinion of VSTS on that XML.

@ethomson If you don't mind and can afford it, I'd love to have the opinion of VSTS on that XML.

Sorry, I missed this notification - can you just push a new commit to this branch (or rebase, or whatever)? The VSTS CI should pick it up and run with it.

Rebased. Though it seems VSTS wasn't triggered ?

@ethomson No luck there. Is it because it predates the VSTS setup ?

I guess not - maybe the github web hook is configured at PR creation time and since this predates the VSTS CI setup, it will never run? I opened #4778, let's party in that branch instead.