Hello! I've recently started running mypy with --junit-xml since our CI system supports parsing JUnit output (and it's been a really useful feature!).
Currently, mypy's JUnit output lumps together all of the error output into one "testcase", like so:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="mypy" skips="0" tests="1" time="0.826">
<testcase classname="mypy" file="mypy" line="1" name="mypy" time="0.826">
<failure message="mypy produced messages">testmodule/bar.py:2: error: The return type of "__init__" must be None
testmodule/bar.py:2: error: Missing return statement
testmodule/foo.py:4: error: Self argument missing for a non-static method (or an invalid type for self)
testmodule/foo.py:5: error: Incompatible return value type (got "str", expected "int")</failure>
</testcase>
</testsuite>Would anyone have any objection to breaking this output up into 1 testcase per file? I imagine it would look something like this:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="mypy" skips="0" tests="3" time="0.826">
<testcase classname="testmodule.bar" file="testmodule/bar.py" line="0" name="testmodule.bar">
<failure message="mypy produced messages">testmodule/bar.py:2: error: The return type of "__init__" must be None
testmodule/bar.py:2: error: Missing return statement</failure>
</testcase>
<testcase classname="testmodule.foo" file="testmodule/foo.py" line="0" name="testmodule.foo">
<failure message="mypy produced messages">testmodule/foo.py:4: error: Self argument missing for a non-static method (or an invalid type for self)
testmodule/foo.py:5: error: Incompatible return value type (got "str", expected "int")</failure>
</testcase>
<testcase classname="testmodule.file_with_no_issues" file="testmodule/file_with_no_issues.py" line="0" name="testmodule.file_with_no_issues"></testcase>
</testsuite>There are a few benefits to splitting it up by file:
- CI systems that support the JUnit format can nicely format each error message per file, making it easier for developers to see which files need fixing
- CI systems that collect and aggregate metrics about which test cases fail most often / how many tests are run can have more detailed information to play with
- We can account for files that don't have typing issues (like
testmodule/file_with_no_issues.py in the example above).
As an alternative, I considered having each typing error surface as a failed test case, but that makes the total number of tests and the number of failing tests vary wildly, making it difficult for CI systems that try to aggregate test case data.
I would love to implement this, just wanted to make sure no one had any objections to this before starting on it!
For implementation, I'm thinking of pulling in https://github.com/kyrus/python-junit-xml to actually generate the XML.
Hello! I've recently started running mypy with
--junit-xmlsince our CI system supports parsing JUnit output (and it's been a really useful feature!).Currently, mypy's JUnit output lumps together all of the error output into one "testcase", like so:
Would anyone have any objection to breaking this output up into 1 testcase per file? I imagine it would look something like this:
There are a few benefits to splitting it up by file:
testmodule/file_with_no_issues.pyin the example above).As an alternative, I considered having each typing error surface as a failed test case, but that makes the total number of tests and the number of failing tests vary wildly, making it difficult for CI systems that try to aggregate test case data.
I would love to implement this, just wanted to make sure no one had any objections to this before starting on it!
For implementation, I'm thinking of pulling in https://github.com/kyrus/python-junit-xml to actually generate the XML.