test: stop four JSON tests taking their verdict from the checkout - #548
Conversation
main went red on the push run for #542 with four failures in TestJsonFormat, all reading 'skip' where they assert 'pass'. Nothing regressed: the tests had been measuring the repository they run in. Each of the four supplies a message, on stdin or in a file, which makes it a prospective commit -- so _resolve_current_author reads `git config user.name` and falls back to HEAD's author. Both are ambient. A GitHub runner configures no git identity (no workflow here sets one), so the fallback always decides, and #542 was a dependabot merge: HEAD's author was dependabot[bot], which cchk.toml lists in [commit] ignore_authors. Every commit check skipped, and overall status is 'skip' when they all do. Reproduced against a clone pinned to 9f12a63 with the global and system git config disabled, which is what the runner looks like: HEAD author : dependabot[bot] user.name : '' -> status 'skip' (the four failures) user.name : set -> status 'pass' (why laptops and PRs were green) So it was never about #542's contents, and it will recur on the next bot-authored merge to main. The four now take a `pinned_author` fixture that fixes both identity sources, leaving the verdict to come from the message under test. The two tests in the class that already passed are the two that happened to patch get_commit_info for other reasons -- the same pin, arrived at by accident. Pinning it in a fixture would hide the fallback everywhere it applies, so it is now asserted directly instead: a new test drives an unconfigured identity with a bot as HEAD's author and expects every check to skip, exit code still 0. What silently decided other tests' results is now a contract of its own. Verified in that clone: 4 failed before, 7 passed after, and the full suite is unchanged in both a configured and an unconfigured environment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #548 +/- ##
=======================================
Coverage 98.13% 98.13%
=======================================
Files 12 12
Lines 1337 1337
=======================================
Hits 1312 1312
Misses 25 25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Warning Review limit reached
Next review available in: 8 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Merging this PR will improve performance by 64.13%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | test_json_format_from_file |
21.1 ms | 11.4 ms | +85.19% |
| ⚡ | test_json_format_valid_message_returns_pass |
23.7 ms | 13.9 ms | +69.94% |
| ⚡ | test_json_format_pass_reports_checked_value |
23.8 ms | 14 ms | +69.58% |
| ⚡ | test_json_format_exit_code_matches_status |
38.8 ms | 28.6 ms | +35.97% |
| 🆕 | test_json_format_skips_when_head_author_is_ignored |
N/A | 17.7 ms | N/A |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/refresh-sample-output-602anc (9cde256) with main (423916a)2
Footnotes
-
121 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
main(9f12a63) during the generation of this report, so 423916a was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩



Fixes the four
TestJsonFormatfailures on the push run for #542, all readingskipwhere they assertpass.Nothing regressed — the tests were measuring the repository they run in
Each of the four supplies a message (on stdin or in a file), which makes it a prospective commit.
_resolve_current_authorthen does:Both sources are ambient. A GitHub runner configures no git identity — no workflow in this repository sets one — so the fallback always decides, and #542 was a dependabot merge:
HEAD's author wasdependabot[bot], whichcchk.tomllists in[commit] ignore_authors. Every commit check skipped, and overall status isskipwhen they all do.Reproduced
A clone pinned to
9f12a63with the global and system git config disabled, which is what the runner looks like:So this was never about #542's contents, and it will recur on the next bot-authored merge to
main— dependabot, pre-commit-ci, copilot or coderabbitai.The fix
The four tests take a new
pinned_authorfixture that fixes both identity sources, so the verdict comes from the message under test. Worth noting: the two tests in the class that kept passing are exactly the two that already patchedget_commit_infofor unrelated reasons — the same pin, arrived at by accident.A fixture that pins this would also hide the fallback everywhere it applies, so the behaviour is now asserted directly rather than left ambient: a new test drives an unconfigured identity with a bot as
HEAD's author and expects every check to skip, exit code still0. What silently decided other tests' results is now a contract of its own.Checks
In the reproduction clone: 4 failed before, 7 passed after. Full suite unchanged in both a configured and an unconfigured environment (
581 passed; the one unrelated failure,test_load_config_file_permission_error, fails only because my sandbox runs as root, wherechmod 000does not deny access — it fails identically on unmodifiedmainand passes in CI).Not changed
Whether the fallback should apply to
ignore_authorsfor a prospective message is a separate design question — the author of the previous commit is arguably irrelevant to a message not yet written. Changing it would alter real behaviour (a bot's PR checked with-mon a runner with no git identity would stop being exempt), so this PR only stops the tests depending on it and leaves the call to you.Generated by Claude Code