Skip to content

test: stop four JSON tests taking their verdict from the checkout - #548

Merged
shenxianpeng merged 1 commit into
mainfrom
claude/refresh-sample-output-602anc
Aug 13, 2026
Merged

test: stop four JSON tests taking their verdict from the checkout#548
shenxianpeng merged 1 commit into
mainfrom
claude/refresh-sample-output-602anc

Conversation

@shenxianpeng

Copy link
Copy Markdown
Member

Fixes the four TestJsonFormat failures on the push run for #542, all reading skip where they assert pass.

Nothing regressed — the tests were measuring the repository they run in

Each of the four supplies a message (on stdin or in a file), which makes it a prospective commit. _resolve_current_author then does:

if context.stdin_text is not None or context.commit_file is not None:
    return get_git_config_value("user.name") or get_commit_info("an")

Both sources are ambient. A GitHub runner configures no git identity — no workflow in this repository sets one — so the fallback always decides, and #542 was a dependabot merge: HEAD's author was dependabot[bot], which cchk.toml lists in [commit] ignore_authors. Every commit check skipped, and overall status is skip when they all do.

Reproduced

A clone pinned to 9f12a63 with the global and system git config disabled, which is what the runner looks like:

HEAD author : dependabot[bot]
user.name   : ''        ->  status: skip     (the four failures, exactly)
user.name   : set       ->  status: pass     (why laptops and PR runs were green)

So this was never about #542's contents, and it will recur on the next bot-authored merge to main — dependabot, pre-commit-ci, copilot or coderabbitai.

The fix

The four tests take a new pinned_author fixture that fixes both identity sources, so the verdict comes from the message under test. Worth noting: the two tests in the class that kept passing are exactly the two that already patched get_commit_info for unrelated reasons — the same pin, arrived at by accident.

A fixture that pins this would also hide the fallback everywhere it applies, so the behaviour is now asserted directly rather than left ambient: a new test drives an unconfigured identity with a bot as HEAD's author and expects every check to skip, exit code still 0. What silently decided other tests' results is now a contract of its own.

Checks

In the reproduction clone: 4 failed before, 7 passed after. Full suite unchanged in both a configured and an unconfigured environment (581 passed; the one unrelated failure, test_load_config_file_permission_error, fails only because my sandbox runs as root, where chmod 000 does not deny access — it fails identically on unmodified main and passes in CI).

Not changed

Whether the fallback should apply to ignore_authors for a prospective message is a separate design question — the author of the previous commit is arguably irrelevant to a message not yet written. Changing it would alter real behaviour (a bot's PR checked with -m on a runner with no git identity would stop being exempt), so this PR only stops the tests depending on it and leaves the call to you.


Generated by Claude Code

main went red on the push run for #542 with four failures in
TestJsonFormat, all reading 'skip' where they assert 'pass'. Nothing
regressed: the tests had been measuring the repository they run in.

Each of the four supplies a message, on stdin or in a file, which makes
it a prospective commit -- so _resolve_current_author reads
`git config user.name` and falls back to HEAD's author. Both are
ambient. A GitHub runner configures no git identity (no workflow here
sets one), so the fallback always decides, and #542 was a dependabot
merge: HEAD's author was dependabot[bot], which cchk.toml lists in
[commit] ignore_authors. Every commit check skipped, and overall status
is 'skip' when they all do.

Reproduced against a clone pinned to 9f12a63 with the global and system
git config disabled, which is what the runner looks like:

    HEAD author : dependabot[bot]
    user.name   : ''        -> status 'skip'   (the four failures)
    user.name   : set       -> status 'pass'   (why laptops and PRs were green)

So it was never about #542's contents, and it will recur on the next
bot-authored merge to main.

The four now take a `pinned_author` fixture that fixes both identity
sources, leaving the verdict to come from the message under test. The
two tests in the class that already passed are the two that happened to
patch get_commit_info for other reasons -- the same pin, arrived at by
accident.

Pinning it in a fixture would hide the fallback everywhere it applies,
so it is now asserted directly instead: a new test drives an
unconfigured identity with a bot as HEAD's author and expects every
check to skip, exit code still 0. What silently decided other tests'
results is now a contract of its own.

Verified in that clone: 4 failed before, 7 passed after, and the full
suite is unchanged in both a configured and an unconfigured environment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
@shenxianpeng
shenxianpeng requested a review from a team as a code owner August 13, 2026 07:05
@github-actions github-actions Bot added the tests Add test related changes label Aug 13, 2026
@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.13%. Comparing base (15a853a) to head (9cde256).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #548   +/-   ##
=======================================
  Coverage   98.13%   98.13%           
=======================================
  Files          12       12           
  Lines        1337     1337           
=======================================
  Hits         1312     1312           
  Misses         25       25           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@shenxianpeng, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 35d2e0e0-31f1-46a7-907f-0f6b909c2b1b

📥 Commits

Reviewing files that changed from the base of the PR and between 9f12a63 and 9cde256.

📒 Files selected for processing (1)
  • tests/main_test.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@shenxianpeng
shenxianpeng merged commit 60d7586 into main Aug 13, 2026
28 checks passed
@shenxianpeng
shenxianpeng deleted the claude/refresh-sample-output-602anc branch August 13, 2026 07:10
@codspeed-hq

codspeed-hq Bot commented Aug 13, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 64.13%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
✅ 485 untouched benchmarks
🆕 1 new benchmark
⏩ 121 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
test_json_format_from_file 21.1 ms 11.4 ms +85.19%
test_json_format_valid_message_returns_pass 23.7 ms 13.9 ms +69.94%
test_json_format_pass_reports_checked_value 23.8 ms 14 ms +69.58%
test_json_format_exit_code_matches_status 38.8 ms 28.6 ms +35.97%
🆕 test_json_format_skips_when_head_author_is_ignored N/A 17.7 ms N/A

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/refresh-sample-output-602anc (9cde256) with main (423916a)2

Open in CodSpeed

Footnotes

  1. 121 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on main (9f12a63) during the generation of this report, so 423916a was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Add test related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant