Skip to content

feat: report a skipped check as skipped, not as passed - #537

Merged
shenxianpeng merged 2 commits into
mainfrom
claude/add-skipped-status-602anc
Aug 7, 2026
Merged

feat: report a skipped check as skipped, not as passed#537
shenxianpeng merged 2 commits into
mainfrom
claude/add-skipped-status-602anc

Conversation

@shenxianpeng

@shenxianpeng shenxianpeng commented Aug 7, 2026

Copy link
Copy Markdown
Member

The problem

A rule that never ran was reported as a pass.

commit-check-action#258 is the visible cost — every check rendered as a green tick under "All 5 checks passed", when nothing had been validated. That PR's author is dependabot[bot], which the org config lists in ignore_authors:

Commit message
  ✔ PR title
  ✔ Commit 1/1
Branch
  ✔ Branch

Measured on that exact case before this change, every rule came back:

{ "rule_id": "CC001", "check": "message", "status": "pass", "value": "" }

The empty value was the only trace that a skip had happened — and an incidental one, not a signal anybody could rely on. A bypassed policy was indistinguishable from an enforced one in the JSON, in the Python API, and in anything rendering them.

What changed

Adds ValidationResult.SKIP, returned from the guards that already make this decision — _should_skip_commit_validation, _should_skip_branch_validation, and the ignored-author branch of _validate_author. Those helpers are named for skipping; they were simply reporting the outcome as PASS.

validate_all_detailed maps SKIP to "skip" and forces value empty, since a rule that did not run examined nothing.

Overall status is "skip" only when every check skipped. One real verdict still yields "pass" or "fail". Only "fail" is an error, so exit codes are unchanged for existing callers and code branching on status == "fail" keeps working.

A duplicated rule, and a latent exit-code bug

The overall-status rule existed in two places — the CLI's --format json and the Python API. That is exactly how the CLI kept printing "pass" for a fully skipped run after the API had been fixed, so it now lives once in engine.overall_status() and both call it.

That also fixes a bug that was waiting to happen. The CLI ended with:

return 0 if overall == "pass" else 1

which would have turned a skipped run into a failing exit code the moment a third status existed. It is now 1 if overall == "fail" else 0.

Verification

End to end in a repository shaped like #258 — same repo, same config, only the author differing:

author overall per-check exit
dependabot[bot] skip all skip 0
a human pass all pass, values reported 0
a human, bad message fail 1
  • 518 passed, ruff clean and formatted.
  • The twelve existing tests that asserted PASS on these paths are all named for skipping (..._ignored_author, ..._skips_validation, ..._skip_conditions); they now assert SKIP.
  • Four new API tests, including a control that changes only the author — without it the skip test would still pass if the rules had quietly stopped running for everyone. Reverting the skip reporting turns the first of them red.
  • The new README example was captured from a live run and compared against the committed text with json.loads, matching the transcript standard set in docs: refresh README sample output to match what commit-check prints #535.

One pre-existing failure is unrelated and untouched: config_test.py::test_load_config_file_permission_error fails on a clean checkout of main in this environment too, because the suite runs as root and a permission-denied path cannot be provoked.

Follow-up

The action still renders skip as a tick, since it only knows pass/fail. That side is a separate PR against commit-check-action.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn


Generated by Claude Code

Summary by CodeRabbit

  • New Features

    • Added skip statuses for overall validation results and individual checks.
    • Ignored or bypassed checks now appear as skipped without checked values.
    • Runs with only skipped checks report skip; failures take precedence when present.
    • Skipped runs now return a successful exit code.
  • Documentation

    • Updated the API schema and examples to explain skipped checks, all-skipped runs, exit codes, and ignored authors.

A rule that never ran was reported as a pass. commit-check-action#258 is
the visible cost: every check on it rendered as a green tick and the
summary announced "All 5 checks passed", when in fact nothing had been
validated -- the author is dependabot[bot], which the org config lists in
ignore_authors. A bypassed policy was indistinguishable from an enforced
one, in the JSON, in the Python API, and in anything rendering them.

Measured on that exact case before the change: every rule came back
"status": "pass" with "value": "", the empty value being the only trace
that a skip had happened, and an incidental one at that.

Adds ValidationResult.SKIP and returns it from the guards that already
decide this -- _should_skip_commit_validation, _should_skip_branch_
validation, and the ignored-author branch of _validate_author. Those
helpers are named for skipping; they were simply reporting it as PASS.
validate_all_detailed maps SKIP to "skip" and forces the value empty,
since a rule that did not run examined nothing.

Overall status is "skip" only when every check skipped; one real verdict
still yields "pass" or "fail". Only "fail" is an error, so the exit code
is unchanged for existing callers and code branching on status == "fail"
keeps working.

The overall-status rule was duplicated between the CLI's --format json
and the Python API, which is how the CLI kept printing "pass" for a fully
skipped run after the API had been fixed. It now lives once, in
engine.overall_status(), used by both. That also fixes a latent bug in
the CLI's exit code: `0 if overall == "pass" else 1` would have turned a
skipped run into a failure.

Verified end to end in a repository shaped like #258 -- same repo, same
config, only the author differing:

    dependabot[bot]  -> overall "skip", every check "skip", exit 0
    a human          -> overall "pass", values reported,    exit 0
    a human, bad msg -> exit 1

The twelve existing tests that asserted PASS on these paths are all named
for skipping (ignored_author, skips_validation, skip_conditions); they now
assert SKIP. Four new API tests pin the behaviour, including a control
that only the author differs so the skip test cannot pass by the rules
having quietly stopped running for everyone. Reverting the skip reporting
turns the first of them red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
@shenxianpeng
shenxianpeng requested a review from a team as a code owner August 7, 2026 10:40
@github-actions github-actions Bot added the enhancement New feature or request label Aug 7, 2026
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@shenxianpeng, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 55 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f25dceca-bc80-4fc4-95ed-ccbaaa172f83

📥 Commits

Reviewing files that changed from the base of the PR and between dcea4d3 and 7710f0d.

📒 Files selected for processing (5)
  • commit_check/api.py
  • commit_check/engine.py
  • commit_check/main.py
  • tests/api_test.py
  • tests/engine_test.py
📝 Walkthrough

Walkthrough

The validation engine now distinguishes skipped checks from passing checks. API and JSON results preserve skip status, skipped checks have empty values, failures take precedence, and skipped JSON runs exit successfully.

Changes

Skipped validation status

Layer / File(s) Summary
Engine skip semantics
commit_check/engine.py, tests/engine_test.py
The engine adds ValidationResult.SKIP, aggregates failure and skip outcomes, and reports bypassed validators as skipped with empty values. Engine tests update ignored-author and bypass expectations.
Result and exit-status integration
commit_check/api.py, commit_check/main.py, README.md
API and JSON execution use shared aggregate status calculation. Schemas and documentation include skip. Skipped JSON results return exit code zero.
Skipped-status validation coverage
tests/api_test.py
API tests cover all-skipped results, empty skipped values, normal passing values, and failure precedence.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant JSONCaller
  participant commit_check_main
  participant commit_check_engine
  JSONCaller->>commit_check_main: run JSON validation
  commit_check_main->>commit_check_engine: overall_status(outcomes)
  commit_check_engine-->>commit_check_main: fail, skip, or pass
  commit_check_main-->>JSONCaller: JSON result and exit code
Loading

Possibly related PRs

Suggested labels: documentation, tests

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: reporting skipped checks as skipped instead of passed.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/add-skipped-status-602anc

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.97%. Comparing base (90c5abe) to head (7710f0d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #537      +/-   ##
==========================================
+ Coverage   97.72%   97.97%   +0.25%     
==========================================
  Files          12       12              
  Lines        1272     1281       +9     
==========================================
+ Hits         1243     1255      +12     
+ Misses         29       26       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@commit_check/api.py`:
- Around line 64-69: Update the combined-result paths in validate_author and
validate_all to use the shared _build_result reducer instead of independently
defaulting to "pass", preserving "skip" when all nested checks skip. Add
coverage for all-skipped calls to both validate_author(name=..., email=...) and
validate_all(...).
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5c0625d5-1288-4280-828a-e1566490519c

📥 Commits

Reviewing files that changed from the base of the PR and between 90c5abe and dcea4d3.

📒 Files selected for processing (6)
  • README.md
  • commit_check/api.py
  • commit_check/engine.py
  • commit_check/main.py
  • tests/api_test.py
  • tests/engine_test.py

Comment thread commit_check/api.py
Review caught a third and fourth copy of the reduce-to-overall rule that
the first commit missed. validate_author(name=..., email=...) merges two
separate runs, and validate_all() merges up to three; each combined its
checks with a private `"fail" if any(...) else "pass"`, so a call in which
every nested check skipped still reported "pass" -- the exact defect the
skip status exists to prevent, surviving in the two entry points most
likely to be called by automation.

overall_status() now takes plain status strings rather than CheckOutcome
objects, which is what lets every caller share it: the CLI, _build_result,
and both combined paths, which hold already-serialised dicts. Four copies
of this rule is how it drifted in the first place, so there is now one.

Both new tests fail if the per-path rule is restored.

Also covers the two skip branches codecov flagged, in BodyValidator and in
CommitTypeValidator's non-ignore_authors path. Each comes with a control
that changes only the author, so neither can pass by the rule having
quietly stopped running for everyone. No line added by this PR is left
uncovered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

@codspeed-hq

codspeed-hq Bot commented Aug 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 439 untouched benchmarks
🆕 10 new benchmarks
⏩ 121 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 test_a_failure_still_outranks_a_skip N/A 3.4 ms N/A
🆕 test_ignored_author_reports_skip_not_pass N/A 3.3 ms N/A
🆕 test_same_message_from_a_listed_author_still_passes N/A 3.4 ms N/A
🆕 test_skipped_checks_carry_no_value N/A 3.3 ms N/A
🆕 test_combined_author_call_preserves_skip N/A 3.6 ms N/A
🆕 test_validate_all_preserves_skip N/A 6.4 ms N/A
🆕 test_body_validator_skips_ignored_author N/A 2.3 ms N/A
🆕 test_body_validator_still_runs_for_other_authors N/A 2.6 ms N/A
🆕 test_commit_type_rule_skips_ignored_author N/A 2.3 ms N/A
🆕 test_commit_type_rule_still_fails_for_other_authors N/A 2.6 ms N/A

Comparing claude/add-skipped-status-602anc (7710f0d) with main (08d14c1)2

Open in CodSpeed

Footnotes

  1. 121 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on main (90c5abe) during the generation of this report, so 08d14c1 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@shenxianpeng
shenxianpeng merged commit 16d328f into main Aug 7, 2026
28 checks passed
@shenxianpeng
shenxianpeng deleted the claude/add-skipped-status-602anc branch August 7, 2026 11:07
shenxianpeng added a commit that referenced this pull request Aug 7, 2026
The pin was v2.13.0, whose requirements.txt installs commit-check 2.13.1.
That engine cannot resolve a rebase target existing only as origin/main in
a pull request checkout, so it reported "not rebased" for a branch that
was -- the false CC202 this workflow has carried since its first run.

v2.13.1 of the action installs commit-check 2.13.4, which carries the
merge-base fix from #532. Confirmed by reading requirements.txt at the tag
rather than assuming the action version tracks the engine version:

    v2.13.0 (124de73) -> commit-check==2.13.1
    v2.13.1 (562a184) -> commit-check==2.13.4

The branch was already rebased before this change -- git merge-base
--is-ancestor origin/main HEAD returned true while CC202 still failed --
so the failure was the engine, not the branch, and rebasing again could
never have fixed it.

Two things ride along on the newer engine. Skipped checks now report as
skipped rather than as passes (#537), so a run bypassed by ignore_authors
says so instead of showing green ticks over nothing. And the imperative
whitelist goes from 396 verbs to 529, retiring a class of false CC003 --
2.13.1 rejected "treat", which #527 had added three releases earlier.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant