* feat: report a skipped check as skipped, not as passed
A rule that never ran was reported as a pass. commit-check-action#258 is
the visible cost: every check on it rendered as a green tick and the
summary announced "All 5 checks passed", when in fact nothing had been
validated -- the author is dependabot[bot], which the org config lists in
ignore_authors. A bypassed policy was indistinguishable from an enforced
one, in the JSON, in the Python API, and in anything rendering them.
Measured on that exact case before the change: every rule came back
"status": "pass" with "value": "", the empty value being the only trace
that a skip had happened, and an incidental one at that.
Adds ValidationResult.SKIP and returns it from the guards that already
decide this -- _should_skip_commit_validation, _should_skip_branch_
validation, and the ignored-author branch of _validate_author. Those
helpers are named for skipping; they were simply reporting it as PASS.
validate_all_detailed maps SKIP to "skip" and forces the value empty,
since a rule that did not run examined nothing.
Overall status is "skip" only when every check skipped; one real verdict
still yields "pass" or "fail". Only "fail" is an error, so the exit code
is unchanged for existing callers and code branching on status == "fail"
keeps working.
The overall-status rule was duplicated between the CLI's --format json
and the Python API, which is how the CLI kept printing "pass" for a fully
skipped run after the API had been fixed. It now lives once, in
engine.overall_status(), used by both. That also fixes a latent bug in
the CLI's exit code: `0 if overall == "pass" else 1` would have turned a
skipped run into a failure.
Verified end to end in a repository shaped like #258 -- same repo, same
config, only the author differing:
dependabot[bot] -> overall "skip", every check "skip", exit 0
a human -> overall "pass", values reported, exit 0
a human, bad msg -> exit 1
The twelve existing tests that asserted PASS on these paths are all named
for skipping (ignored_author, skips_validation, skip_conditions); they now
assert SKIP. Four new API tests pin the behaviour, including a control
that only the author differs so the skip test cannot pass by the rules
having quietly stopped running for everyone. Reverting the skip reporting
turns the first of them red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
* fix: preserve skip when the API merges checks from several runs
Review caught a third and fourth copy of the reduce-to-overall rule that
the first commit missed. validate_author(name=..., email=...) merges two
separate runs, and validate_all() merges up to three; each combined its
checks with a private `"fail" if any(...) else "pass"`, so a call in which
every nested check skipped still reported "pass" -- the exact defect the
skip status exists to prevent, surviving in the two entry points most
likely to be called by automation.
overall_status() now takes plain status strings rather than CheckOutcome
objects, which is what lets every caller share it: the CLI, _build_result,
and both combined paths, which hold already-serialised dicts. Four copies
of this rule is how it drifted in the first place, so there is now one.
Both new tests fail if the per-path rule is restored.
Also covers the two skip branches codecov flagged, in BodyValidator and in
CommitTypeValidator's non-ignore_authors path. Each comes with a control
that changes only the author, so neither can pass by the rule having
quietly stopped running for everyone. No line added by this PR is left
uncovered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9zFxq8V4qxG4aMzJhGBFn
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>