Skip to content

feat: quarantine debt CI summary — surface suppressed tests in every run #5

Description

@EmanueleMinotto

Opportunity

After every run, Skipper knows exactly which tests it suppressed, why, and until when. Today that information goes nowhere. The opportunity: surface a quarantine debt score in every CI run — making the cost of hiding tests visible and creating accountability pressure that no other tool in this space provides.

Proposed feature

Emit a structured quarantine report at the end of every JUnit run (via a TestExecutionListener / Jupiter extension), targeting two outputs:

  • GitHub Actions job summary (via GITHUB_STEP_SUMMARY)
  • A local skipper-report.json artifact for archiving and downstream tooling

The report includes: number of currently suppressed tests, tests expiring this week, tests re-enabled in this run, and a total quarantine-days of debt score (sum of disabledUntil - today across all active rows).

void emitSummary(QuarantineReport report) throws IOException {
    String summaryFile = System.getenv("GITHUB_STEP_SUMMARY");
    String md = buildMarkdownSummary(report);
    if (summaryFile != null) {
        Files.writeString(Path.of(summaryFile), md, StandardOpenOption.APPEND);
    } else {
        System.out.println(md);
    }
    Files.writeString(Path.of("skipper-report.json"), toJson(report));
}

Wire the emission into the existing Skipper listener/extension so it runs automatically under Gradle and Maven without extra config.

Why this is the unique moat

The quarantine-days of debt number grows when tests are disabled and shrinks when they expire. A team with 200 quarantine-days of debt is hiding a lot of reality. That number in every PR comment is a forcing function that costs zero additional tooling — and no other tool in this space does it.

Acceptance criteria

  • After every test run, a markdown summary is written to GITHUB_STEP_SUMMARY if the env var is set
  • If GITHUB_STEP_SUMMARY is not set, the summary is printed to stdout
  • skipper-report.json is written with the full suppression state of the run
  • Report includes: currently suppressed count, expiring this week, re-enabled this run, quarantine-days of debt
  • Quarantine-days of debt = sum of (disabledUntil - today) in days across all active disabled rows
  • Works under both Gradle and Maven without additional configuration

Effort estimate

~150 lines in the core module, no new dependencies (use java.nio.file + existing JSON lib).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions