Collapse spam triage into the agentic issue-triage workflow - #14027

Fixes #14024

Description

gh models was retired on 2026-07-30. The Spam Issue Detection workflow is built on gh models run, so it has been failing on every new issue since.

Porting it as-is would have preserved a worse problem. Three workflows currently fire on issues: opened and interfere with each other:

issues:opened ─┬─ Spam Issue Detection ────── comment + [suspected-spam, invalid] + close   (BROKEN)
               ├─ Issue Triaging ─┬─ label-incoming ──── +needs-triage
               │                  └─ close-single-word ─ comment + labels=[invalid] + close
               └─ Issue Triage (skills-driven) ── suggests labels + comments

• Two bots comment on the same issue. Noey #13974 and x #14020 each got two closure comments seconds apart, one from each closer.
• Labels get clobbered. close-single-word calls issues.update({labels: ['invalid']}), which replaces the label set rather than adding to it. x #14010 lost suspected-spam; probe #14023 lost needs-triage; Noey #13974 kept it. Which labels survive depends on which workflow finishes first.
• The capable classifier never gets to act. Issue Triage (skills-driven) takes ~4 minutes; the deterministic closers finish in ~10 seconds. On probe #14023 the agent fetched the issue 7s after it had already been closed and emitted noop.

This collapses all of it into one path:

issues:opened ─┬─ Issue Triaging ── label-incoming ── +needs-triage
               └─ Issue Triage (skills-driven) ── applies suspected-spam (no comment)
                                                        │
                                    issues:labeled ─────┘
                                                        ▼
                                     close-suspected-spam (shared)
                                     comment + remove needs-triage + close

One comment, one closer, one label writer.

issue-triage.md was already the right home: it loads the issue-classifier skill, whose decision tree opens with "Spam, gibberish, or AI-generated slop", and suspected-spam is already in its add-labels allowlist. What it lacked was the cli/cli-specific spam criteria, which lived in generate-sys-prompt.sh. Those move to shared/spam-criteria.md and reach the agent through gh-aw's imports: key, so they land in the prompt on every run with no tool call.

invalid deliberately stays suggestion-only, because it routes to a different shared workflow that closes with no comment at all.

How did you test this change?

The evals are kept, as #14024 asks. The 273-case corpus carries over verbatim; only the runner changed, from gh models eval to copilot -p.

$ ./.github/workflows/scripts/spam-detection/eval.sh
running 274 cases on gpt-5-mini (effort low, concurrency 8)

cases            274
correct          256 (93.4%)
false positives  1  (legitimate issue judged spam)
false negatives  17  (spam issue judged legitimate)
duration         376s

False positives and negatives are reported separately rather than as one accuracy figure, because closing a real report is the costlier error.

Judged by disagreement set, not pass rate. A re-run of an unchanged prompt moved the aggregate by 0.7 points, so a headline number cannot distinguish a small real change from noise. eval.sh -d before.json,after.json lists which specific cases moved and in which direction.

Old prompt vs. the new criteria file: 256/273 both, 2 cases flipped - at the same-prompt noise floor. The port does not change behaviour.

Compilation and lint:

$ gh aw compile .github/workflows/issue-triage.md
✓ issue-triage.md
$ shellcheck .github/workflows/scripts/spam-detection/eval.sh   # clean
$ ./.github/workflows/scripts/spam-detection/eval.sh -V
FAIL  73
PASS  201
total 274

Not yet verified, and it needs to be before this is considered done: that a label applied by the workflow's GitHub App token fires an issues: labeled event which triggers close-suspected-spam. App installation tokens do trigger workflows (unlike GITHUB_TOKEN) and the shared job's if: condition matches, but issue-triage.md has never applied a label, so there is no direct evidence in this repo, and cross-workflow chaining cannot be tested from a branch. If it does not chain, spam gets labelled but not closed - visible and recoverable.

Key points

The gh --help injection was removed on evidence, not taste. generate-sys-prompt.sh appended the live stdout of gh --help to the prompt. None of the 12 spam indicators reference command knowledge, so I tested whether anything consumed it, using a same-prompt re-run as the noise control:

Re-running an unchanged prompt moved the aggregate more than deleting the section did. It is inert, so it is gone: 3085 bytes, 37% of the prompt.

That also removes a reproducibility hazard. gh --help renders EXTENSION COMMANDS and ALIAS COMMANDS from the invoking user's setup, so the old eval put whatever a developer had installed into the prompt.

A false positive is fixed. #13783 ("missing installation instructions for Amazon Linux 2023", body: a bare PR link) was labelled suspected-spam + invalid by the old classifier despite being a genuine report with a linked fix. It is now in the corpus as a PASS case, sitting deliberately close to the existing spam, a hyperlink FAIL case; the difference is whether the link is coherent with the title.

Issue templates are inlined into spam-criteria.md rather than read at runtime, because a spam indicator refers to them directly. That introduces drift if .github/ISSUE_TEMPLATE/*.md change; they are three files totalling 2775 bytes and change rarely, and the file names its source directory.

Spam will now be closed in ~4 minutes rather than ~10 seconds, since the agent replaces the deterministic closers. That is the price of removing the label-clobbering race, and it seems worth paying.

spam-criteria.md is deliberately role-neutral - it describes what spam looks like and says nothing about what to do about it. The workflow acts on it by applying a label and the eval by emitting a verdict, so an output contract in the shared file would force one consumer to contradict it. The eval-only contract lives in eval-instructions.md.

Notes for reviewers

Start with shared/spam-criteria.md (the content being preserved) and the Step 5 carve-out in issue-triage.md (the behaviour change). eval.sh is a tool, not production code, and can be skimmed.

The one frontmatter subtlety: under issue-intent, omitting suggest applies a label directly while suggest: true makes it a proposal. So the carve-out is a prompt change, not a frontmatter change - suspected-spam is emitted without suggest, everything else keeps it.

Related:

• Update spam detection workflow to use "gh aw" #14024 - the issue this fixes, which explicitly asks that the evals be kept.
• Issue triage agent is told to add gh-* command labels it cannot apply #14026 - filed while working on this: label-taxonomy.md tells the agent to add a command label, but no gh-* label is in the add-labels allowlist, so it cannot. Pre-existing and out of scope here.
• triage-detect-spam.yml is broken: gh models was retired on 2026-07-30 desktop/gh-cli-and-desktop-shared-workflows#36 - that repo has its own copy of the same retired gh models scripts and is equally broken. Different repo, so not fixed here.

Authorship and follow-up

Who wrote this:

• A human wrote it.
• An agent wrote it under close human direction.
• An agent wrote it independently, and no human has guided the implementation beyond the initial prompt.

Who answers review comments:

• @williammartin will read and reply directly.
• An agent will draft replies and @username will read them before they are posted.
• Nobody has explicitly committed to replying.