fix(ci): generate changelog from release tag, not from main - #2530
Conversation
The generate-changelog job in .github/workflows/tags.yaml previously checked out main and ran the AI agent there. The agent followed docs/agents/changelog.md, which instructed it to compute the release range with `git log <previous_version>..HEAD`. For minor releases (vX.Y.0) cut from main this happens to give the right answer; for patch releases (vX.Y.Z, Z>0) cut from a release-X.Y branch, HEAD-on- main is a strict superset of the release tag and contains commits that were merged to main both before and after the tag. The v1.3.1 changelog generated by this workflow (#2480) demonstrated the failure mode: it included 8 PRs merged to main but never shipped in v1.3.1, 6 backport PRs that landed on release-1.3 *after* v1.3.1 was tagged, both originals and their backports as separate entries, a hallucinated 2024 PR, and the cozystack-ci bot in the contributors list. The corrected v1.3.1 changelog contains the single fix that v1.3.1 actually ships. This PR fixes the root cause and tightens the agent guardrails: * tags.yaml: check out the release tag commit (`ref: ${{ steps.tag. outputs.tag }}`) instead of `main`, so HEAD == the release commit and `git log v<prev>..HEAD` corresponds to the actual release. The Create-changelog-branch step still creates the PR branch from origin/main, so PRs continue to merge cleanly into main. * tags.yaml: update the AI prompt to state explicitly that HEAD is the release commit and that the upper bound for the release range is the new tag, not main. * docs/agents/changelog.md: replace `..HEAD` with `..v<new_version>` in every example so the instruction is unambiguous regardless of which branch is checked out. * docs/agents/changelog.md: add a hard rule that backport PRs MUST be merged into the original entry (single line, two PR numbers), never listed as a second entry. Document the edge case where the original PR is outside the range — drop the entry. * docs/agents/changelog.md: forbid using the brief description verbatim as the detailed description, and forbid inventing entries for PRs that are not in the release range. * docs/agents/changelog.md: filter bot/CI accounts (`app/*`, `*[bot]`, cozystack-ci, github-actions, dependabot, renovate) out of the human Contributors list. Bot attribution on individual entries is still valid. The corresponding correction for the v1.3.1 changelog itself is in PR #2480 on the changelog-v1.3.1 branch. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses inaccuracies in the automated changelog generation process for patch releases. By shifting the reference point from the main branch to the specific release tag, the CI workflow now correctly scopes the commit history. Additionally, the documentation has been updated with stricter guidelines and improved filtering logic to ensure higher quality, human-focused changelogs. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe changelog workflow now checks out the pushed release tag commit (instead of Changes
Sequence Diagram(s)sequenceDiagram
participant GH as GitHub Actions
participant Repo as Git (local workspace)
participant Origin as origin/main (remote)
participant Agent as Copilot/Changelog Agent
GH->>Repo: checkout tag ref (v${VERSION})
GH->>Origin: git fetch origin main
GH->>Origin: git cat-file -e refs/heads/main:docs/changelogs/v${VERSION}.md?
alt file exists on origin/main
GH->>GH: abort changelog generation
else file missing
GH->>Agent: run changelog generation (use v<prev>..v<new>)
Agent->>Repo: compute diffs (upper bound = v${VERSION}, never HEAD)
Agent->>GH: produce changelog file
GH->>Origin: commit & push changelog
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 7/8 reviews remaining, refill in 7 minutes and 30 seconds.Comment |
There was a problem hiding this comment.
Code Review
This pull request updates the changelog generation guidelines to improve accuracy and clarity. Key changes include mandating the use of release tags instead of HEAD for commit logs, requiring the consolidation of backport PRs into single entries, and implementing a filter to exclude bot accounts from the contributors list. Review feedback identifies a logic error in the backport exclusion rule—specifically that backports should be included if the original PR has not yet shipped in a stable release—and suggests standardizing placeholder naming and prefixes in the command examples for better consistency.
| - Include both original and backport PR numbers in the changelog entry (e.g., `#1606, #1609`) | ||
| - **MUST** combine the original and backport PR into a **single changelog entry** with both PR numbers (e.g. `#1606, backport #1609`). NEVER list the original and the backport as two separate entries — that produces visible duplicates and was the primary defect that motivated this rule. | ||
| - To find original PR from backport: Check the backport PR description or commit message for "Backport of #ORIGINAL_PR" | ||
| - If a release range contains a backport but its original PR is **not** in the same range (because the original was merged earlier and already shipped in a previous release), drop the entry entirely — it has already been changelog'd. |
There was a problem hiding this comment.
This rule contains a logic error that could lead to missing entries in patch release changelogs. The assumption that an original PR "already shipped" if it is not in the current range is only true if the original PR is in the history of the previous tag (e.g., v1.3.0). If the original PR is on main (a future release), it has not shipped yet in a stable release. In such cases, the backport in the current patch release (e.g., v1.3.1) is the first time the change is being delivered to users and must be included in the changelog. The rule should be refined to only drop the entry if the original PR is an ancestor of the previous release tag.
There was a problem hiding this comment.
Good catch. The rule was too aggressive — a feature backported to a release branch before the original ships in any stable tag would be the first user-visible delivery, and dropping it would lose the entry. Removed the bullet entirely in 0f9198d rather than try to encode the "is the original an ancestor of the previous release tag" check in the agent prompt; combining originals with backports into a single entry (the rule above) already covers the duplicate-entry case that motivated this section.
| # Get all commits including merge commits (backports) | ||
| git log <previous_version>..HEAD --pretty=format:"%h - %s (%an, %ar)" | ||
| # Get all commits including merge commits (backports), bounded by the new tag | ||
| git log <previous_version>..v<new_version> --pretty=format:"%h - %s (%an, %ar)" |
There was a problem hiding this comment.
The placeholder <previous_version> is used here without a v prefix, whereas the example in line 120 and the command for minor releases in line 136 both use the v prefix. For consistency and to avoid confusion for the AI agent, it is better to include the v prefix explicitly in the command template, especially since other parts of the document (e.g., line 178) also use v<previous_version>.
| git log <previous_version>..v<new_version> --pretty=format:"%h - %s (%an, %ar)" | |
| git log v<previous_version>..v<new_version> --pretty=format:"%h - %s (%an, %ar)" |
There was a problem hiding this comment.
Applied in 0f9198d. The four git log invocations in the doc (lines 115, 136, 277, 298) now all use the v prefix on placeholders, matching the rest of the doc (lines 178, 221, 482, 488, 577).
| # Extract all GitHub usernames from the current release changelog, | ||
| # filtering out CI/bot accounts that should not appear as human contributors. | ||
| # This method is simpler and more reliable than extracting from git history. | ||
| grep -oE '\[\*\*@[a-zA-Z0-9_/-]+\*\*\]' docs/changelogs/v<version>.md | \ |
There was a problem hiding this comment.
The placeholder <version> is used here, but <new_version> is used consistently in the rest of the document (lines 115, 136, 524). Using a consistent placeholder name helps the AI agent understand which value to substitute.
| grep -oE '\[\*\*@[a-zA-Z0-9_/-]+\*\*\]' docs/changelogs/v<version>.md | \ | |
| grep -oE '\[\*\*@[a-zA-Z0-9_/-]+\*\*\]' docs/changelogs/v<new_version>.md | \ |
There was a problem hiding this comment.
Step 8 (the contributors-list section) consistently uses <version> as the placeholder for the changelog file being generated — see lines 524, 528, and the existing docs/changelogs/v<version>.md references. <new_version> is reserved for cases where two tags are being compared (lines 115, 136, 277). Changing only this line would create a local inconsistency with the adjacent commands.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
.github/workflows/tags.yaml (1)
334-335: Keep the prompt tag-only here.The checkout now makes
HEADthe release commit, but mentioningHEADin the prompt reintroduces the ambiguity this patch is trying to remove. Usev${VERSION}as the upper bound explicitly so the generated changelog stays aligned withdocs/agents/changelog.md.Proposed prompt cleanup
- copilot \ - --prompt "Generate the release changelog for tag v${VERSION}. The repository is checked out at the v${VERSION} tag commit, so HEAD == the release commit. Use v${VERSION} (or HEAD) as the upper bound when comparing against the previous version — never assume HEAD is on main, because patch releases are cut from release-X.Y branches. Follow the instructions in `@docs/agents/changelog.md` exactly, including the 'Scope and boundaries' section at the top. Your deliverable is the single file docs/changelogs/v${VERSION}.md — write it and exit; this workflow handles branching, committing, pushing, and opening the PR." \ + copilot \ + --prompt "Generate the release changelog for tag v${VERSION}. The repository is checked out at the v${VERSION} tag commit, so HEAD == the release commit. Use v${VERSION} as the upper bound when comparing against the previous version. Follow the instructions in `@docs/agents/changelog.md` exactly, including the 'Scope and boundaries' section at the top. Your deliverable is the single file docs/changelogs/v${VERSION}.md — write it and exit; this workflow handles branching, committing, pushing, and opening the PR." \🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/tags.yaml around lines 334 - 335, The prompt passed to the changelog generator currently mentions "HEAD" which reintroduces ambiguity; update the --prompt string used in the workflow so it uses v${VERSION} explicitly as the upper bound (remove any reference to HEAD or "HEAD ==") and ensure the text still instructs following docs/agents/changelog.md and producing docs/changelogs/v${VERSION}.md; locate the --prompt invocation in the tags.yaml workflow and replace the clause "The repository is checked out at the v${VERSION} tag commit, so HEAD == the release commit. Use v${VERSION} (or HEAD) as the upper bound..." with wording that only references v${VERSION} as the upper bound.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/agents/changelog.md`:
- Around line 502-512: The "Identify new contributors" pass currently scans the
changelog with an unfiltered `@username` regex and can include bot/CI accounts;
reuse the same bot-filter pipeline used for the Contributors list (the grep -oE/
sed -E/ grep -viE pipeline shown and specifically the bot-exclusion regex
'^(app/|.*\[bot\]$|cozystack-ci$|github-actions$|dependabot$|renovate$)') when
detecting first-time contributors so the same accounts are excluded; update the
new-contributors extraction to pipe through the same sed/grep -viE filter (or
call the same helper/command) so bots are never reported as "New Contributors."
---
Nitpick comments:
In @.github/workflows/tags.yaml:
- Around line 334-335: The prompt passed to the changelog generator currently
mentions "HEAD" which reintroduces ambiguity; update the --prompt string used in
the workflow so it uses v${VERSION} explicitly as the upper bound (remove any
reference to HEAD or "HEAD ==") and ensure the text still instructs following
docs/agents/changelog.md and producing docs/changelogs/v${VERSION}.md; locate
the --prompt invocation in the tags.yaml workflow and replace the clause "The
repository is checked out at the v${VERSION} tag commit, so HEAD == the release
commit. Use v${VERSION} (or HEAD) as the upper bound..." with wording that only
references v${VERSION} as the upper bound.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 597886a5-db82-4867-8e75-87e53635ee09
📒 Files selected for processing (2)
.github/workflows/tags.yamldocs/agents/changelog.md
IvanHunters
left a comment
There was a problem hiding this comment.
Correct root cause: HEAD-on-main is a strict superset of any patch tag, so git log v<prev>..HEAD running on main includes commits that never shipped in the release. Checking out the tag commit fixes this at the source. The "Create changelog branch" step still branches from origin/main, so PRs continue to merge cleanly — good separation of concerns.
One unannounced but welcome fix: the old contributor-extraction grep matched [@username] but the actual changelog format uses [**@username**], so the old pattern never matched anything. The new pattern \[\*\*@[a-zA-Z0-9_/-]+\*\*\] is correct. The bot-filter regex is also correct — \[bot\] escapes the brackets properly in ERE, so renovate[bot] is caught by .*\[bot\]$.
LGTM
- Drop the "orphan backport, drop entry" rule: a backport whose original is on main but hasn't shipped in a stable tag is the first delivery and must stay in the changelog. - Add `v` prefix to `<previous_version>`/`<new_version>` placeholders in all four `git log` invocations for consistency with the rest of the doc. - Apply the bot/CI account filter to the new-contributors detection (and the standalone previous-contributors extraction) so bot accounts can no longer leak into the human Contributors list. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/agents/changelog.md`:
- Around line 529-533: The pipeline building /tmp/all_previous_contributors.txt
currently greps across docs/changelogs/v*.md and therefore includes the current
release file; update the pipeline that builds the contributors list to
explicitly exclude the current release file (the docs/changelogs/v<version>.md
for this run) before extracting usernames — e.g., after listing
docs/changelogs/v*.md, filter out the current release filename (use grep -v
"$CURRENT_RELEASE_FILE" or remove basename via sed) so that the subsequent
grep/sed/sort -u that writes /tmp/all_previous_contributors.txt does not contain
contributors from the file being released; ensure the same exclusion is applied
where comm -23 is used to detect first-time contributors.
- Around line 106-107: Documentation and the tags workflow disagree about
whether to allow HEAD for comparing releases: update the
.github/workflows/tags.yaml workflow prompt (the usage of "v${VERSION} (or
HEAD)") to match the docs' strict guidance of using the release tag only (e.g.,
"v${VERSION}") so Copilot/CI will never default to HEAD; locate the prompt text
in tags.yaml and replace the ambiguous "or HEAD" phrasing with explicit
instructions to only use the release tag (v${VERSION}) and add a short
clarifying comment if needed to mirror the language in docs/agents/changelog.md.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: f4068cd5-92f9-4ff9-a9d1-47e857b5e497
📒 Files selected for processing (1)
docs/agents/changelog.md
- tags.yaml: drop "(or HEAD)" from the changelog prompt; the doc says "never HEAD" and the workflow now matches. - docs/agents/changelog.md: exclude the current release file from the "previous contributors" set used for first-time-contributor detection. The previous globbing on `docs/changelogs/v*.md` swept up the file being generated, so `comm -23` would never report any newcomers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/tags.yaml:
- Line 334: The CI prompt in .github/workflows/tags.yaml (the --prompt string
that references docs/agents/changelog.md) conflicts with the changelog doc's
Step 2 requiring branch "main" while this job runs detached at tag v${VERSION};
update that prompt to include an explicit CI override that states the workflow
is running on a detached tag and must NOT attempt to switch branches or require
main, and instruct the agent to use the checked-out tag v${VERSION} as the
single source-of-truth/upper bound for comparisons; edit the prompt string (the
--prompt value) to clearly say "CI_OVERRIDE: running on detached tag
v${VERSION}; do not switch branches or require main; use v${VERSION} as upper
bound" so the agent follows the doc's formatting rules but skips branch
validation.
- Around line 289-299: The "Checkout release tag" step (actions/checkout@v4 with
ref: ${{ steps.tag.outputs.tag }}) causes the subsequent check_changelog step to
run against the immutable tag tree so it cannot detect a changelog already
merged on origin/main; change the flow so check_changelog runs against the
branch state on origin/main (either by moving the tag checkout after
check_changelog, or by checking out origin/main (or running git fetch origin
main and pointing check_changelog at origin/main) before invoking
check_changelog), and keep the later "Create changelog branch" step branching
from origin/main as intended.
In `@docs/agents/changelog.md`:
- Around line 106-107: Step 6 still uses HEAD for the release window
(RELEASE_END=$(git log -1 --format=%ai HEAD)), which contradicts the "never use
HEAD" rule; update both Step 6 snippets to compute RELEASE_END from the release
tag variable (the tag you already pass/compute for the release, e.g.,
RELEASE_TAG or RELEASE) instead of HEAD so the end of the release window is
based on the explicit tag (use the tag variable in the git command that sets
RELEASE_END).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: b0986df4-0b92-4e7d-8a1f-bbd28256ea49
📒 Files selected for processing (2)
.github/workflows/tags.yamldocs/agents/changelog.md
… gate
- tags.yaml: the check_changelog step now looks at origin/main rather than
the working tree. After Round 2 we check out the release tag, which never
has the changelog file (the PR opened by this job merges to main after
the tag is cut), so the previous `[ -f ]` test always returned false and
the gate was effectively dead on reruns.
- changelog.md: Step 6's `RELEASE_END=$(git log -1 --format=%ai HEAD)`
(lines 179, 222) was missed in the previous "never HEAD" sweep — now
uses v<new_version>, matching Step 5.
- changelog.md: Step 2 ("on main branch") was incompatible with the CI flow
which now runs detached at the release tag. Relaxed it to list both
configurations (interactive on main; CI detached at v<version>), while
the "Scope and boundaries" section continues to forbid the agent from
switching branches.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin release-1.3
git worktree add -d .worktree/backport-2530-to-release-1.3 origin/release-1.3
cd .worktree/backport-2530-to-release-1.3
git switch --create backport-2530-to-release-1.3
git cherry-pick -x 2420d8949bb4cebc5035be8ad90724ed7eeb3948 0f9198d5108690f510f16ee78d07af93f74e103f ce63bfcdb0839fdcf3a5ef5dade2910d6f428130 39605f213f85a160ae0e639e84b08fbd03d4e108 |
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin release-1.2
git worktree add -d .worktree/backport-2530-to-release-1.2 origin/release-1.2
cd .worktree/backport-2530-to-release-1.2
git switch --create backport-2530-to-release-1.2
git cherry-pick -x 2420d8949bb4cebc5035be8ad90724ed7eeb3948 0f9198d5108690f510f16ee78d07af93f74e103f ce63bfcdb0839fdcf3a5ef5dade2910d6f428130 39605f213f85a160ae0e639e84b08fbd03d4e108 |
What this PR does
The
generate-changelogjob in.github/workflows/tags.yamlpreviously checked outmainand ran the AI agent there. The agent followeddocs/agents/changelog.md, which instructed it to compute the release range withgit log <previous_version>..HEAD. That works for minor releases (cut frommain), but it breaks for any patch release cut from arelease-X.Ybranch —HEAD-on-mainis a strict superset of the tag and contains commits that were merged tomainboth before and after the tag.The v1.3.1 changelog generated by this workflow (#2480) demonstrated the failure mode: 8 PRs that were merged to
mainbut never shipped in v1.3.1, 6 backport PRs that landed onrelease-1.3after v1.3.1 was tagged, both originals and their backports as separate entries, a hallucinated 2024 PR (#435), and thecozystack-cibot in the contributors list. The corrected v1.3.1 changelog is in #2480.This PR fixes the root cause and tightens the agent guardrails:
tags.yaml— check out the release tag commit (ref: ${{ steps.tag.outputs.tag }}) instead ofmain, soHEAD == release commitandgit log v<prev>..HEADcorresponds to what the tag actually contains. The "Create changelog branch" step still creates the PR branch fromorigin/main, so PRs continue to merge cleanly.tags.yaml— the AI prompt now states explicitly thatHEADis the release commit and that the upper bound of the range is the new tag, nevermain.docs/agents/changelog.md— every example replaces..HEADwith..v<new_version>, so the instruction is unambiguous regardless of which branch is checked out.docs/agents/changelog.md— hard rule: backport PRs MUST be combined with the original into a single entry (#1606, backport #1609), never listed as a second entry. Documented edge case: if the original isn't in the range, drop the entry entirely (it shipped in a previous release).docs/agents/changelog.md— forbid using the brief description verbatim as the detailed description (the failure was* **fix(foo): X**: fix(foo): X (...)), and forbid inventing entries for PRs outside the release range.docs/agents/changelog.md— filter bot/CI accounts (app/*,*[bot],cozystack-ci,github-actions,dependabot,renovate) out of the human Contributors list. Bot attribution on individual entries is still valid.The v1.3.1 changelog correction itself lives in #2480 (separate scope).
Release note
Summary by CodeRabbit
Chores
Documentation