Skip to content

feat: add opt-in singleBranch option to speed up git scm checkout - #9590

Merged
olblak merged 7 commits into
updatecli:mainfrom
kuisathaverat:kuisathaverat-optimize-scm-checkout-depth
Jul 16, 2026
Merged

olblak merged 7 commits into
updatecli:mainfrom
kuisathaverat:kuisathaverat-optimize-scm-checkout-depth

Conversation

@kuisathaverat

Copy link
Copy Markdown
Contributor

Problem

Updatecli pipelines targeting repositories with a very large number of branches, tags, and other refs (hundreds of branches/tags, 350k+ commits) can spend a very long time (measured up to ~19 minutes) just checking out the git scm, even when depth is set to a low value.

Two issues were found in the shared native git implementation used by every git-based scm (github, gitlab, gitea, bitbucket, stash, azuredevops, git) — pkg/plugins/utils/gitgeneric (GoGit, built on go-git):

  1. The initial clone fetches every branch on the remote. git.CloneOptions is built without SingleBranch/ReferenceName, so go-git falls back to its default refspec +refs/heads/*:refs/remotes/<remote>/* (all branches) — depth only limits the commit history per ref, not the number of refs fetched.
  2. A forced, unconditional "mirror" fetch runs after every clone, using RefSpecs: []config.RefSpec{"refs/*:refs/*"}. This matches every ref namespace on the remote — not just branches and tags, but also refs/pull/*, refs/notes/*, and any other custom ref — on every single Updatecli execution, whether the local clone was fresh or reused.

For a repository with hundreds of branches/tags and a long history, this second point in particular is extremely expensive and explains slow checkouts even with depth configured.

Solution

This PR adds a new opt-in singleBranch option to every git-based scm's spec (github, gitlab, gitea, bitbucket, stash, azuredevops, git), following the same pattern as the existing submodules/depth options.

When singleBranch: true is set:

  • The initial clone only fetches the configured branch (SingleBranch: true + ReferenceName on git.CloneOptions), instead of every branch on the remote.
  • The post-clone refs/*:refs/* reconciliation fetch is skipped entirely.

When the option is left unset (default), behavior is 100% unchanged — this is purely opt-in.

Trade-off

Skipping the full ref reconciliation fetch means Updatecli may, in some edge cases, fail to detect an already-published working branch and create a duplicate pull request. This is documented on the new option and is an acceptable trade-off compared to multi-minute checkouts on very large repositories.

Example

scms:
  default:
    kind: github
    spec:
      owner: updatecli
      repository: updatecli
      branch: main
      token: '{{ requiredEnv "GITHUB_TOKEN" }}'
      depth: 1
      singleBranch: true

Impact

Measured on a repository with hundreds of branches/tags and ~350k commits, enabling singleBranch reduced the scm checkout time from ~19 minutes to ~43 seconds.

Validation

  • go build ./...
  • go vet ./pkg/plugins/scms/... ./pkg/plugins/utils/gitgeneric/...
  • go test ./pkg/plugins/scms/... ./pkg/plugins/utils/gitgeneric/... — all pass (one pre-existing, unrelated failure in TestGoGit_RemoteURLs confirmed present on main as well, due to the local worktree environment)

Repositories with a very large number of branches, tags, and other refs
(e.g. hundreds of branches/tags and hundreds of thousands of commits) can
make Updatecli's git scm checkout very slow, even when a shallow `depth`
is configured. There are two contributing factors in
`pkg/plugins/utils/gitgeneric` (GoGit, shared by the github, gitlab,
gitea, bitbucket, stash, azuredevops and git scms):

- The initial `git.PlainClone` fetches every branch on the remote by
  default (go-git's default clone refspec is
  `+refs/heads/*:refs/remotes/<remote>/*`), regardless of `depth`.
- Right after cloning, an unconditional fetch is issued with
  `RefSpecs: []config.RefSpec{"refs/*:refs/*"}`, which mirrors every ref
  on the remote (branches, tags, pull request refs, notes, etc.) on every
  single Updatecli execution.

This change adds a new opt-in `singleBranch` spec field. When enabled:
- the clone only fetches the configured branch
  (`SingleBranch`+`ReferenceName` on `git.CloneOptions`), and
- the post-clone `refs/*:refs/*` reconciliation fetch is skipped
  entirely.

Default behavior (flag unset) is unchanged. As a documented trade-off,
enabling `singleBranch` skips the full ref reconciliation, which in some
edge cases can make Updatecli miss an already published working branch
and create a duplicate pull request.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
olblak
olblak previously approved these changes Jul 15, 2026

@olblak olblak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the great pull request, covering all scm provider plus the documentation.
I assume you tested it so I am happy to merge as is

@kuisathaverat

Copy link
Copy Markdown
Contributor Author

I made a test with the repository that hit the issue, we moved from 19 minutes to process the pipeline to 43 seconds

@mergify

mergify Bot commented Jul 15, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

kuisathaverat and others added 2 commits July 15, 2026 12:28
The typos CI check fails on main (e.g. PR updatecli#9565) due to a pre-existing
spelling typo unrelated to the singleBranch scm change. Fixing it here
so this PR's CI can pass cleanly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread pkg/plugins/resources/shell/changeIf.go
Comment thread pkg/plugins/resources/shell/main.go
@kuisathaverat

Copy link
Copy Markdown
Contributor Author

There are some issues with the Check Spell, but they are in files that I did not touch, so I will not fix them. As a reference, this is the fix 6183807. I will revert now.

@kuisathaverat
kuisathaverat requested a review from olblak July 15, 2026 10:35
@olblak

olblak commented Jul 15, 2026

Copy link
Copy Markdown
Member

It now complains because of this pullrequest that I merged this morning https://github.com/updatecli/updatecli/pull/9565/checks

I can fix it in a different pr

@olblak olblak mentioned this pull request Jul 15, 2026
1 task
@olblak
olblak merged commit 6b3fff2 into updatecli:main Jul 16, 2026
8 checks passed
@olblak olblak added enhancement New feature or request scm-github SCM of type GiHhub scm-git SCM of kind "Git" scm-gittea scm-gitlab scm-stash scm-bitbucket All things related to the bitbucket integration scm-githubsearch scm-gitlabsearch scm-azuredevops labels Jul 22, 2026
olblak added a commit to updatecli/website that referenced this pull request Aug 6, 2026
Adds a new 'Performance on large repositories' subsection to each scm
plugin doc (github, gitlab, gitea, bitbucket, stash, azuredevops, git)
explaining the new singleBranch option (paired with depth) that lets
Updatecli skip fetching every branch, tag, and ref on the remote,
along with a runnable example for each plugin.

Related to updatecli/updatecli#9590.

Co-authored-by: kuisathaverat <kuisathaverat@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Olblak <me@olblak.com>
Co-authored-by: Olivier Vernin <olivier@vernin.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request scm-azuredevops scm-bitbucket All things related to the bitbucket integration scm-git SCM of kind "Git" scm-github SCM of type GiHhub scm-githubsearch scm-gitlab scm-gitlabsearch scm-gittea scm-stash

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants