worktree: copy-on-write creation and shared-branch worktrees#2317
worktree: copy-on-write creation and shared-branch worktrees#2317nevion wants to merge 2 commits into
Conversation
Welcome to GitGitGadgetHi @nevion, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests. Please make sure that either:
You can CC potential reviewers by adding a footer to the PR description with the following syntax: NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description, Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:
It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code. Contributing the patchesBefore you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form Both the person who commented An alternative is the channel Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment If you want to see what email(s) would be sent for a After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail). If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the curl -g --user "<EMailAddress>:<Password>" \
--url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txtTo iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description): To send a new iteration, just add another PR comment with the contents: Need help?New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join. You may also be able to find help in real time in the developer IRC channel, |
|
this feature is used/reimplemented within grok build https://x.com/theskory/status/2059729539287167068 (I have nothing to do with that) but the best place for this to be done is inside git so all tools can benefit. |
|
Benchmark — Linux-kernel fork (93k tracked files, ~33 GB working tree incl. build artifacts, btrfs):
Left: real disk written vs. number of worktrees (normal grows ~1.3 GB each; |
Creating many worktrees from the same base -- for example to run a
fleet of automated agents in parallel -- is expensive today: every
"git worktree add" materializes the entire working tree by writing
each tracked file out from the object store. The objects are shared
via the common directory, but the working tree is not: N worktrees
mean N full checkouts on disk and N times the file I/O.
Add a "--reflink" option that, on copy-on-write filesystems, populates
the new worktree by reflinking the current worktree's files and index
instead. The subsequent "git reset --hard" then only rewrites the
paths that actually differ between the current worktree and
<commit-ish>; everything else (including untracked files such as build
outputs) keeps sharing storage with the source until modified. Because
the cloned index still carries the source files' stat data, it is
refreshed against the reflinked files first so that reset recognizes
the unchanged paths as up to date and leaves them sharing extents
rather than rewriting them.
The clones are made by a new reflink_file() helper in copy.c, which
uses the FICLONE ioctl on Linux and clonefile() on macOS and reports
an error otherwise so callers fall back to a normal copy. Support is
probed up front; when unavailable -- including on filesystems without
copy-on-write and on platforms such as Windows that lack a reflink
primitive -- "--reflink" transparently falls back to an ordinary
checkout, so the worst case is no slower than today rather than a
byte-for-byte copy of the source tree. The directory walk skips the
new worktree itself when it lives inside the source one, and preserves
symlinks and modes.
The behavior can be made the default with the worktree.reflink
configuration ("true", "false" or "auto", the last suppressing the
unsupported-filesystem warning), and turned off per-invocation with
--no-reflink. A configured default degrades quietly in modes that
cannot reflink (--orphan, --no-checkout) instead of erroring, so
enabling it never breaks those commands. The checkout step continues
to honor checkout.workers, so parallel checkout composes with
--reflink for the paths that do need rewriting.
Signed-off-by: Jason Newton <nevion@gmail.com>
When spinning up several worktrees on the same checkout for parallel work (for example a fleet of agents working from one branch), git's refusal to check out a branch that is already checked out elsewhere is just in the way. The restriction exists to stop two worktrees from moving the same branch underneath each other, but plain parallel checkouts do not need that protection. Drop the restriction: "git worktree add <branch>" now checks out a branch even if it is in use by another worktree. The genuinely dangerous case is kept -- a branch that another worktree is in the middle of rebasing or bisecting is still refused, because a second checkout could corrupt that operation. die_if_branch_busy() performs that narrower check in place of the old die_if_checked_out(). The separate guard against force-updating (e.g. with -B) a branch in use elsewhere is left untouched. Signed-off-by: Jason Newton <nevion@gmail.com>
4e56589 to
97f0fda
Compare
|
The non-Linux CI failures here are unrelated to this change (which only touches
Every job that actually exercises this change (all |
|
/allow |
|
Yes, sadly Git's CI is not always in a pristine shape :-( |
|
User nevion is now allowed to use GitGitGadget. |
|
@dscho do I do anything else? |
|
Why, yes, you |
|
/preview |
|
Preview email sent as pull.2317.git.git.1780684356.gitgitgadget@gmail.com |
|
/submit |
|
Submitted as pull.2317.git.git.1780685368.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
"brian m. carlson" wrote on the Git mailing list (how to reply to this email): On 2026-06-05 at 18:49:26, Jason Newton via GitGitGadget wrote:
> When many worktrees share one repository -- e .g. a fleet of agents each
> needing an isolated checkout -- "git worktree add" is costly at scale.
> Objects are shared via the common dir, but the working tree is not: each add
> rewrites every tracked file, so N worktrees cost N full checkouts of disk
> and I/O. And a branch can only be checked out in one worktree.
>
> Patch 1 adds "git worktree add --reflink": on a copy-on-write filesystem it
> populates the new worktree by reflinking the current worktree's files and
> index, then "git reset --hard" rewrites only the paths that differ from . A
> reflink_file() helper in copy.c uses FICLONE (Linux) and clonefile()
> (macOS); elsewhere (other filesystems, Windows) it is probed up front and
> falls back to a normal checkout. Defaulting is via the worktree.reflink
> config (true/false/auto); --no-reflink overrides.
Windows apparently has CoW functionality if you use ReFS. I believe Git
LFS has code to do this and you may be interested in checking it out.
Also, how does this work if worktree A is dirty (but `git update-index`
and `git status` have not been run) when the reflink occurs? Does B
have stale files from the working tree? If not, how do we plan on
detecting that? (While I'm curious, this should also be explained in
your commit message because we want to know that you have thought about
this problem and have a good answer for it.)
I was curious as to how this would work with containers, which typically
use overlayfs, but some searching reveals that overlayfs does indeed
support reflinks. Thanks for the opportunity to learn something new
today.
> Patch 2 lets a branch be checked out in several worktrees, for parallel work
> on one checkout. A branch mid-rebase or mid-bisect elsewhere is still
> refused.
So how does this work if you have two worktrees for the same branch, A
and B, and A commits, and then B does? What we don't want to happen is
that because B's worktree is not up to date, it effectively reverts the
changes that A made when adding objects to the index to commit. (Again,
this is a good thing to explain in your commit message, since reviewers
will be curious.)
My personal approach, if I needed many worktrees of the same commit,
would be to create many refs pointing to the same object ID and check
those out. `git update-ref` can perform a single ref transaction with
many refs, which is especially efficient with reftable. That would
avoid the need for multiple checkout support, although I could still see
the utility of reflinking if it can be done safely. If that's a
solution that you think would be valuable, you could propose it as a FAQ
entry or an edit to the manual page, since I'm sure there are other
people with your use case.
--
brian m. carlson (they/them)
Toronto, Ontario, CA |
|
User |
|
Junio C Hamano wrote on the Git mailing list (how to reply to this email): "Jason Newton via GitGitGadget" <gitgitgadget@gmail.com> writes:
> When many worktrees share one repository -- e .g. a fleet of agents each
> needing an isolated checkout -- "git worktree add" is costly at scale.
> Objects are shared via the common dir, but the working tree is not: each add
> rewrites every tracked file, so N worktrees cost N full checkouts of disk
> and I/O.
Are the "CoW" semantics offered by these underlying mechanisms,
which may differ per operating system and possibly filesystem type,
all meant as mere storage-space optimization, or do some of them
trade potential space saving with some limitation of the features,
i.e., what you can do in the CoW copy and original, or increased
runtime cost, either at the clone time or the time of first
modification?
What I am trying to get at is why should this be even an opt-in
feature. If "cp treeA treeB" at the shell level would make all the
files in treeA under identical names and contents in treeB, and let
you edit/update/delete copies in either tree without affecting the
other tree, then in practice you would not even be able to _tell_ if
CoW is in use, no?
It may tilt the scale if there is a downside associated with the use
of CoW, like at the first modification of one copy, the system may
need to do real copies of other copies, but even such a cost should
not be outrageously worse than the cost of copying everything once
at the worktree creation time.
So I would understand "whenever we say git_copy_file(A, B), we
always use CoW facility under the hood if available, regardless of
the purpose of the operation to copy one file to another location---
it may include, but does not have to be limited to, populating
working file trees in a new worktree", and I think it is a welcome
change.
But I do not quite get "... only if the user gives --reflink
option". Why is it even necessary to offer a choice? Especially
since you seem to have auto-probe, we should be able to implement a
low-level operation to materialize contents identified by a_hash at
a_path in the working tree in two different ways, switching on the
availablity of CoW, e.g.,
if (CoW available && we can find existing path with a_hash) {
copy-cow the found path to a_path;
} else {
write object identified by a_hash to a_path;
}
> And a branch can only be checked out in one worktree.
This is a safety feature that has nothing to do with shared files
across worktrees, no? Your two worktrees may think they have a
checkout of the same branch (thus the same commit), one worktree
makes changes and commits, the other worktree suddenly starts seeing
a totally different output from its "git diff HEAD" that mixes what
it did relative to where it started (which is what we want) plus the
reversion of what was done in the other worktree (which is definitely
not what we want). |
When many worktrees share one repository -- e .g. a fleet of agents each
needing an isolated checkout -- "git worktree add" is costly at scale.
Objects are shared via the common dir, but the working tree is not: each
add rewrites every tracked file, so N worktrees cost N full checkouts of
disk and I/O. And a branch can only be checked out in one worktree.
Patch 1 adds "git worktree add --reflink": on a copy-on-write filesystem
it populates the new worktree by reflinking the current worktree's files
and index, then "git reset --hard" rewrites only the paths that differ
from . A reflink_file() helper in copy.c uses FICLONE (Linux)
and clonefile() (macOS); elsewhere (other filesystems, Windows) it is
probed up front and falls back to a normal checkout. Defaulting is via
the worktree.reflink config (true/false/auto); --no-reflink overrides.
Patch 2 lets a branch be checked out in several worktrees, for parallel
work on one checkout. A branch mid-rebase or mid-bisect elsewhere is
still refused.
Benchmark (Linux-kernel fork, 93k files, ~33 GB tree incl. build output,
btrfs): a normal add allocates ~0.9 GB of real disk per worktree (~5.3 GB
for four, linear); --reflink allocates ~0 at any count and also carries
the untracked build tree. ("Real disk" = btrfs exclusive bytes.)
Note: patch 2 changes a default (same-branch checkout now allowed); two
t2400 assertions were updated accordingly.
cc: "brian m. carlson" sandals@crustytoothpaste.net