rerere: keep a background gc from killing a rebase - #2214
Open
thomasbachem wants to merge 1 commit into
Open
Conversation
Welcome to GitGitGadgetHi @thomasbachem, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests. Please make sure that either:
You can CC potential reviewers by adding a footer to the PR description with the following syntax: NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description, Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:
It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code. Contributing the patchesBefore you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form Both the person who commented An alternative is the channel Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment If you want to see what email(s) would be sent for a After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail). If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the curl -g --user "<EMailAddress>:<Password>" \
--url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txtTo iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description): To send a new iteration, just add another PR comment with the contents: Need help?New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join. You may also be able to find help in real time in the developer IRC channel, |
Member
|
/allow |
|
User thomasbachem is now allowed to use GitGitGadget. |
Author
|
/preview |
|
Preview email sent as pull.2214.git.1788337239398.gitgitgadget@gmail.com |
Author
|
/submit |
|
Submitted as pull.2214.git.1788337897490.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
Phillip Wood wrote on the Git mailing list (how to reply to this email): Hi Thomas
On 02/09/2026 09:31, Thomas Bachem via GitGitGadget wrote:
> From: Thomas Bachem <mail@thomasbachem.com>
> > Since 2.54 unscheduled maintenance uses the "geometric" strategy, so
That change really is the gift that keeps on giving
> the "git maintenance run --auto --detach" behind every "git commit"
> runs "git rerere gc" in the background whenever rr-cache has an entry.
> That includes the "git commit" the sequencer runs for a resolved pick
> on "git rebase --continue".
> > rerere_gc() takes MERGE_RR.lock through setup_rerere(), which uses
> LOCK_DIE_ON_ERROR, and so does the sequencer's repo_rerere() at the
> next conflict a few milliseconds later. Whichever comes second dies.
To me this is another reason why we should disable gc.auto while rebasing. To do that we need to pass "-c gc.auto=false -c maintenance.auto=false" when running "git commit" in run_git_commit() and also when running "git merge" in do_merge(). We should also pass those settings via GIT_CONFIG_PARAMETERS when running a exec command in do_exec(). That is largly papering over the cracks but until we have a systematic solution it does at least stop exposing users to this bug.
> When it is the rebase, it dies in do_pick_commit() That's a bug us well - we should be returning errors, not dying -rerere_setup() should be returning an error, so we can clean up and reschedule the pick.
There is a lot of detail here about what causes the problem which is helpful, but there is very little discussion about the fix. As I understand it we now block the sequencer until the background maintenance has completed, or continue to die in an inconvenient state we timeout before the background maintenance finishes. That seems rather unfortunate as the idea of running the maintenance in the background is to prevent it from interfering with other commands.
I think my preferred solution is to disable gc while rebasing. Returning an error from rerere_setup() would also help in the case where the user runs "git commit" and then continues the rebase. I'd be interested to hear what Junio and Patrick think about that. I'm also not clear why gc.auto has to fork a separate process just to check if it needs to run or not, I've not been following closely but my impression is that that is the cause of quite a lot of the lock contention bugs we've seen.
Thanks
Phillip
> with the index
> written but before make_patch() writes rebase-merge/{message,patch,
> stopped-sha}, and every later "git rebase --continue" refuses with
> "you have staged changes in your working tree". When it is the "git
> commit" of a later continue, that one dies in its post-commit
> repo_rerere() after the commit was made. Before 2.54 the same
> collision needed an auto gc to actually run, since gc runs
> "rerere gc" at its end.
> > A rebase with two conflicts in a row shows it. The filler makes the
> pick slower than the ~5 ms the background task needs to take the
> lock, and keeps the lock held for about 0.4 s. It hit 6 of 6 runs
> here on 2.55.0, and a test suite driving rebases on toy repositories
> with a single rr-cache entry hit it in both runs that were traced:
> > git init -q -b main r && cd r
> git config rerere.enabled true
> git config maintenance.auto false
> mkdir pad && seq 20000 | (cd pad && split -l 1 -a 5)
> echo base >f && git add -A && git commit -qm base
> git checkout -q -b topic
> echo b >f && git commit -qam B
> echo c >f && git commit -qam C
> git checkout -q main
> echo a >f && git commit -qam A
> git repack -adq
> seq 20000 | awk '{printf ".git/rr-cache/%040x\n", $1}' \
> | xargs mkdir -p
> for d in .git/rr-cache/*/; do echo x >$d/preimage; done
> git config --unset maintenance.auto
> git checkout -q topic
> git rebase main
> echo ab >f && git add f
> GIT_EDITOR=true git rebase --continue
> > The second continue dies with "Unable to create '.git/MERGE_RR.lock':
> File exists" while the gc spawned by its own commit holds the lock,
> and after resolving C every further continue refuses. Maintenance
> stays off during the setup so that no repack is pending: a repack due
> at that commit runs ahead of rerere-gc in the task list and would
> spend the window.
> > The gc needs the lock: it removes every rr-cache directory it finds
> empty, and a rerere that has just created its directory but not yet
> written the preimage looks exactly like that. So keep the lock and fix
> both orders. When the gc finds the lock busy, let it warn and do
> nothing this time, the way "maintenance run" treats its own lock, so a
> manual "git rerere gc" sees the warning and the maintenance task and
> "git gc" see a clean exit. When the gc holds the lock, let every other
> caller wait it out instead of dying at once, for rerere.lockTimeout
> milliseconds with the semantics of core.packedRefsTimeout: 1000 by
> default, 0 for the old behaviour, -1 for an unbounded wait. Walking a
> 20000-entry rr-cache takes about 0.4 s here.
> > That rebase now completes. The tests cover the gc under a held lock,
> directly and through the maintenance task, a merge that waits a lock
> out within a five second rerere.lockTimeout, and one that fails at
> once with a timeout of 0.
> > Assisted-by: Claude Fable 5.1
> Signed-off-by: Thomas Bachem <mail@thomasbachem.com>
> ---
> rerere: keep a background gc from killing a rebase
> > Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2214%2Fthomasbachem%2Frerere-gc-lock-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2214/thomasbachem/rerere-gc-lock-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/2214
> > Documentation/config/rerere.adoc | 8 +++++++
> Documentation/git-rerere.adoc | 4 +++-
> rerere.c | 27 +++++++++++++++++----
> rerere.h | 1 +
> t/t4200-rerere.sh | 40 ++++++++++++++++++++++++++++++++
> t/t7900-maintenance.sh | 8 +++++++
> 6 files changed, 82 insertions(+), 6 deletions(-)
> > diff --git a/Documentation/config/rerere.adoc b/Documentation/config/rerere.adoc
> index 3a78b5ebb1..8041a1587b 100644
> --- a/Documentation/config/rerere.adoc
> +++ b/Documentation/config/rerere.adoc
> @@ -10,3 +10,11 @@ rerere.enabled::
> enabled if there is an `rr-cache` directory under the
> `$GIT_DIR`, e.g. if "rerere" was previously used in the
> repository.
> +
> +rerere.lockTimeout::
> + The length of time, in milliseconds, to retry when trying to
> + take the rerere lock while another process holds it, typically
> + a background `git rerere gc`. Value 0 means not to retry at
> + all; -1 means to try indefinitely. Default is 1000 (i.e.,
> + retry for 1 second). `git rerere gc` itself does not wait and
> + skips its run instead.
> diff --git a/Documentation/git-rerere.adoc b/Documentation/git-rerere.adoc
> index 4e6ab9a27c..05935b0603 100644
> --- a/Documentation/git-rerere.adoc
> +++ b/Documentation/git-rerere.adoc
> @@ -70,7 +70,9 @@ occurred a long time ago. By default, unresolved conflicts older
> than 15 days and resolved conflicts older than 60
> days are pruned. These defaults are controlled via the
> `gc.rerereUnresolved` and `gc.rerereResolved` configuration
> -variables respectively.
> +variables respectively. If another process holds the lock on the
> +recorded resolutions, for example a merge or rebase that is recording
> +a conflict, `gc` does nothing and reports so.
> > > DISCUSSION
> diff --git a/rerere.c b/rerere.c
> index 8232542585..22d114262b 100644
> --- a/rerere.c
> +++ b/rerere.c
> @@ -32,6 +32,7 @@ static int rerere_enabled = -1;
> > /* automatically update cleanly resolved paths to the index */
> static int rerere_autoupdate;
> +static int rerere_lock_timeout_ms = 1000;
> > #define RR_HAS_POSTIMAGE 1
> #define RR_HAS_PREIMAGE 2
> @@ -876,6 +877,8 @@ static void git_rerere_config(void)
> {
> repo_config_get_bool(the_repository, "rerere.enabled", &rerere_enabled);
> repo_config_get_bool(the_repository, "rerere.autoupdate", &rerere_autoupdate);
> + repo_config_get_int(the_repository, "rerere.locktimeout",
> + &rerere_lock_timeout_ms);
> repo_config(the_repository, git_default_config, NULL);
> }
> > @@ -908,12 +911,26 @@ int setup_rerere(struct repository *r, struct string_list *merge_rr, int flags)
> > if (flags & (RERERE_AUTOUPDATE|RERERE_NOAUTOUPDATE))
> rerere_autoupdate = !!(flags & RERERE_AUTOUPDATE);
> - if (flags & RERERE_READONLY)
> + if (flags & RERERE_READONLY) {
> fd = 0;
> - else
> + } else if (flags & RERERE_SKIP_LOCKED) {
> fd = hold_lock_file_for_update(&write_lock,
> - git_path_merge_rr(r),
> - LOCK_DIE_ON_ERROR);
> + git_path_merge_rr(r), 0);
> + if (fd < 0) {
> + warning_errno(_("unable to lock '%s', skipping"),
> + git_path_merge_rr(r));
> + return -1;
> + }
> + } else {
> + /*
> + * A background "rerere gc" holds the lock for as long as it
> + * takes to walk rr-cache, so wait it out rather than die.
> + */
> + fd = hold_lock_file_for_update_timeout(&write_lock,
> + git_path_merge_rr(r),
> + LOCK_DIE_ON_ERROR,
> + rerere_lock_timeout_ms);
> + }
> read_rr(r, merge_rr);
> return fd;
> }
> @@ -1237,7 +1254,7 @@ void rerere_gc(struct repository *r, struct string_list *rr)
> timestamp_t cutoff_resolve = now - 60 * 86400;
> struct strbuf buf = STRBUF_INIT;
> > - if (setup_rerere(r, rr, 0) < 0)
> + if (setup_rerere(r, rr, RERERE_SKIP_LOCKED) < 0)
> return;
> > repo_config_get_expiry_in_days(the_repository, "gc.rerereresolved",
> diff --git a/rerere.h b/rerere.h
> index d4b5f7c932..87964bb3c5 100644
> --- a/rerere.h
> +++ b/rerere.h
> @@ -10,6 +10,7 @@ struct repository;
> #define RERERE_AUTOUPDATE 01
> #define RERERE_NOAUTOUPDATE 02
> #define RERERE_READONLY 04
> +#define RERERE_SKIP_LOCKED 010
> > /*
> * Marks paths that have been hand-resolved and added to the
> diff --git a/t/t4200-rerere.sh b/t/t4200-rerere.sh
> index 1717f407c8..6b90294435 100755
> --- a/t/t4200-rerere.sh
> +++ b/t/t4200-rerere.sh
> @@ -242,6 +242,46 @@ test_expect_success 'old records rest in peace' '
> test_path_is_missing $rr2/preimage
> '
> > +test_expect_success 'gc does nothing while MERGE_RR is locked' '
> + mkdir -p $rr2 &&
> + echo Hello >$rr2/preimage &&
> + test-tool chmtime =$just_over_15_days_ago $rr2/preimage &&
> +
> + test_when_finished "rm -f .git/MERGE_RR.lock" &&
> + >.git/MERGE_RR.lock &&
> + git rerere gc 2>err &&
> + test_grep "MERGE_RR" err &&
> + test_path_is_file $rr2/preimage &&
> +
> + rm .git/MERGE_RR.lock &&
> + git rerere gc &&
> + test_path_is_missing $rr2/preimage
> +'
> +
> +test_expect_success 'a held lock is waited out within rerere.lockTimeout' '
> + git reset --hard &&
> + rm -rf $rr &&
> + test_when_finished "rm -f .git/MERGE_RR.lock" &&
> + >.git/MERGE_RR.lock &&
> + {
> + (sleep 1 && rm -f .git/MERGE_RR.lock) &
> + } &&
> + test_must_fail git -c rerere.lockTimeout=5000 merge first 2>err &&
> + wait &&
> + test_grep ! "Unable to create" err &&
> + grep "^=======\$" $rr/preimage
> +'
> +
> +test_expect_success 'rerere.lockTimeout=0 fails at once on a held lock' '
> + git reset --hard &&
> + rm -rf $rr &&
> + test_when_finished "rm -f .git/MERGE_RR.lock" &&
> + >.git/MERGE_RR.lock &&
> + test_must_fail git -c rerere.lockTimeout=0 merge first 2>err &&
> + test_grep "Unable to create" err &&
> + test_path_is_missing $rr/preimage
> +'
> +
> rerere_gc_custom_expiry_test () {
> five_days="$1" right_now="$2"
> test_expect_success "rerere gc with custom expiry ($five_days, $right_now)" '
> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
> index d7f82e1bec..a55ca2e829 100755
> --- a/t/t7900-maintenance.sh
> +++ b/t/t7900-maintenance.sh
> @@ -885,6 +885,14 @@ test_expect_success 'rerere-gc task with --auto honors maintenance.rerere-gc.aut
> test_expect_rerere_gc ! git -c maintenance.rerere-gc.auto=0 maintenance run --auto --task=rerere-gc
> '
> > +test_expect_success 'rerere-gc task succeeds while MERGE_RR is locked' '
> + test_when_finished "rm -rf .git/rr-cache .git/MERGE_RR.lock" &&
> + mkdir .git/rr-cache &&
> + : >.git/rr-cache/entry &&
> + >.git/MERGE_RR.lock &&
> + test_expect_rerere_gc git maintenance run --task=rerere-gc
> +'
> +
> test_expect_success '--auto and --schedule incompatible' '
> test_must_fail git maintenance run --auto --schedule=daily 2>err &&
> test_grep "cannot be used together" err
> > base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc |
|
User |
|
Thomas Bachem wrote on the Git mailing list (how to reply to this email): Hi Phillip,
On 02/09/2026 15:27, Phillip Wood wrote:
> To me this is another reason why we should disable gc.auto while
> rebasing. To do that we need to pass "-c gc.auto=false -c
> maintenance.auto=false" when running "git commit" in run_git_commit()
> and also when running "git merge" in do_merge(). We should also pass
> those settings via GIT_CONFIG_PARAMETERS when running a exec command in
> do_exec(). That is largly papering over the cracks but until we have a
> systematic solution it does at least stop exposing users to this bug.
OK, I'll do that. It is also more consistent than it looks: the
commits the sequencer creates in-process via try_to_commit() don't run
auto maintenance at all, only the "git commit" child does (for a
resolved, reworded or squashed commit). What surprised me is that a
rebase with the merge backend then never runs maintenance, not even at
the end, because it doesn't go through finish_rebase() where the apply
backend runs it. Do you want a single run at the end of the sequence
in that patch, or keep it minimal?
FWIW, the tool I hit this with has been setting both for its whole
process tree since, and the failures stopped.
>> When it is the rebase, it dies in do_pick_commit()
>
> That's a bug us well - we should be returning errors, not dying
> -rerere_setup() should be returning an error, so we can clean up and
> reschedule the pick.
Yes. I don't think we even need to reschedule: when repo_rerere() is
called there, the merge result is already in the index and worktree,
the error and advice have been printed, and the return value is
ignored. If setup_rerere() reports the lock and returns -1, the pick
just stops at the conflict like any other, minus rerere's recording
and replay, and --continue works. I went through the callers of
setup_rerere(): all of them handle a negative return, because that is
what a disabled rerere returns, so this is close to a one-branch
change. It also fixes the stale-lock case (crashed process), which
disabling gc can't.
> As I understand it we now block the sequencer until the background
> maintenance has completed, or continue to die in an inconvenient state
> we timeout before the background maintenance finishes. That seems rather
> unfortunate as the idea of running the maintenance in the background is
> to prevent it from interfering with other commands.
Right, that's what it does. I copied the timeout from
core.packedRefsTimeout, but a ref update can't be skipped and a rerere
can, so the wait buys little. I'll drop rerere.lockTimeout.
What it did buy: the gc spawned by the continue's own commit needs
~5ms to take the lock, the next pick usually longer to reach its
rerere, so the gc is normally holding it by then. With only the
gc-side skip my repro still died 3 of 3 times; with the error return
those runs would survive but lose rerere at that stop. Tolerable, but
it is why I'd rather have the sequencer patch in the same series than
leave it for later.
> I think my preferred solution is to disable gc while rebasing. Returning
> an error from rerere_setup() would also help in the case where the user
> runs "git commit" and then continues the rebase. I'd be interested to
> hear what Junio and Patrick think about that.
So v2 would be two patches: rerere returning an error on a busy lock
(with "rerere gc" still warning and skipping as in v1, and a commit
message that talks about the fix instead of the trace), and the
sequencer disabling gc.auto/maintenance.auto for "git commit", "git
merge" and exec. I'll wait for Junio and Patrick before rerolling in
case they see it differently.
Patrick, one thing I noticed on the way: since 452b12c2e0
(builtin/maintenance: use "geometric" strategy by default, 2026-02-24)
every "maintenance run --auto" runs rerere-gc as soon as rr-cache has
even a single entry, stale or not. The doc for
maintenance.rerere-gc.auto says the heuristic may be refined; that
would make this rare for every command, not only the sequencer. Not
touching it in this series, just mentioning it.
Thanks,
Tom
Am Mi., 2. Sept. 2026 um 15:27 Uhr schrieb Phillip Wood
<phillip.wood123@gmail.com>:
>
> Hi Thomas
>
> On 02/09/2026 09:31, Thomas Bachem via GitGitGadget wrote:
> > From: Thomas Bachem <mail@thomasbachem.com>
> >
> > Since 2.54 unscheduled maintenance uses the "geometric" strategy, so
>
> That change really is the gift that keeps on giving
>
> > the "git maintenance run --auto --detach" behind every "git commit"
> > runs "git rerere gc" in the background whenever rr-cache has an entry.
> > That includes the "git commit" the sequencer runs for a resolved pick
> > on "git rebase --continue".
> >
> > rerere_gc() takes MERGE_RR.lock through setup_rerere(), which uses
> > LOCK_DIE_ON_ERROR, and so does the sequencer's repo_rerere() at the
> > next conflict a few milliseconds later. Whichever comes second dies.
>
> To me this is another reason why we should disable gc.auto while
> rebasing. To do that we need to pass "-c gc.auto=false -c
> maintenance.auto=false" when running "git commit" in run_git_commit()
> and also when running "git merge" in do_merge(). We should also pass
> those settings via GIT_CONFIG_PARAMETERS when running a exec command in
> do_exec(). That is largly papering over the cracks but until we have a
> systematic solution it does at least stop exposing users to this bug.
>
> > When it is the rebase, it dies in do_pick_commit()
>
> That's a bug us well - we should be returning errors, not dying
> -rerere_setup() should be returning an error, so we can clean up and
> reschedule the pick.
>
> There is a lot of detail here about what causes the problem which is
> helpful, but there is very little discussion about the fix. As I
> understand it we now block the sequencer until the background
> maintenance has completed, or continue to die in an inconvenient state
> we timeout before the background maintenance finishes. That seems rather
> unfortunate as the idea of running the maintenance in the background is
> to prevent it from interfering with other commands.
>
> I think my preferred solution is to disable gc while rebasing. Returning
> an error from rerere_setup() would also help in the case where the user
> runs "git commit" and then continues the rebase. I'd be interested to
> hear what Junio and Patrick think about that. I'm also not clear why
> gc.auto has to fork a separate process just to check if it needs to run
> or not, I've not been following closely but my impression is that that
> is the cause of quite a lot of the lock contention bugs we've seen.
>
> Thanks
>
> Phillip
>
> > with the index
> > written but before make_patch() writes rebase-merge/{message,patch,
> > stopped-sha}, and every later "git rebase --continue" refuses with
> > "you have staged changes in your working tree". When it is the "git
> > commit" of a later continue, that one dies in its post-commit
> > repo_rerere() after the commit was made. Before 2.54 the same
> > collision needed an auto gc to actually run, since gc runs
> > "rerere gc" at its end.
> >
> > A rebase with two conflicts in a row shows it. The filler makes the
> > pick slower than the ~5 ms the background task needs to take the
> > lock, and keeps the lock held for about 0.4 s. It hit 6 of 6 runs
> > here on 2.55.0, and a test suite driving rebases on toy repositories
> > with a single rr-cache entry hit it in both runs that were traced:
> >
> > git init -q -b main r && cd r
> > git config rerere.enabled true
> > git config maintenance.auto false
> > mkdir pad && seq 20000 | (cd pad && split -l 1 -a 5)
> > echo base >f && git add -A && git commit -qm base
> > git checkout -q -b topic
> > echo b >f && git commit -qam B
> > echo c >f && git commit -qam C
> > git checkout -q main
> > echo a >f && git commit -qam A
> > git repack -adq
> > seq 20000 | awk '{printf ".git/rr-cache/%040x\n", $1}' \
> > | xargs mkdir -p
> > for d in .git/rr-cache/*/; do echo x >$d/preimage; done
> > git config --unset maintenance.auto
> > git checkout -q topic
> > git rebase main
> > echo ab >f && git add f
> > GIT_EDITOR=true git rebase --continue
> >
> > The second continue dies with "Unable to create '.git/MERGE_RR.lock':
> > File exists" while the gc spawned by its own commit holds the lock,
> > and after resolving C every further continue refuses. Maintenance
> > stays off during the setup so that no repack is pending: a repack due
> > at that commit runs ahead of rerere-gc in the task list and would
> > spend the window.
> >
> > The gc needs the lock: it removes every rr-cache directory it finds
> > empty, and a rerere that has just created its directory but not yet
> > written the preimage looks exactly like that. So keep the lock and fix
> > both orders. When the gc finds the lock busy, let it warn and do
> > nothing this time, the way "maintenance run" treats its own lock, so a
> > manual "git rerere gc" sees the warning and the maintenance task and
> > "git gc" see a clean exit. When the gc holds the lock, let every other
> > caller wait it out instead of dying at once, for rerere.lockTimeout
> > milliseconds with the semantics of core.packedRefsTimeout: 1000 by
> > default, 0 for the old behaviour, -1 for an unbounded wait. Walking a
> > 20000-entry rr-cache takes about 0.4 s here.
> >
> > That rebase now completes. The tests cover the gc under a held lock,
> > directly and through the maintenance task, a merge that waits a lock
> > out within a five second rerere.lockTimeout, and one that fails at
> > once with a timeout of 0.
> >
> > Assisted-by: Claude Fable 5.1
> > Signed-off-by: Thomas Bachem <mail@thomasbachem.com>
> > ---
> > rerere: keep a background gc from killing a rebase
> >
> > Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2214%2Fthomasbachem%2Frerere-gc-lock-v1
> > Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2214/thomasbachem/rerere-gc-lock-v1
> > Pull-Request: https://github.com/gitgitgadget/git/pull/2214
> >
> > Documentation/config/rerere.adoc | 8 +++++++
> > Documentation/git-rerere.adoc | 4 +++-
> > rerere.c | 27 +++++++++++++++++----
> > rerere.h | 1 +
> > t/t4200-rerere.sh | 40 ++++++++++++++++++++++++++++++++
> > t/t7900-maintenance.sh | 8 +++++++
> > 6 files changed, 82 insertions(+), 6 deletions(-)
> >
> > diff --git a/Documentation/config/rerere.adoc b/Documentation/config/rerere.adoc
> > index 3a78b5ebb1..8041a1587b 100644
> > --- a/Documentation/config/rerere.adoc
> > +++ b/Documentation/config/rerere.adoc
> > @@ -10,3 +10,11 @@ rerere.enabled::
> > enabled if there is an `rr-cache` directory under the
> > `$GIT_DIR`, e.g. if "rerere" was previously used in the
> > repository.
> > +
> > +rerere.lockTimeout::
> > + The length of time, in milliseconds, to retry when trying to
> > + take the rerere lock while another process holds it, typically
> > + a background `git rerere gc`. Value 0 means not to retry at
> > + all; -1 means to try indefinitely. Default is 1000 (i.e.,
> > + retry for 1 second). `git rerere gc` itself does not wait and
> > + skips its run instead.
> > diff --git a/Documentation/git-rerere.adoc b/Documentation/git-rerere.adoc
> > index 4e6ab9a27c..05935b0603 100644
> > --- a/Documentation/git-rerere.adoc
> > +++ b/Documentation/git-rerere.adoc
> > @@ -70,7 +70,9 @@ occurred a long time ago. By default, unresolved conflicts older
> > than 15 days and resolved conflicts older than 60
> > days are pruned. These defaults are controlled via the
> > `gc.rerereUnresolved` and `gc.rerereResolved` configuration
> > -variables respectively.
> > +variables respectively. If another process holds the lock on the
> > +recorded resolutions, for example a merge or rebase that is recording
> > +a conflict, `gc` does nothing and reports so.
> >
> >
> > DISCUSSION
> > diff --git a/rerere.c b/rerere.c
> > index 8232542585..22d114262b 100644
> > --- a/rerere.c
> > +++ b/rerere.c
> > @@ -32,6 +32,7 @@ static int rerere_enabled = -1;
> >
> > /* automatically update cleanly resolved paths to the index */
> > static int rerere_autoupdate;
> > +static int rerere_lock_timeout_ms = 1000;
> >
> > #define RR_HAS_POSTIMAGE 1
> > #define RR_HAS_PREIMAGE 2
> > @@ -876,6 +877,8 @@ static void git_rerere_config(void)
> > {
> > repo_config_get_bool(the_repository, "rerere.enabled", &rerere_enabled);
> > repo_config_get_bool(the_repository, "rerere.autoupdate", &rerere_autoupdate);
> > + repo_config_get_int(the_repository, "rerere.locktimeout",
> > + &rerere_lock_timeout_ms);
> > repo_config(the_repository, git_default_config, NULL);
> > }
> >
> > @@ -908,12 +911,26 @@ int setup_rerere(struct repository *r, struct string_list *merge_rr, int flags)
> >
> > if (flags & (RERERE_AUTOUPDATE|RERERE_NOAUTOUPDATE))
> > rerere_autoupdate = !!(flags & RERERE_AUTOUPDATE);
> > - if (flags & RERERE_READONLY)
> > + if (flags & RERERE_READONLY) {
> > fd = 0;
> > - else
> > + } else if (flags & RERERE_SKIP_LOCKED) {
> > fd = hold_lock_file_for_update(&write_lock,
> > - git_path_merge_rr(r),
> > - LOCK_DIE_ON_ERROR);
> > + git_path_merge_rr(r), 0);
> > + if (fd < 0) {
> > + warning_errno(_("unable to lock '%s', skipping"),
> > + git_path_merge_rr(r));
> > + return -1;
> > + }
> > + } else {
> > + /*
> > + * A background "rerere gc" holds the lock for as long as it
> > + * takes to walk rr-cache, so wait it out rather than die.
> > + */
> > + fd = hold_lock_file_for_update_timeout(&write_lock,
> > + git_path_merge_rr(r),
> > + LOCK_DIE_ON_ERROR,
> > + rerere_lock_timeout_ms);
> > + }
> > read_rr(r, merge_rr);
> > return fd;
> > }
> > @@ -1237,7 +1254,7 @@ void rerere_gc(struct repository *r, struct string_list *rr)
> > timestamp_t cutoff_resolve = now - 60 * 86400;
> > struct strbuf buf = STRBUF_INIT;
> >
> > - if (setup_rerere(r, rr, 0) < 0)
> > + if (setup_rerere(r, rr, RERERE_SKIP_LOCKED) < 0)
> > return;
> >
> > repo_config_get_expiry_in_days(the_repository, "gc.rerereresolved",
> > diff --git a/rerere.h b/rerere.h
> > index d4b5f7c932..87964bb3c5 100644
> > --- a/rerere.h
> > +++ b/rerere.h
> > @@ -10,6 +10,7 @@ struct repository;
> > #define RERERE_AUTOUPDATE 01
> > #define RERERE_NOAUTOUPDATE 02
> > #define RERERE_READONLY 04
> > +#define RERERE_SKIP_LOCKED 010
> >
> > /*
> > * Marks paths that have been hand-resolved and added to the
> > diff --git a/t/t4200-rerere.sh b/t/t4200-rerere.sh
> > index 1717f407c8..6b90294435 100755
> > --- a/t/t4200-rerere.sh
> > +++ b/t/t4200-rerere.sh
> > @@ -242,6 +242,46 @@ test_expect_success 'old records rest in peace' '
> > test_path_is_missing $rr2/preimage
> > '
> >
> > +test_expect_success 'gc does nothing while MERGE_RR is locked' '
> > + mkdir -p $rr2 &&
> > + echo Hello >$rr2/preimage &&
> > + test-tool chmtime =$just_over_15_days_ago $rr2/preimage &&
> > +
> > + test_when_finished "rm -f .git/MERGE_RR.lock" &&
> > + >.git/MERGE_RR.lock &&
> > + git rerere gc 2>err &&
> > + test_grep "MERGE_RR" err &&
> > + test_path_is_file $rr2/preimage &&
> > +
> > + rm .git/MERGE_RR.lock &&
> > + git rerere gc &&
> > + test_path_is_missing $rr2/preimage
> > +'
> > +
> > +test_expect_success 'a held lock is waited out within rerere.lockTimeout' '
> > + git reset --hard &&
> > + rm -rf $rr &&
> > + test_when_finished "rm -f .git/MERGE_RR.lock" &&
> > + >.git/MERGE_RR.lock &&
> > + {
> > + (sleep 1 && rm -f .git/MERGE_RR.lock) &
> > + } &&
> > + test_must_fail git -c rerere.lockTimeout=5000 merge first 2>err &&
> > + wait &&
> > + test_grep ! "Unable to create" err &&
> > + grep "^=======\$" $rr/preimage
> > +'
> > +
> > +test_expect_success 'rerere.lockTimeout=0 fails at once on a held lock' '
> > + git reset --hard &&
> > + rm -rf $rr &&
> > + test_when_finished "rm -f .git/MERGE_RR.lock" &&
> > + >.git/MERGE_RR.lock &&
> > + test_must_fail git -c rerere.lockTimeout=0 merge first 2>err &&
> > + test_grep "Unable to create" err &&
> > + test_path_is_missing $rr/preimage
> > +'
> > +
> > rerere_gc_custom_expiry_test () {
> > five_days="$1" right_now="$2"
> > test_expect_success "rerere gc with custom expiry ($five_days, $right_now)" '
> > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
> > index d7f82e1bec..a55ca2e829 100755
> > --- a/t/t7900-maintenance.sh
> > +++ b/t/t7900-maintenance.sh
> > @@ -885,6 +885,14 @@ test_expect_success 'rerere-gc task with --auto honors maintenance.rerere-gc.aut
> > test_expect_rerere_gc ! git -c maintenance.rerere-gc.auto=0 maintenance run --auto --task=rerere-gc
> > '
> >
> > +test_expect_success 'rerere-gc task succeeds while MERGE_RR is locked' '
> > + test_when_finished "rm -rf .git/rr-cache .git/MERGE_RR.lock" &&
> > + mkdir .git/rr-cache &&
> > + : >.git/rr-cache/entry &&
> > + >.git/MERGE_RR.lock &&
> > + test_expect_rerere_gc git maintenance run --task=rerere-gc
> > +'
> > +
> > test_expect_success '--auto and --schedule incompatible' '
> > test_must_fail git maintenance run --auto --schedule=daily 2>err &&
> > test_grep "cannot be used together" err
> >
> > base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
> |
|
Patrick Steinhardt wrote on the Git mailing list (how to reply to this email): On Wed, Sep 02, 2026 at 08:31:37AM +0000, Thomas Bachem via GitGitGadget wrote:
> From: Thomas Bachem <mail@thomasbachem.com>
>
> Since 2.54 unscheduled maintenance uses the "geometric" strategy, so
> the "git maintenance run --auto --detach" behind every "git commit"
> runs "git rerere gc" in the background whenever rr-cache has an entry.
> That includes the "git commit" the sequencer runs for a resolved pick
> on "git rebase --continue".
I think this hints that we should tweak the default value of
"maintenance.rerere-gc.auto". The way it's currently written we indeed
are quite aggressive with spawning `git rerere gc`, and I agree that we
should tweak it. And in the best case we'd not only respect whether we
have a specific number of entries, but we should also respect whether
those would be garbage collected in the first place.
I'll send a patch series later today to do this.
[snip]
> The gc needs the lock: it removes every rr-cache directory it finds
> empty, and a rerere that has just created its directory but not yet
> written the preimage looks exactly like that. So keep the lock and fix
> both orders. When the gc finds the lock busy, let it warn and do
> nothing this time, the way "maintenance run" treats its own lock, so a
> manual "git rerere gc" sees the warning and the maintenance task and
> "git gc" see a clean exit. When the gc holds the lock, let every other
> caller wait it out instead of dying at once, for rerere.lockTimeout
> milliseconds with the semantics of core.packedRefsTimeout: 1000 by
> default, 0 for the old behaviour, -1 for an unbounded wait. Walking a
> 20000-entry rr-cache takes about 0.4 s here.
Having a locking timeout is sensible anyway, I think. It does not only
solve races with a concurrent maintenance run, but also with concurrent
writers.
> diff --git a/rerere.c b/rerere.c
> index 8232542585..22d114262b 100644
> --- a/rerere.c
> +++ b/rerere.c
> @@ -32,6 +32,7 @@ static int rerere_enabled = -1;
>
> /* automatically update cleanly resolved paths to the index */
> static int rerere_autoupdate;
> +static int rerere_lock_timeout_ms = 1000;
>
> #define RR_HAS_POSTIMAGE 1
> #define RR_HAS_PREIMAGE 2
> @@ -876,6 +877,8 @@ static void git_rerere_config(void)
> {
> repo_config_get_bool(the_repository, "rerere.enabled", &rerere_enabled);
> repo_config_get_bool(the_repository, "rerere.autoupdate", &rerere_autoupdate);
> + repo_config_get_int(the_repository, "rerere.locktimeout",
> + &rerere_lock_timeout_ms);
> repo_config(the_repository, git_default_config, NULL);
> }
>
> @@ -908,12 +911,26 @@ int setup_rerere(struct repository *r, struct string_list *merge_rr, int flags)
>
> if (flags & (RERERE_AUTOUPDATE|RERERE_NOAUTOUPDATE))
> rerere_autoupdate = !!(flags & RERERE_AUTOUPDATE);
> - if (flags & RERERE_READONLY)
> + if (flags & RERERE_READONLY) {
> fd = 0;
> - else
> + } else if (flags & RERERE_SKIP_LOCKED) {
> fd = hold_lock_file_for_update(&write_lock,
> - git_path_merge_rr(r),
> - LOCK_DIE_ON_ERROR);
> + git_path_merge_rr(r), 0);
> + if (fd < 0) {
> + warning_errno(_("unable to lock '%s', skipping"),
> + git_path_merge_rr(r));
> + return -1;
> + }
We should instead pass `LOCK_REPORT_ON_ERROR`, as the lockfile machinery
knows better why exactly locking has failed.
> + } else {
> + /*
> + * A background "rerere gc" holds the lock for as long as it
> + * takes to walk rr-cache, so wait it out rather than die.
> + */
> + fd = hold_lock_file_for_update_timeout(&write_lock,
> + git_path_merge_rr(r),
> + LOCK_DIE_ON_ERROR,
> + rerere_lock_timeout_ms);
> + }
I think we can easily combine those two branches and simply set the
timeout value to 0 in case we see the flag.
Patrick |
|
Thomas Bachem wrote on the Git mailing list (how to reply to this email): Hi Patrick,
On Thu, Sep 03, 2026 at 09:40:04AM +0200, Patrick Steinhardt wrote:
> I think this hints that we should tweak the default value of
> "maintenance.rerere-gc.auto". The way it's currently written we indeed
> are quite aggressive with spawning `git rerere gc`, and I agree that we
> should tweak it. And in the best case we'd not only respect whether we
> have a specific number of entries, but we should also respect whether
> those would be garbage collected in the first place.
>
> I'll send a patch series later today to do this.
Thanks. Checking whether anything would actually be pruned sounds
right to me. It takes the frequency away, not the race, so I'd still
do the sequencer part Phillip asked for.
> Having a locking timeout is sensible anyway, I think. It does not only
> solve races with a concurrent maintenance run, but also with concurrent
> writers.
Phillip found the wait unfortunate and I offered to drop it. You would
keep it. I think the two fit together: wait up to rerere.lockTimeout,
then warn and return -1 instead of dying, so the caller goes on
without rerere this once. The gc passes 0 and does not wait. That
takes the die out, which is what broke the rebase. The wait stays,
bounded to a second, but skipping rerere is not free either: it can
mean resolving a conflict again that rerere had already recorded, and
a second is cheap next to that. With the sequencer no longer spawning
the gc and your heuristic change, it should rarely come to either.
Phillip, would that work for you?
> We should instead pass `LOCK_REPORT_ON_ERROR`, as the lockfile machinery
> knows better why exactly locking has failed.
Agreed on the text, which also names a stale lock. But the callers
that go on without rerere then exit as if it were disabled, "git
commit" with 0, so for them I'd print it as a warning through
unable_to_lock_message() rather than let LOCK_REPORT_ON_ERROR call it
an error. An explicit "git rerere forget" or "clear" fails as before.
> I think we can easily combine those two branches and simply set the
> timeout value to 0 in case we see the flag.
Yes, that folds into one call.
So v2: setup_rerere() waits up to rerere.lockTimeout, 0 for the gc,
then warns and returns -1 where the caller can go on, with the
sequencer patch on top. I'll reroll once Phillip has had a look.
Thanks,
Tom |
|
Patrick Steinhardt wrote on the Git mailing list (how to reply to this email): On Thu, Sep 03, 2026 at 10:11:05AM +0200, Thomas Bachem wrote:
> Hi Patrick,
>
> On Thu, Sep 03, 2026 at 09:40:04AM +0200, Patrick Steinhardt wrote:
> > I think this hints that we should tweak the default value of
> > "maintenance.rerere-gc.auto". The way it's currently written we indeed
> > are quite aggressive with spawning `git rerere gc`, and I agree that we
> > should tweak it. And in the best case we'd not only respect whether we
> > have a specific number of entries, but we should also respect whether
> > those would be garbage collected in the first place.
> >
> > I'll send a patch series later today to do this.
>
> Thanks. Checking whether anything would actually be pruned sounds
> right to me. It takes the frequency away, not the race, so I'd still
> do the sequencer part Phillip asked for.
Yes. Ideally, I'd think that we should both introduce the grace period
for locking the file and adapting the heuristic used by the maintenance
strategy. Whether we should completely disable auto-maintenance when in
the sequencer... I dunno. In any case, that feels like another separate
topic that should probably be discussed in its own series.
> > Having a locking timeout is sensible anyway, I think. It does not only
> > solve races with a concurrent maintenance run, but also with concurrent
> > writers.
>
> Phillip found the wait unfortunate and I offered to drop it. You would
> keep it. I think the two fit together: wait up to rerere.lockTimeout,
> then warn and return -1 instead of dying, so the caller goes on
> without rerere this once. The gc passes 0 and does not wait. That
> takes the die out, which is what broke the rebase. The wait stays,
> bounded to a second, but skipping rerere is not free either: it can
> mean resolving a conflict again that rerere had already recorded, and
> a second is cheap next to that. With the sequencer no longer spawning
> the gc and your heuristic change, it should rarely come to either.
> Phillip, would that work for you?
I think that having the wait is a sensible thing to do, as the race was
a preexisting one that was only uncovered by the change to the default
maintenance strategy. It can also happen with two concurrent processes
that both happen to write rerere entries. You wouldn't normally see the
wait anyway, so in the happy path nobody will really care. And in the
cases where you would see it the user is probably more happy to wait a
bit than having Git die (or just not write a rerere entry at all).
Patrick |
|
Thomas Bachem wrote on the Git mailing list (how to reply to this email): Hi Patrick,
On Thu, Sep 03, 2026 at 10:32:36AM +0200, Patrick Steinhardt wrote:
> Yes. Ideally, I'd think that we should both introduce the grace period
> for locking the file and adapting the heuristic used by the maintenance
> strategy. Whether we should completely disable auto-maintenance when in
> the sequencer... I dunno. In any case, that feels like another separate
> topic that should probably be discussed in its own series.
Phillip, this is the part I said I'd do in this series, so I'd
rather answer it here than just drop it. I think Patrick is right
that it's a topic of its own. My reason for wanting it in the same
series was the recording lost at a stop while the gc holds the lock,
and that was for the variant without the wait. With the wait kept,
the next pick waits the gc out and records as before, so the
sequencer patch no longer buys the rebase anything the rerere patch
doesn't, short of a prune that outlasts the timeout.
What it would still decide is whether a rebase with the merge backend
runs maintenance at all, the question from my last mail, and that is
a discussion of its own. So I'd make v2 the rerere patch alone and
send the sequencer change separately if you still want it. Say if
you would rather keep them together.
> I think that having the wait is a sensible thing to do, as the race was
> a preexisting one that was only uncovered by the change to the default
> maintenance strategy. It can also happen with two concurrent processes
> that both happen to write rerere entries. You wouldn't normally see the
> wait anyway, so in the happy path nobody will really care. And in the
> cases where you would see it the user is probably more happy to wait a
> bit than having Git die (or just not write a rerere entry at all).
Agreed, and that is the order v2 keeps: wait first, skip only once
the wait has run out. Since your series means the gc now only runs
when there is something to prune, I measured how long that wait can
get: pruning 20000 stale entries holds the lock for 2.7 s here,
walking 20000 fresh ones takes 0.4 s, so the one second default
covers a prune of roughly 7000 entries if it scales. I'd keep the
default. A backlog that size is a one-off, and where it does hit,
the timeout now skips one recording where it used to kill the
rebase.
My patch is based on maint since the bug is there, and I'd keep it
that way unless Junio would rather have it on master. Merged up it
conflicts with d43f701d32 (lockfile: add
repo_hold_lock_file_for_update{,_timeout}{,_mode}(), 2026-07-14) in
setup_rerere(). The resolution is to take the repo-scoped helper, and
with that t4200 and t7900 pass on top of your series. I'll wait a
day or two for Phillip before rerolling.
Thanks,
Tom |
|
Phillip Wood wrote on the Git mailing list (how to reply to this email): Hi Patrick and Thomas
On 03/09/2026 09:32, Patrick Steinhardt wrote:
> On Thu, Sep 03, 2026 at 10:11:05AM +0200, Thomas Bachem wrote:
>> Hi Patrick,
>>
>> On Thu, Sep 03, 2026 at 09:40:04AM +0200, Patrick Steinhardt wrote:
>>> I think this hints that we should tweak the default value of
>>> "maintenance.rerere-gc.auto". The way it's currently written we indeed
>>> are quite aggressive with spawning `git rerere gc`, and I agree that we
>>> should tweak it. And in the best case we'd not only respect whether we
>>> have a specific number of entries, but we should also respect whether
>>> those would be garbage collected in the first place.
>>>
>>> I'll send a patch series later today to do this.
>>
>> Thanks. Checking whether anything would actually be pruned sounds
>> right to me. It takes the frequency away, not the race, so I'd still
>> do the sequencer part Phillip asked for.
> > Yes. Ideally, I'd think that we should both introduce the grace period
> for locking the file and adapting the heuristic used by the maintenance
> strategy. I agree
> Whether we should completely disable auto-maintenance when in
> the sequencer... I dunno. In any case, that feels like another separate
> topic that should probably be discussed in its own series.
We've seen other bugs reported related to auto-maintenance triggered during a rebase such as the one dscho fixed recently. While I can see repacking might be helpful during a very large rebase, I do not think garbage collection is useful - all the objects and rerere entries that are created during the rebase are going to be too fresh to be collected. So I think it would be a good idea to disable auto maintenance in a rebase and see if anyone complains. If it turns out to be a problem we can figure out how to make it repack incrementally.
>>> Having a locking timeout is sensible anyway, I think. It does not only
>>> solve races with a concurrent maintenance run, but also with concurrent
>>> writers.
>>
>> Phillip found the wait unfortunate and I offered to drop it. You would
>> keep it. I think the two fit together: wait up to rerere.lockTimeout,
>> then warn and return -1 instead of dying, so the caller goes on
>> without rerere this once. The gc passes 0 and does not wait. That
>> takes the die out, which is what broke the rebase. The wait stays,
>> bounded to a second, but skipping rerere is not free either: it can
>> mean resolving a conflict again that rerere had already recorded, and
>> a second is cheap next to that. With the sequencer no longer spawning
>> the gc and your heuristic change, it should rarely come to either.
>> Phillip, would that work for you?
> > I think that having the wait is a sensible thing to do, as the race was
> a preexisting one that was only uncovered by the change to the default
> maintenance strategy. It can also happen with two concurrent processes
> that both happen to write rerere entries. You wouldn't normally see the
> wait anyway, so in the happy path nobody will really care. And in the
> cases where you would see it the user is probably more happy to wait a
> bit than having Git die (or just not write a rerere entry at all).
I don't object to the timeout as part of the solution. My objection was based on it being the only solution as it is inconvenient to the user if they have to wait for background maintenance jobs and it does not stop the rebase from failing if the timeout is too short.
Thanks
Phillip |
|
Phillip Wood wrote on the Git mailing list (how to reply to this email): Hi Thomas
On 02/09/2026 16:07, Thomas Bachem wrote:
> On 02/09/2026 15:27, Phillip Wood wrote:
>> To me this is another reason why we should disable gc.auto while
>> rebasing. To do that we need to pass "-c gc.auto=false -c
>> maintenance.auto=false" when running "git commit" in run_git_commit()
>> and also when running "git merge" in do_merge(). We should also pass
>> those settings via GIT_CONFIG_PARAMETERS when running a exec command in
>> do_exec(). That is largly papering over the cracks but until we have a
>> systematic solution it does at least stop exposing users to this bug.
> > OK, I'll do that. It is also more consistent than it looks: the
> commits the sequencer creates in-process via try_to_commit() don't run
> auto maintenance at all, only the "git commit" child does (for a
> resolved, reworded or squashed commit). What surprised me is that a
> rebase with the merge backend then never runs maintenance, not even at
> the end, because it doesn't go through finish_rebase() where the apply
> backend runs it. Do you want a single run at the end of the sequence
> in that patch, or keep it minimal?
We should be consistent between the backends, so yes we should be calling run_auto_maintenance() at the end of a rebase with the merge backend.
> FWIW, the tool I hit this with has been setting both for its whole
> process tree since, and the failures stopped.
> >>> When it is the rebase, it dies in do_pick_commit()
>>
>> That's a bug us well - we should be returning errors, not dying
>> -rerere_setup() should be returning an error, so we can clean up and
>> reschedule the pick.
> > Yes. I don't think we even need to reschedule: when repo_rerere() is
> called there, the merge result is already in the index and worktree,
> the error and advice have been printed, and the return value is
> ignored. If setup_rerere() reports the lock and returns -1, the pick
> just stops at the conflict like any other, minus rerere's recording
> and replay, and --continue works. Oh good point, if we get an error then we'll write the files to get "git rebase --continue" to commit the conflict resolution so we don't need to reschedule.
Thanks
Phillip
>> I think my preferred solution is to disable gc while rebasing. Returning
>> an error from rerere_setup() would also help in the case where the user
>> runs "git commit" and then continues the rebase. I'd be interested to
>> hear what Junio and Patrick think about that.
> > So v2 would be two patches: rerere returning an error on a busy lock
> (with "rerere gc" still warning and skipping as in v1, and a commit
> message that talks about the fix instead of the trace), and the
> sequencer disabling gc.auto/maintenance.auto for "git commit", "git
> merge" and exec. I'll wait for Junio and Patrick before rerolling in
> case they see it differently.
> > Patrick, one thing I noticed on the way: since 452b12c2e0
> (builtin/maintenance: use "geometric" strategy by default, 2026-02-24)
> every "maintenance run --auto" runs rerere-gc as soon as rr-cache has
> even a single entry, stale or not. The doc for
> maintenance.rerere-gc.auto says the heuristic may be refined; that
> would make this rare for every command, not only the sequencer. Not
> touching it in this series, just mentioning it.
> > Thanks,
> Tom
> > > Am Mi., 2. Sept. 2026 um 15:27 Uhr schrieb Phillip Wood
> <phillip.wood123@gmail.com>:
>>
>> Hi Thomas
>>
>> On 02/09/2026 09:31, Thomas Bachem via GitGitGadget wrote:
>>> From: Thomas Bachem <mail@thomasbachem.com>
>>>
>>> Since 2.54 unscheduled maintenance uses the "geometric" strategy, so
>>
>> That change really is the gift that keeps on giving
>>
>>> the "git maintenance run --auto --detach" behind every "git commit"
>>> runs "git rerere gc" in the background whenever rr-cache has an entry.
>>> That includes the "git commit" the sequencer runs for a resolved pick
>>> on "git rebase --continue".
>>>
>>> rerere_gc() takes MERGE_RR.lock through setup_rerere(), which uses
>>> LOCK_DIE_ON_ERROR, and so does the sequencer's repo_rerere() at the
>>> next conflict a few milliseconds later. Whichever comes second dies.
>>
>> To me this is another reason why we should disable gc.auto while
>> rebasing. To do that we need to pass "-c gc.auto=false -c
>> maintenance.auto=false" when running "git commit" in run_git_commit()
>> and also when running "git merge" in do_merge(). We should also pass
>> those settings via GIT_CONFIG_PARAMETERS when running a exec command in
>> do_exec(). That is largly papering over the cracks but until we have a
>> systematic solution it does at least stop exposing users to this bug.
>>
>>> When it is the rebase, it dies in do_pick_commit()
>>
>> That's a bug us well - we should be returning errors, not dying
>> -rerere_setup() should be returning an error, so we can clean up and
>> reschedule the pick.
>>
>> There is a lot of detail here about what causes the problem which is
>> helpful, but there is very little discussion about the fix. As I
>> understand it we now block the sequencer until the background
>> maintenance has completed, or continue to die in an inconvenient state
>> we timeout before the background maintenance finishes. That seems rather
>> unfortunate as the idea of running the maintenance in the background is
>> to prevent it from interfering with other commands.
>>
>> I think my preferred solution is to disable gc while rebasing. Returning
>> an error from rerere_setup() would also help in the case where the user
>> runs "git commit" and then continues the rebase. I'd be interested to
>> hear what Junio and Patrick think about that. I'm also not clear why
>> gc.auto has to fork a separate process just to check if it needs to run
>> or not, I've not been following closely but my impression is that that
>> is the cause of quite a lot of the lock contention bugs we've seen.
>>
>> Thanks
>>
>> Phillip
>>
>>> with the index
>>> written but before make_patch() writes rebase-merge/{message,patch,
>>> stopped-sha}, and every later "git rebase --continue" refuses with
>>> "you have staged changes in your working tree". When it is the "git
>>> commit" of a later continue, that one dies in its post-commit
>>> repo_rerere() after the commit was made. Before 2.54 the same
>>> collision needed an auto gc to actually run, since gc runs
>>> "rerere gc" at its end.
>>>
>>> A rebase with two conflicts in a row shows it. The filler makes the
>>> pick slower than the ~5 ms the background task needs to take the
>>> lock, and keeps the lock held for about 0.4 s. It hit 6 of 6 runs
>>> here on 2.55.0, and a test suite driving rebases on toy repositories
>>> with a single rr-cache entry hit it in both runs that were traced:
>>>
>>> git init -q -b main r && cd r
>>> git config rerere.enabled true
>>> git config maintenance.auto false
>>> mkdir pad && seq 20000 | (cd pad && split -l 1 -a 5)
>>> echo base >f && git add -A && git commit -qm base
>>> git checkout -q -b topic
>>> echo b >f && git commit -qam B
>>> echo c >f && git commit -qam C
>>> git checkout -q main
>>> echo a >f && git commit -qam A
>>> git repack -adq
>>> seq 20000 | awk '{printf ".git/rr-cache/%040x\n", $1}' \
>>> | xargs mkdir -p
>>> for d in .git/rr-cache/*/; do echo x >$d/preimage; done
>>> git config --unset maintenance.auto
>>> git checkout -q topic
>>> git rebase main
>>> echo ab >f && git add f
>>> GIT_EDITOR=true git rebase --continue
>>>
>>> The second continue dies with "Unable to create '.git/MERGE_RR.lock':
>>> File exists" while the gc spawned by its own commit holds the lock,
>>> and after resolving C every further continue refuses. Maintenance
>>> stays off during the setup so that no repack is pending: a repack due
>>> at that commit runs ahead of rerere-gc in the task list and would
>>> spend the window.
>>>
>>> The gc needs the lock: it removes every rr-cache directory it finds
>>> empty, and a rerere that has just created its directory but not yet
>>> written the preimage looks exactly like that. So keep the lock and fix
>>> both orders. When the gc finds the lock busy, let it warn and do
>>> nothing this time, the way "maintenance run" treats its own lock, so a
>>> manual "git rerere gc" sees the warning and the maintenance task and
>>> "git gc" see a clean exit. When the gc holds the lock, let every other
>>> caller wait it out instead of dying at once, for rerere.lockTimeout
>>> milliseconds with the semantics of core.packedRefsTimeout: 1000 by
>>> default, 0 for the old behaviour, -1 for an unbounded wait. Walking a
>>> 20000-entry rr-cache takes about 0.4 s here.
>>>
>>> That rebase now completes. The tests cover the gc under a held lock,
>>> directly and through the maintenance task, a merge that waits a lock
>>> out within a five second rerere.lockTimeout, and one that fails at
>>> once with a timeout of 0.
>>>
>>> Assisted-by: Claude Fable 5.1
>>> Signed-off-by: Thomas Bachem <mail@thomasbachem.com>
>>> ---
>>> rerere: keep a background gc from killing a rebase
>>>
>>> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2214%2Fthomasbachem%2Frerere-gc-lock-v1
>>> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2214/thomasbachem/rerere-gc-lock-v1
>>> Pull-Request: https://github.com/gitgitgadget/git/pull/2214
>>>
>>> Documentation/config/rerere.adoc | 8 +++++++
>>> Documentation/git-rerere.adoc | 4 +++-
>>> rerere.c | 27 +++++++++++++++++----
>>> rerere.h | 1 +
>>> t/t4200-rerere.sh | 40 ++++++++++++++++++++++++++++++++
>>> t/t7900-maintenance.sh | 8 +++++++
>>> 6 files changed, 82 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/Documentation/config/rerere.adoc b/Documentation/config/rerere.adoc
>>> index 3a78b5ebb1..8041a1587b 100644
>>> --- a/Documentation/config/rerere.adoc
>>> +++ b/Documentation/config/rerere.adoc
>>> @@ -10,3 +10,11 @@ rerere.enabled::
>>> enabled if there is an `rr-cache` directory under the
>>> `$GIT_DIR`, e.g. if "rerere" was previously used in the
>>> repository.
>>> +
>>> +rerere.lockTimeout::
>>> + The length of time, in milliseconds, to retry when trying to
>>> + take the rerere lock while another process holds it, typically
>>> + a background `git rerere gc`. Value 0 means not to retry at
>>> + all; -1 means to try indefinitely. Default is 1000 (i.e.,
>>> + retry for 1 second). `git rerere gc` itself does not wait and
>>> + skips its run instead.
>>> diff --git a/Documentation/git-rerere.adoc b/Documentation/git-rerere.adoc
>>> index 4e6ab9a27c..05935b0603 100644
>>> --- a/Documentation/git-rerere.adoc
>>> +++ b/Documentation/git-rerere.adoc
>>> @@ -70,7 +70,9 @@ occurred a long time ago. By default, unresolved conflicts older
>>> than 15 days and resolved conflicts older than 60
>>> days are pruned. These defaults are controlled via the
>>> `gc.rerereUnresolved` and `gc.rerereResolved` configuration
>>> -variables respectively.
>>> +variables respectively. If another process holds the lock on the
>>> +recorded resolutions, for example a merge or rebase that is recording
>>> +a conflict, `gc` does nothing and reports so.
>>>
>>>
>>> DISCUSSION
>>> diff --git a/rerere.c b/rerere.c
>>> index 8232542585..22d114262b 100644
>>> --- a/rerere.c
>>> +++ b/rerere.c
>>> @@ -32,6 +32,7 @@ static int rerere_enabled = -1;
>>>
>>> /* automatically update cleanly resolved paths to the index */
>>> static int rerere_autoupdate;
>>> +static int rerere_lock_timeout_ms = 1000;
>>>
>>> #define RR_HAS_POSTIMAGE 1
>>> #define RR_HAS_PREIMAGE 2
>>> @@ -876,6 +877,8 @@ static void git_rerere_config(void)
>>> {
>>> repo_config_get_bool(the_repository, "rerere.enabled", &rerere_enabled);
>>> repo_config_get_bool(the_repository, "rerere.autoupdate", &rerere_autoupdate);
>>> + repo_config_get_int(the_repository, "rerere.locktimeout",
>>> + &rerere_lock_timeout_ms);
>>> repo_config(the_repository, git_default_config, NULL);
>>> }
>>>
>>> @@ -908,12 +911,26 @@ int setup_rerere(struct repository *r, struct string_list *merge_rr, int flags)
>>>
>>> if (flags & (RERERE_AUTOUPDATE|RERERE_NOAUTOUPDATE))
>>> rerere_autoupdate = !!(flags & RERERE_AUTOUPDATE);
>>> - if (flags & RERERE_READONLY)
>>> + if (flags & RERERE_READONLY) {
>>> fd = 0;
>>> - else
>>> + } else if (flags & RERERE_SKIP_LOCKED) {
>>> fd = hold_lock_file_for_update(&write_lock,
>>> - git_path_merge_rr(r),
>>> - LOCK_DIE_ON_ERROR);
>>> + git_path_merge_rr(r), 0);
>>> + if (fd < 0) {
>>> + warning_errno(_("unable to lock '%s', skipping"),
>>> + git_path_merge_rr(r));
>>> + return -1;
>>> + }
>>> + } else {
>>> + /*
>>> + * A background "rerere gc" holds the lock for as long as it
>>> + * takes to walk rr-cache, so wait it out rather than die.
>>> + */
>>> + fd = hold_lock_file_for_update_timeout(&write_lock,
>>> + git_path_merge_rr(r),
>>> + LOCK_DIE_ON_ERROR,
>>> + rerere_lock_timeout_ms);
>>> + }
>>> read_rr(r, merge_rr);
>>> return fd;
>>> }
>>> @@ -1237,7 +1254,7 @@ void rerere_gc(struct repository *r, struct string_list *rr)
>>> timestamp_t cutoff_resolve = now - 60 * 86400;
>>> struct strbuf buf = STRBUF_INIT;
>>>
>>> - if (setup_rerere(r, rr, 0) < 0)
>>> + if (setup_rerere(r, rr, RERERE_SKIP_LOCKED) < 0)
>>> return;
>>>
>>> repo_config_get_expiry_in_days(the_repository, "gc.rerereresolved",
>>> diff --git a/rerere.h b/rerere.h
>>> index d4b5f7c932..87964bb3c5 100644
>>> --- a/rerere.h
>>> +++ b/rerere.h
>>> @@ -10,6 +10,7 @@ struct repository;
>>> #define RERERE_AUTOUPDATE 01
>>> #define RERERE_NOAUTOUPDATE 02
>>> #define RERERE_READONLY 04
>>> +#define RERERE_SKIP_LOCKED 010
>>>
>>> /*
>>> * Marks paths that have been hand-resolved and added to the
>>> diff --git a/t/t4200-rerere.sh b/t/t4200-rerere.sh
>>> index 1717f407c8..6b90294435 100755
>>> --- a/t/t4200-rerere.sh
>>> +++ b/t/t4200-rerere.sh
>>> @@ -242,6 +242,46 @@ test_expect_success 'old records rest in peace' '
>>> test_path_is_missing $rr2/preimage
>>> '
>>>
>>> +test_expect_success 'gc does nothing while MERGE_RR is locked' '
>>> + mkdir -p $rr2 &&
>>> + echo Hello >$rr2/preimage &&
>>> + test-tool chmtime =$just_over_15_days_ago $rr2/preimage &&
>>> +
>>> + test_when_finished "rm -f .git/MERGE_RR.lock" &&
>>> + >.git/MERGE_RR.lock &&
>>> + git rerere gc 2>err &&
>>> + test_grep "MERGE_RR" err &&
>>> + test_path_is_file $rr2/preimage &&
>>> +
>>> + rm .git/MERGE_RR.lock &&
>>> + git rerere gc &&
>>> + test_path_is_missing $rr2/preimage
>>> +'
>>> +
>>> +test_expect_success 'a held lock is waited out within rerere.lockTimeout' '
>>> + git reset --hard &&
>>> + rm -rf $rr &&
>>> + test_when_finished "rm -f .git/MERGE_RR.lock" &&
>>> + >.git/MERGE_RR.lock &&
>>> + {
>>> + (sleep 1 && rm -f .git/MERGE_RR.lock) &
>>> + } &&
>>> + test_must_fail git -c rerere.lockTimeout=5000 merge first 2>err &&
>>> + wait &&
>>> + test_grep ! "Unable to create" err &&
>>> + grep "^=======\$" $rr/preimage
>>> +'
>>> +
>>> +test_expect_success 'rerere.lockTimeout=0 fails at once on a held lock' '
>>> + git reset --hard &&
>>> + rm -rf $rr &&
>>> + test_when_finished "rm -f .git/MERGE_RR.lock" &&
>>> + >.git/MERGE_RR.lock &&
>>> + test_must_fail git -c rerere.lockTimeout=0 merge first 2>err &&
>>> + test_grep "Unable to create" err &&
>>> + test_path_is_missing $rr/preimage
>>> +'
>>> +
>>> rerere_gc_custom_expiry_test () {
>>> five_days="$1" right_now="$2"
>>> test_expect_success "rerere gc with custom expiry ($five_days, $right_now)" '
>>> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
>>> index d7f82e1bec..a55ca2e829 100755
>>> --- a/t/t7900-maintenance.sh
>>> +++ b/t/t7900-maintenance.sh
>>> @@ -885,6 +885,14 @@ test_expect_success 'rerere-gc task with --auto honors maintenance.rerere-gc.aut
>>> test_expect_rerere_gc ! git -c maintenance.rerere-gc.auto=0 maintenance run --auto --task=rerere-gc
>>> '
>>>
>>> +test_expect_success 'rerere-gc task succeeds while MERGE_RR is locked' '
>>> + test_when_finished "rm -rf .git/rr-cache .git/MERGE_RR.lock" &&
>>> + mkdir .git/rr-cache &&
>>> + : >.git/rr-cache/entry &&
>>> + >.git/MERGE_RR.lock &&
>>> + test_expect_rerere_gc git maintenance run --task=rerere-gc
>>> +'
>>> +
>>> test_expect_success '--auto and --schedule incompatible' '
>>> test_must_fail git maintenance run --auto --schedule=daily 2>err &&
>>> test_grep "cannot be used together" err
>>>
>>> base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
>> |
Since 2.54 unscheduled maintenance uses the "geometric" strategy, so
the "git maintenance run --auto --detach" behind every "git commit"
runs "git rerere gc" in the background whenever rr-cache has an entry.
That includes the "git commit" the sequencer runs for a resolved pick
on "git rebase --continue".
rerere_gc() takes MERGE_RR.lock through setup_rerere(), which uses
LOCK_DIE_ON_ERROR, and so does the sequencer's repo_rerere() at the
next conflict a few milliseconds later. Whichever comes second dies.
When it is the rebase, it dies in do_pick_commit() with the index
written but before make_patch() writes rebase-merge/{message,patch,
stopped-sha}, and every later "git rebase --continue" refuses with
"you have staged changes in your working tree". When it is the "git
commit" of a later continue, that one dies in its post-commit
repo_rerere() after the commit was made. Before 2.54 the same
collision needed an auto gc to actually run, since gc runs
"rerere gc" at its end.
A rebase with two conflicts in a row shows it. The filler makes the
pick slower than the ~5 ms the background task needs to take the
lock, and the stale entries give the gc something to prune, which
keeps the lock held for about half a second. It hit 6 of 6 runs here
on 2.55.0, and a test suite driving rebases on toy repositories with
a single rr-cache entry hit it in both runs that were traced:
git init -q -b main r && cd r
git config rerere.enabled true
git config maintenance.auto false
mkdir pad && seq 20000 | (cd pad && split -l 1 -a 5)
echo base >f && git add -A && git commit -qm base
git checkout -q -b topic
echo b >f && git commit -qam B
echo c >f && git commit -qam C
git checkout -q main
echo a >f && git commit -qam A
git repack -adq
git ls-files -s pad | head -n 5000 |
awk '{print ".git/rr-cache/" $2}' | xargs mkdir -p
for d in .git/rr-cache/*/; do echo x >$d/preimage; done
touch -t 202001010000 .git/rr-cache/*/preimage
git config --unset maintenance.auto
git checkout -q topic
git rebase main
echo ab >f && git add f
GIT_EDITOR=true git rebase --continue
The continue dies with "Unable to create '.git/MERGE_RR.lock': File
exists" while the gc spawned by its own commit holds the lock, and
after resolving C every further continue refuses. Maintenance stays
off during the setup so that no repack is pending: a repack due at
that commit runs ahead of rerere-gc in the task list and would spend
the window.
The gc needs the lock: it removes every rr-cache directory it finds
empty, and a rerere that has just created its directory but not yet
written the preimage looks exactly like that. So keep the lock and
stop dying over it. A caller that finds the lock held now waits for
rerere.lockTimeout milliseconds, with the semantics of
core.packedRefsTimeout and the same default of 1000, and then warns
and goes on without rerere: a merge, a commit or a pick loses one
recording or replay, which is nothing next to a rebase that cannot
continue. The gc itself never waits, since it has nothing to lose
from a skipped run, and "git rerere", "git rerere forget" and "git
rerere clear" keep dying, since they exist for nothing but the state
behind the lock. The clearing "git am" and "git rebase" do on --abort
and --skip goes on without it: the cleanup that follows removes
MERGE_RR anyway, and the unresolved entries it would have dropped are
left for the gc. A stale MERGE_RR.lock, which used to stop every
merge, now costs each command a second and a warning until it is
removed.
That rebase now stops at C the normal way, with its preimage recorded
once the prune is over, and continues once C is resolved. The tests
cover the gc under a held lock, directly and through the maintenance
task, a merge that waits a lock out within rerere.lockTimeout, a
merge, a commit and a rebase that go on without rerere once it is
up, "git rebase --abort" doing the same, and the three explicit
commands failing.
Assisted-by: Claude Fable 5.1
Signed-off-by: Thomas Bachem <mail@thomasbachem.com>
thomasbachem
force-pushed
the
rerere-gc-lock
branch
from
September 4, 2026 07:39
ecd9e5b to
bbb2338
Compare
Author
|
/preview |
|
Preview email sent as pull.2214.v2.git.1788507662888.gitgitgadget@gmail.com |
Author
|
/submit |
|
Submitted as pull.2214.v2.git.1788507876543.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes since v1:
instead of dying, so a merge, commit or pick goes on without rerere
(Phillip). Only "git rerere", "git rerere forget" and "git rerere
clear" keep dying, through a RERERE_LOCK_OR_DIE flag. rerere_clear()
takes flags for that, since "am" and "rebase" call it too and go on.
does, not waiting is what sets the gc apart. The wait and its default
stay (Patrick).
(Patrick): the lockfile message tells the user to terminate other
git processes and try again, which a command that goes on without
rerere should not say. It names the lock file, says that rerere is
skipped, and carries the errno, so the reason still shows.
something to prune and the run still triggers a gc that only prunes
what is due, as with Patrick's heuristic series. The prune holds the
lock for about half a second here, within the default timeout, so
the fixed rebase records the second conflict rather than skip it.
ones cover a commit and a rebase going on under a held lock, "git
rebase --abort", and the three explicit commands. test_grep
throughout.
and its script (Phillip). What waits, what goes on and what keeps
dying is spelled out, and the script has one continue, not two.
Still based on maint, where the bug ships (2.54.0 and 2.55.0). Merged
up it conflicts with d43f701 (lockfile: add
repo_hold_lock_file_for_update{,_timeout}{,_mode}(), 2026-07-14) in
setup_rerere(), where the resolution takes the repo-scoped helper.
With that, t4200 and t7900 pass on top of Patrick's series and with
the sequencer series that keeps auto maintenance out of a rebase, sent
separately.
Cc: Patrick Steinhardt ps@pks.im
Cc: Phillip Wood phillip.wood@dunelm.org.uk
Cc: Junio C Hamano gitster@pobox.com