Skip to content

bootstrap: push the trunk first and group small branches - #130

Merged
nodo merged 3 commits into
mainfrom
nodo/ent-3617-trunk-first-grouped-branches
Oct 27, 2026
Merged

nodo merged 3 commits into
mainfrom
nodo/ent-3617-trunk-first-grouped-branches

Conversation

@nodo

@nodo nodo commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Batched bootstrap now plans and pushes the trunk first, plans every other branch in one commit-graph fetch, and pushes small branches in shared packs (ENT-3617).

Problem

planBatches fetched each branch's commit graph with its own request, one at a time, and pushed nothing until every branch was planned. On a repository with thousands of branches, planning outlasted a 30-minute sync budget, so every attempt timed out with nothing on the target and each retry started over.

Change

  • The trunk is planned and pushed first, so a timeout later in the sync keeps the default branch.
  • The remaining branches share one tree:0 commit-graph fetch (all tips as wants, trunk as have) instead of one request each.
  • Small branches are pushed in shared packs that create their refs directly. A group over the pack budget splits in half; a branch too big to share a pack keeps its own checkpointed batch.
  • FetchCommitParents now takes its wants as a list of hashes.
  • Test harness: the git-http-backend test server buffers chunked request bodies, which net/http/cgi rejects.

Measurements

Against github.com/openclaw/openclaw (3,904 branches), from a laptop:

  • Before (v0.10.0, per-branch fetch): ~450 ms per branch request, ~25 minutes of planning.
  • After: one fetch with 3,904 wants and the trunk as have returned 21,831 commits in 19.5 s, no error. GitHub accepts the large want list.

Tests

go test ./internal/strategy/bootstrap/ ./internal/gitproto/ ./internal/syncer/ passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FU3noNxsCP6TVRKqEmGsqS


Note

Medium Risk
Rewrites batched bootstrap planning and execution for multi-thousand-branch repos; behavior changes are substantial but scoped to bootstrap with extensive new tests and an integration path.

Overview
Batched bootstrap now makes progress on branch-heavy repositories (ENT-3617) by changing when planning and pushing happen, instead of fetching every branch’s commit graph up front before any data reaches the target.

The trunk is planned and pushed first; non-trunk branches are planned only after trunk batches run (planAfterTrunk), so a timeout mid-sync still leaves the default branch on the target. FetchCommitParents accepts multiple want hashes in one protocol v2 tree:0 fetch, so remaining branches share a single commit-graph round trip.

Small branches are batched into shared packs (branch_group.go): one fetch and one push create many refs directly (no temp ref). Groups over the pack budget split in half; branches with resume markers or over the size budget keep the existing per-branch checkpoint path.

Tests add unit coverage for trunk ordering, grouping, splits, and standalone fallbacks, plus a git-http-backend integration test that buffers chunked push bodies so CGI receive-pack works with streamed uploads.

Reviewed by Cursor Bugbot for commit b716b71. Configure here.

Batched bootstrap fetched the commit graph of every branch, one request
per branch, before it pushed anything. On a repository with thousands
of branches that planning outlasted the 30-minute sync budget, so every
attempt timed out with nothing on the target (ENT-3617).

The trunk is now planned and pushed first. The remaining branches share
one commit-graph fetch. Small branches are pushed in shared packs that
create their refs directly. A group over budget splits in half, and a
branch too big to share a pack keeps its own checkpointed batch.

FetchCommitParents takes its wants as a list of hashes. The
git-http-backend test server buffers chunked request bodies, which
net/http/cgi rejects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU3noPxGFpQ3TEX4SRyccF
Copilot AI balanced review requested due to automatic review settings October 27, 2026 21:08
pfleidi
pfleidi previously approved these changes Oct 27, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved pack-limit handling and retry-state issues can fail grouped fetches, replay confirmed ref creations, and stall checkpointed or resumed bootstrap.

4 open issues
What changed in this PR

Improves batched bootstrap for branch-heavy repositories by pushing the trunk before planning other branches and sharing fetches and packs across small branches.

Changes:

  • Defers remaining branch planning until after the trunk is pushed, using one multi-tip commit-graph fetch.
  • Groups small branches into shared packs, splitting oversized groups and retaining checkpointed fallbacks.
  • Adds grouping tests and a many-branch integration test, with chunked-request buffering in the test server.
File Description
internal/​syncer/​git_http_backend_test.go Buffers chunked requests and tests many-branch bootstrap.
internal/​strategy/​bootstrap/​github_preflight_test.go Updates the commit-graph fetch mock signature.
internal/​strategy/​bootstrap/​branch_group.go Adds shared branch planning, grouped pushes, and splitting.
internal/​strategy/​bootstrap/​branch_group_test.go Tests trunk ordering, grouping, splitting, and resume fallbacks.
internal/​strategy/​bootstrap/​bootstrap.go Integrates deferred planning and grouped execution.
internal/​strategy/​bootstrap/​bootstrap_test.go Adapts mocks and graph helpers to multiple wants.
internal/​gitproto/​fetch.go Supports multiple commit-graph wants in one request.
internal/​gitproto/​fetch_test.go Updates capability tests for the new fetch signature.
CHANGELOG.md Documents the bootstrap improvements.

Review effort: Balanced

Comment on lines +465 to +469
if i == len(batches) && deferred != nil {
more, err := planAfterTrunk(ctx, p, *deferred)
deferred = nil
if err != nil {
return result, err

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit b716b71. Configure here.

A group push used a copy of the batch budget. When the target cut or
rejected it, the group split but the run kept the old budget, the old
calibration and no announced limit, so every later group and every
level of splitting re-sent close to the cutoff.

A rejected group push now updates the shared budget, its provenance,
the calibration and the announced limit by the same rules as a
checkpoint push. It splits into as many parts as the observed size
needs, not always two. planAfterTrunk sizes groups with the current
budget, which the trunk may already have lowered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU3noPxGFpQ3TEX4SRyccF
- A branch too big to share a pack now runs on its own checkpoints
  before the groups still pending. When branches share a large history,
  it lands once and becomes a have, instead of being re-fetched by
  every smaller split.
- A group whose union passes MaxPackBytes splits instead of failing the
  run. gitproto.ErrSourcePackTooLarge identifies that read error. A
  source-side limit does not change the target budget.
- A failed group push records the refs it still created, and the retry
  leaves them out. The pusher splits ref updates across requests, so an
  earlier request can land before a later one fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU3noPxGFpQ3TEX4SRyccF
@nodo
nodo merged commit ca4404a into main Oct 27, 2026
5 checks passed
@nodo
nodo deleted the nodo/ent-3617-trunk-first-grouped-branches branch October 27, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants