Skip to content

fix(store): uniform compose part sizes for Cloudflare R2 - #33

Open
gonewx wants to merge 2 commits into
tobi:mainfrom
gonewx:fix/r2-compose-uniform-parts
Open

fix(store): uniform compose part sizes for Cloudflare R2#33
gonewx wants to merge 2 commits into
tobi:mainfrom
gonewx:fix/r2-compose-uniform-parts

Conversation

@gonewx

@gonewx gonewx commented Aug 29, 2026

Copy link
Copy Markdown

Problem

Bundle builds against Cloudflare R2 fail forever with:

bundle task: store error: s3 complete multipart: service error

(repeats every maintenance pass; the server log shows receive-pack-side pushes are fine, only compose fails at CompleteMultipartUpload).

Root cause

R2 requires every non-trailing part of a multipart upload to have the same length (it answers InvalidPart: All non-trailing parts must have the same length.), whereas S3 only requires non-trailing parts to be >= 5 MiB.

compose cut copy ranges at COPY_PART = 1 GiB. For the standard bundle shape — a small bundle header in front of a large pack — the layout is one 5 MiB uploaded part (header + pack prefix) followed by one large copied part (the rest of the pack), so R2 rejects the Complete. The contract test's compose case only exercises a ~1 MiB trailing copy, a layout that happens to be legal, which is why this never showed up before.

Fix

Cut copy parts at MIN_PART instead of 1 GiB, so every non-trailing part — uploaded or copied — is exactly 5 MiB and only the trailing part may differ. S3 accepts uniform parts just as well; part counts stay small (a 5 GiB compose needs <= 1024 copy requests, a 30 GB one ~6k, below the 10k part limit).

Verification

  • Reproduced against a real R2 bucket with a raw-SDK probe (31-byte header + 46 MiB source → UploadPart(5 MiB) + UploadPartCopy(41 MB)): before the fix Complete fails with InvalidPart, after it succeeds and the content round-trips (head, tail, total size).
  • Full store contract suite against R2 (virtual-hosted style, region auto): all cases pass, including test_compose.
  • Production: a 48.7 MiB weekly bundle now builds successfully on R2 (bundles/list.pb published).

Second commit is a small test affordance: s3_contract hardcodes path-style addressing and us-east-1, which makes it impossible to run the suite against R2/OSS; the two env overrides keep rustfs defaults unchanged.

…non-trailing part sizes

R2 rejects CompleteMultipartUpload when non-trailing parts differ in
length ("InvalidPart: All non-trailing parts must have the same
length."), while S3 only requires >= 5 MiB. The 1 GiB COPY_PART produced
a 5 MiB uploaded part followed by an arbitrarily-sized copied part for
the common bundle shape (small header + large pack), so every bundle
build against R2 failed at Complete.

Cut copy parts at MIN_PART so every non-trailing part — uploaded or
copied — is exactly 5 MiB and only the trailing part may differ. S3
accepts uniform parts equally; a 5 GiB compose now needs <= 1024 copy
requests, a 30 GB one ~6k (below the 10k part limit).
WALGIT_TEST_S3_REGION and WALGIT_TEST_S3_FORCE_PATH_STYLE override the
hardcoded us-east-1 / path-style defaults, so the contract suite can run
against endpoints that only accept virtual-hosted addressing (R2, OSS).
Defaults unchanged for rustfs.
@0bserver07

Copy link
Copy Markdown

The R2 constraint is real (every non-trailing part the same size), but making COPY_PART 5 MiB everywhere changes the cost of the thing compose exists for: the weekly full bundle of a 32 GB base goes from ~32 UploadPartCopy requests at 1 GiB parts to ~6,500, and the docs/ROUNDTRIPS.md budget table still lists compose at about one request per source. S3 and GCS don't need uniform parts, so this pays a ~200x request increase on stores that didn't ask for it. Maybe a store.s3.compose_part_size (default 1 GiB, 5 MiB for R2) plus a ROUNDTRIPS row? The env-var additions to the contract test look handy either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants