Skip to content

Add content-idempotent path upload coordinator - #413

Draft
Hossain007 wants to merge 2 commits into
boto:developfrom
Hossain007:content-idempotent-path-upload
Draft

Add content-idempotent path upload coordinator#413
Hossain007 wants to merge 2 commits into
boto:developfrom
Hossain007:content-idempotent-path-upload

Conversation

@Hossain007

Copy link
Copy Markdown

Summary

Add a small synchronous coordinator for path uploads that compares full-object SHA-256 content before submitting an existing classic or CRT managed transfer.

The coordinator:

  • hashes the local file in bounded chunks;
  • calls HeadObject with checksum mode enabled;
  • returns False without starting a transfer when content matches;
  • creates a missing key with IfNoneMatch='*';
  • replaces changed content with IfMatch set to the observed ETag;
  • supplies the computed ChecksumSHA256 to the transfer;
  • rejects caller-provided checksum and conditional-write arguments;
  • propagates conditional conflicts without hidden retries; and
  • raises a dedicated exception when an existing object has no comparable full-object SHA-256.

This PR intentionally supports paths only. It does not add a file-object API.

Stacked dependency

This draft is stacked on #371. The first commit is a rebase of that PR's conditional-write patch onto current develop; it is not a separate implementation of, or claim of authorship over, #371. After #371 merges, this branch should be rebased so the upstream diff contains only the coordinator and its tests.

The proposed Boto3 API is tracked in boto/boto3#4823.

Tests

  • python -m pytest tests/unit tests/functional -q — 758 passed
  • pre-commit run --all-files — passed
  • Classic single-part and multipart requests assert the exact HEAD, PUT, and multipart completion parameters.
  • CRT functional coverage verifies full-checksum conditional headers.
  • Live CRT single-part and multipart integration tests are included but were not run locally because AWS credentials were unavailable.

The supported Python matrix and live AWS coverage remain CI/reviewer gates.

Development note

AI tools assisted with implementation and test iteration. Human review is pending while this PR remains a draft.

amartani and others added 2 commits July 30, 2026 00:50
Adds `IfMatch` and `IfNoneMatch` to the list of allowed arguments for S3 uploads. This enables users to perform conditional uploads, such as preventing the overwrite of existing files by using `IfNoneMatch: "*"`.

These parameters are not supported by the `CreateMultipartUpload` API call, so they are added to the blocklist for that specific operation to prevent validation errors. For multipart uploads, these parameters are passed to the `CompleteMultipartUpload` operation.

Functional tests have been added for both single-part and multipart uploads to verify that the arguments are passed correctly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants