Skip to content

feat: add S3 gateway service - #163

Merged
forhappy merged 3 commits into
mainfrom
codex/crab-s3-gateway
Sep 8, 2026
Merged

feat: add S3 gateway service#163
forhappy merged 3 commits into
mainfrom
codex/crab-s3-gateway

Conversation

@forhappy

@forhappy forhappy commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a runnable crab-s3-gateway service backed by configured Crab repositories
  • support authenticated S3 object, listing, copy, delete, checksum, conditional read, range, and durable multipart workflows
  • stream request bodies, multipart parts/completion, LFS-backed reads, ranges, and copies through bounded memory
  • publish mutations through canonical Crab locks, generations, Git objects, attributes, and readiness evidence
  • add configuration, initialization, container packaging, operator documentation, and a frozen protocol contract
  • share publication readiness between the gateway and crab-http-server

Compatibility

Existing S3 clients can connect by configuring the gateway endpoint, region, access key, and secret key. The initial contract uses path-style addressing. Single PUTs, copies, and multipart parts support up to 5 GiB; multipart completion supports S3 general-purpose bucket objects up to 50 TB. Large objects use the verified LFS content path while preserving logical S3 sizes and ETags. Unsupported S3 facilities are rejected explicitly and documented in crab/docs/architecture/s3-gateway-contract.md.

Validation

  • cargo test -p crab-s3-gateway --locked
  • cargo clippy -p crab-s3-gateway --all-targets --locked -- -D warnings
  • cargo test -p crab-write --locked
  • cargo test -p crab-http-server maintenance --lib --locked
  • cargo check -p crab-http-server --locked
  • cargo fmt --all -- --check
  • make architecture-check
  • live AWS CLI v2 smoke against a RustFS-backed Crab repository, covering signed and presigned requests, object CRUD, listing pagination, ranges, conditions, checksums, copy, multi-delete, multipart durability and retry, restart recovery, and independent Git visibility
  • live 300 MiB single PUT and three-part multipart upload, with HEAD/LIST/full GET/range GET and byte-identical MD5 verification; gateway RSS was about 98 MiB after transfers

All Cargo verification used /Volumes/Workspace/crabbuild-target/crab-5623-s3-gateway.

@forhappy
forhappy force-pushed the codex/crab-s3-gateway branch from a43293e to 1b549ed Compare September 8, 2026 16:15
@forhappy
forhappy merged commit 9edc4ca into main Sep 8, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant