Classify transient S3 failures as retryable

[ethanstoner]

AGENTS.md §5 says S3 and GCS are both first class and that "GCS only" is a bug. This is one.

What is wrong

classify_put_error maps everything that is not a precondition failure to StoreError::Other, and classify_list_error is unconditionally Other. head, delete and the five multipart paths do the same. The result is that no S3 operation can produce StoreError::Retryable — the only exception is the presigned-GET path (s3.rs:195), which classifies 5xx/429 correctly because it reads the raw HTTP status.

GCS does classify them, at gcs.rs:1186: gRPC Unavailable/DeadlineExceeded/ResourceExhausted/Internal/Aborted, HTTP 503/504/429/500, plus connect/IO/transient faults.

Why it matters

Two places read StoreError::is_retryable:

coord::cas_update — the manifest CAS, which per Principle II is the only commit point. On Retryable it sleeps with jittered backoff and re-reads; on anything else it returns CoordError::Store and gives up. So a throttled manifest PUT fails the push outright on S3, while the same throttle on GCS is absorbed. S3 partitions by key prefix, so a hot manifest key under a monorepo push rate is exactly the shape that draws 503 SlowDown.

smart::wal_err — maps retryable store errors to 503 ServiceUnavailable and everything else to 500 Internal. Its comment reads "A store call that timed out / was throttled: fail fast, let the client retry (never hang the request on the bucket)" — on S3 that branch is unreachable, so the git client gets a hard 500 instead.

The SDK's own retries sit underneath this, not in place of it. I confirmed against a fake service that the client makes three attempts and then surfaces SdkError::ServiceError with code: "SlowDown" and raw status 503 — which walgit then discarded as permanent.

The change

is_retryable(&SdkError<E>): dispatch failures and timeouts (no response at all), the transient AWS error codes, and the transient HTTP statuses — the status check matters because rustfs and other S3-compatible stores do not all use AWS error codes. Every SdkError site routes through it: put, list, head, delete, and multipart create/upload/upload-copy/complete/abort.

Precondition handling is untouched, and there is a regression test pinning it.

Tests

A fake S3 on an ephemeral port, driving real SdkError values through the classifiers with SDK retries disabled so each test observes exactly the error the service produced. Hermetic, no bucket, no network, runs in 0.01s.

• throttling (503 SlowDown), server fault (500 InternalError), unrecognised transient status (504), and an unreachable endpoint → Retryable
• access denied (403) → Other
• failed precondition (412) → PreconditionFailed, unchanged
• the code and status tables, as table-driven unit tests

Reverting just the two classifier arms turns the five behavioural tests red, so they gate the fix rather than describing it.

Verification

Gate	Result
cargo test -p walgit-store	56 + 4 pass, 0 fail
just test-s3 (contract vs rustfs in Docker)	s3_contract passes
cargo test -p walgit-server --test sim	20/20 pass
cargo fmt --check	clean
clippy on walgit-store	40 findings on main → 39 on this branch; none new, one needless_pass_by_value removed

Three gates fail identically on pristine main (6d8fa54) in my environment, so I could not use them as signal and did not try to fix them here:

• just clippy fails at walgit-server/build.rs on two clippy::expect_used in the build script — the strict gate from Add a strict clippy gate to the justfile and CI #15 does not spare build scripts. walgit-proto's generated code adds 27 more under -D warnings.
• just test — three setup::tests installer tests fail on main.
• just e2e — 7 fail on main, 6 on this branch (the difference is fetch_from_front_that_serves_the_base_remotely, which AGENTS.md documents as ~1-in-3 flaky). My git is 2.43; the README asks for ≥2.46, which likely explains these.

Happy to split the multipart sites out if you would rather keep the diff to the CAS path.