Go SDK 4/8: resilience core

[bkeroack]

Fourth of eight PRs implementing the Go SDK per SATD_GO_SDK_PLAN.md. Stacked on #514 — merge order: #512 → #513 → #514 → this.

Adds the reconnect-and-replay layer: CursorStore (+ FileCursorStore), Backoff, LagPolicy, ResilientSubscription.

Durability

Commit-on-poll, matching the Rust SDK: an event's cursor is persisted only once the caller comes back for the next one, which is an implicit ack. The store never advances past an event the caller has not finished with, so a crash mid-processing replays that event rather than skipping it — at-least-once, not at-most-once. Arming happens in Next rather than on the pump goroutine, so an immediately following Commit checkpoints the event just returned.

FileCursorStore writes via os.CreateTemp + fsync + rename. The temp name must be unique rather than derived — two subscriptions sharing a cursor path in one process would otherwise collide and rename each other's partial file (caught by a test that runs eight concurrent writers). The on-disk format is byte-identical to the Rust SDK's, so a Go and a Rust consumer can share a cursor file.

Cancel safety

From the shape rather than an explicit state machine: the reconnect loop runs on its own goroutine and hands events over an unbuffered channel, so it is never more than one event ahead of the caller and a cancelled Next cannot consume one. A gRPC Recv cannot be abandoned mid-flight, which is why the pump exists at all.

Lag

A lag re-anchor bumps a generation counter. The pump cannot simply clear an armed cursor because the caller may not have armed it yet; stamping generations lets the superseded arm be recognized and dropped whenever it lands. Under LagAutoResume the notice is consumed and the recovery cursor persisted immediately; under LagSurface it reaches the caller unchanged.

Behavior change: Subscribe waits for stream acceptance

Subscribe now waits for the server's response headers before returning. grpc-go starts a server-streaming call without waiting for the server to accept it, so "subscribe, then mine a block" could silently miss the block — a bad default for an event API. tonic awaits headers as a matter of course, so this is parity with the Rust SDK, not a new invention; it costs one round trip (~1ms measured). Deliberately not done for the bidirectional Watch stream, where a server is free to withhold headers until it has something to send.

Tests

Unit: backoff growth/clamping, cursor-store round trip and corruption rejection (each field parsed at its real width, so a wrapped u32 errors instead of silently resuming from the wrong height), concurrent writers, reconnect replay anchoring, commit-on-poll ordering, store seeding and write elision, ReplayGap synthesis on a clamped replay, both lag policies, non-retryable surfacing, retry budget, cancel safety, store-failure surfacing without losing the held-back event.

E2E: restart-resume across a real satd restart (new port, new publisher instance id), reconnect through a cut TCP proxy, and both lag policies — forced deterministically with a shrunk node broadcast buffer (SATD_EVENT_BROADCAST_CAPACITY) plus pinned HTTP/2 windows, since grpc-go floors the window at 64 KiB where tonic can go lower. The auto-resume test asserts no block is lost, not merely that no Lagged surfaced.

All 16 new assertions were perturbed once and observed to fail.

Harness fixes

Per-node environment (preserved across restart) for node-side test knobs; the RPC cookie re-read on every readiness poll — satd mints a fresh one per start, so caching it made a restarted node look dead; mineUntilSeen for the Watch control path, which has no per-message ack.