Go SDK 5/8: resilient watch - #516
Open
bkeroack wants to merge 1 commit into
Open
Conversation
Adds ResilientWatch: a Watch stream that reconnects, re-registers its watch-set, and re-anchors its cursor without the consumer noticing. The client-side mirror is the existing WatchSet, which already holds exactly the right shape (net sets, keyed so a re-assertion overwrites). This adds the removals that balance its Add* methods, a rendering back into the control messages that reconstruct it on a fresh stream, and a diff for the reload counts. The replay rendering has two hazards worth naming: - Lifecycles and depth alarms are the same wire message, and the node tells them apart ONLY by whether min_depths is empty. Getting that backwards would silently convert every lifecycle watch into a depth alarm, so it has its own test. - min_values is either empty or exactly parallel to scripthashes; a ragged pair misaligns every floor. Rendering is sorted rather than left to Go's map order, so a replay is byte-identical run to run and diffable against the Rust mirror in PR 7. One-shot watches the node auto-evicts when they fire - a depth alarm that reached its depth, a finalized lifecycle - are pruned from the mirror as those events arrive, so a reconnect does not re-register a completed watch and burn quota on it. The WatchSetLoader closes the two gaps a pure in-memory mirror leaves: after a process restart the mirror is empty, and it goes stale if the integrator's truth changes while the stream is down. With a loader the mirror becomes a cache of that truth, rebuilt on every connect before any event is pumped. A loader failure is transient and retried on the next connect; a retry budget converts a permanently failing one into a real error rather than a consumer that silently never yields. Reload applies the truth to a live stream as one atomic SetWatchSet, letting the node reconcile under its own lock rather than sequencing client-computed Add*/Remove* messages whose ordering could strand coverage or double-charge a quota mid-swap. Two deliberate divergences from the Rust SDK, both simplifications the language allows: - The transient-reject retry runs on the pump goroutine, so there is no cancellable caller future that could strand a charged-but-unsent retry. Rust needs an explicit PendingReanchor state machine for this. - Cancel safety comes from the unbuffered handoff, as in ResilientSubscription. Also documents a sharp edge the Rust SDK leaves implicit: a reconnect re-anchors the CHAIN stream but does NOT replay watch MATCHES, because the node's cursor replay synthesizes events from the block index and never runs the matcher. A payment confirming while the stream is down produces no ScriptMatched on reconnect - Rescan is the remedy. An E2E test pins both halves, so the day the node changes, the docs change with it. E2E covers reconnect re-registration through a cut TCP proxy, loader rebuild from a truth that moved while disconnected, atomic reload (including that the replaced watch is genuinely gone), restart-resume plus rescan recovery, and a fired alarm not being re-armed. Every new assertion was perturbed once and observed to fail.
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fifth of eight PRs implementing the Go SDK per
SATD_GO_SDK_PLAN.md. Stacked on #515 — merge order: #512 → #513 → #514 → #515 → this.Adds
ResilientWatch: aWatchstream that reconnects, re-registers its watch-set, and re-anchors its cursor without the consumer noticing.The mirror
The client-side mirror is the existing
WatchSet— it already holds exactly the right shape (net sets, keyed so a re-assertion overwrites rather than duplicates, which is the node's own semantics). This PR adds the removals that balance itsAdd*methods, a rendering back into the control messages that reconstruct it on a fresh stream, and a diff for the reload counts.Two hazards in that rendering get their own tests:
AddTransactions), and the node tells them apart only by whethermin_depthsis empty. Getting it backwards would silently convert every lifecycle watch into a depth alarm.min_valuesis either empty or exactly parallel toscripthashes— a ragged pair misaligns every floor.Rendering is sorted rather than left to Go's randomized map order, so a replay is byte-identical run to run and diffable against the Rust mirror in PR 7.
One-shot watches the node auto-evicts when they fire — a depth alarm that reached its depth, a finalized lifecycle — are pruned from the mirror as those events arrive, so a reconnect does not re-register a completed watch and burn quota on a done txid.
The loader
WatchSetLoadercloses the two gaps a pure in-memory mirror leaves: after a process restart the mirror is empty, and it goes stale if the integrator's truth changes while the stream is down. With a loader configured the mirror becomes a cache of that truth, rebuilt on every connect before any event is pumped. A loader failure is transient and retried on the next connect (a momentary database outage must not kill an at-least-once consumer); aMaxRetriesbudget converts a permanently failing loader into a real error rather than a consumer that silently never yields.Reloadapplies the truth to a live stream as one atomicSetWatchSet, letting the node reconcile under its own lock — no client-computedAdd*/Remove*sequence whose ordering could strand coverage or double-charge a quota mid-swap.SetWatchSetdoes not carry the raw-tx opt-in, so that is reconciled separately and in both directions.Divergences from the Rust SDK
Both are simplifications Go allows, not behavior changes:
PendingReanchorstate machine for exactly this.ResilientSubscription.A documented sharp edge
A reconnect re-anchors the chain stream but does not replay watch matches — the node's cursor replay synthesizes events from the block index and deliberately never runs the watch matcher (that needs full block bodies and undo, which is what
RescanBlocksis for). A payment confirming while the stream is down produces noScriptMatchedon reconnect;Rescanover the gap is the remedy.The Rust SDK behaves the same way but does not say so anywhere.
ResilientWatch's doc comment now spells it out, andTestE2EMissedMatchesNeedARescanpins both halves — the match does not arrive on its own, and the rescan recovers it — so the day the node changes, the docs change with it.Tests
Unit (against a scripted bidi server): replay of every kind and its ordering, the lifecycle/alarm distinction, parallel
min_values, determinism, removals not replaying, offline edits landing on the next connection, re-anchoring to the resume cursor, transient reject retried internally vs terminal reject surfaced,CursorAcceptedadopted as the resume anchor, one-shot pruning, loader canonicality/transience/budget, all four reload paths, cursor seeding, commit-on-poll, cancel safety, and a race-detector exercise drivingNextand registrations concurrently.E2E: reconnect re-registration through a cut TCP proxy, loader rebuild from a truth that moved while disconnected, atomic reload (including that the replaced watch is genuinely gone, not merely unmentioned), restart-resume plus rescan recovery of the missed matches, and a fired alarm not being re-armed.
All 28 new assertions were perturbed once and observed to fail.