feat: fail-fast retries: jittered backoff bounded by time, not error type

CLAUDE.md

@@ -97,6 +97,10 @@ Don't document deprecated flags, options, or APIs. User-facing docs (`/doc`), `-
 
 The rename/removal rationale lives in the commit message and PR description, not in docs that users read. Warning someone who *uses* the deprecated path is not just fine but encouraged -- at compile time (Rust's `#[deprecated(note = "...")]`) or at runtime (a log line). Those fire on use, so they reach the one person who needs them and nobody else; they aren't documentation. A standing note in the docs that advertises the dead name is what's banned.
 
+## Retries
+
+Fail fast; retry only what a few seconds can fix. Every retry loop uses capped exponential backoff with jitter, inlined at the call site (no shared `Backoff` type), and is bounded by *time*, not by error type: a short budget (~10s), then surface the last real error. Don't classify errors as retryable (`is_retryable()`); an ephemeral failure is one that clears within the budget, so the budget is the classifier. The one exception is an answer a peer actually sent, where the protocol defines the meaning: an HTTP status short-circuits (a `404` fails immediately; `408`/`429`/gateway statuses ride the backoff) via the `status()` accessors on `moq_native::Error` / `moq_hls::Error`. The only unbounded loops are process-lifetime supervisors with nobody to return an error to (cluster peers, device reopen, accept loops); they retry forever but cap the delay at seconds and warn per attempt, loudly broken rather than silently parked. Exactly one layer owns a retry: an outer loop that rebuilds an inner one resets its escalation, so watch the inner loop's terminal signal instead.
+
 ## Root Cause First
 
 - Before fixing a bug, reproduce it and explain the mechanism. A fix that adds a retry, sleep, widened timeout, defensive check, or call-site special case without a stated mechanism is a symptom patch, not a fix.

doc/bin/gstreamer.md

@@ -41,9 +41,13 @@ signal when it changes, so you can poll it via `g_object_get` or connect to `not
 | `estimated-send-bitrate` | uint64 | Estimated send bitrate in bits per second (congestion controller); 0 when unavailable |
 | `estimated-recv-bitrate` | uint64 | Estimated receive bitrate in bits per second; 0 when unavailable |
 
-`status` distinguishes a transient drop (`disconnected`, the reconnect loop is still retrying) from a
-permanent give-up (`failed`, a non-retryable error such as an auth rejection), which a bare
-`connected` bool cannot.
+`status` distinguishes a drop the reconnect loop is still retrying (`disconnected`) from a permanent
+give-up (`failed`), which a bare `connected` bool cannot. The sink retries for as long as the
+pipeline runs, so a relay outage of any length is ridden out; it goes `failed` only on an answer the
+relay actually gave that redialing cannot change: a rejected token, or a CONNECT answered with a
+status that isn't an invitation to retry. Everything else keeps retrying, so watch the logs when a
+sink stays `disconnected` from the very first attempt: a pipeline that has never connected once is
+far more likely misconfigured than waiting out an outage.
 
 ## Prerequisites
 

doc/bin/relay/cluster.md

@@ -117,6 +117,12 @@ Cluster peers must authenticate to each other:
 
 See [Authentication](/bin/relay/auth) for the full setup.
 
+Peers are redialed indefinitely, with exponential backoff and jitter so a restarting cluster doesn't
+reconnect in lockstep. That includes a peer that rejects us: a bad token logs `cluster peer error;
+will retry` on every attempt rather than giving up, so watch for a peer that never reaches
+`cluster peer session closed`. The delay escalates to ten seconds at most, so a dead or rejecting
+peer stays loudly visible in the logs and a returning one is picked up within seconds.
+
 ## Migration from older configs
 
 `cluster.root` was removed. To dial cluster peers use `cluster.connect`; to advertise this relay's own address set `cluster.node` and enable `cluster.mesh`. `cluster.mesh` is now a boolean gossip toggle (it used to take this relay's URL); the URL moved to `cluster.node`. The old `mesh = "<url>"` form still works for backwards compatibility: it enables gossip and is treated as `cluster.node`, with a deprecation warning (or an error if it conflicts with an explicit `cluster.node`).

js/CLAUDE.md

@@ -80,6 +80,7 @@ Plain custom elements built directly on `@moq/signals`, no framework (except moq
 
 ## Conventions
 
+- **Retry loops inline their backoff** (root Retries has the policy). Escalate a local delay toward a `max`, jitter each wait (`delay * (0.5 + Math.random() / 2)`), and hand it to `effect.timer`; a short time or attempt budget stops the loop, never a classification of the thrown value.
 - **Avoid callback parameters.** A function taking a `fn`/`create`/`onXxx` to invoke later reads poorly and hides control flow. Prefer returning a value the caller acts on, exposing a method or getter, or splitting into a couple of small calls the caller sequences itself (e.g. a cache `get()` then `insert(value)`, not `getOrCreate(key, () => value)`). Reserve callbacks for genuine event/subscription sinks where there is no alternative (`effect.subscribe`, DOM listeners, `Signal` subscriptions).
 - ESM only (`"type": "module"`). Relative imports include the `.ts`/`.tsx` extension in the lower-level packages (`net`, `signals`, `hang`); `rewriteRelativeImportExtensions` in `tsconfig.json` rewrites them to `.js` on build. Some higher-level packages (watch/publish) still omit extensions, so match the file you are editing.
 - Document every exported symbol and add a top-of-file `@module` doc block to each entrypoint (root convention; the published JSR/`.d.ts` docs render these). Use `@public` on the load-bearing classes.

js/net/src/connection/reload.ts

@@ -1,26 +1,31 @@
 import { Effect, type Getter, Signal } from "@moq/signals";
 import * as Announce from "../announced.ts";
+import { error } from "../error.ts";
 import type * as Path from "../path.ts";
 import { empty as emptyPath } from "../path.ts";
 import { type ConnectProps, connect, type WebSocketOptions, type WebTransportProps } from "./connect.ts";
 import type { Established } from "./established.ts";
 import type { Probe, Stats } from "./stats.ts";
 
-/** Exponential backoff settings for {@link Reload}'s reconnect loop. */
+/**
+ * Exponential backoff settings for {@link Reload}'s reconnect loop.
+ *
+ * The delays carry jitter, so a fleet of tabs knocked offline together doesn't reconnect in
+ * lockstep. Every failure is retried; {@link ReloadDelay.timeout} is what stops the loop.
+ */
 export type ReloadDelay = {
 	/** The delay in milliseconds before reconnecting (default: 1000). */
 	initial: DOMHighResTimeStamp;
 
 	/** The multiplier for the delay (default: 2). */
 	multiplier: number;
 
-	/** The maximum delay in milliseconds (default: 30000). */
+	/** The maximum delay in milliseconds (default: 5000). */
 	max: DOMHighResTimeStamp;
 
 	/**
 	 * Maximum total time in milliseconds to spend retrying before giving up (default:
-	 * 300000, 5 minutes). Resets after each successful connection. Set to 0 for
-	 * unlimited retries.
+	 * 10000). Resets after each successful connection. Set to 0 for unlimited retries.
 	 */
 	timeout?: DOMHighResTimeStamp;
 };
@@ -37,6 +42,14 @@ export type ReloadProps = Omit<ConnectProps, "signal"> & {
 	delay?: ReloadDelay;
 };
 
+/**
+ * How long to keep retrying before giving up, when {@link ReloadDelay.timeout} is unset.
+ *
+ * Short on purpose: a failure that clears within it was transient, and one that doesn't should
+ * surface as an error rather than leave the page silently reconnecting for minutes.
+ */
+const DEFAULT_TIMEOUT = 10000;
+
 /** Current state of a {@link Reload} connection. */
 export type ReloadStatus = "connecting" | "connected" | "disconnected";
 
@@ -80,15 +93,20 @@ export class Reload {
 	/** The reactive effect scope driving the connect loop; closed by {@link Reload.close}. */
 	#signals = new Effect();
 
-	/** Resolves when the reconnect loop stops via {@link Reload.close} or the retry timeout. */
+	/**
+	 * Resolves when the reconnect loop stops via {@link Reload.close}.
+	 *
+	 * Rejects when the loop gives up instead, carrying the failure that was in flight when the
+	 * retry window expired.
+	 */
 	closed: Promise<void>;
 	#closedResolve!: () => void;
 	#closedReject!: (err: Error) => void;
 
-	#delay: DOMHighResTimeStamp;
-
-	// Timestamp when the current retry sequence started (for timeout).
-	#retryStart: DOMHighResTimeStamp | undefined;
+	// The current wait between attempts, doubling per failure, and when the retry window expires.
+	// Both are undefined between sequences, so a later edit to `delay` applies to the next one.
+	#delay: DOMHighResTimeStamp | undefined;
+	#deadline: DOMHighResTimeStamp | undefined;
 
 	// Increased by 1 each time to trigger a reload.
 	#tick = new Signal(0);
@@ -102,18 +120,21 @@ export class Reload {
 	constructor(props?: ReloadProps) {
 		this.url = Signal.from(props?.url);
 		this.enabled = Signal.from(props?.enabled ?? false);
-		this.delay = props?.delay ?? { initial: 1000, multiplier: 2, max: 30000 };
+		this.delay = props?.delay ?? { initial: 1000, multiplier: 2, max: 5000 };
 		this.webtransport = props?.webtransport;
 		this.websocket = props?.websocket;
 		this.discovery = props?.discovery;
 
-		this.#delay = this.delay.initial;
-
 		this.closed = new Promise((resolve, reject) => {
 			this.#closedResolve = resolve;
 			this.#closedReject = reject;
 		});
 
+		// A caller is free to never await `closed`, and giving up rejects it unprompted. Marking the
+		// rejection handled here keeps that from surfacing as an `unhandledrejection`; a consumer
+		// awaiting the same promise still receives it.
+		this.closed.catch(() => {});
+
 		if (typeof window !== "undefined" && typeof document !== "undefined") {
 			this.#signals.event(window, "pagehide", () => this.#suspended.set(true));
 			this.#signals.event(window, "pageshow", () => this.#suspended.set(false));
@@ -190,9 +211,9 @@ export class Reload {
 	}
 
 	/**
-	 * Schedule the next connect attempt after the current backoff, or give up when the
-	 * retry window has expired. `connected` is when the dead session was established, if
-	 * it ever was, and `cause` the error that killed it, if it died with one.
+	 * Schedule the next connect attempt after the current backoff, or stop once the retry window
+	 * has expired. `connected` is when the dead session was established, if it ever was, and
+	 * `cause` the error that killed it, if it died with one.
 	 */
 	#retry(effect: Effect, connected: DOMHighResTimeStamp | undefined, cause?: unknown): void {
 		// Any session is dead now: report disconnected during the backoff rather than
@@ -205,29 +226,29 @@ export class Reload {
 		// shorter is a peer that accepts and immediately severs, which has to keep
 		// escalating or we hammer it forever at the initial delay.
 		if (connected !== undefined && performance.now() - connected >= this.delay.initial) {
-			this.#delay = this.delay.initial;
-			this.#retryStart = undefined;
+			this.#delay = undefined;
+			this.#deadline = undefined;
 		}
 
-		// Track retry start for timeout.
-		this.#retryStart ??= performance.now();
-
-		const timeout = this.delay.timeout ?? 300000;
-		if (timeout > 0) {
-			const elapsed = performance.now() - this.#retryStart;
-			if (elapsed >= timeout) {
-				console.warn("reconnect timed out");
-				// A graceful close has no error, so report the timeout itself.
-				if (cause === undefined) this.#closedReject(new Error("reconnect timed out"));
-				else this.#closedReject(cause instanceof Error ? cause : new Error(String(cause)));
-				return;
-			}
-		}
+		const now = performance.now();
+		const timeout = this.delay.timeout ?? DEFAULT_TIMEOUT;
+		this.#delay ??= this.delay.initial;
+		this.#deadline ??= timeout > 0 ? now + timeout : Number.POSITIVE_INFINITY;
 
-		const tick = this.#tick.peek() + 1;
-		effect.timer(() => this.#tick.update((prev) => Math.max(prev, tick)), this.#delay);
+		if (now >= this.#deadline) {
+			console.warn("reconnect timed out");
+			// A graceful close has no error, so report the timeout itself.
+			this.#closedReject(cause === undefined ? new Error("reconnect timed out") : error(cause));
+			return;
+		}
 
+		// Equal jitter, so a fleet of tabs knocked offline together doesn't reconnect on the same
+		// tick, and never past the deadline the retry window promised.
+		const wait = Math.min(this.#delay * (0.5 + Math.random() / 2), this.#deadline - now);
 		this.#delay = Math.min(this.#delay * this.delay.multiplier, this.delay.max);
+
+		const tick = this.#tick.peek() + 1;
+		effect.timer(() => this.#tick.update((prev) => Math.max(prev, tick)), wait);
 	}
 
 	/**

js/publish/src/source/camera.ts

@@ -76,9 +76,10 @@ export class Camera {
 		const constraints = effect.get(this.constraints);
 
 		if (!this.#retry.begin(effect, [device, constraints])) {
-			// Out of budget with the same settings. Only a change to what is plugged in is new
-			// information worth another attempt, so watch the device list here and not while
-			// healthy, where a rerun would restart a working capture for unrelated device churn.
+			// Waiting out a backoff, or out of budget entirely, with the same settings. Either way
+			// a change to what is plugged in is new information worth acting on now, so watch the
+			// device list here and not while healthy, where a rerun would restart a working capture
+			// for unrelated device churn.
 			const spent = this.device.out.available.peek();
 			effect.subscribe(this.device.out.available, (available) => {
 				if (available !== spent) this.#retry.refund();

js/publish/src/source/microphone.ts

@@ -64,9 +64,10 @@ export class Microphone {
 		const constraints = effect.get(this.constraints);
 
 		if (!this.#retry.begin(effect, [device, constraints])) {
-			// Out of budget with the same settings. Only a change to what is plugged in is new
-			// information worth another attempt, so watch the device list here and not while
-			// healthy, where a rerun would restart a working capture for unrelated device churn.
+			// Waiting out a backoff, or out of budget entirely, with the same settings. Either way
+			// a change to what is plugged in is new information worth acting on now, so watch the
+			// device list here and not while healthy, where a rerun would restart a working capture
+			// for unrelated device churn.
 			const spent = this.device.out.available.peek();
 			effect.subscribe(this.device.out.available, (available) => {
 				if (available !== spent) this.#retry.refund();