Add deferred WebSocket upgrade materialization (deferUpgradeMaterialization + materializeUpgrade)

[jonastemplestein]

What this is

Base: main (the #6 rebase has landed — this is now a normal single-commit PR, rebased onto upstream 2de5871, which includes cloudflare#241's new StubHook arg-disposal contract).

Implements the Level 2 ("fix it in the fork") recommendation from project-worker/docs/live-capnweb-ws-handler.md (iterate/iterate R3b): stop materializing a tunneled WebSocket at the capnweb session endpoint, and instead let infrastructure code carry the tunnel onward in stream form, materializing a real socket exactly once — at the hop that actually returns the 101 to the eyeball. This composes with the topology Kenton describes as the long-term plan in capnweb#36: capnweb terminates in a stateless Worker, and the DO sees only regular Workers RPC.

What existing users need to know

One deliberate behavior change on Workers; everything else just works. In detail:

• The default flipped on Cloudflare Workers. A tunneled upgrade Response received over a capnweb session on workerd now arrives with webSocket as the transportable pair by default, instead of an eagerly materialized WebSocketPair end. Rationale: at a session endpoint, eager materialization was only ever correct when that endpoint itself served the upgrade — in every other topology it produced the DataCloneError on the next internal hop — and materialization is a one-way door (the pump consumes the tunnel streams), so the receiver must build the forwardable form up front. The universally-safe shape is now the default; an endpoint that serves the socket itself either calls materializeUpgrade(response.webSocket) on the next line or sets deferUpgradeMaterialization: false on the session.
• Who could that affect? Only code that (a) runs on workerd, (b) receives tunneled upgrades over a capnweb session, and (c) used response.webSocket as a native socket at the session endpoint. The deferred feature itself has never been published; the only published surface is the base tunnel in @iterate-com/capnweb@0.10.0, and we know of no consumer of its workerd auto-materialize branch (iterate's tunnel consumers — captun, the e2e harnesses — run on Node, where nothing changes). Ships as a documented minor with a migration note in the changeset.
• Node and browsers are unchanged. response.webSocket is still a usable TunneledWebSocket by default; the session option opts a non-Workers receiver into the pair form (that's how a Node relay — or our own tests — get it).
• No wire change. The capnweb protocol is identical, so version mixing on the wire is a non-issue: a new receiver works against an old sender and vice versa — the sending peer can't tell the difference.
• API surface. Three new names: the deferUpgradeMaterialization session option (now an override in either direction rather than an opt-in), the materializeUpgrade() export, and the DeferredWebSocketUpgrade type.
• One version-coupling rule. The pair's internal byte framing is spoken by exactly two parties: the deferring session and materializeUpgrade(). Both must run this version or later (they can be different deployments — the format is frozen with append-only evolution, and a mismatch tears the tunnel down loudly rather than desyncing). Peers on the far side of the capnweb wire are unaffected either way.
• The pair has four rules (all enforced or documented, all tested): treat it as opaque (never parse or partially consume it); forward the whole object — destructuring {readable, writable} drops init and with it the negotiated subprotocol; detect upgrades via response.webSocket != null — the deferred Response's own status is 200, and the Response itself must not cross a native boundary (its webSocket property is a JS expando that silently vanishes); and if you receive the pair as a call result, keep the delivering Response referenced (undisposed) while the tunnel is in use — a pair received in call params can't be kept past the call.

What's newly possible (with example code)

What already worked (the base WebSocket-over-RPC feature, unchanged): a fetch target on the far side of a capnweb session can return an upgrade Response, and the receiving endpoint gets a working socket —

const backend = newWebSocketRpcSession<Backend>("wss://device.example/rpc");
const response = await backend.fetch(new Request("https://x/", { headers: { Upgrade: "websocket" } }));
response.webSocket;  // a real, working socket… but ONLY in this isolate

The wall this PR removes: that socket was welded to the isolate holding the capnweb session. Handing it to any other worker or Durable Object over native Workers RPC — or returning the Response through an internal RPC hop toward a fetch handler elsewhere — died with DataCloneError: Could not serialize object of type "WebSocket". So you could not split "where the capnweb session terminates" from "where the eyeball's WebSocket is served", which is exactly the split the recommended architecture wants (capnweb in a stateless Worker, DOs speaking only native RPC — capnweb#36).

Now — the session endpoint defers, the raw pair crosses the native hop like any serializable value, and the far end mints the real socket:

import { newWebSocketRpcSession, materializeUpgrade,
         type DeferredWebSocketUpgrade } from "@iterate-com/capnweb";

// ── Relay worker: terminates the capnweb session to the device/provider. ──────────
export class Relay extends WorkerEntrypoint<Env> {
  // Keep delivering Responses referenced while their tunnels are in use (they are
  // released when the session ends, or sooner if you drop/dispose them).
  #live = new Set<Response>();

  async openBackend(request: Request): Promise<DeferredWebSocketUpgrade> {
    // No options needed: on Workers, tunneled upgrades arrive in forwardable form by default.
    const backend = newWebSocketRpcSession<Backend>(this.env.PROVIDER_URL);
    const response = await backend.fetch(request);
    this.#live.add(response);
    // response.webSocket is { readable, writable, init }: two byte streams plus the
    // provider's upgrade headers as plain data — all of it serializable over native
    // Workers RPC, params or return values alike.
    return (response as any).webSocket;
  }
}

// ── Durable Object (any other isolate): the fetch-lane exit. ──────────────────────
export class Session extends DurableObject<Env> {
  async fetch(request: Request): Promise<Response> {
    if (request.headers.get("Upgrade")?.toLowerCase() !== "websocket") {
      return new Response("Expected a WebSocket upgrade.", { status: 426 });
    }
    const pair = await this.env.RELAY.openBackend(request);
    // A real 101 with a native socket, negotiated Sec-WebSocket-Protocol included
    // (it rode across on pair.init). Returnable straight to the eyeball.
    return materializeUpgrade(pair);
  }
}

Both halves of that flow are what the workerd tests exercise verbatim: the pair crossing a service-binding hop in a call param and in a return payload, and a real HTTP upgrade completing through a fetch handler with the subprotocol echoed on the actual 101. End to end, the eyeball's frames travel: eyeball socket ⇄ WebSocketPair pump ⇄ framed byte streams over native RPC ⇄ capnweb tunnel ⇄ provider's socket — with capnweb's flow-control window bounding what any middle hop buffers, and every failure mode (death of either socket, the session, or the transit streams) collapsing the whole chain closed.

The same option also unblocks the apps/os mesh case (the quarantined live-capability-websocket e2e): the capability-host isolate defers, invokeCapability returns the pair over its internal hop, and the dynamic worker calls materializeUpgrade(pair) in its own fetch handler.

API

• Deferred delivery, default on Workers — a tunneled upgrade Response deserializes with Response.webSocket as an opaque DeferredWebSocketUpgrade — { readable, writable, init } — on workerd by default, and as a usable socket elsewhere. RpcSessionOptions.deferUpgradeMaterialization?: boolean overrides the default in either direction (false on Workers restores eager materialization for an endpoint that serves the socket itself; true opts a Node relay into the pair form). init carries the provider's upgrade headers (e.g. a negotiated Sec-WebSocket-Protocol), so infrastructure that forwards the whole pair object preserves them with zero extra plumbing. The deferred Response itself has status 200 (constructed Responses can't be 1xx) — detect upgrades via response.webSocket != null, never by status. Receive-side only; the wire format is unchanged.
• materializeUpgrade(pair, init?): Response — rebuilds the real upgrade Response (on workerd: WebSocketPair + pump + status 101 + the carried headers; an explicit init replaces pair.init wholesale). The pair is consumed: both streams lock synchronously, so a second call on the same pair throws instead of corrupting the first.

The option threads to the deserializer through the Importer interface — the same route upstream added for RpcLimits — as an optional method, so only the real RPC session implements it and map.ts stays byte-identical with upstream.

The load-bearing design finding: the pair is byte-oriented

The design doc assumed the deferred pair could be the tunnel's raw value streams ("streams serialize across Workers RPC" was listed as known). Empirically false for value streams — probed against a real service-binding hop: reads fail with TypeError: This ReadableStream did not return bytes, string writes are refused; Uint8Array chunks flow fine in both directions. So the pair carries the tunnel's text/binary/close frames in a length-prefixed framing (1 byte type, 4 bytes big-endian length, payload). Because the deferring worker and the materializing worker deploy independently, the framing is a frozen cross-deployment format: header layout fixed, evolution append-only (unknown frame types tear the tunnel down loudly). The layout is pinned byte-for-byte in the tests.

Robustness properties (each one tested)

• Ownership: the pair's inner ends stay payload-owned for the tunnel's whole life; first use locks them but takes no ownership (outer streams use highWaterMark: 0, so nothing pulls — or locks — before a real read). An untouched deferred Response in params releases the tunnel on payload disposal. A pair received in params cannot be kept past the call (same as upstream's plain-stream semantics — documented); the intended flow receives it as a result and keeps the delivering Response undisposed while the tunnel is in use.
• Fail closed: malformed frames, oversized frames (128 MiB cap enforced on both the encode and decode sides, so the error is attributed at the offending sender), and truncation at end-of-stream all abort the sender's socket; truncation surfaces as error + close 1006, never a fake clean close. RPC-session death under a materialized socket yields error + close 1006, not a hang.
• Transport-shape tolerance: frames split at arbitrary byte boundaries (including mid-header) and frames coalesced into one chunk both decode correctly, in linear time.
• Flow control, inherited end to end: capnweb stream acks fire as the pair is read, so an unconsumed or in-transit pair keeps the provider throttled to the flow-control window — asserted by a deterministic test (32 MiB offered, ~one window crosses while unread; reading resumes flow). A materialized endpoint behaves like a non-deferred one. Measured overhead of the framing: ~12% in-process on a throughput microbench; no added stalls.
• Mixed traffic: plain (non-upgrade) Responses on a deferring session deserialize exactly as before.
• Headers at a real exit: reserved handshake headers riding pair.init (Connection, Sec-WebSocket-Accept, …) are recomputed/dropped by workerd/kj on a genuine upgrade — verified during review with a raw-socket RFC 6455 probe against a poisoned init, which produced a fully sanitized 101 with a correctly computed Sec-WebSocket-Accept — so only non-reserved headers like the subprotocol reach the wire. Documented on materializeUpgrade.

Two-round multi-agent adversarial review

At Jonas's request this PR went through two staged multi-agent reviews (eleven lenses total), with every bug claim independently re-reproduced by adversarial verifiers running live experiments, and Kenton's actual review comments on our PR #1 plus his capnweb#36 architecture comments as the design yardstick.

Round 1 (six lenses → 28 findings): dropped TunneledWebSocket's parallel writable mode in favor of upstream's WritableStreamStubHook; collapsed four stream wrappers into two; made the deferred branch a one-line call; reverted map.ts to upstream-identical. Fixed reproduced defects: eager HWM-1 pull locking the tunnel readable at deserialization; double-materializeUpgrade silently killing the first socket's writes; truncation-at-EOS masquerading as a clean close; quadratic decoder buffering with no length cap. The round's own re-chunking tests then caught a deadlock in its own verified simplification (a pull() that enqueues nothing is never re-invoked) — exactly the "hung tunnel" failure its test-gap lens predicted.

Round 2 (five lenses, requirements-first → 18 findings, zero surviving bug claims): a requirement-by-requirement audit against the design doc, which drove the real fetch-lane-exit test (an actual HTTP upgrade served from a fetch handler, subprotocol echoed on the literal 101 — independently confirmed with a raw-wire RFC 6455 handshake probe); init asserted across the native hop; the mixed-traffic and window-bounded-backpressure pins; the symmetric encode-side frame cap; and a clarity pass that corrected every remaining stale or imprecise sentence in the shipped prose (ownership model, wholesale-init-replacement, the framing banner, test rationales).

Round 3 (architecture challenge): pressure-tested against the "why not just proxy natively?" alternative. Empirical outcome: workerd's distinguished-fetch chain (which does proxy WebSockets natively) cannot be extended into a stateless session-holding instance — ctx.exports loopback stubs refuse to serialize (DataCloneError: ... "LoopbackServiceStub"), so the only flag-free design was to make deferral the default on Workers, which this PR now does. The relay-side configuration disappeared entirely, and materializeUpgrade remains confined to platform code at the serving exits.

Verification

• npm run build, npm run test:types — clean. Full suite: 1258/1258 across node, workerd, Chromium, Firefox, WebKit, plus Bun 16/16.
• Node (34 tests in websocket-tunnel.test.ts): frame format pinned byte-for-byte; split/coalesced/large/empty frames; header carriage; mixed traffic; unread-pair backpressure; double-materialize fail-fast; malformed/oversized/truncated frames failing closed; abort + lazy-cancel teardown; ignored-params release; session-death → 1006; full session battery over a deferred-then-materialized socket.
• workerd (19 tests): in-isolate defer→materialize; the pair crossing a native service-binding hop as a param and in a return payload (the relay→DO shape) with init and close propagation asserted through both boundaries; and a real HTTP upgrade served from a fetch handler via materializeUpgrade.

Notes for the iterate/iterate integration

The design doc (project-worker/docs/live-capnweb-ws-handler.md) has been amended in the iterate worktree to match the shipped contract: a "Status: shipped in the fork" section records the byte-framing correction, the { readable, writable, init } pair shape and its four usage rules, the resolved lifetime story (keep the delivering Response referenced; no dup/claim needed), and two operational notes (a live materialized tunnel keeps its DO resident — the pump is in-memory listeners, not the hibernatable API; per-tunnel reclamation before session end = observing pass-throughs around the pair's streams, or retention tied to the Invoker/Pager connection). All four of the doc's open spikes are marked resolved with their fork-test evidence; the 25-consecutive-parallel-run criterion for the re-enabled e2e remains on the apps/os side.

What this unblocks

The clean-room relay requests deferred delivery, returns the pair over the Invoker Workers-RPC leg, and ItxDurableObject.#fetchCapability replaces its 501 ("needs a frame bridge") with materializeUpgrade(pair) → 101 to the eyeball. Same shape flips the quarantined live-capability-websocket e2e in apps/os.

🤖 Generated with Claude Code

[jonastemplestein]

Note on checks: the Test suite (full matrix) passed on this branch via workflow_dispatch (run 32038698280), and Semgrep is green. The pkg.pr.new check failure is a service-side issue, not this PR: the publish step 404s with There is no workflow defined for ySlB0FJ6VC — the same workflow last succeeded on 2026-07-17, so the pkg.pr.new GitHub App registration for this repo appears to have lapsed and needs re-installing by an org admin (https://github.com/apps/pkg-pr-new). A rerun reproduces the failure.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/iterate/capnweb/@iterate-com/capnweb@7

commit: ca3da33

[jonastemplestein]

Force-pushed a reviewed revision (48166d0 → 79f771e) after a staged multi-agent adversarial review — six finder lenses, 28 findings, every bug claim independently re-reproduced by verifier agents running live experiments. Full story in the updated PR description; highlights:

• Simplified toward upstream idiom (guided by kentonv's review comments on Add support for sending WebSockets over RPC, as part of a fetch upgrade Response. #1 and his capnweb#36 architecture comments): TunneledWebSocket's parallel writable mode is gone (reuses upstream's WritableStreamStubHook), four stream wrappers collapsed to two, the deferred branch in serialize.ts is one line, and map.ts is byte-identical with upstream again (Importer method now optional).
• Real bugs fixed (each empirically reproduced first): eager HWM-1 pull locked the tunnel readable at deserialization; double-materializeUpgrade silently killed the first socket's writes (now throws synchronously — the pair is consumed); truncation-at-EOS masqueraded as a clean close (now error + 1006); quadratic decoder buffering + unbounded declared frame length (now chunk-list + 128 MiB cap, fail closed).
• API: the pair is now DeferredWebSocketUpgrade { readable, writable, init } — init carries the provider's upgrade headers so a negotiated Sec-WebSocket-Protocol survives to the materialized 101 with no extra plumbing.
• New coverage: split/coalesced/large/empty frames, return-payload cross-isolate transfer (the exact relay→DO shape), close propagation asserted at the origin through both boundaries, session-death fail-closed, ignored-params release. Full suite 1255/1255 + Bun 16/16; the dispatched Test run on this branch is green.

Also: the earlier pkg.pr.new registration failure has resolved itself — the check now passes, so my previous comment's ask (reinstall the app) is moot.

[jonastemplestein]

Round 2 (79f771e → 8f9077e): a second staged multi-agent pass, this time requirements-first against the R3b design doc, plus a fresh adversarial round on the round-1 code and an editor lens over every shipped sentence. Zero bug claims survived verification this round. What changed:

• The fetch-lane exit is now proven literally: a new test serves a real HTTP upgrade from a fetch handler via materializeUpgrade (status 101 + subprotocol from pair.init echoed on the actual response), and a raw-socket RFC 6455 probe during review confirmed workerd/kj sanitizes reserved handshake headers and computes the correct Sec-WebSocket-Accept even from a poisoned init — so header carriage is safe by construction, now documented.
• Backpressure is now pinned, not assumed (design-doc spike 3): a deterministic test asserts an unread deferred pair throttles the provider to ~one flow-control window (32 MiB offered, ~0.3 MiB crosses), with flow resuming on read. The measured model is documented in the module banner.
• init asserted across the native hop; mixed upgrade/plain traffic on one deferring session pinned; the 128 MiB frame cap is now enforced on the encode side too, so oversized sends fail at their source.
• Every stale sentence fixed: the ownership model ("first use locks but takes no ownership; disposing the payload releases the tunnel"), status-200 detection, wholesale init replacement, the "raw pair" misnomers, and a format-pinning test rationale that now states the real threat (cross-version drift, to which round-trip tests are structurally blind).

Full suite 1258/1258 + Bun 16/16; dispatched CI Test run green. The PR description gained a "Notes for the iterate/iterate integration" section with three design-doc deltas the consuming author needs (forward the whole pair object; detect upgrades by .webSocket, not status; apps/os needs the exit-hop materializeUpgrade call).

[jonastemplestein]

Description updated with a "What existing users need to know" section (TL;DR: nothing changes unless you opt in — no wire change, additive API, semver minor; opt-in users get a four-rule checklist for the pair) and a "What's newly possible" section with a complete relay + Durable Object example showing the previously-impossible split: capnweb session terminating in one worker, the eyeball's WebSocket served from another isolate via materializeUpgrade(). The R3b design doc has also been amended in the iterate worktree (new "Status: shipped in the fork" section; all four spikes marked resolved with evidence).

[jonastemplestein]

Round 3 (8f9077e → ef4af85), from Jonas's architecture challenge ("why should the edge need a flag at all?"): deferral is now the default on Cloudflare Workers, and the session option becomes an override in either direction rather than an opt-in.

The reasoning, empirically grounded: a workerd Response must receive its socket at construction (expandos/getters are invisible to the upgrade machinery), and materialization is a one-way door (the pump consumes the tunnel streams) — so the receiver must choose pair-vs-socket at deserialization time, and no fully automatic scheme can exist. We also probed the last conceivable flag-free alternative — extending the distinguished-fetch chain into the session-holding stateless instance via a lent ctx.exports loopback Fetcher — and workerd refuses: DataCloneError: Could not serialize object of type "LoopbackServiceStub". Given eager materialization at a session endpoint is only ever correct when that endpoint serves the socket itself (and is the exact source of the original DataCloneError confusion), the universally-safe shape is now the default.

Net effect: the relay/edge session is created with no options at all; materializeUpgrade stays confined to platform code at the serving exits (the DO's fetch handler / the ITX wrapper), so app authors and callers never see any of it — itx.wsbackend.fetch(req) just works, like fetch on a Workers binding. Node/browsers unchanged (still a usable TunneledWebSocket; true opts a Node relay into the pair). The only behavior change for any published user is the workerd auto-materialize branch of the 0.10.0 tunnel, which has no known consumers; deferUpgradeMaterialization: false restores it, and the changeset carries the migration note.

Suites: Node 34/34, workerd 19/19, full run 1257/1258 (the one failure is the long-documented WebKit backpressure flake), Bun 16/16.