Enable browser G2G latency measurement by default

[AdirAmsalem]

Human TL;DR

Glass-to-glass latency (g2gMs / ttffMs) now measures automatically in browser
realtime sessions instead of requiring debugQuality: true. This is the number that
actually reflects how laggy a session feels — network RTT hides model inference time,
so a session can read "good" while feeling bad. When present, g2g drives the latency
verdict instead of RTT.

Enabling it by default surfaced that the measurement had never worked: a unit-conversion
bug meant it silently produced zero samples in every browser, so g2gMs was always
null. That's fixed here and verified against a live lucy-2.5 session (g2gMs 651,
ttffMs 2476, 300-sample window).

Note for viewers: browser publishers now advertise frame timing, so anyone using
realtime.subscribe should be on this SDK version or newer.

Summary

• Remove the debugQuality opt-in gate from browser glass-to-glass latency measurement; the option remains accepted as a deprecated no-op.
• Enable LiveKit frame metadata automatically in supported browser publisher sessions and report rolling G2G latency/drop snapshots through the existing stats pipeline.
• Preserve frame-timing compatibility for watchers by encoding frame_timing in subscribe tokens, attaching the frame-metadata worker for browser subscribers, and explicitly rejecting unsupported/RN frame-timed subscriptions.
• Add package/test coverage for the frame-metadata worker asset, runtime support predicate, frequent rolling-window G2G sampling, subscribe worker cleanup, and failure paths.

Details

• G2G now samples on LiveKit timeSyncUpdate events rather than a one-time startup probe. Stats snapshots expose a 300-sample rolling window after warm-up and are emitted on the existing WebRTC stats tick.
• Publisher frame timing is only advertised once the SDK can create the LiveKit frame-metadata worker in a runtime that supports encoded transforms, so we do not ask the server to append frame trailers when the client cannot strip them.
• Legacy subscribe tokens still work without creating a worker. New frame-timed tokens require browser frame-metadata support so viewers can decode streams produced with frame metadata enabled.

Validation

• pnpm test tests/realtime.unit.test.ts tests/prepare-connection.unit.test.ts tests/frame-metadata-diagnostics.unit.test.ts tests/react-native-compat.unit.test.ts (passed 10 files / 289 tests)
• pnpm typecheck
• pnpm biome check packages/sdk/src/realtime/browser/frame-metadata-diagnostics.ts packages/sdk/src/realtime/browser/index.ts packages/sdk/src/realtime/browser/preflight.ts packages/sdk/src/realtime/browser/prepare-connection.ts packages/sdk/src/realtime/client.ts packages/sdk/src/realtime/media-channel.ts packages/sdk/src/realtime/observability/connection-quality.ts packages/sdk/src/realtime/observability/webrtc-stats.ts packages/sdk/src/realtime/react-native/prepare-connection.ts packages/sdk/src/realtime/stream-session.ts packages/sdk/src/realtime/subscribe-client.ts packages/sdk/tests/built-package.mjs packages/sdk/tests/frame-metadata-diagnostics.unit.test.ts packages/sdk/tests/prepare-connection.unit.test.ts packages/sdk/tests/react-native-compat.unit.test.ts packages/sdk/tests/realtime.unit.test.ts
• pnpm build (passes; existing tsdown define option warning still prints)
• pnpm test:package
• git diff --check
• Live end-to-end run against prod lucy-2.5 (Playwright + Chrome fake camera): publisher
  reports non-null g2g, same-version viewer decodes cleanly (354 frames, 0 dropped).
• Each new test verified to fail under its target mutation (no tautological coverage).
• Independent Codex review found issues around subscribe compatibility, worker failure handling, runtime support checks, and cleanup; addressed and re-reviewed with no blocking findings. Claude Code review was unavailable because its OAuth token is revoked.

Attribution

Requested by @roekatz / Roe Katz in Slack (message link).

Companion server telemetry PR: https://github.com/DecartAI/api-platform/pull/717

---

Note

High Risk
Default frame timing changes the on-wire format and subscribe-token contract; older viewers can connect but show black video, and worker creation failures now abort publisher/subscribe connects.

Overview
Browser realtime publishers now turn on glass-to-glass (G2G) latency automatically when LiveKit frame metadata and encoded transforms are available, instead of requiring connect({ debugQuality: true }). The legacy debugQuality flag remains accepted but is deprecated and no longer gates measurement.

Publisher path: prepareBrowserConnection probes runtime support (isFrameMetadataRuntimeSupported), pre-creates a frame-metadata worker, and only then advertises frame_timing to the server. Subscribe tokens can include frame_timing: true. If the worker cannot be created after timing was requested, LiveKit connect fails permanently rather than joining without a strip transform (server packet trailers would break decode).

Viewer path: realtime.subscribe decodes frame_timing from the token, attaches the same worker to LiveKitRoom when required, and rejects unsupported environments (including React Native) with UNSUPPORTED_PLATFORM_FEATURE instead of connecting to a black video.

Measurement fix: G2G diagnostics treat LiveKit timeSyncUpdate timestamps as epoch ms (not performance.timeOrigin-relative), fixing silent zero-sample measurement. Rolling latency uses a 300-sample window; deep preflight no longer passes debugQuality.

Docs and the SDK demo page drop the G2G opt-in UI and warn that viewers must upgrade before publishers on this SDK version.

^{Reviewed by Cursor Bugbot for commit f84959c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@decartai/sdk@191

commit: f84959c

[AdirAmsalem]

Review + live verification (Claude Code)

Reviewed the diff and ran it end-to-end against lucy-2.5 (headless Chrome, fake camera, real backend). Found one blocker that I've fixed in f84959c, plus one open item that needs a product decision.

🔴 G2G measured nothing — fixed

RTCRtpContributingSource.timestamp is spec'd as performance.timeOrigin + performance.now(), i.e. already epoch ms. The frame reader added performance.timeOrigin on top, putting playout ~56 years in the future so every sample tripped the MAX_PLAUSIBLE_MS guard in recordFrame.

Instrumented the live session — the LiveKit chain was perfectly healthy, only the SDK arithmetic was wrong:

• timeSyncUpdate fired 481×, lookupFrameMetadata hit 481/481
• rawSrcTimestamp 1785322433661 vs Date.now() 1785322433664 → already epoch
• computed latencyMs ≈ 1.785e12 ms → rejected every time

Same session, before → after the one-line fix:

	before	after
g2gMs	null	651
ttffMs	null	2476
sampleCount	0 (forever)	33 → 300
limitingFactor	bandwidth	latency

The unit test encoded the same wrong premise — it drove the event with a relative timestamp — which is why this survived review. It now uses an epoch timestamp like a real browser.

Also in f84959c

• subscribe-client — flattened the frame-timing guard; the logger.warn fallback was unreachable (the outer condition already required frameTiming). RN now reports this platform has no frame-metadata worker rather than wrongly blaming absent encoded transforms.
• subscribe-client — documented that LiveKit's FrameMetadataManager owns the worker after handoff and terminates it on RoomEvent.Disconnected. I'd initially flagged disconnect() as leaking; it doesn't. The comment prevents someone "fixing" it into a double-free.
• media-channel — worker-creation failure is now an identifiable error classified as permanent, so a reconnect fails fast instead of burning every retry on an environment-deterministic failure.
• 4 new tests, each verified to fail under its target mutation: Safari's script-transform path (deleting scriptTransformSupported || previously kept the suite green), the pre-created worker handoff + per-reconnect refresh, frameTiming round-tripping into the sessionStarted token, and RN rejection routed through the real RN entry point.

294/294 tests, typecheck, biome, build, test:package all green. Re-verified live after the changes: publisher G2G works and a same-version viewer decodes cleanly (354 frames, 0 dropped).

⚠️ Open — needs a decision, not a code change

Viewers on an older @decartai/sdk (≤ 0.1.16) ignore the token's unknown frame_timing flag, join without a strip worker, and silently decode nothing. Reproduced:

SUBSCRIBE {"ok":true}                      ← connects, state "connected", no error
VIEW readyState:0 w:0 h:0 totalFrames:0    ← for 20s straight

Connects "successfully", no error raised, permanent black screen. Since this PR flips the publisher default on, the blast radius goes from ~0 (opt-in) to every browser-published stream. RN viewers now hard-fail with UNSUPPORTED_PLATFORM_FEATURE where they previously worked.

The SDK can't fix already-published viewers. I've added a README warning, but the real options are:

1. Split this PR (my recommendation) — the viewer side (token decode, worker attach, rejection) and the publisher side (isFrameMetadataRuntimeSupported() gating in prepare-connection) are independent. Ship viewer support first, let it propagate, flip the publisher default in a later release.
2. Server-side kill switch so it can be rolled back without an SDK release.
3. Accept the break with a major bump + comms.

🤖 Generated with Claude Code