H-6763: Add OpenAI Realtime voice input to Brunch

[kostandinang]

🌟 What is the purpose of this PR?

Add disabled-by-default OpenAI Realtime voice input to Brunch inside the Petrinaut website. OpenAI transcribes microphone audio; each completed transcript is submitted through Petrinaut's generic composer API and the existing AI SDK/Brunch transport.

Brunch remains authoritative for tools, pending asks, evidence, completion, and projection. OpenAI is used only for transcription in this PR.

🔗 Related links

• Linear: H-6763 (internal)
• Depends on #9355
• Stack: #9346 → #9355 → H-6763: Add OpenAI Realtime voice input to Brunch #9356 → #9357
• #9297 is historical evidence only and is unchanged.

🚫 Blocked by

• #9355, transitively #9345 and #9346

🔍 What does this change?

• Adds one app-owned, same-origin WebRTC initialization boundary.
  • The browser sends only its SDP offer.
  • The server atomically adds the OpenAI key and fixed transcription policy before calling /v1/realtime/calls.
  • Provider credentials, model, language, vocabulary, and VAD policy never reach the browser.
• Uses transcription-only gpt-live-transcribe with English process-modeling vocabulary and semantic VAD at low eagerness.
  • No Realtime response generation is enabled.
  • Voice is unavailable unless the server flag and dedicated key are both present.
  • Public production fails closed; this remains a deployment-protected preview feature.
• Implements the browser WebRTC lifecycle: microphone permission, media tracks, peer connection, data channel, connection timeout, cleanup, reconnect epochs, and stale-callback rejection.
• Admits only completed transcript events.
  • Partial transcripts are visible but never submitted.
  • Completed turns are deduplicated by connection epoch, OpenAI item ID, and content index.
  • Stable message IDs preserve one semantic submission across transport retries.
• Routes finalized text through H-6763: Add a generic Petrinaut composer submission API #9355 and the existing Brunch transport.
  • A pending brunch_ask remains correlated and receives the answer through its tool call.
  • Explicit corrections are separate normal messages and cannot accidentally answer a pending ask.
• Adds a half-duplex microphone controller and accessible UI for start, end, reconnect, provisional text, correction, and status.
  • The microphone stays closed while connecting, finalizing, delivering, or waiting for Brunch.
  • Keyboard chat remains available on every failure path.
• Installs voice only for the real Brunch endpoint; the generic fallback chat remains voice-free.
• Updates preview configuration, deployment limits, and user/developer documentation without adding a provider abstraction or SDK dependency.

Stack record: direct base 3152049cabe28aed98c4e80560366f16fcaae0e4; PR head 7c042206e919b80ef27ce46649bfe5202005b27c.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

No existing documentation screenshot covers the new voice control.

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Voice is disabled by default and intentionally unavailable in public production.
• Real provider/browser validation still requires a dedicated OpenAI project, microphone permission, and a real remote Brunch endpoint.
• The inherited Brunch docs-index check remains 6/7 because FE-1506: Show captures and completion while a panel elicitation runs #9346 references ADR-0008 from the parallel prerequisite stack.

🐾 Next steps

• H-6763: Speak finalized Brunch responses with OpenAI #9357 speaks finalized canonical Brunch text through OpenAI's dedicated Speech API.
• Voice PR4 will add delivery/reload recovery, stale-ask handling, authentication, distributed quotas, telemetry, retention cleanup, a kill switch, and real completion/projection evidence after the required platform contracts land.

🛡 What tests cover this?

• Server tests cover fail-closed configuration, production blocking, same-origin/size/content validation, server-owned policy, sanitized upstream failures, browser aborts, and timeouts.
• WebRTC tests cover complete lifecycle cleanup, microphone permission failures, reconnect epochs, strict OpenAI event parsing, and stale events.
• Turn-controller tests cover partial/final separation, stable identity, duplicate/out-of-order events, half-duplex races, stop/reconnect, pending-ask routing, and corrections.
• Component/host tests cover accessible status and installation only on the real Brunch path.

Validation run

yarn workspace @hashintel/petrinaut test:unit --run
yarn workspace @hashintel/petrinaut lint:eslint
yarn workspace @hashintel/petrinaut lint:tsc
yarn workspace @hashintel/petrinaut build
yarn workspace @apps/petrinaut-website test:unit
yarn workspace @apps/petrinaut-website lint:eslint
yarn workspace @apps/petrinaut-website lint:tsc
yarn workspace @apps/petrinaut-website build
yarn workspace @hashintel/brunch-agent-transport-aisdk test:unit
yarn workspace @hashintel/brunch-agent-transport-aisdk lint:eslint
yarn workspace @hashintel/brunch-agent-transport-aisdk lint:tsc
yarn workspace @hashintel/brunch-agent-transport-aisdk build
yarn workspace @apps/brunch-agent test:unit
yarn workspace @apps/brunch-agent lint:eslint
yarn workspace @apps/brunch-agent lint:tsc
yarn workspace @apps/brunch-agent build
yarn workspace @local/petrinaut-arch-docs lint:arch-docs
yarn oxfmt --check apps/petrinaut-website apps/brunch-agent libs/@hashintel/petrinaut libs/@hashintel/brunch-agent

Results: 214 Petrinaut tests, 49 website tests, 12 transport tests, and 35 Brunch tests passed. Lint, typecheck, build, formatting, and architecture-doc checks passed. Transport emitted 4 inherited no-await-in-loop warnings and Brunch emitted 12 inherited warnings.

The direct-base no-ElevenLabs audit and git diff --check passed. No package manifest or lockfile changed.

Credential-dependent checks not run: live WebRTC negotiation, representative-audio semantic VAD, one finalized item reaching remote Brunch/pending ask, live reconnect rejection, desktop Chrome/Edge/Safari accessibility, and deployed production unavailability.

❓ How to test this?

1. Run yarn workspace @apps/petrinaut-website test:unit plus its lint, typecheck, and build commands above.
2. In a protected preview with PETRINAUT_OPENAI_VOICE_ENABLED=true, a dedicated server-side OPENAI_VOICE_API_KEY, and a real Brunch endpoint, start voice input and speak one answer.
3. Confirm provisional text is labelled “not sent,” only the completed transcript reaches the current Brunch turn, and the microphone remains off until Brunch is ready again.
4. End/reconnect voice and confirm late events do not submit; disable the feature and confirm keyboard chat still works.

📹 Demo

Not included: a meaningful demo requires provider credentials, microphone permission, and the remote Brunch environment described above.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
petrinaut	Ready	Preview	Sep 1, 2026 12:39pm UTC
petrinaut-docs	Ready	Preview	Sep 1, 2026 12:39pm UTC

2 Skipped Deployments

Project	Deployment	Actions	Updated
hash	Ignored	Preview	Sep 1, 2026 12:39pm UTC
hashdotdesign-tokens	Ignored	Preview	Sep 1, 2026 12:39pm UTC

[kostandinang]

• FE-1576: Ground Brunch interviews in the current Petrinaut net #9523
• H-6800: Improve Petrinaut Voice turn-taking and answer provenance #9512
• H-6763: Add temporary Brunch ask shim for Voice preview #9507
• H-6763: Submit only completed Voice transcripts as interview answers #9500
• FE-1570: Optimize spoken Brunch responses for Voice mode #9496
• H-6763: Integrate Voice mode into the AI assistant #9467
• H-6763: Migrate voice interviews to GPT-Realtime-2 #9399
• H-6763: Add the voice interview experience #9380
• H-6763: Improve voice preview reliability diagnostics #9378
• H-6763: Speak finalized Brunch responses with OpenAI #9357
• H-6763: Add OpenAI Realtime voice input to Brunch #9356 👈 (View in Graphite)
• H-6763: Add a generic Petrinaut composer submission API #9355
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

The base branch was changed.

[cursor]

Merge activity

• Sep 1, 11:59 AM UTC: This pull request can not be added to the Graphite merge queue. Please try rebasing and resubmitting to merge when ready.
• Sep 1, 11:59 AM UTC: Graphite disabled "merge when ready" on this PR due to: a merge conflict with the target branch; resolve the conflict and try again..

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 24df8d4. Configure here.}