FE-1663: Experiment Live Full-Duplex Migration

.changeset/stable-composer-controls.md

@@ -30,10 +30,13 @@ per-session Show transcription in chat action write those turns into the convers
 instead. Keep every session control -- transcription, the microphone toggle, Resume, Reconnect, and
 End -- in the dock, leaving the canvas toolbar untouched. Add `setMicrophoneMuted` to the Voice mode
 controls and a `muted` session phase, so muting stops capture without interrupting what the assistant
-is saying, unlike pausing. Surface voice recovery failures as toasts with privacy-safe diagnostic
-references, and request one-time consent before the host starts the microphone. Mark persisted spoken
-messages and the exact interactive-tool answer completed by Voice with an inline Voice chip ahead of
-the words themselves.
+is saying, unlike pausing. Keep the complete legacy Voice controls and context registration contract
+required while a separate optional registration seam accepts provider-specific capabilities. Offer
+an explicit playback retry when a browser blocks session audio, keeping the warning visible until
+that user gesture succeeds. Surface voice recovery failures as toasts with privacy-safe diagnostic
+references, and request one-time consent before the host starts the microphone. Mark persisted
+spoken messages and the exact interactive-tool answer completed by Voice with an inline Voice chip
+ahead of the words themselves.
 
 End Voice mode before submitting typed text exactly once through the shared composer, preserving the
 draft if handoff fails. Pause active media before the AI panel closes and reopen the mounted session

.changeset/tidy-voice-dock.md

@@ -0,0 +1,5 @@
+---
+"@hashintel/petrinaut": patch
+---
+
+Allow host-owned Voice sessions to report a neutral connected state and omit unsupported microphone, resume, reconnect, and playback controls from the dock. Keep the viewport controls above a collapsed Voice dock instead of reserving full-height chat panel space.

apps/petrinaut-website/MISSION.md

@@ -0,0 +1,92 @@
+# Experiment Live Full-Duplex Migration
+
+## Status
+
+Live experiment scope; local implementation prepared for Kostandin's manual
+witness, not conversational-quality acceptance.
+[FE-1663](https://linear.app/hash/issue/FE-1663/experiment-live-full-duplex-migration)
+is related to FE-1661, which remains the migration-effort assessment.
+This website experiment uses Petrinaut's existing Voice dock, with optional
+session controls and local audio-activity indicators, not inferred turn boundaries.
+It does not recut Brunch's existing Mission 7a or promote its future-planning drafts.
+
+## Imperative
+
+Learn whether GPT-Live-1 natively makes process interviewing feel fluid:
+concise relevant follow-ups, room for hesitation and elaboration, and natural
+interruption/correction without losing meaning. Smooth audio alone is not success.
+
+## Throughline
+
+Existing website Voice entry → explicit experimental consent/Start → trusted
+website `/api/voice/live-session` → client-delegated GPT-Live-1 WebRTC → microphone
+and speaker → existing Voice dock → local End/Exit. Local disconnection is
+not a remote-closure claim.
+`main → FE-1663 → Experiment Live Full Brunch Integration` is the intended stack;
+only FE-1663 is authorized now. No relay, rephrasing, or harness code is a donor.
+
+## Proof
+
+- `openai-live-session.test.ts` checks provider/default selection, enablement,
+  trusted session payload, origin/content validation, credential privacy, and
+  no retry. Existing Realtime policy and transport tests remain regression oracles.
+- `live-conversation.test.ts` checks waiting for `session.started`, no duplicate
+  start, no transcript/delegation execution, remote audio attachment, connection
+  failures, late microphone permission, and capture/playback/transport cleanup.
+- `live-conversation-control.test.tsx` checks explicit consent/Start, provider
+  pinning, canonical isolation, host state reporting, panel closure and stale callbacks.
+- Petrinaut's assistant-panel and contents tests check optional action visibility
+  and retain regression coverage for Realtime's playback and lifecycle controls.
+- Rendered real-editor consent, simulated error, and mocked active controls
+  must be inspected without a real microphone or provider session.
+- Kostandin's [10–15 minute manual procedure](README.md#manual-test--1015-minutes)
+  is the oracle for conversational quality and actual media behavior. Provider-free
+  tests and screenshots do not establish these or end-to-end Brunch compatibility.
+
+## Constraints
+
+Unset `PETRINAUT_VOICE_PROVIDER` means `realtime`; only `realtime` and `live` are
+valid. Keep existing enablement/credential plumbing. Pin configuration per
+mounted conversation; never switch providers or resubmit input automatically.
+Keep Realtime's prompts, admission, queue, settlement and playback unchanged.
+
+Live has no access to canonical submission, Brunch/Petrinaut tools or chat
+history. Its guidance does not grant domain authority. Transcript deltas are
+not finalized utterances; delegation events are metadata; commentary (unused
+here) is paraphrasable context limited to 500 tokens, and append acknowledgements
+are not speech/playback completion. Do not invent response terminals or infer
+authoritative completion from silence. Stop ends local media, not canonical work.
+
+No paid sessions, synthetic recordings, audio evaluation harness, deployment,
+Notion writes, second issue, or integration implementation are authorized.
+The follow-up permits cleanup, a clean commit without Amp thread IDs, and a draft PR.
+Preserve other worktrees and uncommitted work.
+
+## Fog-line
+
+Native finalization and enforceable output gating remain unresolved for PR 2.
+Observe whether interruption preserves corrections, whether the user feels
+heard, whether questions advance process elicitation, whether pauses/one-word
+answers survive, and whether output is concise. Observe transport failures and
+remote closure separately. A pleasant standalone conversation cannot answer
+whether canonical admission/settlement delays retain this quality.
+
+## Stop or reorient
+
+Stop at an authority change: autonomous domain speech, local substantive
+follow-ups in the integrated workflow, hybrid authoritative transcription, or
+buffered/gated output requires an explicit recut for Lu's approval. Neither
+this prompt nor success here reopens FE-1624. Provider-free verification cannot
+be replaced with paid calls or a synthetic audio campaign.
+
+## Deferred
+
+PR 2 must preserve: finalized input → existing composer → Flue → Brunch →
+authorized Petrinaut execution → complete-turn settlement → frozen canonical
+reply → Live delivery. Brunch alone authors substantive domain answers and
+questions; Flue is canonical conversation; Petrinaut is execution authority.
+Proposal approval, full-turn settlement and playback eligibility remain distinct.
+Unknown execution outcomes must not be reapplied or spoken as success. Workpiece,
+basis and tool payloads are not assistant prose. See the existing Brunch
+[future spine](../../libs/@hashintel/brunch-agent/MISSION.next.md#voice-after-the-live-transport-cut)
+for retained context, not implementation authority.

apps/petrinaut-website/README.md

@@ -134,20 +134,115 @@ from jsDelivr and Optuna from PyPI; later runs use the browser cache.
 
 ## Environment variables
 
-| Name                             | Required         | Used by          | Notes                                                      |
-| -------------------------------- | ---------------- | ---------------- | ---------------------------------------------------------- |
-| `OPENAI_API_KEY`                 | for chat to work | `api/chat.ts`    | OpenAI key the function uses to call `streamText`.         |
-| `OPENAI_VOICE_API_KEY`           | for voice        | voice API        | Dedicated OpenAI key used to create Realtime WebRTC calls. |
-| `PETRINAUT_OPENAI_VOICE_ENABLED` | no               | voice API        | Set to `true` to enable voice, including in production.    |
-| `PETRINAUT_AI_MODEL`             | no               | `api/chat.ts`    | Overrides the default OpenAI model id.                     |
-| `VITE_BRUNCH_CHAT_ENDPOINT`      | for Brunch       | website          | Base URL of the mounted Brunch Flue route.                 |
-| `SENTRY_DSN`                     | no               | `vite.config.ts` | Wired into the bundle via `__SENTRY_DSN__` at build time.  |
+| Name                             | Required         | Used by          | Notes                                                                                         |
+| -------------------------------- | ---------------- | ---------------- | --------------------------------------------------------------------------------------------- |
+| `OPENAI_API_KEY`                 | for chat to work | `api/chat.ts`    | OpenAI key the function uses to call `streamText`.                                            |
+| `OPENAI_VOICE_API_KEY`           | for voice        | voice API        | Dedicated OpenAI key used to create Voice WebRTC sessions.                                    |
+| `PETRINAUT_OPENAI_VOICE_ENABLED` | no               | voice API        | Set to `true` to enable voice, including in production.                                       |
+| `PETRINAUT_VOICE_PROVIDER`       | no               | voice API        | `realtime` (default) or `live` (detached experiment). Invalid values disable Voice discovery. |
+| `PETRINAUT_AI_MODEL`             | no               | `api/chat.ts`    | Overrides the default OpenAI model id.                                                        |
+| `VITE_BRUNCH_CHAT_ENDPOINT`      | for Brunch       | website          | Base URL of the mounted Brunch Flue route.                                                    |
+| `SENTRY_DSN`                     | no               | `vite.config.ts` | Wired into the bundle via `__SENTRY_DSN__` at build time.                                     |
 
 Local values live in `.env.local`; Vite's `loadEnv` (see [`vite.config.ts`](vite.config.ts)) copies them into `process.env` for both the dev server and the API functions. In production, set these in the Vercel project settings.
 
+### Experimental Live interview (FE-1663)
+
+`PETRINAUT_VOICE_PROVIDER=live` selects a standalone GPT-Live-1 conversation
+inside the existing Voice entry. **It is not Brunch output.** It cannot see or
+submit chat, execute tools, or change the model/workpiece. No experimental
+transcripts are displayed or saved. The server uses client delegation, not
+managed Responses; transcript deltas and delegation metadata are ignored.
+The short process-interview prompt is conversational guidance, not a domain
+system or a guarantee of model compliance.
+
+From the repository root, with `OPENAI_VOICE_API_KEY` already exported (or in
+this worktree's `apps/petrinaut-website/.env.local`):
+
+```sh
+# Initial local preparation, without inference:
+turbo run build --filter '@apps/brunch-agent^...' --filter '@apps/petrinaut-website^...'
+yarn workspace @apps/petrinaut-website codegen
+yarn workspace @apps/petrinaut-website examples:generate
+
+# Standalone Live needs only the existing panel launcher, not a running Brunch server.
+PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=live yarn dev:brunch:panel
+```
+
+Open [http://localhost:4915/new](http://localhost:4915/new), dismiss the tour if shown, open the AI panel,
+and select the waveform **Start voice mode** action in the empty composer.
+Read the experimental label, check consent, then choose **Start voice**.
+Only that last action requests microphone access and a billable Live session.
+Use headphones for the first trial. HTTPS or localhost and an OpenAI project
+with GPT-Live-1 access are required.
+
+Once connected, the existing Voice dock replaces the composer with
+**Listening** or **Speaking**, collapse/expand and **End voice mode**.
+The experimental label appears only during consent. Local WebRTC audio levels
+drive the microphone ribbon and Speaking indicator; Listening means the session
+is open for input, including while output is active. These are activity indicators,
+not authoritative turn boundaries or proof of heard playback. Browsers without
+audio-level telemetry retain Listening without an animated input level.
+There is no separate experiment panel, replay menu, or microphone toggle.
+Connection errors return to setup; starting again requires fresh consent.
+Brief WebRTC interruptions show **Connecting** while the existing session has up
+to the connection timeout (15 seconds by default) to recover. Media stays open;
+no new session is created and no input is replayed. End still stops both directions
+immediately. A failed connection or an expired recovery deadline ends the session.
+
+**End voice mode**, **Exit experiment** during setup, closing the panel, switching to text,
+changing conversation, and leaving the page stop local experimental capture
+and playback. Stop requests `session.close` only after `session.started` and
+waits up to two seconds for `session.closed` before releasing the transport.
+Local silence is not proof of remote closure or final usage. Connection failures are not retried.
+Starting again creates a new session with fresh consent; there is no resume,
+replay, "Your turn", or fabricated Realtime terminal lifecycle.
+
+To return to the **unchanged integrated Realtime path**, Exit, stop the panel
+dev command with Ctrl-C, configure the existing local Brunch environment, and run:
+
+```sh
+PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=realtime yarn dev:brunch
+```
+
+Reload the page before starting a new session. Unsetting
+`PETRINAUT_VOICE_PROVIDER` also selects Realtime. Provider/config selection is
+pinned for the mounted conversation; there is no provider switching or input
+resubmission mid-session. The launcher sets the existing `/agents/chat` route;
+the ordinary website launcher still needs `VITE_BRUNCH_CHAT_ENDPOINT` configured
+to expose Voice. Export variables to the launcher directly or use `.env.local`;
+the website's generic Turbo `dev` task does not forward arbitrary shell variables.
+
+#### Manual test — 10–15 minutes
+
+1. **3 minutes:** explain a familiar process. Let Live ask relevant follow-ups.
+2. **2 minutes:** hesitate, pause, answer with one word, then elaborate. Note
+   whether it leaves room and follows the meaning rather than guessing a turn end.
+3. **2 minutes:** interrupt mid-sentence and correct an earlier detail. Note
+   whether the correction is retained and the interview advances.
+4. **2 minutes:** speak while Live responds. Listen for lost words, overlap,
+   unwanted acknowledgements, and long monologues. A button click is not an
+   acoustic-interruption test.
+5. **2–4 minutes:** End voice mode and check the browser microphone indicator and actual
+   speaker silence. Change to Realtime using the commands above, reload,
+   and begin a fresh session. Confirm the experiment added no canonical chat
+   messages or model changes.
+
+Record **feeling heard and advancing the interview** separately from transport
+correctness. Note browser/headset, representative pauses/corrections, and which
+responses were excessive or useful. Prior relay/rephrasing/harness evidence is
+not a passing comparative baseline. This experiment has no audio harness or
+synthetic recordings.
+
+The intended successor is **Experiment Live Full Brunch Integration**, stacked
+on this branch; it is not implemented here. See [MISSION.md](MISSION.md) for
+the authority boundary and unresolved finalization/output-control questions.
+The existing unauthenticated Voice endpoint risk below also applies to Live;
+do not expose this local experiment publicly without addressing that boundary.
+
 ### Brunch Voice mode
 
-Voice mode is disabled by default. To enable it, configure a real
+The following describes Realtime, the default provider. Voice mode is disabled by default. To enable it, configure a real
 `VITE_BRUNCH_CHAT_ENDPOINT`, set `PETRINAUT_OPENAI_VOICE_ENABLED=true`, and
 provide a dedicated `OPENAI_VOICE_API_KEY`.
 

apps/petrinaut-website/api/voice/live-session.ts

@@ -0,0 +1,10 @@
+import { createOpenAILiveSessionHandler } from "../../src/server/voice/openai-live-session.js";
+
+declare const process: { env: Record<string, string | undefined> };
+
+export default {
+  fetch: createOpenAILiveSessionHandler({
+    environment: process.env,
+    fetch: globalThis.fetch.bind(globalThis),
+  }),
+};