FE-1664: Experiment Live Full Brunch Integration

apps/petrinaut-website/MISSION.md

@@ -1,92 +1,15 @@
-# Experiment Live Full-Duplex Migration
+# Experiment Live Full Brunch Integration
 
-## Status
+The child branch's sole execution authority is the
+[Brunch mission](../../libs/@hashintel/brunch-agent/MISSION.md).
+This file is a pointer, not a second mission.
 
-Live experiment scope; local implementation prepared for Kostandin's manual
-witness, not conversational-quality acceptance.
-[FE-1663](https://linear.app/hash/issue/FE-1663/experiment-live-full-duplex-migration)
-is related to FE-1661, which remains the migration-effort assessment.
-This website experiment uses Petrinaut's existing Voice dock, with optional
-session controls and local audio-activity indicators, not inferred turn boundaries.
-It does not recut Brunch's existing Mission 7a or promote its future-planning drafts.
+The accepted experiment uses native Live with best-effort speech instructions and
+separate authoritative transcription. Brunch retains canonical answers and tool
+authority; settlement gates Brunch context sent to Live, not all audible speech.
+Independent questions and unsupported claims are manual-test failures.
 
-## Imperative
-
-Learn whether GPT-Live-1 natively makes process interviewing feel fluid:
-concise relevant follow-ups, room for hesitation and elaboration, and natural
-interruption/correction without losing meaning. Smooth audio alone is not success.
-
-## Throughline
-
-Existing website Voice entry → explicit experimental consent/Start → trusted
-website `/api/voice/live-session` → client-delegated GPT-Live-1 WebRTC → microphone
-and speaker → existing Voice dock → local End/Exit. Local disconnection is
-not a remote-closure claim.
-`main → FE-1663 → Experiment Live Full Brunch Integration` is the intended stack;
-only FE-1663 is authorized now. No relay, rephrasing, or harness code is a donor.
-
-## Proof
-
-- `openai-live-session.test.ts` checks provider/default selection, enablement,
-  trusted session payload, origin/content validation, credential privacy, and
-  no retry. Existing Realtime policy and transport tests remain regression oracles.
-- `live-conversation.test.ts` checks waiting for `session.started`, no duplicate
-  start, no transcript/delegation execution, remote audio attachment, connection
-  failures, late microphone permission, and capture/playback/transport cleanup.
-- `live-conversation-control.test.tsx` checks explicit consent/Start, provider
-  pinning, canonical isolation, host state reporting, panel closure and stale callbacks.
-- Petrinaut's assistant-panel and contents tests check optional action visibility
-  and retain regression coverage for Realtime's playback and lifecycle controls.
-- Rendered real-editor consent, simulated error, and mocked active controls
-  must be inspected without a real microphone or provider session.
-- Kostandin's [10–15 minute manual procedure](README.md#manual-test--1015-minutes)
-  is the oracle for conversational quality and actual media behavior. Provider-free
-  tests and screenshots do not establish these or end-to-end Brunch compatibility.
-
-## Constraints
-
-Unset `PETRINAUT_VOICE_PROVIDER` means `realtime`; only `realtime` and `live` are
-valid. Keep existing enablement/credential plumbing. Pin configuration per
-mounted conversation; never switch providers or resubmit input automatically.
-Keep Realtime's prompts, admission, queue, settlement and playback unchanged.
-
-Live has no access to canonical submission, Brunch/Petrinaut tools or chat
-history. Its guidance does not grant domain authority. Transcript deltas are
-not finalized utterances; delegation events are metadata; commentary (unused
-here) is paraphrasable context limited to 500 tokens, and append acknowledgements
-are not speech/playback completion. Do not invent response terminals or infer
-authoritative completion from silence. Stop ends local media, not canonical work.
-
-No paid sessions, synthetic recordings, audio evaluation harness, deployment,
-Notion writes, second issue, or integration implementation are authorized.
-The follow-up permits cleanup, a clean commit without Amp thread IDs, and a draft PR.
-Preserve other worktrees and uncommitted work.
-
-## Fog-line
-
-Native finalization and enforceable output gating remain unresolved for PR 2.
-Observe whether interruption preserves corrections, whether the user feels
-heard, whether questions advance process elicitation, whether pauses/one-word
-answers survive, and whether output is concise. Observe transport failures and
-remote closure separately. A pleasant standalone conversation cannot answer
-whether canonical admission/settlement delays retain this quality.
-
-## Stop or reorient
-
-Stop at an authority change: autonomous domain speech, local substantive
-follow-ups in the integrated workflow, hybrid authoritative transcription, or
-buffered/gated output requires an explicit recut for Lu's approval. Neither
-this prompt nor success here reopens FE-1624. Provider-free verification cannot
-be replaced with paid calls or a synthetic audio campaign.
-
-## Deferred
-
-PR 2 must preserve: finalized input → existing composer → Flue → Brunch →
-authorized Petrinaut execution → complete-turn settlement → frozen canonical
-reply → Live delivery. Brunch alone authors substantive domain answers and
-questions; Flue is canonical conversation; Petrinaut is execution authority.
-Proposal approval, full-turn settlement and playback eligibility remain distinct.
-Unknown execution outcomes must not be reapplied or spoken as success. Workpiece,
-basis and tool payloads are not assistant prose. See the existing Brunch
-[future spine](../../libs/@hashintel/brunch-agent/MISSION.next.md#voice-after-the-live-transport-cut)
-for retained context, not implementation authority.
+Keep `PETRINAUT_VOICE_PROVIDER=realtime|live`, with unset meaning `realtime`.
+The Brunch mission records current proof, risks and acceptance blockers. No
+agent-started paid sessions, automatic microphone access, merge or deployment are
+authorized; Kostandin performs live testing manually.

apps/petrinaut-website/README.md

@@ -146,15 +146,14 @@ from jsDelivr and Optuna from PyPI; later runs use the browser cache.
 
 Local values live in `.env.local`; Vite's `loadEnv` (see [`vite.config.ts`](vite.config.ts)) copies them into `process.env` for both the dev server and the API functions. In production, set these in the Vercel project settings.
 
-### Experimental Live interview (FE-1663)
+### Experimental Brunch-backed Live interview (FE-1664)
 
-`PETRINAUT_VOICE_PROVIDER=live` selects a standalone GPT-Live-1 conversation
-inside the existing Voice entry. **It is not Brunch output.** It cannot see or
-submit chat, execute tools, or change the model/workpiece. No experimental
-transcripts are displayed or saved. The server uses client delegation, not
-managed Responses; transcript deltas and delegation metadata are ignored.
-The short process-interview prompt is conversational guidance, not a domain
-system or a guarantee of model compliance.
+`PETRINAUT_VOICE_PROVIDER=live` uses GPT-Live-1 for conversational audio while a
+separate `gpt-4o-transcribe` session supplies finalized user text to Brunch.
+Brunch remains the canonical conversation, domain and tool authority. Settled
+Brunch prose is offered to Live as delegation-correlated commentary; Live has no
+tools and must not answer domain questions independently. These instructions are
+best effort, not an enforced speech boundary.
 
 From the repository root, with `OPENAI_VOICE_API_KEY` already exported (or in
 this worktree's `apps/petrinaut-website/.env.local`):
@@ -165,38 +164,38 @@ turbo run build --filter '@apps/brunch-agent^...' --filter '@apps/petrinaut-webs
 yarn workspace @apps/petrinaut-website codegen
 yarn workspace @apps/petrinaut-website examples:generate
 
-# Standalone Live needs only the existing panel launcher, not a running Brunch server.
-PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=live yarn dev:brunch:panel
+# Live input and answers use the existing Brunch route.
+PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=live yarn dev:brunch
 ```
 
 Open [http://localhost:4915/new](http://localhost:4915/new), dismiss the tour if shown, open the AI panel,
 and select the waveform **Start voice mode** action in the empty composer.
-Read the experimental label, check consent, then choose **Start voice**.
+Read the short audio-processing disclosure, allow microphone audio for voice and
+transcription, then choose **Start voice**. **Cancel** returns to text without starting a session.
 Only that last action requests microphone access and a billable Live session.
 Use headphones for the first trial. HTTPS or localhost and an OpenAI project
 with GPT-Live-1 access are required.
 
-Once connected, the existing Voice dock replaces the composer with
-**Listening** or **Speaking**, collapse/expand and **End voice mode**.
-The experimental label appears only during consent. Local WebRTC audio levels
+Once connected, the Voice dock replaces the composer. It shows **Thinking** while
+a Brunch request is submitted or streaming, **Speaking** during active playback,
+and **Listening** when both are idle. Connection and error states take precedence.
+Thinking is a local work indicator, not a spoken progress update.
+The consent panel uses a plain voice-permission heading. Local WebRTC audio levels
 drive the microphone ribbon and Speaking indicator; Listening means the session
 is open for input, including while output is active. These are activity indicators,
 not authoritative turn boundaries or proof of heard playback. Browsers without
-audio-level telemetry retain Listening without an animated input level.
+audio-level telemetry omit the animated input level; Brunch work still shows Thinking.
 There is no separate experiment panel, replay menu, or microphone toggle.
 Connection errors return to setup; starting again requires fresh consent.
 Brief WebRTC interruptions show **Connecting** while the existing session has up
 to the connection timeout (15 seconds by default) to recover. Media stays open;
 no new session is created and no input is replayed. End still stops both directions
 immediately. A failed connection or an expired recovery deadline ends the session.
 
-**End voice mode**, **Exit experiment** during setup, closing the panel, switching to text,
-changing conversation, and leaving the page stop local experimental capture
-and playback. Stop requests `session.close` only after `session.started` and
-waits up to two seconds for `session.closed` before releasing the transport.
-Local silence is not proof of remote closure or final usage. Connection failures are not retried.
-Starting again creates a new session with fresh consent; there is no resume,
-replay, "Your turn", or fabricated Realtime terminal lifecycle.
+**End voice mode**, **Cancel** during setup, closing the panel, switching to text,
+changing conversation, and leaving the page stop both Voice transports and local
+capture. Starting again creates fresh Live and transcription sessions; there is
+no resume, replay, automatic retry, or input resubmission.
 
 To return to the **unchanged integrated Realtime path**, Exit, stop the panel
 dev command with Ctrl-C, configure the existing local Brunch environment, and run:
@@ -215,28 +214,21 @@ the website's generic Turbo `dev` task does not forward arbitrary shell variable
 
 #### Manual test — 10–15 minutes
 
-1. **3 minutes:** explain a familiar process. Let Live ask relevant follow-ups.
-2. **2 minutes:** hesitate, pause, answer with one word, then elaborate. Note
-   whether it leaves room and follows the meaning rather than guessing a turn end.
-3. **2 minutes:** interrupt mid-sentence and correct an earlier detail. Note
-   whether the correction is retained and the interview advances.
-4. **2 minutes:** speak while Live responds. Listen for lost words, overlap,
-   unwanted acknowledgements, and long monologues. A button click is not an
-   acoustic-interruption test.
-5. **2–4 minutes:** End voice mode and check the browser microphone indicator and actual
-   speaker silence. Change to Realtime using the commands above, reload,
-   and begin a fresh session. Confirm the experiment added no canonical chat
-   messages or model changes.
-
-Record **feeling heard and advancing the interview** separately from transport
-correctness. Note browser/headset, representative pauses/corrections, and which
-responses were excessive or useful. Prior relay/rephrasing/harness evidence is
-not a passing comparative baseline. This experiment has no audio harness or
-synthetic recordings.
-
-The intended successor is **Experiment Live Full Brunch Integration**, stacked
-on this branch; it is not implemented here. See [MISSION.md](MISSION.md) for
-the authority boundary and unresolved finalization/output-control questions.
+1. Explain a familiar process and answer one Brunch follow-up.
+2. Hesitate, elaborate, and correct a consequential detail. Confirm each retained
+   finalized utterance appears once in canonical history and later questions use
+   the correction.
+3. Request one available model operation. Compare the spoken result with settled
+   Brunch text and inspect the actual workpiece/model effect.
+4. Speak while Live responds and interrupt it acoustically. Record lost input,
+   overlap, unsupported acknowledgements, independent questions, or unsupported
+   completion claims separately from canonical Brunch behavior.
+5. End Voice mode and confirm microphone capture and speaker playback stop.
+
+Use headphones while the known phantom-input risk is investigated. Server VAD can
+split hesitation into multiple finalized items, and the existing one-waiting-input
+policy may not retain all of them. See [MISSION.md](MISSION.md) for the current
+acceptance limits and manual proof obligations.
 The existing unauthenticated Voice endpoint risk below also applies to Live;
 do not expose this local experiment publicly without addressing that boundary.
 

apps/petrinaut-website/api/voice/transcription-session.ts

@@ -0,0 +1,10 @@
+import { createOpenAITranscriptionSessionHandler } from "../../src/server/voice/openai-transcription-session.js";
+
+declare const process: { env: Record<string, string | undefined> };
+
+export default {
+  fetch: createOpenAITranscriptionSessionHandler({
+    environment: process.env,
+    fetch: globalThis.fetch.bind(globalThis),
+  }),
+};

apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.test.tsx

@@ -334,10 +334,20 @@ describe("local storage demo Brunch voice integration", () => {
     expect(getBrunchVoiceMode(null)).toBeUndefined();
   });
 
-  test("installs the app-owned voice control for a configured Brunch transport", () => {
+  test("installs the app-owned voice control for a configured Brunch transport", async () => {
     const config = { available: true as const, connectionTimeoutMs: 15_000 };
     const tracker = new BrunchPanelConversationTracker();
-    const voiceMode = getBrunchVoiceMode(config, tracker);
+    const snapshot = {
+      conversationId: "petrinaut-preview:net-1",
+      messages: [],
+      settlements: [],
+    };
+    const voiceMode = getBrunchVoiceMode(
+      config,
+      tracker,
+      snapshot.settlements,
+      snapshot,
+    );
     const renderControl = () =>
       voiceMode?.({
         canAcceptVoiceInput: true,
@@ -366,6 +376,20 @@ describe("local storage demo Brunch voice integration", () => {
     if (!isValidElement(control)) {
       throw new Error("Expected the configured composer control to render.");
     }
+    expect(control.props).toHaveProperty("snapshot", snapshot);
+    let finishSubmission = () => {};
+    const pending = tracker.trackSubmission(
+      new Promise<void>((resolve) => {
+        finishSubmission = resolve;
+      }),
+    );
+    const whilePending = renderControl();
+    expect(isValidElement(whilePending) && whilePending.props).toHaveProperty(
+      "snapshot",
+      snapshot,
+    );
+    finishSubmission();
+    await pending;
     const failureListener = vi.fn();
     const responseCompletedListener = vi.fn();
     const responseStartedListener = vi.fn();
@@ -2055,7 +2079,7 @@ describe("assistant selection", () => {
     const brunch = currentAssistant();
     // Brunch reads Flue history, which holds none of the stock turn; and a
     // Brunch-side message write never reaches the local store.
-    expect(brunch.messages).not.toContainEqual(stockMessage);
+    expect(brunch.messages ?? []).not.toContainEqual(stockMessage);
     act(() =>
       brunch.onMessages?.([
         { id: "brunch-1", role: "user", parts: [] } as PetrinautAiMessage,

apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx

@@ -3,7 +3,11 @@
  * @role Editable demo shell: nets in local storage, one live document handle
  */
 
-import { createFlueClient, type FlueConversationSettlement } from "@flue/sdk";
+import {
+  createFlueClient,
+  type FlueConversationSettlement,
+  type FlueConversationState,
+} from "@flue/sdk";
 import {
   use,
   useCallback,
@@ -134,6 +138,7 @@ export const getBrunchVoiceMode = (
   config: OpenAIVoiceConfig | null | undefined,
   tracker?: BrunchPanelConversationTracker,
   settlements?: readonly FlueConversationSettlement[],
+  snapshot?: FlueConversationState,
 ): PetrinautAiVoiceMode | undefined => {
   if (!config) return undefined;
 
@@ -161,6 +166,9 @@ export const getBrunchVoiceMode = (
       {...context}
       config={config}
       settlements={settlements}
+      // Voice only observes this snapshot. Message replacement remains gated
+      // independently by followMessages.canReplace below.
+      snapshot={snapshot}
       resolveInputSubmission={resolveInputSubmission}
       resolveResponseSubmission={resolveResponseSubmission}
       subscribeToResponseMessageCompleted={subscribeToResponseMessageCompleted}
@@ -787,11 +795,13 @@ export const LocalStorageDemoApp = ({
         brunchSelected ? openAIVoiceConfig : null,
         conversationTracker,
         flueHistory.settlements,
+        flueHistory.snapshot,
       ),
     [
       brunchSelected,
       conversationTracker,
       flueHistory.settlements,
+      flueHistory.snapshot,
       openAIVoiceConfig,
     ],
   );