H-6763: Integrate Voice mode into the AI assistant

.changeset/stable-composer-controls.md

@@ -2,16 +2,40 @@
 "@hashintel/petrinaut": patch
 ---
 
-Add generic host-rendered AI composer controls and a persistent interview stage with docked and
-detached placements, protected active conversations, keyboard fallback, and one-answer buffering
-while the normal chat stream settles. Include stable finalized-text submission, conversation
-identity, stop handling, schema-validated interactive-tool text mapping, explicit separate-message
-targeting for corrections, and a queue-aware voice submission path. Add the Chat / Interview mode
-switch and export `PetrinautAiInteractionMode`, with the selected interaction mode and mode-change
-callback available to host-rendered interview stages. `renderComposerControl` remains a supported
-public seam for hosts that only need their own control beside the message box, independently of the
-interview stage.
+Add generic host-rendered AI composer controls and a persistent inline Voice session, protected
+active conversations, keyboard fallback, and one-answer buffering while the normal chat stream
+settles. Add the provider-neutral `renderVoiceMode` contract and export `PetrinautAiInputMode`, with
+the selected input mode and mode-change callback available to host-rendered Voice modes. Include
+stable finalized-text submission, conversation identity, stop handling, schema-validated
+interactive-tool text mapping, explicit separate-message targeting for corrections, and a
+queue-aware voice submission path. Present text and voice through one transcript and composer whose
+trailing action switches between waveform, Send, and Stop. `renderComposerControl` remains a
+supported public seam for hosts that only need their own control beside the message box,
+independently of Voice mode. Surface assistant request failures as error toasts instead of
+transcript entries.
 
-Simplify Interview mode with a circular microphone waveform, compact transcript states that
-distinguish recording, sending, sent, and undelivered answers, phase-specific icon controls, and
-recovery that names the kind of failure before offering reconnect.
+Render every live Voice surface from a session snapshot the host reports through
+`reportVoiceSessionState`, so hosts describe their session while Petrinaut owns its chrome. Replace
+the composer with a low-profile Voice dock -- a canvas ribbon of three travelling waves, one leading
+and two trailing it more faintly, fading out at both ends, opening with the microphone while
+listening and moving under its own power through the same drawing path while the assistant speaks,
+near flat while neither holds the turn, and naming one phase at a time -- with an announced phase and
+reduced-motion behavior. Curve the raw microphone level before it drives the waves, so an ordinary
+speaking level is plainly visible rather than a flicker above the line and a raised voice still has
+somewhere to go. Sample the level per animation frame rather than through React, so drawing costs no
+re-renders, and ease it against the frame delta so it looks the same at any refresh rate. Hold spoken
+turns out of the transcript until the session ends, then reveal them together under a turn-count
+divider, while typed messages and interactive tools awaiting an answer stay visible throughout. Let a
+per-session Show transcription in chat action write those turns into the conversation as they land
+instead. Keep every session control -- transcription, the microphone toggle, Resume, Reconnect, and
+End -- in the dock, leaving the canvas toolbar untouched. Add `setMicrophoneMuted` to the Voice mode
+controls and a `muted` session phase, so muting stops capture without interrupting what the assistant
+is saying, unlike pausing. Surface voice recovery failures as toasts with privacy-safe diagnostic
+references, and request one-time consent before the host starts the microphone. Mark persisted spoken
+messages and the exact interactive-tool answer completed by Voice with an inline Voice chip ahead of
+the words themselves.
+
+End Voice mode before submitting typed text exactly once through the shared composer, preserving the
+draft if handoff fails. Pause active media before the AI panel closes and reopen the mounted session
+paused. Provisional transcription and Realtime audio remain ephemeral rather than becoming
+persisted chat history.

apps/petrinaut-website/README.md

@@ -74,60 +74,73 @@ provides a fake optimizer for isolated UI development.
 
 Local values live in `.env.local`; Vite's `loadEnv` (see [`vite.config.ts`](vite.config.ts)) copies them into `process.env` for both the dev server and the API functions. In production, set these in the Vercel project settings.
 
-### Brunch voice preview
+### Brunch Voice mode preview
 
-Voice input is disabled by default and always unavailable when `VERCEL_ENV` is
+Voice mode is disabled by default and always unavailable when `VERCEL_ENV` is
 `production`. To exercise the preview locally or in a Vercel preview, set a
 real `VITE_BRUNCH_CHAT_ENDPOINT`, `PETRINAUT_OPENAI_VOICE_ENABLED=true`, and a
-dedicated `OPENAI_VOICE_API_KEY`. The browser sends its SDP offer to this app;
-the server initializes a trusted `gpt-realtime-2` audio-input/audio-output
-session through OpenAI's unified Realtime call endpoint. The provider key,
-model, instructions, tools, language and vocabulary policy stay server-side.
-The session uses semantic VAD with low eagerness so natural thinking pauses are
-less likely to end an answer early.
+dedicated `OPENAI_VOICE_API_KEY`.
+
+Text and Voice mode use one assistant transcript and composer. When Voice mode
+is available, the empty first-run prompt and empty composer show a waveform
+action; non-whitespace text replaces it with **Send**, and a busy assistant
+shows **Stop**. Starting Voice mode opens an inline, versioned consent
+disclosure before requesting microphone access. The disclosure also provides a
+microphone check and is remembered in browser storage only after Voice mode
+starts.
+
+An active session stays at the end of the transcript. Its compact divider shows
+a waveform and **Connecting**, **Listening**, **Speaking**, **Paused**, or a
+recovery state. Listening levels follow microphone input; provisional words
+appear immediately above the divider in an ephemeral user-style bubble. The
+bubble is replaced by the finalized message or pending-question tool output,
+which retains a waveform indicator without duplicating the answer. Provisional
+transcription and Realtime audio are not persisted as chat history.
+
+The text composer remains available. Sending typed text ends Voice mode first,
+then submits the draft exactly once through the same conversation; a failed
+handoff restores the draft. Closing the assistant pauses capture and speech
+before hiding it. Reopening preserves the mounted session in **Paused** state.
+**Pause** and **End voice mode** live under **Voice mode actions**, while
+**Resume** or **Reconnect** appears as the primary action when applicable.
+
+The browser sends its SDP offer to this app; the server initializes a trusted
+`gpt-realtime-2` audio-input/audio-output session through OpenAI's unified
+Realtime call endpoint. The provider key, model, instructions, tools, language,
+and vocabulary policy stay server-side. The session uses semantic VAD with low
+eagerness so natural thinking pauses are less likely to end an answer early.
 
 Realtime is the disposable media plane: it carries continuous microphone and
-remote audio, detects complete turns and handles barge-in. Brunch remains the
-control plane and sole authority for interview questions, captures, state,
-completion and durable history. The browser bridge accepts only the configured
+remote audio, detects complete turns, and handles barge-in. Brunch remains the
+control plane and sole authority for questions, captures, state, completion,
+and durable history. The browser bridge accepts only the configured
 `continue_interview` function, validates and serializes its arguments, rejects
-duplicate or stale calls and submits the answer through Petrinaut's existing
+duplicate or stale calls, and submits the answer through Petrinaut's shared
 composer path with pending-`brunch_ask` correlation.
 
 The bridge waits for the correlated Brunch turn before returning canonical
 speech segments to Realtime. It then requests audio with tools disabled and
-instructs Realtime to speak only those segments. Generated audio is not treated
-as a verbatim record: canonical Brunch text remains visible and authoritative.
-Partial input transcripts are display-only and never submit or disable capture.
-
-The microphone remains active while the interviewer speaks and while Brunch is
+instructs Realtime to speak only those segments. Generated audio is not a
+verbatim record: canonical Brunch text remains visible and authoritative. The
+microphone stays active while the interviewer speaks and while Brunch is
 working. Speaking over assistant audio interrupts playback automatically;
 WebRTC truncates provider-side unheard audio without changing Brunch history.
-**Pause** temporarily turns off input, while **End interview**, failure and
-reconnect release or replace all media and invalidate events from old
-connections. There is no separate Speech API, MP3 playback queue, required
-**Done speaking** action or manual **Interrupt and speak** action.
 
 The Brunch deployment must allow the website origin through its
-`BRUNCH_PETRINAUT_ORIGINS` setting. Starting voice input requests browser
-microphone permission. Denying permission leaves the existing text composer
-available and does not submit anything to Brunch.
-
-When the preview cannot continue, the status panel distinguishes microphone
-permission, microphone device, interrupted request, network, timeout, invalid
-response, and unavailable/disabled failures. Permission and device failures
-identify what to fix; network, timeout, and interrupted requests offer a
-reconnect; invalid responses include a diagnostic reference for an operator;
-and unavailable voice leaves the text composer as the fallback. Voice failures
-always leave the canonical response visible to read.
-
-Realtime connection, transcription and canonical speech timings use random
+`BRUNCH_PETRINAUT_ORIGINS` setting. Denying microphone permission leaves the
+text composer available and submits nothing to Brunch. When Voice mode cannot
+continue, the inline recovery state distinguishes microphone, connection, and
+other Voice failures, explains the next action, and offers **Reconnect** where
+appropriate. Sanitized error codes and diagnostic references remain collapsed
+under **Technical details**.
+
+Realtime connection, transcription, and canonical speech timings use random
 request IDs, and the existing Brunch transport provides its own request
 correlation. Browser and server diagnostics report only operation, stage,
-outcome, duration, request ID and—where applicable—status or a sanitized error
+outcome, duration, request ID, and—where applicable—status or a sanitized error
 code. Voice responses also expose privacy-safe `Server-Timing` metrics. These
 diagnostics never record audio, SDP, transcript or prompt contents, canonical
-speech text, credentials or provider response bodies. This controlled-preview
+speech text, credentials, or provider response bodies. This controlled-preview
 evidence does not enable production: production remains unconditionally
 disabled by the server policy.
 

apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.test.tsx

@@ -5,7 +5,7 @@ import { isValidElement, type ReactNode } from "react";
 import { describe, expect, test, vi } from "vitest";
 
 import { VoiceInterviewControl } from "../voice-interview/voice-interview-control";
-import { getBrunchVoiceInterviewStage } from "./local-storage-demo-app";
+import { getBrunchVoiceMode } from "./local-storage-demo-app";
 
 const defaultTransportOptions = vi.hoisted(() => ({
   current: null as unknown,
@@ -28,28 +28,24 @@ vi.mock("@hashintel/petrinaut/ui", () => ({
 
 describe("local storage demo Brunch voice integration", () => {
   test("does not install voice on the generic local chat fallback", () => {
-    expect(getBrunchVoiceInterviewStage(null)).toBeUndefined();
+    expect(getBrunchVoiceMode(null)).toBeUndefined();
   });
 
   test("installs the app-owned voice control for a configured Brunch transport", () => {
     const config = { available: true as const, connectionTimeoutMs: 15_000 };
-    const stage = getBrunchVoiceInterviewStage(config);
-    const control = stage?.({
-      canAcceptInterviewAnswer: true,
+    const voiceMode = getBrunchVoiceMode(config);
+    const control = voiceMode?.({
+      canAcceptVoiceInput: true,
       conversationId: "petrinaut-preview:net-1",
-      focusComposer: vi.fn(),
-      interactionMode: "chat",
+      inputMode: "text",
+      isAiAssistantOpen: true,
       messages: [],
-      openSidebar: vi.fn(),
-      placement: "sidebar",
-      setActive: vi.fn(),
-      setInteractionMode: vi.fn(),
+      registerVoiceModeControls: vi.fn(() => () => undefined),
+      reportVoiceSessionState: vi.fn(),
+      setInputMode: vi.fn(),
+      setVoiceActive: vi.fn(),
       status: "ready",
       stop: vi.fn(async () => undefined),
-      submitInterviewAnswer: vi.fn(async () => ({
-        kind: "message" as const,
-        messageId: "message-1",
-      })),
       submitText: vi.fn(async () => ({
         kind: "message" as const,
         messageId: "message-1",

apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx

@@ -17,9 +17,9 @@ import {
 import {
   DefaultChatTransport,
   Petrinaut,
-  type PetrinautAiInterviewStage,
-  type PetrinautAiInterviewStageContext,
   type PetrinautAiMessage,
+  type PetrinautAiVoiceMode,
+  type PetrinautAiVoiceModeContext,
   WalkthroughProvider,
 } from "@hashintel/petrinaut/ui";
 
@@ -95,11 +95,17 @@ const brunchPreviewConfig = resolveBrunchPreviewConfig(
   import.meta.env.VITE_BRUNCH_CHAT_ENDPOINT,
 );
 
-export const getBrunchVoiceInterviewStage = (
+// Only Brunch keeps a conversation to hydrate from; the generic fallback route
+// has no history door.
+const brunchHistoryEndpoint = brunchPreviewConfig.isBrunchConfigured
+  ? brunchPreviewConfig.chatEndpoint
+  : null;
+
+export const getBrunchVoiceMode = (
   config: OpenAIVoiceConfig | null | undefined,
-): PetrinautAiInterviewStage | undefined =>
+): PetrinautAiVoiceMode | undefined =>
   config
-    ? (context: PetrinautAiInterviewStageContext) => (
+    ? (context: PetrinautAiVoiceModeContext) => (
         <VoiceInterviewControl {...context} config={config} />
       )
     : undefined;
@@ -181,8 +187,8 @@ export const LocalStorageDemoApp = () => {
     return () => abortController.abort();
   }, []);
 
-  const brunchVoiceInterviewStage = useMemo(
-    () => getBrunchVoiceInterviewStage(openAIVoiceConfig),
+  const brunchVoiceMode = useMemo(
+    () => getBrunchVoiceMode(openAIVoiceConfig),
     [openAIVoiceConfig],
   );
 
@@ -321,7 +327,11 @@ export const LocalStorageDemoApp = () => {
   const conversationId = currentNetId
     ? getOrCreateBrunchConversationId(currentNetId)
     : null;
-  const flueHistory = useFlueChatHistory(conversationId ?? "", brunchPrincipal);
+  const flueHistory = useFlueChatHistory(
+    brunchHistoryEndpoint,
+    conversationId ?? "",
+    brunchPrincipal,
+  );
   const petrinautAiChatTransport = useMemo(
     () =>
       conversationId === null
@@ -361,15 +371,15 @@ export const LocalStorageDemoApp = () => {
           return next;
         });
       },
-      ...(brunchVoiceInterviewStage
+      ...(brunchVoiceMode
         ? {
-            renderInterviewStage: brunchVoiceInterviewStage,
+            renderVoiceMode: brunchVoiceMode,
           }
         : {}),
     }),
     [
       aiMessagesByNetId,
-      brunchVoiceInterviewStage,
+      brunchVoiceMode,
       conversationId,
       currentNetId,
       flueHistory.messages,

apps/petrinaut-website/src/main/app/local-storage-demo/use-flue-chat-history.test.ts

@@ -0,0 +1,69 @@
+/**
+ * @vitest-environment jsdom
+ */
+import { cleanup, renderHook, waitFor } from "@testing-library/react";
+import { afterEach, expect, test, vi } from "vitest";
+
+import { BRUNCH_PRINCIPAL_HEADER } from "@hashintel/brunch-agent-transport-aisdk/headers";
+
+import { useFlueChatHistory } from "./use-flue-chat-history";
+
+const brunchEndpoint = "https://brunch.test/api/chat";
+
+afterEach(() => {
+  cleanup();
+  vi.unstubAllGlobals();
+});
+
+test("hydrates from the configured Brunch endpoint", async () => {
+  const fetchMock = vi.fn(async () =>
+    Response.json({
+      messages: [{ id: "assistant-1", role: "assistant", parts: [] }],
+    }),
+  );
+  vi.stubGlobal("fetch", fetchMock);
+
+  const { result } = renderHook(() =>
+    useFlueChatHistory(brunchEndpoint, "conversation-1", "principal-1"),
+  );
+
+  await waitFor(() => expect(result.current.ready).toBe(true));
+  expect(result.current.messages).toStrictEqual([
+    { id: "assistant-1", role: "assistant", parts: [] },
+  ]);
+
+  const [url, init] = fetchMock.mock.calls[0] as unknown as [
+    URL,
+    { headers: Record<string, string> },
+  ];
+  expect(url.toString()).toBe("https://brunch.test/api/chat?id=conversation-1");
+  expect(init.headers[BRUNCH_PRINCIPAL_HEADER]).toBe("principal-1");
+});
+
+test("asks nothing of the generic chat route, which keeps no history", () => {
+  const fetchMock = vi.fn();
+  vi.stubGlobal("fetch", fetchMock);
+
+  const { result } = renderHook(() =>
+    useFlueChatHistory(null, "conversation-1", "principal-1"),
+  );
+
+  expect(fetchMock).not.toHaveBeenCalled();
+  expect(result.current.ready).toBe(false);
+  expect(result.current.messages).toBeUndefined();
+});
+
+test("leaves the panel on its local cache when hydration fails", async () => {
+  const fetchMock = vi.fn(
+    async () => new Response("Method not allowed", { status: 405 }),
+  );
+  vi.stubGlobal("fetch", fetchMock);
+
+  const { result } = renderHook(() =>
+    useFlueChatHistory(brunchEndpoint, "conversation-1", "principal-1"),
+  );
+
+  await waitFor(() => expect(fetchMock).toHaveBeenCalled());
+  expect(result.current.ready).toBe(false);
+  expect(result.current.messages).toBeUndefined();
+});