FE-1570: Optimize spoken Brunch responses for Voice mode

[kostandinang]

🌟 What is the purpose of this PR?

Improve spoken response quality for the Voice mode introduced by H-6763 without weakening Brunch's authority. Petrinaut continues to render the complete canonical Brunch response, while OpenAI Realtime prepares only shorter explanatory context for disposable audio delivery. Application code then appends every protected brunch_ask question exactly.

This PR is stacked on #9467 so it can reuse the unified Voice session, transcript, and lifecycle contracts without duplicating them.

🔗 Related links

• FE-1570 (internal)
• H-6763 (internal) — parent Voice-mode work
• #9467 — parent unified Voice-mode PR
• #9399 — merged GPT-Realtime-2 transport foundation

🚫 Blocked by

• H-6763: Integrate Voice mode into the AI assistant #9467

🔍 What does this change?

• Splits canonical Brunch output into finalized explanatory context, an exact protected question, and the complete canonical fallback.
• Uses an out-of-band, text-only Realtime response with conversation: "none", tool_choice: "none", and no tools to prepare context.
• Reserves a strict 50-word spoken budget and appends the exact canonical question and choices in application code.
• Validates prepared output, limits preparation to two seconds, and falls back to complete canonical speech after timeout, provider failure, interruption, or invalid output.
• Caches validated preparation only for the active connection epoch and discards stale completions on stop or reconnect.
• Serializes preparation and audio work with the existing Realtime response queue while preserving pause, barge-in, mute, reconnect, typed handoff, provenance, and exactly-once answer submission.
• Adds Repeat question and Read full response Voice-dock actions that bypass preparation and replay canonical text.
• Correlates replay availability with both terminal response and output-buffer completion so overlapping or stale responses cannot re-enable unsafe replay.
• Keeps canonical/prepared text, audio, provider transcripts, tool payloads, IDs, diagnostics, and disposable Realtime history out of persistent history and logs.

Boundaries

• Brunch remains the control plane and sole authority for questions, captures, interview state, completion, and durable history.
• No Brunch Mission 3 runtime, mission, endpoint, capture/session, or tool-result behavior changes are included.
• Generated preparation is experimental delivery text, never canonical evidence.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• modifies an npm-publishable library and I have added a changeset file(s)

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Voice is a non-production preview and is disabled by server policy in production.
• Realtime context preparation is experimental because local validation cannot prove semantic equivalence; canonical fallback is therefore mandatory.
• Physical microphone quality and interruption behavior still need validation in a preview deployment.

🐾 Next steps

• Retarget this PR to main after H-6763: Integrate Voice mode into the AI assistant #9467 merges.
• Validate prepared speech, exact questions, barge-in, pause/resume, reconnect, mute, and canonical replay on a physical device in preview.
• After Brunch Mission 3 lands, write a separate plan for durable brunch_ask human-answer admission against the landed architecture.

🛡 What tests cover this?

• yarn workspace @apps/petrinaut-website test:unit — 31 files / 237 tests passed.
• yarn workspace @hashintel/petrinaut test:unit --run — 46 files / 405 tests passed.
• yarn workspace @local/petrinaut-arch-docs lint:arch-docs — 57 layers / 259 edges / 579 files passed.
• Website and Petrinaut builds, TypeScript checks, ESLint, repository formatting, and git diff --check passed.
• Focused coverage includes canonical selection, preparation validation/timeout/fallback, protected-question assembly, stale epochs, function-call and response correlation, barge-in, pause/reconnect, replay safety, provenance, mute state, and exactly-once submissions.

❓ How to test this?

1. Check out this branch after configuring a real VITE_BRUNCH_CHAT_ENDPOINT, PETRINAUT_OPENAI_VOICE_ENABLED=true, and dedicated OPENAI_VOICE_API_KEY in apps/petrinaut-website/.env.local.
2. Run yarn workspace @apps/petrinaut-website dev and open the local URL.
3. Start Voice mode and progress through a Brunch response containing explanatory context followed by a question with choices.
4. Confirm the transcript retains the complete canonical response while speech uses concise context followed by the exact question and choices.
5. Use Repeat question and Read full response; confirm both replay canonical content rather than prepared wording.
6. Exercise barge-in, pause/resume, reconnect, and mute around preparation and playback; confirm speech does not duplicate and stale work does not resume.
7. End Voice mode and confirm the canonical transcript and accepted answer remain unchanged.

📹 Demo

A short physical-device preview recording should be added before marking this PR ready for review because the primary behavior is audio-dependent.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
petrinaut	Ready	Preview	Sep 2, 2026 8:06pm UTC
petrinaut-docs	Ready	Preview	Sep 2, 2026 8:06pm UTC

2 Skipped Deployments

Project	Deployment	Actions	Updated
hash	Ignored	Preview	Sep 2, 2026 8:06pm UTC
hashdotdesign-tokens	Ignored	Preview	Sep 2, 2026 8:06pm UTC

[kostandinang]

• FE-1576: Ground Brunch interviews in the current Petrinaut net #9523
• H-6800: Improve Petrinaut Voice turn-taking and answer provenance #9512
• H-6763: Add temporary Brunch ask shim for Voice preview #9507
• H-6763: Submit only completed Voice transcripts as interview answers #9500
• FE-1570: Optimize spoken Brunch responses for Voice mode #9496 👈 (View in Graphite)
• H-6763: Integrate Voice mode into the AI assistant #9467
• H-6763: Migrate voice interviews to GPT-Realtime-2 #9399
• H-6763: Add the voice interview experience #9380
• H-6763: Improve voice preview reliability diagnostics #9378
• H-6763: Speak finalized Brunch responses with OpenAI #9357
• H-6763: Add OpenAI Realtime voice input to Brunch #9356
• H-6763: Add a generic Petrinaut composer submission API #9355
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

PR Summary

Medium Risk
Changes Realtime orchestration, speech timing, and interview UX around canonical vs prepared audio; fallbacks and replay gating reduce but do not eliminate risk of wrong or duplicated spoken content.

Overview
Voice mode now uses an Approach D pipeline: after Brunch finishes a turn, OpenAI Realtime may run a text-only, out-of-band step to condense finalized assistant context within a 50-word budget (reserving words for the protected brunch_ask question), then plays verbatim tool-disabled audio with the exact question appended in application code. Invalid, slow (>2s), interrupted, or failed preparation falls back to full canonical context plus question; Brunch transcript text stays authoritative and unchanged.

selectInterviewSpeech separates context segments, the pending question, and full-response segments for automatic delivery. The Realtime session queues preparation and speech together, validates prepared text, caches successful prep per connection epoch, and supports speakPrepared / optional speakResponse: false on function-call completion when speech is cancelled.

The Voice dock gains Voice playback options (Repeat question, Read full response) wired through Petrinaut session state (canRepeatQuestion, canReadFullResponse) so replay uses speakCanonical and skips preparation. Barge-in, pause, and disconnect invalidate pending preparation; replay stays disabled until the active response is safely terminal.

Docs and changesets describe the dock replacing the composer during voice, typed send requiring End voice mode first, and expanded voice diagnostics for the preparation operation.

^{Reviewed by Cursor Bugbot for commit c7fe8a2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 943dfc2. Configure here.}

[lunelson]

The Brunch-side premise has changed enough since this work was designed that we now intend to validate a different Voice integration in FE-1574. This is an architectural coordination note, not a claim that the careful correlation, fallback, cancellation, and replay work in this PR is incorrect.

🏗️ Agent notes

What changed

Mission 4 replaced the older Brunch agent shape with the package-composed useBrunchAgent() + useSdcpnPlugin() agent and showed focused conversational behavior on its successful interactive probes. That does not prove every canonical reply is already suitable for speech, but it removes the premise that a secondary semantic rewrite is necessarily required; we should measure the current agent before making that rewrite a permanent layer.

We also no longer expect the existing Petrinaut AI assistant or its AI SDK useChat request/response format to be the authority for Voice. Flue already exposes the native conversation operations Voice needs: durable send, submission-correlated read, resumable/live observe, canonical history, and abort. The intended first experiment is therefore:

finalized STT
→ FlueClient.send to the owning Brunch conversation
→ canonical Flue response via observe/read
→ visible Voice text + ordinary TTS

A remotely exposed browser path may still need a thin authenticated same-origin proxy, but it should preserve Flue semantics rather than translate Voice through AI SDK UI messages.

Why this matters for this PR

The preparation path here is defensive: it excludes tools/reasoning, protects the exact brunch_ask question, times out, falls back canonically, and keeps generated speech ephemeral. Those are good properties. Even so, prepareInterviewSpeech() adds another semantic actor, up to two seconds before delivery, more cancellation/race state, and possible divergence between what Brunch canonically said and what the user heard. The 50-word result cannot be proven semantically equivalent by the structural validation in this branch.

Mission 5 will initially remove that variable and test exact canonical speech. Its visible proof target is:

• one finalized spoken answer appears exactly once in canonical Flue history;
• the corresponding canonical Brunch response reaches visible Voice text and TTS without the AI SDK chat composer or a secondary generation call;
• local playback cancellation and durable Flue abort remain distinct and visible; and
• reopening the conversation reconstructs the settled turn without duplicate submission or playback.

The draft contract is here.

What we should preserve

The reusable work in this stack—exactly-once finalized answer submission, turn correlation, interruption/barge-in behavior, stale-work rejection, playback controls, answer provenance, and explicit canonical fallback—should inform or be ported into that experiment. For now, please treat the AI SDK/Petrinaut-assistant dependency and secondary 50-word preparation as experimental choices rather than the integration seam we intend to stabilize.

If canonical speech proves unusably long, the next least-lossy options are deterministic question-focused presentation or Brunch-owned spoken-mode output. We should retain a secondary generative preparation step only if the direct experiment demonstrates that those are insufficient and the latency/semantic trade-off is accepted explicitly.