H-6763: Submit only completed Voice transcripts as interview answers

[kostandinang]

🌟 What is the purpose of this PR?

Stop Voice mode from submitting answers the user never said.

The Realtime session used semantic VAD with create_response: true and a required continue_interview({ answer }) tool. Every VAD commit therefore made the model produce a tool call, and the browser bridge submitted the model-written answer to Brunch with source: "voice". The real input_audio_transcription.completed result only drove UI state. Silence or background noise could become invented answers such as "hi", "um", or "one".

This PR makes the completed gpt-4o-transcribe transcript the sole authoritative user answer. Realtime no longer responds or calls tools on its own between turns; it only transcribes the user's audio and renders the speech Petrinaut supplies.

Stacked on #9496 and targets its branch; retarget to main once #9496 merges.

🔗 Related links

• H-6763 (internal)
• FE-1570: Optimize spoken Brunch responses for Voice mode #9496 (FE-1570, base of this stack)

🚫 Blocked by

• FE-1570: Optimize spoken Brunch responses for Voice mode #9496 — retarget this PR to main after it merges

🔍 What does this change?

• Session policy (openai-voice-policy.ts): semantic VAD create_response: false (keeps interrupt_response: true), tools: [], tool_choice: "none", instructions rewritten so the model only speaks supplied response_text; policy version brunch-control-plane-v2.
• Realtime session (openai-realtime-session.ts): removed the tool-argument events, completeFunctionCall(), and function-call correlation. response.function_call_arguments.delta is ignored; a response.done containing a function_call fails closed as invalid-response. Speech requests remain out-of-band with tools disabled, as in FE-1570: Optimize spoken Brunch responses for Voice mode #9496.
• Brunch bridge (realtime-brunch-bridge.ts): consumes only completed / transcription-failed transcript events for the active connection epoch, normalizes whitespace, tracks processed ${epoch}:${itemId}:${contentIndex} identities, and submits each accepted transcript exactly once with ID voice-realtime:<epoch>:<itemId>:<contentIndex>. Rejections emit a new transcript-rejected { reason } event (empty | failed | duplicate | unavailable | too-long); stale epochs are dropped. Brunch responses to transcript submissions now flow through the FE-1570 prepared-speech path (concise context + exact canonical question, canonical fallback, caching) or verbatim canonical speech. The inbound function-call delivery plumbing is removed; the preparation lifecycle is otherwise unchanged.
• Turn controller (voice-turn-controller.ts): new snapshot field inputNotice: "none" | "not-heard". Empty or failed transcripts set not-heard, clear the provisional transcript, and leave the interview and lastCommittedText unchanged; the notice clears when the user speaks again or an answer is submitted.
• README: describes the transcript-authoritative inbound path.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• No transcription confidence gate. OpenAI's conversation.item.input_audio_transcription.completed documents transcript, optional languages, and opt-in per-token logprobs (include: ["item.input_audio_transcription.logprobs"]), but no confidence score, so no threshold was invented. The protections here are: authoritative transcript only, empty-input rejection, per-item idempotency, and diagnostics.
• Transcripts that arrive while an answer is already in flight (unavailable) are dropped without a UI notice, to avoid a misleading "not heard" message; they remain visible as bridge events.

🐾 Next steps

• Surface inputNotice: "not-heard" in the Petrinaut Voice UI. PetrinautAiVoiceSessionState in @hashintel/petrinaut has no notice field yet; that is a published-library change with a changeset.
• Optionally enable transcription logprobs for diagnostics to gather data before deciding on any confidence gating.
• Retarget to main after FE-1570: Optimize spoken Brunch responses for Voice mode #9496 merges.

🛡 What tests cover this?

apps/petrinaut-website: yarn vitest run src/server/voice src/main/app/voice-interview — 11 files, 152 tests.

New or rewritten regression tests:

• Policy / call handler: create_response: false, tools: [], tool_choice: "none"; Realtime neither answers for the user nor calls tools between turns.
• Session: legacy function-call argument events never surface as user speech; a fabricated function_call response fails closed without sending function_call_output; keyed transcripts (including empty ones) are emitted verbatim.
• Bridge: completed transcript submitted verbatim; whitespace normalized; empty/whitespace input not submitted; duplicate completion events submit once; stale connection-epoch events ignored; failed transcription returns to listening; fabricated continue_interview arguments cannot be submitted; overlapping transcripts rejected as unavailable; over-limit transcripts rejected; Brunch responses to transcript submissions use FE-1570 preparation and exact-question assembly; preparation fallback speaks canonical content exactly once; cancelled preparation speaks nothing.
• Controller: empty/failed rejections return to listening with inputNotice: "not-heard" and no committed text; duplicate/unavailable leave state unchanged; notice clears on speech or submission, including while paused. Existing pause, mute, barge-in, replay, reconnect, and typed-handoff tests still pass.
• Integration (voice-preview.integration.test.ts): end-to-end from browser SDP through a completed transcript to canonical duplex audio, including silence, a legacy tool-argument delta, duplicate completion, prepared speech, replay, fallback, and a fabricated tool call failing closed.

Also run: yarn lint:tsc, yarn lint:eslint (0 errors), oxfmt --check, git diff --check.

❓ How to test this?

1. Check out the branch and run the Petrinaut website with PETRINAUT_OPENAI_VOICE_ENABLED=true and an OpenAI key.
2. Start a Voice interview and stay silent, cough, or make background noise after the question is asked.
3. Confirm no user message appears in the transcript and the session keeps listening (the controller snapshot reports inputNotice: "not-heard").
4. Speak a real answer and confirm it is submitted verbatim as the transcribed text, then the next Brunch response is spoken through the prepared-speech path.

📹 Demo

None yet.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
petrinaut	Ready	Preview	Sep 2, 2026 8:30pm UTC
petrinaut-docs	Ready	Preview	Sep 2, 2026 8:30pm UTC

2 Skipped Deployments

Project	Deployment	Actions	Updated
hash	Ignored	Preview	Sep 2, 2026 8:30pm UTC
hashdotdesign-tokens	Ignored	Preview	Sep 2, 2026 8:30pm UTC

[kostandinang]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• FE-1576: Ground Brunch interviews in the current Petrinaut net #9523
• H-6800: Improve Petrinaut Voice turn-taking and answer provenance #9512
• H-6763: Add temporary Brunch ask shim for Voice preview #9507
• H-6763: Submit only completed Voice transcripts as interview answers #9500 👈 (View in Graphite)
• FE-1570: Optimize spoken Brunch responses for Voice mode #9496
• H-6763: Integrate Voice mode into the AI assistant #9467
• H-6763: Migrate voice interviews to GPT-Realtime-2 #9399
• H-6763: Add the voice interview experience #9380
• H-6763: Improve voice preview reliability diagnostics #9378
• H-6763: Speak finalized Brunch responses with OpenAI #9357
• H-6763: Add OpenAI Realtime voice input to Brunch #9356
• H-6763: Add a generic Petrinaut composer submission API #9355
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

PR Summary

Medium Risk
Changes the authoritative path for spoken interview answers and Realtime session policy; incorrect transcript handling could drop or mis-submit user input, though the design is fail-closed on model tool calls and heavily tested.

Overview
Voice answers now come only from completed gpt-4o-transcribe transcripts, not from a required Realtime continue_interview tool. Server session policy moves to half-duplex: semantic VAD with create_response: false, no tools, and instructions that forbid the model from inferring or restating what the user said (brunch-control-plane-v3).

The Realtime session mutes the microphone during assistant playback, tags and drops input that overlaps playback, removes tool-argument plumbing and completeFunctionCall, treats unexpected function_call output as fail-closed invalid-response, and keeps all interviewer speech as out-of-band response.create with tools disabled.

The Brunch bridge normalizes transcript text, submits each (epoch, item, content index) at most once via voice-realtime:… IDs, emits transcript-rejected for empty, failed, duplicate, unavailable, or over-limit input, and speaks Brunch replies through prepared/canonical speech only (no function-call round trip). restoreCancelledSpeech replays the question after recoverable “not heard” cases.

The turn controller adds inputNotice: "not-heard" for empty/failed rejections; docs/README describe waiting for playback instead of barge-in submitting echo as an answer.

^{Reviewed by Cursor Bugbot for commit 935aa9f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5ed5a61. Configure here.}