FE-1604: Allow Voice interruption by speaking

[kostandinang]

🌟 What is the purpose of this PR?

This PR lets a person interrupt Voice assistant playback by speaking without losing the interrupting utterance. It keeps Your turn as an optional half-duplex fallback and rejects likely prompt-regurgitation or assistant-echo transcripts before they become answers.

This is an independent semantic recut of the interruption-only delta from the stale, conflicting #9550, based on post-deployment main at ef0f444987. It does not merge or carry the old branch's unrelated Brunch stack. The automated proof establishes protocol ordering, admission, retention, UI preference, and package integrity; it does not establish real microphone latency or acoustic echo behavior.

🔗 Related links

• FE-1604 (internal)
• Replaces FE-1604: Allow Voice interruption by speaking #9550
• Independent settlement sibling: FE-1580: Port omitted Voice settlement behavior #9588

🚫 Blocked by

• FE-1580: Port omitted Voice settlement behavior #9588 landing and a final update from main
• Real browser/microphone witness for speaker feedback and false interruption
• Review completion

🔍 What does this change?

• Keeps microphone capture active during canonical playback when Interruption by speaking is enabled.
• Preserves immediate speech_started → response.cancel → output_audio_buffer.clear; interruption never clears the input buffer or waits for transcript validation.
• Carries the same input item through completed transcription and submits it exactly once through the existing Brunch admission path.
• Retains an accepted interruption while the previous Brunch turn settles, including across follow-on canonical speech and later rejected or failed transcripts.
• Rejects completed interruption transcripts with strong ordered overlap against the configured transcription vocabulary or the exact canonical text active when interruption began.
• Uses medium Semantic VAD eagerness while keeping response creation and interruption under Petrinaut control.
• Adds a browser-saved, default-on Interruption by speaking checkbox to the playback menu and shows Your turn only when it is disabled.
• Preserves mute, pause, Stop, end, reconnect, exact question replay, and exact full-response replay.
• Updates the website README, Petrinaut Voice guide, Voice architecture decision record, and Petrinaut patch changeset.

🏗️ Agent notes

Recut provenance

The stale PR head f69ac17034 contained four interruption product commits after its FE-1580 parent. Those commits were applied onto current main; range comparison found one cancellation-settlement hunk already present on main, with the remaining semantic patches unchanged. The current NodeNext website boundary additionally required the new shared transcription import to use its .js extension.

Constraints retained

• Completed provider transcription remains the only Voice-answer authority.
• Cancellation is immediate, output-only, and never gated by classification.
• Only playback-overlapping input is classified; ordinary capture is unchanged.
• Self-echo compares only with canonical text active when speech began, not queued speech or history.
• Rejection diagnostics contain only operational metadata and a reason.
• Short novel answers such as “stop”, “no”, and “wait” remain admissible.
• FE-1580: Port omitted Voice settlement behavior #9588 remains an independent sibling and is not included in this branch.

Fog-line and stop conditions

Browser echo cancellation may still permit a false VAD event to stop playback before the completed transcript is rejected. Stop or reorient if current-main behavior clears the interrupting input, delays cancellation, admits an interruption twice, erases an earlier retained answer, or fails to restore acknowledged half-duplex handoff when the preference is disabled.

The complete live mission contract is in libs/@hashintel/brunch-agent/MISSION.md.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

• Modifies an npm-publishable library and includes a patch changeset.

📜 Does this require a change to the docs?

• Requires docs changes, made in apps/petrinaut-website/README.md, libs/@hashintel/petrinaut/docs/ai-assistant.md, and libs/@hashintel/brunch-agent/docs/adr/0009-openai-voice-ui-turn-shell.md.

🕸️ Does this require a change to the Turbo Graph?

• Does not affect the execution graph.

⚠️ Known issues

• A false VAD event can still stop playback even when the completed transcript is discarded.
• The deterministic classifier is conservative and heuristic; real speaker/microphone behavior still needs a human witness.
• Protocol tests do not benchmark audible interruption latency.
• Website ESLint reports one pre-existing set-state-in-effect warning in the Voice control component; there are zero errors.

🛡 What tests cover this?

Coverage includes immediate cancellation and input preservation; duplicate completion; delayed and pending admission; follow-on canonical speech; retained-answer display; prompt-regurgitation and active-playback self-echo rejection; short novel answers; unchanged ordinary capture; preference/UI behavior; and lifecycle cleanup.

Fresh verification on this recut:

• Focused Voice suites: 13 files, 246 tests passed.
• Full @apps/petrinaut-website suite: 41 files, 417 tests passed.
• Full @hashintel/petrinaut suite: 94 files, 800 tests passed.
• Dependency-aware TypeScript checks, ESLint, and builds passed for both affected workspaces (19/19 Turbo tasks).
• Petrinaut architecture lint passed: 78 layers, 409 edges, 786 files.
• Repository formatting and git diff --check passed.

❓ How to test this?

1. Start Voice and open Voice playback options. Confirm Interruption by speaking is checked by default.
2. While Brunch is speaking, begin answering, including a short answer such as “stop”, “no”, or “wait”. Confirm playback stops immediately and the same utterance appears once, including if follow-on canonical speech is queued before transcription completes.
3. Interrupt while the previous Brunch turn is still settling. Confirm Answer captured. Waiting for Brunch. remains visible and submits once input reopens. Confirm a later rejected echo does not erase that notice.
4. Uncheck Interruption by speaking. Confirm Your turn returns and the microphone stays closed during playback until the acknowledged handoff completes. Reload and confirm the preference is remembered.
5. Exercise mute, pause, Stop, end, reconnect, Repeat question, and Read full response in both modes.

📹 Demo

The original interruption-only capture remains representative of this semantic recut:

fe-1604-voice-interruption-by-speaking.mp4

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 9, 2026 2:09pm UTC
petrinaut	Ready	Preview	Sep 9, 2026 2:09pm UTC
petrinaut-docs	Ready	Preview	Sep 9, 2026 2:09pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
hashdotdesign-tokens	Ignored	Preview	Sep 9, 2026 2:09pm UTC

[cursor]

PR Summary

Medium Risk
Changes real-time Voice turn-taking, transcript admission, and Brunch submission timing; mistakes could drop or duplicate answers, though behavior is heavily tested and stays on the existing admission path.

Overview
Adds Interruption by speaking (default on, browser-saved): while Brunch is playing canonical audio, the Realtime session keeps the mic open, immediately sends response.cancel and output_audio_buffer.clear on speech start (without clearing the input buffer), and still submits the completed transcript through the existing Brunch bridge once.

When Brunch is not ready, interrupting answers are retained and surfaced in the dock until admission opens; only those completions run local prompt-regurgitation and active-playback self-echo checks (shared transcription vocabulary, normalized fuzzy bigrams). Rejections emit content-free diagnostics with a rejectionReason. Disabling the preference restores half-duplex capture and the manual Your turn handoff (hidden while interruption is on).

Shared Petrinaut UI exposes the toggle in Voice playback options; server Realtime policy moves semantic VAD to medium eagerness and centralizes the transcription prompt. Docs, ADR, changeset, and broad unit/integration coverage (including durable history reopen) accompany the stack changes.

^{Reviewed by Cursor Bugbot for commit 863ee19. Bugbot is set up for automated code reviews on this repo. Configure here.}

[lunelson]

I found four behavioral blockers and one mission-contract gap:

1. Short exact self-echo is admitted. classify-interruption.ts returns null for every transcript under six tokens before comparing canonical playback. An exact echo such as “Who approves this?” therefore becomes an answer. The contract permits short novel answers, not short exact echoes.

2. Generated audio awaiting playback escapes spoken interruption. #interruptOutputBySpeaking() cancels only active or already-speaking responses. After response.done but before output_audio_buffer.started, speech does not cancel/clear the terminal buffered response. The manual-cancellation suite models this phase, but the spoken-interruption “generated” case starts playback first.

3. Pause or host Stop silently drops a retained answer. VoiceTurnController.pause() / host Stop call cancelPendingSpeech(), which sets RealtimeBrunchBridge.#pendingInterruption = null. The controller can continue showing “Answer captured. Waiting for Brunch” after the answer has become undeliverable.

4. Cancellation before response.created can leave permanent output ownership. The cancelled-created path returns before recording speechRequestId; its terminal event cannot clear #pendingSpeechRequestIds in the bridge. Later ordinary speech can remain classified against phantom pending output. The bridge test fabricates a terminal speechRequestId that the real session does not emit in this path.

Separately, MISSION.next.md requires every later Voice mission to exercise the typed-origin + Voice-origin + durably-stopped second-tab reopen scenario. FE-1604 exercises Voice but does not include that gate. This is also still deferred in FE-1573 Step B, so the ownership/disposition should be explicit rather than silently left there.

The architecture and overall branch scope otherwise align well: provider logic remains website-owned, Petrinaut stays provider-neutral, and admission still uses the existing Brunch path.

[kostandinang]

Addressed and pushed in 268303c8a4, with the ownership follow-up in a16c19ef0f (mission authority changes remain committed separately):

1. Exact normalized playback echoes are now rejected before the six-token fuzzy-overlap threshold. The regression also proves a short novel answer is still admitted unchanged.
2. Spoken interruption clears completed canonical audio buffered between response.done and output_audio_buffer.started; response.cancel is sent only while the response is active. The buffer-clear acknowledgement now carries speechRequestId downstream, so both the bridge and controller release generated-but-unplayed output ownership and restore capture.
3. Output cancellation no longer deletes a finalized retained interruption. Delivery waits for both cancellation handoff and chat readiness. Both Stop/cancellation race orders now assert that the first gate alone cannot drain the retained answer.
4. Response-to-speech correlation is recorded as soon as canonical response.created arrives and retained independently through output cleanup, so a pre-created cancellation's terminal event carries speechRequestId and releases bridge ownership.
5. MISSION.md owns the combined continuity gate. voice-history-continuity.integration.test.tsx exercises typed history, supported Voice client-tool provenance, durable composer Stop, close/reopen through a second mounted Flue observation, stopped rendering, and local End Voice without another durable Stop. Per the owner decision, direct-user Voice source reconstruction is explicitly deferred: Flue 2.0.3 exposes no supported caller-metadata/idempotency-key projection, and the prohibited browser-side/text-encoding alternatives were not introduced.

Verification:

• Petrinaut website: 422/422 tests
• Petrinaut library: 800/800 tests
• Session/bridge/controller Voice suites: 175/175 tests
• Website and Petrinaut TypeScript + ESLint checks pass (website has one pre-existing warning, zero errors)
• Website/library build: 13/13 tasks
• Formatting and git diff --check pass

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a16c19e. Configure here.}