H-6800: Improve Petrinaut Voice turn-taking and answer provenance

[kostandinang]

🌟 What is the purpose of this PR?

Make Voice interruption predictable and keep Voice attribution next to the interactive-tool answer it describes. Users can explicitly take the turn while the assistant prepares or speaks, and the microphone resumes only after Realtime confirms that canceled input and output have cleared.

This PR collects the independently useful UI/UX improvements discovered while testing H-6763. It does not change semantic VAD eagerness, add full duplex, or provide durable elicitation or canvas writes.

🔗 Related links

• H-6800 (internal)
• H-6763 (internal)
• H-6763: Add temporary Brunch ask shim for Voice preview #9507 — immediate base containing the temporary brunch_ask shim

🚫 Blocked by

• H-6763: Add temporary Brunch ask shim for Voice preview #9507 — this PR is stacked on its branch

🔍 What does this change?

• Adds an accessible Your turn action while Voice prepares or speaks, styled consistently with the Voice dock controls.
• Clears input and output audio, cancels queued or active speech without disconnecting Realtime, and waits for cancellation acknowledgements before reopening the microphone.
• Ignores speech captured before the handoff while keeping the interrupted question visible and repeatable.
• Lets interactive tools opt into a backwards-compatible answer-prefix slot so the Voice provenance chip appears immediately before submitted answers; other widgets retain the trailing fallback.
• Updates the Brunch ask widget, Petrinaut AI Assistant documentation, package guidance, release notes, and controller/session/integration/UI tests.

🏗️ Agent notes

Imperative: Preserve the useful Voice interaction improvements exposed by H-6763 as a reviewable change independent of the temporary Brunch compatibility mechanism.

Throughline: Voice dock → composer control → host Voice control → turn controller → OpenAI Realtime cancellation acknowledgements → listening resumes. Interactive-tool rendering independently projects Voice provenance immediately before an opted-in answer.

Proof: Controller, Realtime session, host-control, preview-integration, and UI tests cover successful and failed handoffs, stale audio rejection, cancellation ordering, repeatability, inline provenance, and backwards-compatible fallback rendering.

Constraints: Semantic VAD remains at low eagerness so natural thinking pauses are not prematurely committed. The handoff does not disconnect Realtime or enable full duplex. Other interactive tools retain their existing provenance placement unless they opt into the prefix slot.

Fog-line: The durable elicitation and canvas-write architecture remains outside this PR. These interaction behaviors must survive any later replacement of the temporary parent shim.

Stop or reorient: Restack or migrate these behaviors onto the durable Voice branch before #9507 is removed.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• modifies the npm-publishable @hashintel/petrinaut library and includes its required patch changeset

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Voice remains intentionally half-duplex; Your turn is an explicit handoff rather than simultaneous conversation.
• This PR does not change semantic VAD eagerness; it remains low.
• Durable elicitation and Petrinaut canvas writes remain outside this stack.

🐾 Next steps

• Restack or migrate these behaviors onto the durable Voice implementation before the temporary H-6763: Add temporary Brunch ask shim for Voice preview #9507 shim is retired.

🛡 What tests cover this?

• Focused Voice controller, Realtime session, preview integration, session-state, and host-control tests
• Focused Petrinaut AI Assistant and interactive-tool UI tests
• yarn workspace @apps/petrinaut-website lint:tsc
• yarn workspace @hashintel/petrinaut lint:tsc
• yarn workspace @apps/petrinaut-website lint:eslint — 0 errors; 1 pre-existing warning
• yarn workspace @hashintel/petrinaut lint:eslint — 0 errors or warnings
• Focused oxfmt, Markdown, architecture-doc, and git diff --check validation

❓ How to test this?

1. Start a Voice interview and wait for the assistant to prepare or speak a question.
2. Select Your turn and confirm speech stops without disconnecting.
3. Speak only after the handoff completes and confirm pre-handoff audio is not submitted.
4. Confirm Repeat question remains available for the interrupted question.
5. Submit a Voice answer through the Brunch ask widget and confirm the Voice chip appears immediately before the answer.

📹 Demo

Not recorded.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 3, 2026 12:43pm UTC
petrinaut	Ready	Preview	Sep 3, 2026 12:43pm UTC
petrinaut-docs	Ready	Preview	Sep 3, 2026 12:43pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
hashdotdesign-tokens	Ignored	Preview	Sep 3, 2026 12:43pm UTC

[cursor]

PR Summary

Medium Risk
Changes real-time voice cancellation, microphone gating, and chat metadata attribution—behavioral paths that are easy to get wrong under race conditions, though coverage is extensive in session/controller/UI tests.

Overview
Adds a Your turn control in the Voice dock (and takeTurn / canTakeTurn on host session APIs) so users can interrupt assistant speech or pending delivery without ending the session. The website turn controller drives a stricter Realtime cancelOutput() that clears input/output buffers, awaits provider acknowledgements, and only then re-enables the mic while discarding utterances that started before the handoff.

Voice provenance moves from a text Voice chip to a compact waveform icon on spoken user messages. Interactive tools can opt in with supportsSubmittedOutputProvenance and submittedOutputProvenance so Brunch ask (and similar widgets) place that icon inside the submitted answer bubble; others keep the trailing fallback. Assistant message metadata now tracks voiceToolCallIds (with rollback when a tool submit fails) so multiple voice-answered tools on one turn stay attributed correctly.

Supporting refactors split Brunch/voice modules for Fast Refresh (widget, disclosure, config loader, getBrunchVoiceMode) and replace diagnostics transport refs with DiagnosticsTransportState in the AI assistant panel.

^{Reviewed by Cursor Bugbot for commit d13535d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[kostandinang]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• FE-1576: Ground Brunch interviews in the current Petrinaut net #9523
• H-6800: Improve Petrinaut Voice turn-taking and answer provenance #9512 👈 (View in Graphite)
• H-6763: Add temporary Brunch ask shim for Voice preview #9507
• H-6763: Submit only completed Voice transcripts as interview answers #9500
• FE-1570: Optimize spoken Brunch responses for Voice mode #9496
• H-6763: Integrate Voice mode into the AI assistant #9467
• H-6763: Migrate voice interviews to GPT-Realtime-2 #9399
• H-6763: Add the voice interview experience #9380
• H-6763: Improve voice preview reliability diagnostics #9378
• H-6763: Speak finalized Brunch responses with OpenAI #9357
• H-6763: Add OpenAI Realtime voice input to Brunch #9356
• H-6763: Add a generic Petrinaut composer submission API #9355
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d13535d. Configure here.}

[cursor]

Rollback restores stale voice IDs

Medium Severity

When overlapping Voice tool submissions both reject, the later rollback can restore previousMetadata after sibling IDs were already removed. That puts a failed toolCallId back into voiceToolCallIds, so a later typed retry can still receive Voice provenance.

 

^{Reviewed by Cursor Bugbot for commit d13535d. Configure here.}