H-6763: Integrate Voice mode into the AI assistant - #9467
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
github-actions
Bot
added
area/infra
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
65a210e to
525c69b
Compare
525c69b to
815028c
Compare
Keep text and voice in one conversation with inline controls, provenance, lifecycle recovery, and accessible live feedback. Co-authored-by: Cursor <cursoragent@cursor.com>
Preserve failed speech for recovery and ensure every text-mode transition safely ends the active voice session before submission. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep realtime submissions current without render-time ref access and avoid querying the POST-only local chat fallback for history. Co-authored-by: Cursor <cursoragent@cursor.com>
Hosts report a session snapshot instead of rendering the live Voice UI, so Petrinaut owns the dock, indicator and canvas controls, defers spoken turns until a session ends, and shows recovery failures as toasts. Also carries the local-storage demo's chat-history refinement: an explicit endpoint parameter, URL construction through `new URL()`, and comments for why the generic route has no history door. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep pending question audio untouched when the panel closes during connection, then start the bridge after the user explicitly resumes. Co-authored-by: Cursor <cursoragent@cursor.com>
Remember the notified voice failure until the session recovers so repeated snapshots and callback changes cannot enqueue the same toast again. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Consolidate the session controls onto the assistant dock: a transcription toggle, a round microphone button that mutes input without ending the turn, and a red stop button. Replace the bar indicator with a canvas ribbon driven by the live microphone level, and mark spoken turns with an inline Voice chip. Co-authored-by: Cursor <cursoragent@cursor.com>
The ribbon's shape came from two sine waves beating against each other, which read as decoration rather than as speech. Keep a couple of seconds of sampled levels instead and draw the envelope from that contour, so the newest sound enters at one edge and the shape of what was said drifts across. Ease against the frame delta so the motion is identical at any refresh rate, and drop to one stroke and a faint echo so the dock stays quiet. Co-authored-by: Cursor <cursoragent@cursor.com>
Draw the ribbon as three travelling waves that fade out at both ends, replacing the rolling contour. Curve the raw microphone level before it drives them: an RMS reading puts ordinary speech near 0.15, which the previous linear gain turned into under two pixels of deflection, so a whole sentence barely lifted off the midline. Give the inline Voice chip a legible blue. It was pale blue text on a four percent tint, which at ten pixels read as nothing at all. Quieten the dock to ghost icons at one size, and end a session with a close mark rather than a filled stop. Draw the transcription toggle as lines of transcript; the shared set only offers a serif T, which promises inserting text rather than showing it. Drop the error tone from the muted microphone, so red belongs to the one action that cannot be undone. Keep the composer's trailing action on a single button and swap only the glyph inside it. Rendering a different element per state tore the button down and built it again, so the box blinked on every change, including the send-to-voice swap that happens on the first character typed. Hold the composer and the dock to one height and fade them through each other in place, so trading one for the other leaves the transcript above untouched. Restore the reader's distance from the end for the cases that cannot cover, such as prompt chips returning with the composer. Stop the composer raising a scrollbar on a box that visibly fits. `scrollHeight` is a rounded integer, so the height it yields can land a fraction of a pixel under the real content; allow scrolling only once the content passes the cap. Recompute the height when the panel opens, too, since a measurement taken while it is hidden reads zero and was never revisited.
Keep shared session state types in the React layer so state providers do not depend on the UI components that render them. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b5eba96. Configure here.
Report muting independently from the active turn phase so the dock remains accurate and can unmute while the assistant speaks or thinks. Co-authored-by: Cursor <cursoragent@cursor.com>
PR SummaryMedium Risk Overview The Brunch website voice control is slimmed to inline consent + Realtime/session orchestration—the large Typed send ends Voice once before shared submission (draft restored on failure); closing the panel pauses media and reopens the same mounted session paused. Assistant failures surface as toasts instead of transcript lines. Docs, ADR-0009, integration spec, changeset, and tests align with the new contract. Reviewed by Cursor Bugbot for commit 2a1f1bb. Bugbot is set up for automated code reviews on this repo. Configure here. |
2 participants
🌟 What is the purpose of this PR?
Complete H-6763 by integrating Voice mode into Petrinaut's existing AI assistant instead of maintaining a separate interview surface. Text and voice share one canonical Brunch conversation and transcript, while the merged GPT-Realtime-2 transport remains a disposable media plane for microphone input, interruption, and speech.
🔗 Related links
🚫 Blocked by
🔍 What does this change?
brunch_askcorrelation, ordered answer submission, host/programmatic text handoff, draft recovery, and exactly-once semantics.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
🐾 Next steps
🛡 What tests cover this?
❓ How to test this?
apps/petrinaut-website/.env.local, configure a realVITE_BRUNCH_CHAT_ENDPOINT,PETRINAUT_OPENAI_VOICE_ENABLED=true, and a dedicatedOPENAI_VOICE_API_KEY.yarn workspace @apps/petrinaut-website devand open the local URL.📹 Demo
A short physical-device preview recording is still needed because the primary behavior is audio- and motion-dependent.