From 091ddd609ebc90fde33bcaafb6b66b8c569e3e03 Mon Sep 17 00:00:00 2001 From: Kostandin Angjellari Date: Wed, 9 Sep 2026 11:19:18 +0200 Subject: [PATCH 01/22] Recut Voice interruption mission from deployed main Co-authored-by: Cursor --- libs/@hashintel/brunch-agent/MISSION.md | 231 +++++++++++------------- 1 file changed, 103 insertions(+), 128 deletions(-) diff --git a/libs/@hashintel/brunch-agent/MISSION.md b/libs/@hashintel/brunch-agent/MISSION.md index 086ab0d5705..5cfd0a22bf5 100644 --- a/libs/@hashintel/brunch-agent/MISSION.md +++ b/libs/@hashintel/brunch-agent/MISSION.md @@ -1,159 +1,134 @@ -# Brunch remote browser-origin policy +# Voice interruption by speaking ## Status -**Live as of 2026-09-08** for -[SRE-1042](https://linear.app/hash/issue/SRE-1042/configure-petrinauts-deployment-variables-for-the-brunch-agent-chat) -on `t/sre-1042-allow-wildcard-origins-for-brunch-previews`, cut from `main` after -[FE-1626](https://github.com/hashintel/hash/pull/9583) established the exact-origin allow-list for -`/agents/*`. This file is the branch's sole execution authority. +**Live as of 2026-09-09** for +[FE-1604](https://linear.app/hash/issue/FE-1604/allow-voice-interruption-by-speaking) +on `kostandin/fe-1604-recut-voice-interruption`, cut from post-deployment `main` at +`ef0f4449876d63d82657147fb4e29cdf024e9f79`. -Exact origins alone do not fit the deployment: every Petrinaut preview has its own -`https://petrinaut-git-.stage.hash.ai` origin, so the allow-list additionally accepts a -wildcard for exactly one leading host label. CORS governs whether a conforming browser exposes a cross-origin response -to client code; it does not authenticate or restrict non-browser callers, authorize a -conversation, or make public exposure safe by itself. +This is an independent semantic recut of the interruption-only delta from the +stale, conflicting [PR #9550](https://github.com/hashintel/hash/pull/9550) head +`f69ac17034dfe4290691d34b64930e5a07245480`. It must not merge the old branch or +carry its unrelated Brunch stack. The open settlement port in +[PR #9588](https://github.com/hashintel/hash/pull/9588) is a separate sibling and +is not part of this branch. ## Imperative -Let a deployed Petrinaut website use the Brunch `/agents/*` Flue routes from an explicitly trusted -browser origin while causing browsers to withhold cross-origin access from unlisted origins. Do -this now because the deployed website and Brunch service are separate origins and -[SRE-1042](https://linear.app/hash/issue/SRE-1042/configure-petrinauts-deployment-variables-for-the-brunch-agent-chat) -cannot point the browser at the deployed Brunch route until preflight and response headers work. +Let a person interrupt Voice assistant playback by speaking without losing the +interrupting utterance. Keep the existing **Your turn** handoff as a +browser-saved half-duplex fallback, and reject likely prompt regurgitation or +assistant self-echo before a completed interruption transcript becomes an +answer. + +The interruption must stop playback immediately while preserving completed +transcription as the sole answer authority and the existing Brunch admission +path as the sole submission authority. ## Throughline ```text -Petrinaut browser at one configured exact origin -→ OPTIONS /agents// with requested method and headers -→ route-scoped Hono CORS middleware before ownership middleware -→ 204 preflight carrying the matching origin, GET/POST/OPTIONS, and Flue request headers -→ browser FlueClient GET/POST with x-brunch-principal + x-brunch-conversation -→ existing agentOwnershipGuard and createAgentRouter -→ response exposes the Flue/Durable Streams headers the browser SDK reads +OpenAI Realtime microphone input remains enabled during canonical playback +→ input_audio_buffer.speech_started +→ response.cancel + output_audio_buffer.clear, without input_audio_buffer.clear +→ completed transcription for the same input item +→ interruption-only prompt-regurgitation and active-playback self-echo checks +→ retain while the previous Brunch turn settles, if necessary +→ existing Voice bridge and panel admission path exactly once +→ canonical Brunch turn and ordinary Voice lifecycle ``` -`BRUNCH_CORS_ALLOWED_ORIGINS` is read once at startup as a comma-separated list of HTTP(S) -origins, each either exact or with a wildcard as the whole leading host label in front of a domain -with at least two labels (`https://*.stage.hash.ai`). A wildcard matches exactly one label, like a -wildcard TLS certificate. Parsing trims whitespace, normalizes an optional trailing slash through -`URL.origin`, and deduplicates values. Credentials, non-root paths, queries, fragments, wildcards in -any other position, opaque origins, and non-HTTP(S) schemes are startup configuration errors. Missing or blank configuration means an -empty allowlist: same-origin and non-browser callers continue through the existing route, but -browser code at another origin receives no CORS grant. See the -[Brunch application README](../../../apps/brunch-agent/README.md#production-container) for -operator configuration details. - -The middleware applies only to `/agents/*` and runs before `agentOwnershipGuard`, so a valid -preflight does not need conversation headers. It permits `GET`, `POST`, and `OPTIONS`; permits -`Content-Type`, `x-brunch-principal`, and `x-brunch-conversation`; does not permit credentials; and -uses a 600-second preflight cache. It exposes the non-safelisted response headers read by the -installed Flue 2.0.3 and Durable Streams 0.2.6 clients: - -- `flue-error-ref` -- `Stream-Next-Offset` -- `Stream-Cursor` -- `Stream-Up-To-Date` -- `Stream-Closed` -- `stream-sse-data-encoding` - -Hono's maintained CORS middleware owns header emission, `Vary` handling, and the `OPTIONS` response. -Non-browser callers can still send requests and receive ordinary HTTP responses because CORS is -enforced by browsers, not by the service as caller authentication. A response to an unlisted -browser origin carries no `Access-Control-Allow-Origin`, so the browser withholds that response -from client code. +The playback menu owns a default-on **Interruption by speaking** preference. +Disabling it restores the existing half-duplex microphone closure and +acknowledged **Your turn** handoff. ## Proof -This mission establishes the application-side CORS contract required by the deployed browser -transport. It does **not** establish authentication, authorization, rate limiting, infrastructure -configuration, a deployed endpoint, or end-to-end remote verification. - -1. **Configuration is exact and fail-closed.** Missing and blank configuration produce no allowed - origins; whitespace, trailing slashes, duplicates, and multiple exact origins normalize - deterministically; malformed or broader-than-origin entries fail with the offending variable - named. Oracle: focused unit cases in `apps/brunch-agent/test/cors.test.ts`. -2. **Allowed browser traffic receives the complete grant.** An allowed origin receives its exact - value on an `/agents/*` response. Its preflight receives 204 before ownership, the three allowed - methods, the three allowed request headers, the six exposed response headers, no credentials - grant, and the required `Vary` values. Oracle: in-process Hono requests in - `apps/brunch-agent/test/cors.test.ts`. -3. **Rejected origins receive no grant.** An unlisted origin's preflight and ordinary response omit - `Access-Control-Allow-Origin`; an allowed origin does not make another origin pass. Oracle: - focused negative cases in `apps/brunch-agent/test/cors.test.ts`. -4. **The policy cannot widen unrelated routes.** `/health`, `/`, and `/assets/*` carry no Brunch - CORS grant. Existing ownership checks still return 401/403 for actual agent requests with - missing or mismatched identity. Oracle: CORS route-scope tests plus the existing - `apps/brunch-agent/test/agent-ownership.test.ts`. -5. **The shipped artifact and operator contract agree.** Brunch's README documents the variable, - exact-origin configuration, empty-list behavior, and the fact that CORS governs browser access - rather than authenticating or restricting non-browser callers. Oracle: - `yarn workspace @apps/brunch-agent test:unit`, - `yarn workspace @apps/brunch-agent lint:tsc`, - `yarn workspace @apps/brunch-agent lint:eslint`, and - `yarn workspace @apps/brunch-agent build`. - -## Constraints - -- Use Hono's built-in CORS middleware; do not create a parallel HTTP server or hand-maintain generic - CORS response logic. -- Keep one Flue product route and the existing ownership guard. CORS must not add, proxy, rename, or - reinterpret an agent route. -- The origin list is explicit: exact origins or one-label wildcards, matched by scheme, host and - port. Do not hard-code Petrinaut domains, reflect arbitrary `Origin` values, or silently skip - malformed entries. -- Keep credentials disabled. The current browser client uses explicit ownership headers, not - cookies, and those headers are not authentication. -- Answer preflight before ownership while preserving ownership enforcement on every non-preflight - agent request. -- Read configuration once at startup. Dynamic policy storage or hot reload is not earned by this - deployment. -- Preserve local same-origin proxying when the variable is unset. -- No implementation begins until this authority cut is committed separately. Material changes to - this contract require owner review and another focused authority commit. +1. **Immediate, input-preserving cancellation.** Session tests observe + `speech_started → response.cancel → output_audio_buffer.clear`, no input + buffer clear, and completion of the same input item. +2. **Exactly-once admission.** Bridge and controller tests cover duplicate + completions, delayed Brunch admission, an unsettled previous turn, follow-on + canonical speech, queued playback, and lifecycle cleanup. +3. **Local false-transcript rejection.** Tests cover configured transcription + prompt regurgitation and exact active canonical self-echo, while preserving + short novel answers and leaving ordinary non-interruption capture unchanged. +4. **Retained-answer visibility.** Controller tests prove that later empty, + failed, prompt-regurgitated, or self-echo transcripts cannot erase an + earlier retained answer or submit a replacement. +5. **User control.** Shared Petrinaut tests prove the preference is default-on, + browser-saved, exposed in the existing playback menu, and controls whether + **Your turn** is visible. +6. **Package integrity.** Focused Voice unit tests, Petrinaut unit tests, + TypeScript checks, ESLint, the website and library builds, architecture-doc + lint, repository formatting, and `git diff --check` distinguish a working + recut from code presence alone. + +Mocked protocol tests establish event ordering and state behavior; they do not +establish real microphone latency, speaker echo cancellation, or acoustic +classifier accuracy. ### Expected touched paths ```text -~ libs/@hashintel/brunch-agent/MISSION.md branch authority -~ apps/brunch-agent/src/http/cors.ts exact and one-label wildcard origins, Hono middleware -~ apps/brunch-agent/src/app.ts mount CORS before ownership on /agents/* -+ apps/brunch-agent/test/cors.test.ts parser, allowed, rejected, preflight, route-scope tests -~ apps/brunch-agent/README.md deployment variable and security boundary -~ apps/brunch-agent/turbo.json pass the variable into the local dev task +~ apps/petrinaut-website/src/main/app/voice-interview/ session, bridge, controller, preference, tests +~ apps/petrinaut-website/src/server/voice/ Realtime VAD and transcription policy +~ apps/petrinaut-website/src/shared/ shared transcription vocabulary +~ apps/petrinaut-website/README.md website behavior +~ libs/@hashintel/petrinaut/ shared state, playback control, user guide +~ libs/@hashintel/brunch-agent/docs/adr/ Voice turn-shell decision ++ .changeset/ Petrinaut patch release note ``` +## Constraints + +- A completed provider transcription is the only Voice-answer authority. + Provisional text remains display-only. +- Interruption cancellation is immediate and is never gated on transcript + classification. It clears output, never the interrupting input buffer. +- Only input that began while canonical playback was active is classified as an + interruption. Ordinary capture behavior remains unchanged. +- Self-echo compares only with the exact canonical text active when speech + started, not queued speech or conversation history. +- Comparison may normalize Unicode, case, punctuation, and whitespace, but + admitted wording, casing, and punctuation remain unchanged. +- Rejection diagnostics contain operational metadata and a reason, never the + transcript, transcription prompt, or assistant text. +- Short novel answers such as “stop”, “no”, and “wait” remain admissible. +- Mute, pause, Stop, end, reconnect, exact question replay, and exact full + response replay retain their current behavior in both preference modes. +- Keep FE-1604 independent from PR #9588. If that sibling lands, update from + `main` and resolve overlap semantically instead of importing its branch. +- Update the Petrinaut user guide and retain exactly one Petrinaut patch + changeset. + ## Fog-line -- Infrastructure repository access is unavailable in this worktree, so this branch can prove only - the application contract. Runtime deployment configuration must supply the chosen origins before - remote verification. -- A one-label wildcard admits every host directly under the configured domain, not only Petrinaut - previews. Narrow the deployed pattern or return to exact origins if that breadth becomes a - problem in practice. -- The allowed and exposed headers are pinned to the installed Flue and Durable Streams clients. - Re-evaluate them from client source when either dependency changes. +- Browser echo cancellation may still allow speaker feedback to trigger VAD or + transcription. A deterministic completed-transcript classifier reduces false + admission but cannot prevent playback from stopping after a false VAD event. +- Real interruption latency and acoustic behavior remain unmeasured until a + human browser/microphone witness is retained. +- The classifier is intentionally conservative. Evidence of rejected novel + speech or admitted repeated playback requires threshold or feature + re-evaluation before release. ## Stop or reorient -Stop if the real browser client emits a request method or non-safelisted request header outside the -pinned contract, reads another non-safelisted response header, or needs cookie credentials. Bring -that evidence back to the contract before broadening the grant. - -Stop if middleware ordering bypasses ownership for a non-`OPTIONS` request, if an invalid -configuration widens access or is ignored, if an unlisted origin receives -`Access-Control-Allow-Origin`, or if `/health`, `/`, or `/assets/*` inherit the policy. +Stop if the recut requires unrelated stale-branch files, a second Voice +submission path, delayed cancellation, input-buffer clearing, transcript +logging, or assistant-generated classification. -Do not represent a green CORS test as permission for unauthenticated public exposure. Authentication, -per-conversation authorization, rate/spend controls, and the infrastructure ingress boundary remain -separate release gates. +Stop if current-main APIs cannot preserve the same input item through +cancellation and completion, if an interruption can submit twice, if a rejected +completion can replace or erase a retained answer, or if disabling the +preference no longer restores the acknowledged half-duplex handoff. ## Deferred -- SRE-1013 owns injection of the allowlist into the Brunch runtime deployment. SRE-1042 owns - `VITE_BRUNCH_CHAT_ENDPOINT`, Voice deployment variables, and the deployed browser verification - after this application contract lands. -- FE-1615 and FE-1616 retain authentication and rate-limit work. CORS does not discharge either. -- A same-origin Petrinaut proxy stays deferred; the one-label wildcard covers the preview - deployments the exact list could not. +- PR #9588 owns the omitted Voice settlement port and remains a separate + mainline update. +- A human browser/microphone witness owns claims about speaker feedback, + acoustic false interruption, and audible interruption latency. +- Preventing a false VAD event from stopping playback is outside FE-1604. From ecf0027555aacff64f72775a0308467fae7afb18 Mon Sep 17 00:00:00 2001 From: Kostandin Angjellari Date: Sun, 6 Sep 2026 12:19:13 +0200 Subject: [PATCH 02/22] Allow Voice interruption by speaking Co-authored-by: Cursor --- .changeset/petrinaut-voice-interruption.md | 5 + .../openai-realtime-session.test.ts | 232 ++++++++++++++++++ .../openai-realtime-session.ts | 77 +++++- .../realtime-brunch-bridge.test.ts | 194 +++++++++++++++ .../voice-interview/realtime-brunch-bridge.ts | 73 +++++- .../voice-interview-control.test.tsx | 34 +++ .../voice-interview-control.tsx | 37 +++ .../voice-session-state.test.ts | 2 + .../voice-interview/voice-session-state.ts | 15 +- .../voice-turn-controller.test.ts | 30 +++ .../voice-interview/voice-turn-controller.ts | 88 +++++-- .../@hashintel/petrinaut/docs/ai-assistant.md | 21 +- .../src/react/voice-session/store.ts | 1 + .../src/react/voice-session/types.ts | 2 + .../react/voice-session/use-voice-session.ts | 9 + .../ui/types/ai-assistant-composer-control.ts | 2 + .../Editor/components/voice-session-labels.ts | 1 + .../Editor/panels/ai-assistant-panel.tsx | 6 + .../ai-assistant-contents.test.tsx | 63 +++++ .../ai-assistant-contents/voice-dock.tsx | 8 +- .../voice-dock/playback-menu.tsx | 14 ++ 21 files changed, 867 insertions(+), 47 deletions(-) create mode 100644 .changeset/petrinaut-voice-interruption.md diff --git a/.changeset/petrinaut-voice-interruption.md b/.changeset/petrinaut-voice-interruption.md new file mode 100644 index 00000000000..65e2f4b7561 --- /dev/null +++ b/.changeset/petrinaut-voice-interruption.md @@ -0,0 +1,5 @@ +--- +"@hashintel/petrinaut": patch +--- + +Allow Voice users to interrupt assistant playback by speaking, with a browser-saved preference and optional manual handoff. diff --git a/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.test.ts b/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.test.ts index 245058d31d2..a6cd2c75557 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.test.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.test.ts @@ -4,6 +4,8 @@ import { OpenAIRealtimeSession, type OpenAIRealtimeSessionEvent, } from "./openai-realtime-session"; +import { RealtimeBrunchBridge } from "./realtime-brunch-bridge"; +import { VoiceTurnController } from "./voice-turn-controller"; import type { CanonicalSpeechSegment } from "./canonical-speech"; @@ -175,6 +177,236 @@ describe("OpenAIRealtimeSession", () => { vi.useRealTimers(); }); + test.each(["playing", "generated", "creating"] as const)( + "preserves an interrupting answer through the real Voice stack while %s", + async (phase) => { + const harness = createHarness(); + const submitInterviewAnswer = vi.fn< + ConstructorParameters< + typeof RealtimeBrunchBridge + >[0]["submitInterviewAnswer"] + >(async (input) => ({ kind: "message", messageId: input.id })); + const bridge = new RealtimeBrunchBridge({ + session: harness.session, + submitInterviewAnswer, + }); + const controller = new VoiceTurnController({ + bridge, + session: harness.session, + submitText: vi.fn(async () => undefined), + }); + controller.setInterruptionBySpeaking(true); + await controller.start(); + const channel = harness.channels.at(-1)!; + const question = canonicalSegment("question", "Who approves this?"); + controller.updateChat({ + canAcceptInterviewAnswer: true, + canonicalSegments: [question], + questionSegment: question, + status: "ready", + }); + if (phase !== "creating") { + authorizeLatestSpeechResponse(channel, "question-response"); + channel.receive({ + type: "output_audio_buffer.started", + response_id: "question-response", + }); + } + if (phase === "generated") { + channel.receive({ + type: "response.done", + response: { + id: "question-response", + status: "completed", + output: [], + }, + }); + } + expect(harness.localTracks.at(-1)?.enabled).toBe(true); + const pendingResponse = sentEvents(channel).findLast( + ({ type }) => type === "response.create", + )?.response as Record; + harness.session.speakCanonical([ + canonicalSegment("queued", "This must not play."), + ]); + channel.send.mockClear(); + channel.receive({ + type: "input_audio_buffer.speech_started", + item_id: "interrupting-answer", + audio_start_ms: 100, + }); + if (phase !== "creating") { + expect(sentEvents(channel).map(({ type }) => type)).toEqual([ + "response.cancel", + "output_audio_buffer.clear", + ]); + } + expect(harness.localTracks.at(-1)?.enabled).toBe(true); + channel.receive({ + type: "conversation.item.input_audio_transcription.delta", + item_id: "interrupting-answer", + content_index: 0, + delta: "The supervisor", + }); + expect(controller.getSnapshot().partialText).toBe("The supervisor"); + if (phase === "creating") { + channel.receive({ + type: "response.created", + response: { + id: "question-response", + metadata: pendingResponse.metadata, + }, + }); + expect(sentEvents(channel).map(({ type }) => type)).toEqual([ + "response.cancel", + "output_audio_buffer.clear", + ]); + } + const terminal = { + type: "response.done", + response: { id: "question-response", status: "cancelled", output: [] }, + }; + const cleared = { + type: "output_audio_buffer.cleared", + response_id: "question-response", + }; + channel.receive(phase === "playing" ? terminal : cleared); + channel.receive(phase === "playing" ? cleared : terminal); + channel.receive({ + type: "output_audio_buffer.started", + response_id: "question-response", + }); + expect(controller.getSnapshot().connection).toBe("connected"); + expect(harness.localTracks.at(-1)?.enabled).toBe(true); + expect( + sentEvents(channel).some(({ type }) => type === "response.create"), + ).toBe(false); + channel.receive({ + type: "conversation.item.input_audio_transcription.completed", + item_id: "interrupting-answer", + content_index: 0, + transcript: "The supervisor approves it.", + }); + await vi.waitFor(() => + expect(submitInterviewAnswer).toHaveBeenCalledOnce(), + ); + expect(submitInterviewAnswer).toHaveBeenCalledWith( + expect.objectContaining({ + id: "voice-realtime:1:interrupting-answer:0", + text: "The supervisor approves it.", + }), + ); + expect( + sentEvents(channel).some( + ({ type }) => type === "input_audio_buffer.clear", + ), + ).toBe(false); + expect(controller.getSnapshot().lastCommittedText).toBe( + "The supervisor approves it.", + ); + await controller.end(); + }, + ); + + test("reopens capture for a streamed reply and submits the retained interruption after settlement", async () => { + const harness = createHarness(); + const submitInterviewAnswer = vi.fn< + ConstructorParameters< + typeof RealtimeBrunchBridge + >[0]["submitInterviewAnswer"] + >(async (input) => { + input.onAdmission("submission-1"); + return { + kind: "message", + messageId: input.id, + submissionId: "submission-1", + }; + }); + const bridge = new RealtimeBrunchBridge({ + session: harness.session, + submitInterviewAnswer, + }); + const controller = new VoiceTurnController({ + bridge, + session: harness.session, + submitText: vi.fn(async () => undefined), + }); + controller.setInterruptionBySpeaking(true); + await controller.start(); + controller.updateChat({ + canAcceptInterviewAnswer: true, + canonicalSegments: [], + status: "ready", + }); + const channel = harness.channels.at(-1)!; + channel.receive({ + type: "input_audio_buffer.speech_started", + item_id: "first-answer", + audio_start_ms: 0, + }); + channel.receive({ + type: "conversation.item.input_audio_transcription.completed", + item_id: "first-answer", + content_index: 0, + transcript: "We need an approval.", + }); + await vi.waitFor(() => + expect(controller.getSnapshot().lastAnswerDelivery).toBe("delivered"), + ); + const reply = { + ...canonicalSegment("reply", "Who approves this?"), + submissionIds: ["submission-1"], + }; + bridge.notifyResponseMessageCompleted({ + messageId: reply.messageId, + submissionId: "submission-1", + position: { batch: 1, index: 0 }, + }); + controller.updateChat({ + canAcceptInterviewAnswer: false, + canonicalSegments: [reply], + status: "streaming", + }); + authorizeLatestSpeechResponse(channel, "streamed-reply"); + channel.receive({ + type: "output_audio_buffer.started", + response_id: "streamed-reply", + }); + expect(harness.localTracks.at(-1)?.enabled).toBe(true); + controller.setMicrophoneMuted(true); + expect(harness.localTracks.at(-1)?.enabled).toBe(false); + controller.setMicrophoneMuted(false); + expect(harness.localTracks.at(-1)?.enabled).toBe(true); + channel.receive({ + type: "input_audio_buffer.speech_started", + item_id: "second-answer", + audio_start_ms: 100, + }); + channel.receive({ + type: "conversation.item.input_audio_transcription.completed", + item_id: "second-answer", + content_index: 0, + transcript: "The supervisor approves it.", + }); + expect(submitInterviewAnswer).toHaveBeenCalledOnce(); + expect(controller.getSnapshot().inputNotice).toBe("answer-pending"); + controller.updateChat({ + canAcceptInterviewAnswer: true, + canonicalSegments: [reply], + status: "ready", + }); + await vi.waitFor(() => + expect(submitInterviewAnswer).toHaveBeenCalledTimes(2), + ); + expect(submitInterviewAnswer).toHaveBeenLastCalledWith( + expect.objectContaining({ + id: "voice-realtime:1:second-answer:0", + text: "The supervisor approves it.", + }), + ); + await controller.end(); + }); + test("negotiates duplex WebRTC, attaches remote audio, and cleans all media", async () => { const harness = createHarness(); diff --git a/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.ts b/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.ts index 0051a56fdb9..d665b827150 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.ts @@ -34,6 +34,8 @@ export type OpenAIRealtimeSessionEvent = readonly connectionEpoch: number; readonly itemId: string; readonly type: "input-speech-started"; + /** Capture was accepted with interruption by speaking enabled. */ + readonly interruptionBySpeaking?: true; } | { readonly connectionEpoch: number; @@ -221,6 +223,7 @@ export class OpenAIRealtimeSession { #meterHasSample = false; #meterLevel = 0; #meterSamples: Uint8Array | null = null; + #interruptionBySpeaking = false; #microphoneRequested = false; #microphoneTrack: MediaStreamTrack | null = null; #peerConnection: RTCPeerConnection | null = null; @@ -396,6 +399,39 @@ export class OpenAIRealtimeSession { this.#syncMicrophoneTrack(); } + public setInterruptionBySpeaking(enabled: boolean): void { + this.#interruptionBySpeaking = enabled; + this.#syncMicrophoneTrack(); + } + + /** Cancel only assistant output; the utterance which caused this stays alive. */ + #interruptOutputBySpeaking(): void { + for (const request of this.#canonicalSpeechQueue.splice(0)) { + this.#cancelPendingSpeechRequest(request.speechRequestId); + } + if (this.#responseCreateEventId !== null) { + const pending = this.#pendingClientEvents.get( + this.#responseCreateEventId, + ); + if (pending?.kind === "response-create") { + this.#cancelledSpeechRequestIds.add(pending.request.speechRequestId); + } + } + let cancelledOutput = false; + for (const responseId of this.#canonicalResponseIds) { + if ( + (this.#activeResponseIds.has(responseId) || + this.#speakingResponseId === responseId) && + !this.#cancelledCanonicalResponseIds.has(responseId) + ) { + this.#cancelledCanonicalResponseIds.add(responseId); + this.#cancelResponse(responseId); + cancelledOutput = true; + } + } + if (cancelledOutput) this.#send({ type: "output_audio_buffer.clear" }); + } + public speakCanonical(segments: CanonicalSpeechSegment[]): void { this.#requestCanonicalSpeech(segments, true); } @@ -586,10 +622,12 @@ export class OpenAIRealtimeSession { request, responseTerminalSequence: this.#responseTerminalSequence, }); - for (const itemId of this.#acceptedInputItemIds) { - this.#playbackOverlappingInputItemIds.add(itemId); + if (!this.#interruptionBySpeaking) { + for (const itemId of this.#acceptedInputItemIds) { + this.#playbackOverlappingInputItemIds.add(itemId); + } + this.#acceptedInputItemIds.clear(); } - this.#acceptedInputItemIds.clear(); this.#syncMicrophoneTrack(); try { this.#send({ @@ -650,12 +688,26 @@ export class OpenAIRealtimeSession { if (parsed.type === "input_audio_buffer.speech_started") { const itemId = nonEmptyString(parsed.item_id); if (!itemId || nonNegativeInteger(parsed.audio_start_ms) === null) return; - if (this.#speakingResponseId || !this.#microphoneTrack?.enabled) { + if ( + (!this.#interruptionBySpeaking && this.#speakingResponseId) || + !this.#microphoneTrack?.enabled + ) { this.#playbackOverlappingInputItemIds.add(itemId); return; } this.#acceptedInputItemIds.add(itemId); + if (this.#interruptionBySpeaking) { + try { + this.#interruptOutputBySpeaking(); + } catch { + this.#handleConnectionFailure("network", "speech"); + return; + } + } this.#emit({ + ...(this.#interruptionBySpeaking + ? { interruptionBySpeaking: true as const } + : {}), connectionEpoch, itemId, type: "input-speech-started", @@ -925,10 +977,12 @@ export class OpenAIRealtimeSession { this.#handleConnectionFailure("invalid-response", "connection"); return; } - for (const itemId of this.#acceptedInputItemIds) { - this.#playbackOverlappingInputItemIds.add(itemId); + if (!this.#interruptionBySpeaking) { + for (const itemId of this.#acceptedInputItemIds) { + this.#playbackOverlappingInputItemIds.add(itemId); + } + this.#acceptedInputItemIds.clear(); } - this.#acceptedInputItemIds.clear(); this.#speakingResponseId = responseId; this.#syncMicrophoneTrack(); const speechRequestId = this.#speechRequestIds.get(responseId); @@ -1268,10 +1322,11 @@ export class OpenAIRealtimeSession { this.#microphoneRequested && this.#connected && this.#cancelOutputPromise === null && - this.#authorizedResponseIds.size === 0 && - this.#canonicalSpeechQueue.length === 0 && - this.#responseCreateEventId === null && - this.#speakingResponseId === null; + (this.#interruptionBySpeaking || + (this.#authorizedResponseIds.size === 0 && + this.#canonicalSpeechQueue.length === 0 && + this.#responseCreateEventId === null && + this.#speakingResponseId === null)); this.#microphoneTrack.enabled = enabled; if (enabled) { this.#startMeter(); diff --git a/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.test.ts b/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.test.ts index 3f2596e316d..a7eecb29ba6 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.test.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.test.ts @@ -115,6 +115,200 @@ const startReady = ( }; describe("RealtimeBrunchBridge", () => { + test("retains an interrupting transcript until the previous Brunch submission settles", async () => { + const harness = createHarness(); + startReady(harness); + harness.emit(completedTranscript(3)); + await vi.waitFor(() => + expect( + harness.events.some(({ type }) => type === "submission-accepted"), + ).toBe(true), + ); + harness.bridge.updateChat({ + canAcceptInterviewAnswer: false, + canonicalSegments: [], + status: "streaming", + }); + harness.emit({ + type: "canonical-speech-requested", + connectionEpoch: 3, + speechRequestId: "speech-1", + }); + harness.emit({ + type: "input-speech-started", + connectionEpoch: 3, + itemId: "interruption", + interruptionBySpeaking: true, + }); + harness.emit( + completedTranscript( + 3, + "Actually, the manager approves it.", + "interruption", + ), + ); + expect(harness.submitInterviewAnswer).toHaveBeenCalledOnce(); + const update = { + canAcceptInterviewAnswer: true, + canonicalSegments: [ + segment("reply", "Who is informed?", "submission-voice-1"), + ], + status: "ready" as const, + }; + harness.bridge.updateChat(update); + await vi.waitFor(() => + expect(harness.submitInterviewAnswer).toHaveBeenCalledTimes(2), + ); + expect(harness.submitInterviewAnswer).toHaveBeenLastCalledWith( + expect.objectContaining({ + id: "voice-realtime:3:interruption:0", + text: "Actually, the manager approves it.", + }), + ); + harness.emit( + completedTranscript( + 3, + "Actually, the manager approves it.", + "interruption", + ), + ); + harness.bridge.updateChat(update); + expect(harness.submitInterviewAnswer).toHaveBeenCalledTimes(2); + expect(harness.session.speakCanonical).not.toHaveBeenCalled(); + }); + + test.each(["stop", "cancelPendingSpeech", "reconnect"] as const)( + "clears a retained interruption on %s", + (action) => { + const harness = createHarness(); + startReady(harness); + harness.bridge.updateChat({ + canAcceptInterviewAnswer: false, + canonicalSegments: [], + status: "streaming", + }); + harness.emit({ + type: "input-speech-started", + connectionEpoch: 3, + itemId: "pending", + interruptionBySpeaking: true, + }); + harness.emit(completedTranscript(3, "Pending answer", "pending")); + expect(harness.events).toContainEqual({ + type: "transcript-retained", + answer: "Pending answer", + }); + if (action === "reconnect") harness.bridge.start(4); + else harness.bridge[action](); + harness.bridge.updateChat({ + canAcceptInterviewAnswer: true, + canonicalSegments: [], + status: "ready", + }); + expect(harness.submitInterviewAnswer).not.toHaveBeenCalled(); + }, + ); + + test("drains a retained interruption once the panel reopens voice input", async () => { + const harness = createHarness(); + startReady(harness); + harness.bridge.updateChat({ + canAcceptInterviewAnswer: false, + canonicalSegments: [], + status: "streaming", + }); + harness.emit({ + type: "input-speech-started", + connectionEpoch: 3, + itemId: "interruption", + interruptionBySpeaking: true, + }); + harness.emit( + completedTranscript(3, "The auditor approves it.", "interruption"), + ); + + // The panel reports ready before it releases the queued voice input, so + // the first ready update cannot deliver the retained answer. + const heldSegments = [ + segment("held", "Who signs it off?", "submission-held"), + ]; + harness.bridge.updateChat({ + canAcceptInterviewAnswer: false, + canonicalSegments: heldSegments, + status: "ready", + }); + expect(harness.submitInterviewAnswer).not.toHaveBeenCalled(); + + harness.bridge.updateChat({ + canAcceptInterviewAnswer: true, + canonicalSegments: heldSegments, + status: "ready", + }); + await vi.waitFor(() => + expect(harness.submitInterviewAnswer).toHaveBeenCalledOnce(), + ); + expect(harness.submitInterviewAnswer).toHaveBeenCalledWith( + expect.objectContaining({ text: "The auditor approves it." }), + ); + expect(harness.session.speakCanonical).not.toHaveBeenCalled(); + }); + + test("refuses ordinary capture while a submission is active", async () => { + const harness = createHarness(); + startReady(harness); + harness.emit(completedTranscript(3)); + await vi.waitFor(() => + expect( + harness.events.some(({ type }) => type === "submission-accepted"), + ).toBe(true), + ); + harness.bridge.updateChat({ + canAcceptInterviewAnswer: false, + canonicalSegments: [], + status: "streaming", + }); + + harness.emit(completedTranscript(3, " ", "second-item")); + + expect(harness.events).toContainEqual({ + type: "transcript-rejected", + reason: "unavailable", + }); + expect(harness.submitInterviewAnswer).toHaveBeenCalledOnce(); + }); + + test("retains only the first pending interruption and reports the extra utterance", () => { + const harness = createHarness(); + startReady(harness); + harness.bridge.updateChat({ + canAcceptInterviewAnswer: false, + canonicalSegments: [], + status: "streaming", + }); + for (const itemId of ["first", "second"]) { + harness.emit({ + type: "input-speech-started", + connectionEpoch: 3, + itemId, + interruptionBySpeaking: true, + }); + harness.emit(completedTranscript(3, itemId, itemId)); + } + expect(harness.events).toContainEqual({ + type: "transcript-rejected", + reason: "pending", + }); + harness.bridge.updateChat({ + canAcceptInterviewAnswer: true, + canonicalSegments: [], + status: "ready", + }); + expect(harness.submitInterviewAnswer).toHaveBeenCalledOnce(); + expect(harness.submitInterviewAnswer).toHaveBeenCalledWith( + expect.objectContaining({ text: "first" }), + ); + }); + test("rehydrates settled canonical speech without submission or playback", () => { const harness = createHarness(); harness.bridge.updateChat({ diff --git a/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.ts b/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.ts index 3172b93b15d..439af407358 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/realtime-brunch-bridge.ts @@ -105,9 +105,11 @@ export type RealtimeTranscriptRejectionReason = | "empty" | "failed" | "over-limit" + | "pending" | "unavailable"; export type RealtimeBrunchBridgeEvent = + | { readonly answer: string; readonly type: "transcript-retained" } | { readonly answer: string; readonly deliveryId: string; @@ -214,6 +216,8 @@ export class RealtimeBrunchBridge { input: SubmitInterviewAnswerInput, ) => Promise; readonly #seenSegmentIds = new Set(); + readonly #interruptionInputItemIds = new Set(); + #pendingInterruption: { answer: string; deliveryId: string } | null = null; #activeEpoch: number | null = null; #activeSubmission: ActiveSubmission | null = null; #chat: ChatUpdate = { @@ -240,6 +244,8 @@ export class RealtimeBrunchBridge { public cancelPendingSpeech(): void { this.#outputCancellationPending = true; + this.#pendingInterruption = null; + this.#interruptionInputItemIds.clear(); if (this.#activeSubmission) { this.#activeSubmission.speechCancelled = true; } @@ -296,6 +302,8 @@ export class RealtimeBrunchBridge { this.#activeEpoch = connectionEpoch; this.#activeSubmission = null; this.#acceptedInputItemIds.clear(); + this.#interruptionInputItemIds.clear(); + this.#pendingInterruption = null; this.#playbackOverlappingInputItemIds.clear(); this.#processedTranscripts.clear(); this.#activeOutputResponseIds.clear(); @@ -313,6 +321,8 @@ export class RealtimeBrunchBridge { this.#activeEpoch = null; this.#activeSubmission = null; this.#acceptedInputItemIds.clear(); + this.#interruptionInputItemIds.clear(); + this.#pendingInterruption = null; this.#playbackOverlappingInputItemIds.clear(); this.#processedTranscripts.clear(); this.#activeOutputResponseIds.clear(); @@ -348,6 +358,7 @@ export class RealtimeBrunchBridge { if (update.status !== "ready") { return; } + if (this.#drainPendingInterruption()) return; const newSegments = update.canonicalSegments.filter( ({ id }) => !this.#seenSegmentIds.has(id), @@ -382,6 +393,8 @@ export class RealtimeBrunchBridge { ++this.#generation; this.#activeSubmission?.abortController.abort(); this.#activeSubmission = null; + this.#pendingInterruption = null; + this.#interruptionInputItemIds.clear(); this.#emit({ code, message, type: "error" }); } @@ -389,6 +402,8 @@ export class RealtimeBrunchBridge { ++this.#generation; this.#activeSubmission?.abortController.abort(); this.#activeSubmission = null; + this.#pendingInterruption = null; + this.#interruptionInputItemIds.clear(); this.#emit({ code: admissionErrorCode(error.failure), failure: error.failure, @@ -405,7 +420,13 @@ export class RealtimeBrunchBridge { return; } if (event.type === "input-speech-started") { - if (this.#ownsOutputTurn()) { + if (event.interruptionBySpeaking) { + this.#interruptionInputItemIds.add(event.itemId); + if (this.#activeSubmission) { + this.#activeSubmission.speechCancelled = true; + } + } + if (!event.interruptionBySpeaking && this.#ownsOutputTurn()) { this.#playbackOverlappingInputItemIds.add(event.itemId); } else { this.#acceptedInputItemIds.add(event.itemId); @@ -424,9 +445,11 @@ export class RealtimeBrunchBridge { this.#pendingSpeechRequestIds.delete(event.speechRequestId); this.#activeOutputResponseIds.add(event.responseId); for (const itemId of this.#acceptedInputItemIds) { - this.#playbackOverlappingInputItemIds.add(itemId); + if (!this.#interruptionInputItemIds.has(itemId)) { + this.#playbackOverlappingInputItemIds.add(itemId); + this.#acceptedInputItemIds.delete(itemId); + } } - this.#acceptedInputItemIds.clear(); return; } if ( @@ -466,11 +489,13 @@ export class RealtimeBrunchBridge { this.#rejectTranscript("failed"); return; } - if ( - this.#activeSubmission || - !this.#chat.canAcceptInterviewAnswer || - this.#chat.status !== "ready" - ) { + + // An interrupting utterance is retained rather than refused when Brunch is + // still busy, so only ordinary capture answers a closed submission window. + const interruptionBySpeaking = this.#interruptionInputItemIds.delete( + event.key.itemId, + ); + if (!interruptionBySpeaking && !this.#canSubmitAnswerNow()) { this.#rejectTranscript("unavailable"); return; } @@ -486,6 +511,36 @@ export class RealtimeBrunchBridge { } const deliveryId = createRealtimeSubmissionId(event.key); + if (this.#pendingInterruption) { + this.#rejectTranscript("pending"); + return; + } + if (!this.#canSubmitAnswerNow()) { + this.#pendingInterruption = { answer, deliveryId }; + this.#emit({ answer, type: "transcript-retained" }); + return; + } + this.#submitAnswer(answer, deliveryId); + } + + #canSubmitAnswerNow(): boolean { + return ( + !this.#activeSubmission && + this.#chat.canAcceptInterviewAnswer && + this.#chat.status === "ready" + ); + } + + #drainPendingInterruption(): boolean { + const pending = this.#pendingInterruption; + if (!pending) return false; + if (this.#activeEpoch === null || !this.#canSubmitAnswerNow()) return true; + this.#pendingInterruption = null; + this.#submitAnswer(pending.answer, pending.deliveryId); + return true; + } + + #submitAnswer(answer: string, deliveryId: string): void { const generation = this.#generation; this.#activeSubmission = { abortController: new AbortController(), @@ -728,6 +783,7 @@ export class RealtimeBrunchBridge { ...(active.speechCancelled ? { speechCancelled: true as const } : {}), type: "canonical-response-ready", }); + this.#drainPendingInterruption(); } /** @@ -754,5 +810,6 @@ export class RealtimeBrunchBridge { outcome: settlement.outcome, type: "submission-stopped", }); + this.#drainPendingInterruption(); } } diff --git a/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.test.tsx b/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.test.tsx index 632a4332269..17f7317a30a 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.test.tsx +++ b/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.test.tsx @@ -19,6 +19,8 @@ import { acknowledgeVoiceInterviewDisclosure, isVoiceInterviewDisclosureAcknowledged, loadOpenAIVoiceConfig, + readInterruptionBySpeakingPreference, + saveInterruptionBySpeakingPreference, submitVoiceInputWithAdmission, VOICE_INTERVIEW_DISCLOSURE_STORAGE_KEY, VoiceInterviewControl, @@ -670,3 +672,35 @@ describe("voice interview control", () => { ).toBe("acknowledged"); }); }); + +describe("interruption by speaking preference", () => { + test("defaults on and remembers both settings across reads", () => { + const values = new Map(); + const storage = { + getItem: (key: string) => values.get(key) ?? null, + setItem: (key: string, value: string) => { + values.set(key, value); + }, + }; + expect(readInterruptionBySpeakingPreference(storage)).toBe(true); + saveInterruptionBySpeakingPreference(false, storage); + expect(readInterruptionBySpeakingPreference(storage)).toBe(false); + saveInterruptionBySpeakingPreference(true, storage); + expect(readInterruptionBySpeakingPreference(storage)).toBe(true); + }); + test("works when browser storage is unavailable", () => { + const storage = { + getItem: () => { + throw new Error("denied"); + }, + setItem: () => { + throw new Error("denied"); + }, + }; + expect(readInterruptionBySpeakingPreference(storage)).toBe(true); + expect(() => + saveInterruptionBySpeakingPreference(false, storage), + ).not.toThrow(); + expect(readInterruptionBySpeakingPreference(null)).toBe(true); + }); +}); diff --git a/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.tsx b/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.tsx index 7094a09f9aa..ac362a52fdc 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.tsx +++ b/apps/petrinaut-website/src/main/app/voice-interview/voice-interview-control.tsx @@ -149,6 +149,36 @@ const getVoiceInterviewDisclosureStorage = (): Storage | null => { } }; +const interruptionBySpeakingStorageKey = + "petrinaut:interruption-by-speaking:v1"; + +export const readInterruptionBySpeakingPreference = ( + storage: Pick< + Storage, + "getItem" + > | null = getVoiceInterviewDisclosureStorage(), +): boolean => { + try { + return storage?.getItem(interruptionBySpeakingStorageKey) !== "false"; + } catch { + return true; + } +}; + +export const saveInterruptionBySpeakingPreference = ( + enabled: boolean, + storage: Pick< + Storage, + "setItem" + > | null = getVoiceInterviewDisclosureStorage(), +): void => { + try { + storage?.setItem(interruptionBySpeakingStorageKey, String(enabled)); + } catch { + // The preference still applies to this session when storage is unavailable. + } +}; + export const isVoiceInterviewDisclosureAcknowledged = ( storage: Pick< Storage, @@ -485,6 +515,9 @@ const AvailableVoiceInterviewControl = ({ session, submitText: (input) => latestSubmitVoiceInput(input), }); + controller.setInterruptionBySpeaking( + readInterruptionBySpeakingPreference(), + ); return { bridge, controller, @@ -599,6 +632,10 @@ const AvailableVoiceInterviewControl = ({ resume: () => { void store.controller.resume(); }, + setInterruptionBySpeaking: (enabled) => { + store.controller.setInterruptionBySpeaking(enabled); + saveInterruptionBySpeakingPreference(enabled); + }, setMicrophoneMuted: (muted) => store.controller.setMicrophoneMuted(muted), takeTurn: () => store.controller.takeTurn(), diff --git a/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.test.ts b/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.test.ts index 912282facab..4d5033a88e7 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.test.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.test.ts @@ -16,6 +16,7 @@ const listeningSnapshot = { errorRequestId: "", input: "listening", inputNotice: "none", + interruptionBySpeaking: false, lastAnswerDelivery: "none", lastCommittedText: "", microphoneEnabled: true, @@ -38,6 +39,7 @@ describe("toVoiceSessionState", () => { canRepeatQuestion: false, canTakeTurn: false, errorMessage: null, + interruptionBySpeaking: false, microphoneLevel: 0.24, microphoneMuted: false, notice: null, diff --git a/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.ts b/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.ts index 5d8bfce2219..5a713fc2440 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/voice-session-state.ts @@ -97,6 +97,7 @@ export const toVoiceSessionState = ({ canReadFullResponse: snapshot.canReadFullResponse, canRepeatQuestion: snapshot.canRepeatQuestion, canTakeTurn: snapshot.canTakeTurn, + interruptionBySpeaking: snapshot.interruptionBySpeaking, errorMessage: snapshot.connection === "error" ? errorMessageOf(snapshot) : null, microphoneMuted: @@ -105,11 +106,15 @@ export const toVoiceSessionState = ({ !snapshot.microphoneEnabled, microphoneLevel: snapshot.microphoneLevel, notice: - snapshot.inputNotice === "not-heard" - ? "We didn't catch that. Please try again." - : snapshot.inputNotice === "too-long" - ? "That answer is too long. Please try a shorter response." - : null, + snapshot.inputNotice === "answer-pending" + ? "Answer captured. Waiting for Brunch." + : snapshot.inputNotice === "answer-already-pending" + ? "Previous answer waiting. Please try again after it is sent." + : snapshot.inputNotice === "not-heard" + ? "We didn't catch that. Please try again." + : snapshot.inputNotice === "too-long" + ? "That answer is too long. Please try a shorter response." + : null, phase: phaseOf(snapshot), }; }; diff --git a/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.test.ts b/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.test.ts index 7152451dcd0..d7bc160b633 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.test.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.test.ts @@ -19,6 +19,7 @@ const createHarness = () => { cancelOutput: vi.fn<() => Promise>(async () => undefined), connect: vi.fn(async () => ++epoch), disconnect: vi.fn(async () => undefined), + setInterruptionBySpeaking: vi.fn(), setMicrophoneEnabled: vi.fn(), speakCanonical: vi.fn(), subscribe: vi.fn( @@ -89,6 +90,35 @@ const markedQuestion = ( }); describe("VoiceTurnController", () => { + test("keeps interruption preference through end and reconnect and disables manual handover", async () => { + const harness = createHarness(); + harness.controller.setInterruptionBySpeaking(true); + await harness.controller.start(); + harness.controller.updateChat({ + canAcceptInterviewAnswer: true, + canonicalSegments: [], + questionSegment: markedQuestion("question"), + status: "ready", + }); + harness.emitSession({ + type: "output-started", + connectionEpoch: 1, + responseId: "response", + speechRequestId: "speech", + }); + await harness.controller.takeTurn(); + expect(harness.session.cancelOutput).not.toHaveBeenCalled(); + await harness.controller.reconnect(); + expect(harness.controller.getSnapshot().interruptionBySpeaking).toBe(true); + await harness.controller.end(); + await harness.controller.start(); + expect(harness.controller.getSnapshot().interruptionBySpeaking).toBe(true); + harness.controller.setInterruptionBySpeaking(false); + expect(harness.session.setInterruptionBySpeaking).toHaveBeenLastCalledWith( + false, + ); + }); + test("records the content-free Voice lifecycle once in causal order", async () => { const harness = createHarness(); await harness.controller.start(); diff --git a/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.ts b/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.ts index 4c8c5950c3a..51145a3f13c 100644 --- a/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.ts +++ b/apps/petrinaut-website/src/main/app/voice-interview/voice-turn-controller.ts @@ -25,7 +25,12 @@ export type VoiceOutputState = | "speaking" | "interrupted"; export type VoiceAnswerDelivery = "none" | "pending" | "delivered" | "failed"; -export type VoiceInputNotice = "none" | "not-heard" | "too-long"; +export type VoiceInputNotice = + | "none" + | "not-heard" + | "too-long" + | "answer-pending" + | "answer-already-pending"; export interface VoiceTurnSnapshot { readonly canReadFullResponse: boolean; @@ -39,6 +44,7 @@ export interface VoiceTurnSnapshot { readonly errorRequestId: string; readonly input: VoiceInputState; readonly inputNotice: VoiceInputNotice; + readonly interruptionBySpeaking: boolean; readonly lastAnswerDelivery: VoiceAnswerDelivery; readonly lastCommittedText: string; readonly microphoneEnabled: boolean; @@ -66,6 +72,7 @@ interface RealtimeSession { cancelOutput(): Promise; connect(): Promise; disconnect(): Promise; + setInterruptionBySpeaking(enabled: boolean): void; setMicrophoneEnabled(enabled: boolean): void; speakCanonical(segments: CanonicalSpeechSegment[]): void; subscribe(listener: (event: OpenAIRealtimeSessionEvent) => void): () => void; @@ -125,6 +132,7 @@ const initialSnapshot: VoiceTurnSnapshot = { errorRequestId: "", input: "paused", inputNotice: "none", + interruptionBySpeaking: false, lastAnswerDelivery: "none", lastCommittedText: "", microphoneEnabled: false, @@ -184,6 +192,19 @@ export class VoiceTurnController { bridge.subscribe((event) => this.#handleBridgeEvent(event)); } + public setInterruptionBySpeaking(enabled: boolean): void { + this.#session.setInterruptionBySpeaking(enabled); + this.#update({ interruptionBySpeaking: enabled }); + if ( + this.#snapshot.connection === "connected" && + this.#snapshot.input !== "paused" && + !this.#takingTurnPromise && + !this.#outputCancellationPromise + ) { + this.#session.setMicrophoneEnabled(this.#snapshot.microphoneEnabled); + } + } + public getSnapshot(): VoiceTurnSnapshot { return this.#snapshot; } @@ -296,7 +317,10 @@ export class VoiceTurnController { this.#session.setMicrophoneEnabled(false); const teardownPromise = this.#teardownPromise ?? this.#session.disconnect(); this.#teardownPromise = teardownPromise; - this.#update({ ...initialSnapshot }); + this.#update({ + ...initialSnapshot, + interruptionBySpeaking: this.#snapshot.interruptionBySpeaking, + }); try { await teardownPromise; } finally { @@ -364,9 +388,10 @@ export class VoiceTurnController { if ( this.#takingTurnPromise === null && this.#outputCancellationPromise === null && - this.#activeSpeechResponseId === null && - (this.#snapshot.output === "idle" || - this.#snapshot.output === "interrupted") + (this.#snapshot.interruptionBySpeaking || + (this.#activeSpeechResponseId === null && + (this.#snapshot.output === "idle" || + this.#snapshot.output === "interrupted"))) ) { this.#session.setMicrophoneEnabled(!muted); } @@ -486,7 +511,8 @@ export class VoiceTurnController { */ public takeTurn(): Promise { if (this.#takingTurnPromise) return this.#takingTurnPromise; - if (!this.#snapshot.canTakeTurn) return Promise.resolve(); + if (this.#snapshot.interruptionBySpeaking || !this.#snapshot.canTakeTurn) + return Promise.resolve(); const generation = this.#generation; this.#bridge.cancelPendingSpeech(); @@ -591,6 +617,13 @@ export class VoiceTurnController { }); return; } + if (event.type === "transcript-retained") { + this.#update({ + inputNotice: "answer-pending", + partialText: event.answer, + }); + return; + } if (event.type === "transcript-rejected") { if (event.reason === "duplicate" || event.reason === "unavailable") { return; @@ -598,7 +631,12 @@ export class VoiceTurnController { this.#transcriptItemId = null; this.#transcriptKey = null; this.#update({ - inputNotice: event.reason === "over-limit" ? "too-long" : "not-heard", + inputNotice: + event.reason === "pending" + ? "answer-already-pending" + : event.reason === "over-limit" + ? "too-long" + : "not-heard", partialText: "", }); return; @@ -695,10 +733,20 @@ export class VoiceTurnController { } if (event.type === "canonical-speech-requested") { this.#pendingSpeechRequestIds.add(event.speechRequestId); - this.#session.setMicrophoneEnabled(false); - this.#inputTurnPending = false; - this.#transcriptItemId = null; - this.#transcriptKey = null; + if ( + this.#snapshot.interruptionBySpeaking && + this.#snapshot.input !== "paused" && + !this.#takingTurnPromise && + !this.#outputCancellationPromise + ) { + this.#session.setMicrophoneEnabled(this.#snapshot.microphoneEnabled); + } + if (!this.#snapshot.interruptionBySpeaking) { + this.#session.setMicrophoneEnabled(false); + this.#inputTurnPending = false; + this.#transcriptItemId = null; + this.#transcriptKey = null; + } this.#update({ output: "waiting-for-tool", partialText: "" }); if ( this.#latencyCorrelationId !== null && @@ -715,9 +763,11 @@ export class VoiceTurnController { this.#activeSpeechResponseId = event.responseId; this.#activeSpeechResponseTerminal = this.#terminalSpeechRequestIds.delete(event.speechRequestId); - this.#inputTurnPending = false; - this.#transcriptItemId = null; - this.#transcriptKey = null; + if (!this.#snapshot.interruptionBySpeaking) { + this.#inputTurnPending = false; + this.#transcriptItemId = null; + this.#transcriptKey = null; + } if (this.#snapshot.input === "paused") { void this.#cancelOutput(); this.#update({ output: "interrupted", partialText: "" }); @@ -765,11 +815,19 @@ export class VoiceTurnController { if (event.type === "input-speech-started") { if ( this.#takingTurnPromise || - this.#snapshot.output === "speaking" || + this.#snapshot.input === "paused" || + (!event.interruptionBySpeaking && + this.#snapshot.output === "speaking") || this.#snapshot.output === "cancelling" ) { return; } + if (event.interruptionBySpeaking) { + this.#activeSpeechResponseId = null; + this.#activeSpeechOutputEnded = false; + this.#activeSpeechResponseTerminal = false; + this.#update({ output: "interrupted" }); + } this.#inputTurnPending = true; this.#transcriptItemId = event.itemId; this.#transcriptKey = null; diff --git a/libs/@hashintel/petrinaut/docs/ai-assistant.md b/libs/@hashintel/petrinaut/docs/ai-assistant.md index a7209260ae4..52e1e6973b9 100644 --- a/libs/@hashintel/petrinaut/docs/ai-assistant.md +++ b/libs/@hashintel/petrinaut/docs/ai-assistant.md @@ -72,13 +72,20 @@ transcription and Realtime audio are ephemeral. Finalized spoken user messages c carries the same chip, so Voice provenance remains visible without duplicating an answer while the session is mounted. -Voice is half-duplex. The microphone is closed while the interviewer speaks or the assistant is -working, which prevents playback from becoming a false answer. Select **Your turn** to interrupt: -the dock shows the handoff as thinking while it clears pending audio and waits for the provider to -finish cancellation, then opens a fresh input turn. Audio captured before that completed handoff is -discarded. Semantic voice detection finishes your answer automatically after a natural pause, so -there is no required done-speaking action. Duplicate, empty, failed, or unavailable transcripts are -not submitted; the dock asks you to try again. An overlong answer instead asks for a shorter response. +**Interruption by speaking** is on by default. Start speaking while Brunch is talking to stop its +audio and give your answer. Your interrupting words are captured; you do not need to repeat them. +If Brunch is still finishing its previous turn, the dock shows **Answer captured. Waiting for Brunch.** +and sends that answer when it is ready. Wait for that answer to be sent before giving another one. + +Open **Voice playback options** and uncheck **Interruption by speaking** to use manual handover. +This preference is remembered in your browser. In manual mode the microphone closes during +assistant output. Select **Your turn**, wait for cancellation to finish, then speak; audio before +that handover is discarded. The **Your turn** control is hidden when interruption by speaking is on. +If speaker playback causes unwanted interruptions, use headphones or switch to manual handover. + +Semantic voice detection finishes your answer automatically after a natural pause, so there is no +required done-speaking action. Duplicate, empty, failed, or unavailable transcripts are not submitted. +An empty or failed transcript asks you to try again, and an overlong answer asks for a shorter response. Provisional words remain display-only until the provider completes their transcript. Every session control lives in the dock: **Collapse voice session** / **Expand voice session** and diff --git a/libs/@hashintel/petrinaut/src/react/voice-session/store.ts b/libs/@hashintel/petrinaut/src/react/voice-session/store.ts index 73e502c20bb..bc03867ca28 100644 --- a/libs/@hashintel/petrinaut/src/react/voice-session/store.ts +++ b/libs/@hashintel/petrinaut/src/react/voice-session/store.ts @@ -11,6 +11,7 @@ export type VoiceSessionActions = { reconnect: () => void; repeatQuestion?: () => void; resume: () => void; + setInterruptionBySpeaking?: (enabled: boolean) => void; setMicrophoneMuted: (muted: boolean) => void; takeTurn?: () => Promise | void; }; diff --git a/libs/@hashintel/petrinaut/src/react/voice-session/types.ts b/libs/@hashintel/petrinaut/src/react/voice-session/types.ts index b72bee75f8c..78aeacbc2b6 100644 --- a/libs/@hashintel/petrinaut/src/react/voice-session/types.ts +++ b/libs/@hashintel/petrinaut/src/react/voice-session/types.ts @@ -22,6 +22,8 @@ export type PetrinautAiVoiceSessionState = { canRepeatQuestion?: boolean; /** Whether the user can cancel Voice output and start their turn. */ canTakeTurn?: boolean; + /** Whether speaking can interrupt assistant audio. */ + interruptionBySpeaking?: boolean; errorMessage: string | null; /** Whether microphone capture is muted independently of whose turn it is. */ microphoneMuted: boolean; diff --git a/libs/@hashintel/petrinaut/src/react/voice-session/use-voice-session.ts b/libs/@hashintel/petrinaut/src/react/voice-session/use-voice-session.ts index 3f88e964621..4660ec2513e 100644 --- a/libs/@hashintel/petrinaut/src/react/voice-session/use-voice-session.ts +++ b/libs/@hashintel/petrinaut/src/react/voice-session/use-voice-session.ts @@ -102,3 +102,12 @@ export const useVoiceSessionNotice = (): string | null => { () => null, ); }; + +export const useVoiceSessionInterruptionBySpeaking = (): boolean => { + const store = use(VoiceSessionContext); + return useSyncExternalStore( + store.subscribe, + () => store.getSnapshot().state?.interruptionBySpeaking ?? false, + () => false, + ); +}; diff --git a/libs/@hashintel/petrinaut/src/ui/types/ai-assistant-composer-control.ts b/libs/@hashintel/petrinaut/src/ui/types/ai-assistant-composer-control.ts index 0f7f8da56f0..54c70070cd1 100644 --- a/libs/@hashintel/petrinaut/src/ui/types/ai-assistant-composer-control.ts +++ b/libs/@hashintel/petrinaut/src/ui/types/ai-assistant-composer-control.ts @@ -77,6 +77,8 @@ export type PetrinautAiVoiceModeControls = { * whole session when Petrinaut closes the panel. */ setMicrophoneMuted: (muted: boolean) => void; + /** Allows speech to interrupt assistant playback without clearing input. */ + setInterruptionBySpeaking?: (enabled: boolean) => void; /** Cancels Voice output and hands the live microphone turn to the user. */ takeTurn?: () => Promise | void; }; diff --git a/libs/@hashintel/petrinaut/src/ui/views/Editor/components/voice-session-labels.ts b/libs/@hashintel/petrinaut/src/ui/views/Editor/components/voice-session-labels.ts index 0e2f843924b..7dd389c9e4e 100644 --- a/libs/@hashintel/petrinaut/src/ui/views/Editor/components/voice-session-labels.ts +++ b/libs/@hashintel/petrinaut/src/ui/views/Editor/components/voice-session-labels.ts @@ -29,6 +29,7 @@ export const voiceSessionActionLabels = { collapse: "Collapse voice session", end: "End voice mode", expand: "Expand voice session", + interruptionBySpeaking: "Interruption by speaking", mute: "Mute microphone", pause: "Pause voice mode", playbackOptions: "Voice playback options", diff --git a/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel.tsx b/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel.tsx index 0979fd72005..9bef8e9cf93 100644 --- a/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel.tsx +++ b/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel.tsx @@ -624,6 +624,12 @@ const ConversationAiAssistantPanel = ({ ? { repeatQuestion: () => controls.repeatQuestion?.() } : {}), resume: () => controls.resume(), + ...(controls.setInterruptionBySpeaking + ? { + setInterruptionBySpeaking: (enabled: boolean) => + controls.setInterruptionBySpeaking?.(enabled), + } + : {}), setMicrophoneMuted: (muted) => controls.setMicrophoneMuted(muted), ...(controls.takeTurn ? { takeTurn: () => controls.takeTurn?.() } : {}), }); diff --git a/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents.test.tsx b/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents.test.tsx index 7bfb76a8df4..35e4c0fad7d 100644 --- a/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents.test.tsx +++ b/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents.test.tsx @@ -412,6 +412,69 @@ describe("AiAssistantContents", () => { expect(onVoiceDockCollapsedChange).toHaveBeenCalledWith(false); }); + test("toggles interruption by speaking in the playback menu and reveals manual handover", async () => { + const store = createVoiceSessionStore(); + const state = { + canTakeTurn: true, + interruptionBySpeaking: true, + errorMessage: null, + microphoneLevel: 0, + microphoneMuted: false, + phase: "speaking" as const, + }; + const setInterruptionBySpeaking = vi.fn((enabled: boolean) => + store.setState({ ...state, interruptionBySpeaking: enabled }), + ); + store.setActions({ + end: vi.fn(), + pause: vi.fn(), + reconnect: vi.fn(), + resume: vi.fn(), + setMicrophoneMuted: vi.fn(), + takeTurn: vi.fn(), + setInterruptionBySpeaking, + }); + store.setState(state); + render( + + + , + ); + expect(screen.queryByRole("button", { name: "Your turn" })).toBeNull(); + fireEvent.click( + screen.getByRole("button", { name: "Voice playback options" }), + ); + const preference = await screen.findByRole("menuitem", { + name: "Interruption by speaking", + }); + expect(preference.hasAttribute("data-selected")).toBe(true); + const menu = screen.getByRole("menu"); + fireEvent.keyDown(menu, { key: "End" }); + await waitFor(() => + expect(menu.getAttribute("aria-activedescendant")).toBe(preference.id), + ); + fireEvent.keyDown(menu, { key: "Enter" }); + await waitFor(() => + expect(setInterruptionBySpeaking).toHaveBeenCalledWith(false), + ); + expect(screen.getByRole("menu")).not.toBeNull(); + expect(preference.hasAttribute("data-selected")).toBe(false); + expect(screen.getByRole("button", { name: "Your turn" })).not.toBeNull(); + fireEvent.keyDown(menu, { key: "Enter" }); + await waitFor(() => + expect(setInterruptionBySpeaking).toHaveBeenLastCalledWith(true), + ); + expect(screen.queryByRole("button", { name: "Your turn" })).toBeNull(); + }); + test("keeps handoff and canonical playback controls in the Voice dock", async () => { const store = createVoiceSessionStore(); const actions = { diff --git a/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents/voice-dock.tsx b/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents/voice-dock.tsx index 8eb347ea695..fa88dda5029 100644 --- a/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents/voice-dock.tsx +++ b/libs/@hashintel/petrinaut/src/ui/views/Editor/panels/ai-assistant-panel/ai-assistant-contents/voice-dock.tsx @@ -6,6 +6,7 @@ import { useVoiceSessionCanReadFullResponse, useVoiceSessionCanRepeatQuestion, useVoiceSessionCanTakeTurn, + useVoiceSessionInterruptionBySpeaking, useVoiceSessionMicrophoneMuted, useVoiceSessionNotice, useVoiceSessionPhase, @@ -111,6 +112,7 @@ export type VoiceDockProps = { collapsed: boolean; /** Rendered instead of the live indicator when the caller supplies one. */ indicator?: ReactNode; + interruptionBySpeaking?: boolean; microphoneMuted: boolean; notice: string | null; onCollapsedEnd?: () => void; @@ -130,6 +132,7 @@ export const VoiceDock = ({ canTakeTurn, collapsed, indicator, + interruptionBySpeaking = false, microphoneMuted, notice, onCollapsedEnd, @@ -174,6 +177,7 @@ export const VoiceDock = ({ actions={actions} canReadFullResponse={canReadFullResponse} canRepeatQuestion={canRepeatQuestion} + interruptionBySpeaking={interruptionBySpeaking} /> )} @@ -188,7 +192,7 @@ export const VoiceDock = ({ {actions !== null && ( <> - {canTakeTurn && actions.takeTurn && ( + {!interruptionBySpeaking && canTakeTurn && actions.takeTurn && (