feat(openai): support context hints in STT - #2233
feat(openai): support context hints in STT#2233rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 523ce4a The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } else if (event.type === VADEventType.END_OF_SPEECH) { | ||
| this.#emitEndOfSpeech(); | ||
| if (this.#options.turnDetection === null) { | ||
| if (isRealtimeOnly(this.#options.model)) { |
There was a problem hiding this comment.
🔴 Speech is never submitted for transcription when the user turns off the provider's speech detection
The end-of-speech signal that tells the service to transcribe the buffered audio is only sent for two specific model names (isRealtimeOnly(this.#options.model) at plugins/openai/src/stt.ts:846) instead of whenever local speech detection is driving the session, so users who disable the provider's own speech detection get no transcripts at all.
Impact: With turnDetection: null on models such as gpt-4o-transcribe, audio is streamed but never finalized, so the agent never hears anything the user says.
Mechanism: commit condition narrowed from turn-detection state to model name
Before this PR both commit sites used this.#options.turnDetection === null, which exactly matches _requiresRealtimeVad (plugins/openai/src/stt.ts:231-236) — the condition under which a client VAD stream is created and the plugin must send input_audio_buffer.commit.
Now #forwardVadEvents commits only when isRealtimeOnly(model) (plugins/openai/src/stt.ts:846) and the flush path commits only when isRealtimeOnly(model) && !this.#options.vad (plugins/openai/src/stt.ts:826). For a non-realtime-only model configured with turnDetection: null (plus useRealtime: true and a VAD, which _validateRealtimeVad requires), session.update sends turn_detection: null so the server does no endpointing, and neither commit site fires — the input buffer is never committed and no transcription events are produced.
| if (isRealtimeOnly(this.#options.model)) { | |
| if (_requiresRealtimeVad(this.#options.model, this.#options.turnDetection)) { |
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (modelChanged || clearedLanguage) { | ||
| this.#reconnectRequested = true; | ||
| this.#ws.close(); | ||
| return; | ||
| } |
There was a problem hiding this comment.
🔴 Changing settings mid-call can kill the transcription stream instead of reconnecting
When a settings change requires a fresh connection, the old connection is torn down (this.#ws.close() at plugins/openai/src/stt.ts:702) while audio is still being pushed into it, so incoming speech can hit the half-closed connection and abort the whole transcription stream with an error instead of reconnecting.
Impact: Switching model or enabling language detection during an active call can drop the user's transcription entirely.
Mechanism: `#ws` stays set while the socket is CLOSING
#applyOptions closes the socket but leaves this.#ws pointing at it and leaves #wsReady set; #ws/#wsReady are only cleared in run's finally (plugins/openai/src/stt.ts:747-754) which runs later, after the raced tasks settle.
In that window, #forwardInput (plugins/openai/src/stt.ts:817-836) picks up the closing socket and calls #sendAudioFrame, which does ws.send(...) with no readyState check (plugins/openai/src/stt.ts:948-963). The ws library's sendAfterClose emits an 'error' on the socket, which #messages turns into a thrown Error('WebSocket is not open: ...') (plugins/openai/src/stt.ts:928-945), so #forwardEvents rejects and the rejection propagates out of the reconnect loop before #reconnectRequested is examined — the stream fails instead of reconnecting.
Clearing this.#ws and #wsReady in #applyOptions before calling close() (and/or guarding #sendAudioFrame on readyState === OPEN) would make audio wait for the new socket.
| if (modelChanged || clearedLanguage) { | |
| this.#reconnectRequested = true; | |
| this.#ws.close(); | |
| return; | |
| } | |
| if (modelChanged || clearedLanguage) { | |
| this.#reconnectRequested = true; | |
| const ws = this.#ws; | |
| this.#ws = undefined; | |
| this.#wsReady.clear(); | |
| ws.close(); | |
| return; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (opts.model !== undefined && isRealtimeOnly(opts.model)) { | ||
| if (!this.capabilities.streaming) { | ||
| throw new Error( | ||
| `${model} is served only over the realtime API, and this STT was created for the ` + | ||
| 'transcriptions endpoint; pass useRealtime: true to the constructor to reach it', | ||
| ); | ||
| } | ||
| if (!this.#opts.vad && !this.#vadOptedOut) { | ||
| throw new Error( | ||
| `${model} has no server-side endpointing, so it needs a vad to commit the audio buffer`, | ||
| ); | ||
| } | ||
| } |
There was a problem hiding this comment.
🟡 Switching to a realtime-only model is rejected even when the required settings are supplied in the same call
The switch is validated against the previously stored settings (this.capabilities.streaming and this.#opts.vad at plugins/openai/src/stt.ts:543-555) rather than the ones supplied in the same update, so a valid request that enables realtime mode or supplies speech detection at the same time is refused with an error.
Impact: Users cannot move to gpt-live-transcribe/gpt-realtime-whisper in a single settings update; they get a misleading error telling them to pass options they already passed.
Mechanism: stale state used in the guard
updateOptions computes const useRealtime = opts.useRealtime ?? this.#opts.useRealtime at plugins/openai/src/stt.ts:523 but the realtime-only guard checks this.capabilities.streaming, which is only refreshed later via updateCapabilities (plugins/openai/src/stt.ts:588-592). So updateOptions({ model: 'gpt-live-transcribe', useRealtime: true }) on a batch-mode STT throws.
Similarly the VAD guard checks !this.#opts.vad, ignoring opts.vad, so updateOptions({ model: 'gpt-live-transcribe', vad }) throws even though a VAD was provided — note the later _validateRealtimeVad call at plugins/openai/src/stt.ts:567 correctly uses opts.vad ?? this.#opts.vad.
| if (opts.model !== undefined && isRealtimeOnly(opts.model)) { | |
| if (!this.capabilities.streaming) { | |
| throw new Error( | |
| `${model} is served only over the realtime API, and this STT was created for the ` + | |
| 'transcriptions endpoint; pass useRealtime: true to the constructor to reach it', | |
| ); | |
| } | |
| if (!this.#opts.vad && !this.#vadOptedOut) { | |
| throw new Error( | |
| `${model} has no server-side endpointing, so it needs a vad to commit the audio buffer`, | |
| ); | |
| } | |
| } | |
| if (opts.model !== undefined && isRealtimeOnly(opts.model)) { | |
| if (!useRealtime) { | |
| throw new Error( | |
| `${model} is served only over the realtime API, and this STT was created for the ` + | |
| 'transcriptions endpoint; pass useRealtime: true to the constructor to reach it', | |
| ); | |
| } | |
| if (!(opts.vad ?? this.#opts.vad) && !this.#vadOptedOut) { | |
| throw new Error( | |
| `${model} has no server-side endpointing, so it needs a vad to commit the audio buffer`, | |
| ); | |
| } | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6705 to the TypeScript OpenAI plugin.
Adds keyword and code-switched language hints for gpt-transcribe and gpt-live-transcribe, framework keyterm propagation, per-stream language state, detected-language tagging, realtime-only model handling, and live session updates with reconnects only where required.
Source diff coverage
livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/models.py: ported toplugins/openai/src/models.ts; adds the realtime-only and context-hint STT model names to the target model type.livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py: adapted toplugins/openai/src/stt.tsandplugins/openai/src/realtime/api_proto.ts; maps Python dataclasses, aiohttp sockets, and connection pooling to TypeScript option types and the target's per-stream WebSocket lifecycle while preserving context validation, keyterm merging, language normalization/detection, endpointing, livesession.update, reconnect, and detected-language behavior.livekit-plugins/livekit-plugins-openai/pyproject.toml: not applicable; this is Python dependency metadata. The target's OpenAI JS client forwards multipart request fields without requiring a package upgrade, and the newer typed JS SDK release requires Node 22 while this package targets Node 16-compatible output, so the target uses a narrow local request/response type adaptation instead.tests/test_plugin_openai_stt_context.py: adapted toplugins/openai/src/stt.test.ts; ports every behavior with a target counterpart using Vitest, fake OpenAI HTTP responses, local WebSocket servers, and target VAD fixtures. Pythonaiohttp.ConnectionPool-specific mechanics are not applicable because agents-js owns one socket per speech stream; equivalent target lifecycle coverage verifies in-place updates and reconnect behavior.uv.lock: not applicable; Python-only lockfile, with no target dependency change required.Validation
OPENAI_API_KEY= pnpm vitest run plugins/openai/src(93 passed, 7 provider-gated skipped)pnpm --filter @livekit/agents-plugin-openai buildpnpm --filter @livekit/agents-plugin-openai lint(passes with 6 pre-existing warnings)pnpm exec prettier --check "plugins/openai/src/**/*.{ts,tsx,md,json}"pnpm buildcue-clivoice-mode run withgpt-live-transcribe: final user transcription -> assistant conversation item, with STT usage attributed toapi.openai.com/gpt-live-transcribeapi:checkremains blocked by the repository's existing API Extractor limitation onexport * asindist/index.d.ts.Ported from livekit/agents#6705
Original PR description
gpt-transcribeandgpt-live-transcribeaccept context hints:keywordsfor literal terms expected in the audio, and a plurallanguageslist for code-switched speech.languagenow takes astror alist[str]; more than one language raises on models that accept only one, and codes are normalized to ISO-639-1 because the API rejects regional tags such asen-US.keywordsgives the plugin somewhere to put terms, sostt_context_optionsand automatic keyterm detection now reach OpenAI STT. Detected terms merge behind the user's own and apply with asession.updateon the open connection; only a change ofmodelreconnects.gpt-live-transcriberejects anyturn_detectionconfig and emits nospeech_started/speech_stopped, so it joinsgpt-realtime-whisperon the client-commit path.openai>=2.50, wherekeywords/languagesbecame typed on both transcription APIs.