Skip to content

fix(audio): keep the selected command on voice/STT-dictated messages - #2992

Open
RoyBA wants to merge 2 commits into
Chainlit:mainfrom
RoyBA:fix/stt-command-auto-attach
Open

fix(audio): keep the selected command on voice/STT-dictated messages#2992
RoyBA wants to merge 2 commits into
Chainlit:mainfrom
RoyBA:fix/stt-command-auto-attach

Conversation

@RoyBA

@RoyBA RoyBA commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Problem

When a command is selected in the composer (e.g. /search) and the user
dictates their message with the voice/STT feature instead of typing it, the
message is sent without the selected command. Typing the exact same message
correctly includes the command.

Root cause

The selected command lives in frontend state (persistentCommandState) and is
attached to the message only in the text-composer submit path (the
client_message event). The audio path (audio_startaudio_chunk
audio_end) never transmits the command, and the transcribed user message is
created on the backend (in the app's on_audio_end handler), so it has no
knowledge of the UI selection.

Fix

Carry the selected command through the audio turn and auto-attach it to the user
message produced during that turn.

Frontend — send the selected command with audio_start:

  • VoiceButton reads persistentCommandState and passes selectedCommand?.id
    to startConversation.
  • useAudio.startConversation(command?)useChatInteract.startAudioStream(command?)
    socket.emit('audio_start', { command }).

Backend — store it for the turn and attach it:

  • BaseSession gains a current_command field.
  • audio_start stores session.current_command; audio_end clears it, so it
    only applies to that audio turn.
  • Message.__post_init__ auto-attaches current_command to user_messages
    that don't already carry a command.

Backward compatibility

Fully backward compatible. The audio_start payload is optional (old clients
that emit no payload keep working), the new frontend arguments are optional, and
the existing Message.command field is reused. Typed messages and assistant
messages are unchanged.

Testing

  • Added unit tests in backend/tests/test_message.py: auto-attach on
    user_message, explicit command takes precedence, no-op when no command is
    set, and assistant messages never inherit the command.
  • Full backend suite, mypy, ruff, pnpm type-check, ESLint, and Prettier
    all pass.
  • Manual check: select a command, dictate a message via the mic → the resulting
    user message carries the command, identical to typing it.

Out of scope

modes has the same gap (also only attached in the text submit path). Left for a
follow-up to keep this PR focused on commands.


Summary by cubic

Keep the selected command on voice/STT messages so dictated messages behave like typed ones. The composer’s command is sent with the audio session, stored for the turn, and auto-attached to the transcribed user message.

  • Bug Fixes
    • Frontend: VoiceButton reads persistentCommandState and passes selectedCommand?.id to useAudio.startConversation; @chainlit/react-client hooks (useAudio, useChatInteract) emit audio_start with { command }.
    • Backend: Store session.current_command on audio_start, clear on audio_end; Message.__post_init__ attaches it to user_message when missing; Message.from_dict keeps payload command authoritative so typed/resumed messages never inherit an active audio turn’s command.
    • Backward compatible: audio_start payload is optional; typed messages and existing Message.command behavior are unchanged.

Written for commit 05a82f5. Summary will update on new commits.

Review in cubic

Auto-attach the UI-selected command to the user message produced during an audio turn, so voice-dictated messages carry the command just like typed ones. The command is sent with audio_start, stored on the session for the turn, cleared on audio_end, and attached in Message.__post_init__.

Co-Authored-By: GitHub Copilot <noreply@github.com>
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. backend Pertains to the Python backend. bug Something isn't working frontend Pertains to the frontend. unit-tests Has unit tests. labels Jul 28, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread backend/chainlit/message.py
Message.from_dict deserializes incoming client messages and resumed thread history. Because the audio-command fallback lives in the shared __post_init__, a command-less typed message (or resumed step) created during an active audio turn would wrongly inherit the turn's command. Reset the command from the payload in from_dict so deserialized messages stay authoritative; app-constructed transcription messages still inherit as intended.

Addresses PR review feedback.

Co-Authored-By: GitHub Copilot <noreply@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Pertains to the Python backend. bug Something isn't working frontend Pertains to the frontend. size:M This PR changes 30-99 lines, ignoring generated files. unit-tests Has unit tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant