feat(examples): 音声パイプラインを MoQT で可視化し、ターンごとの遅延を計測する - #358
Open
yuki-uchida wants to merge 15 commits into
Open
Conversation
yuki-uchida
changed the title
…line Each audio track now feeds a VoicePipeline whose stages are chosen by environment variables: PIPELINE_VAD (silero via the ONNX model bundled with faster-whisper, or energy), PIPELINE_STT (whisper, deepgram, openai, wav), PIPELINE_LLM (gemini, echo, none) and PIPELINE_TTS (gemini, none). Stages implement the protocols in pipeline/base.py and are registered in factory tables in pipeline/__init__.py, so a new service is one class plus one table entry. LLM and TTS are optional; 'none' stops the pipeline after the previous stage. Because the VAD now segments upstream, the speech-to-text stages are per-utterance requests: Deepgram uses its pre-recorded endpoint and OpenAI the audio transcriptions endpoint instead of websockets; Whisper keeps its no_speech_prob filter. Utterances are processed by a worker task so the track reader never waits for a model or a remote service. Results go back over MoQT: transcripts and replies as JSON objects on <namespace>/transcript, synthesized speech re-encoded to Opus on <namespace>/reply, one group per reply. Verified with fakes in pytest and, with the real Silero and Whisper models, on synthesized speech; Deepgram, OpenAI and Gemini are implemented against their documented APIs but not called by tests.
The page now subscribes to <namespace>/reply as well, decodes each Opus object with WebCodecs and schedules it on an AudioContext, and renders transcript and reply objects as 'you:' / 'assistant:' lines.
…g review Merge TranscriptEvent/ReplyTextEvent into TextEvent(kind), drop the mono-only PcmFormat.channels, let Conversation own its counters and Opus encoder instead of threading a state object through the sink, key the result-track fanouts by (namespace, track) in one dict, let google-genai read GEMINI_API_KEY itself, return faster-whisper's own segments instead of mirroring them, turn never-passed constructor arguments into module constants, and share one subscribeResultTrack helper in the browser example.
OpusEncoder became encode_opus(): its only caller encoded and flushed in one expression, and flush() threw the codec state away, so the instance never carried state. SpeechToText.transcribe drops its pcm_format argument, which only ever held the PIPELINE_PCM constant. The per-namespace TrackFanout objects become one dict of writer lists with a module-level write_group. The in-process event log and GET /transcripts go away: transcripts already leave on the MoQT transcript track and through the log. The STT factory loses its track_label argument, which only the wav debug stage read and which it did not need. Track-name environment knobs nothing can set (the browser and tests hardcode the names) become constants, never-passed constructor arguments become module constants, and the tests build tones with numpy and reuse encode_opus instead of repeating the encoder.
A pipeline is built per audio track, so each concurrent track loaded its own copy of the Whisper weights (hundreds of megabytes); the model is now cached per (model, device, compute_type). VoiceHub also kept every Conversation, and with it the track's pipeline and model handles, after the track ended; the entry is removed when run() returns.
PcmFormat had one field left, so the stages pass sample rates as ints and PIPELINE_SAMPLE_RATE documents the shared format. ReplyAudioEvent only wrapped a SynthesizedSpeech and a timestamp nobody read, so the synthesized speech is the event. VoicePipeline starts its worker in the constructor instead of an async start() with one caller, EnergyVad's silence window and the test tone's shape become constants, and the Whisper stage transcribes in one executor hop now that the model itself is cached. The Silero test no longer shells out to macOS `say` (it was skipped everywhere else, so nothing covered the emitting path in CI): the ONNX session is injected, and queued probabilities drive the state machine through push(), which also pins down the pre-roll and the pause. The stage-failure test now asserts that the utterances after the failure are transcribed rather than reading a private field.
…-roll The reply branch encoded Opus even when nobody subscribed to the reply track, and looking the writers up with setdefault inserted empty lists, so stats() reported subscriber entries for tracks nobody had subscribed to. It now reads the writers without inserting and returns early when there are none. SileroVad reset its pre-roll history to the single window that ended an utterance, so every utterance after the first started with ~32 ms of audio before the speech instead of SPEECH_PAD_MS; the history now keeps the tail of the utterance just emitted.
AudioDecoder's target sample rate and the wav stage's TRANSCRIPT_DIR were parameters no caller set; the decoder reads PIPELINE_SAMPLE_RATE and the wav stage defaults to recordings/. The README no longer claims a test that synthesizes speech with macOS `say`.
The Gemini stages need an API key in the environment, and the ignore rule only covered a .env at the repository root, so a key file next to an example could be committed. Tracked .env.example files stay visible.
A sourced .env needs export on every line or the child process does not inherit the variable; uv reads plain KEY=value lines instead.
Verified with a real API key and fixed what the first call exposed. GeminiLlm discarded the genai.Client after creating the chat, and the SDK's httpx client closes itself when collected, so the first send_message failed with "Cannot send a request, as the client has been closed"; the client is now kept alive by the stage. The default models did not work with a newly issued key: gemini-2.5-flash answers 404 "no longer available to new users" and gemini-flash-latest returned 503 after a minute of retries, so the chat default is gemini-3.6-flash (the model the 404 recommends). gemini-2.5-flash-preview-tts rejected two of four short inputs with "Model tried to generate text, but it should only be used for TTS", while gemini-3.1-flash-tts-preview answered all four, so that is the TTS default. The synthesized PCM rate is read from the response mime type instead of being assumed to be 24 kHz.
start_group() and write_group() only ever used the next id in sequence, so an application that numbers groups by something of its own — a conversation turn, a source frame — could not put its objects in the matching group. Both now take an optional id and reject one that would go backwards, since draft-14 §10.4.1 has group ids increase within a track.
start_group() and write_group() take an optional group_id so Python can number groups by its own key.
The server described itself only in its logs and published transcripts as loose JSON objects, so a client could not tell which reply belonged to which utterance or where the time went. The transcript track becomes <namespace>/pipeline. Group 0 carries the pipeline's nodes and edges, sent to every new subscriber, so a client can draw the graph without knowing which implementations are configured. Group N carries conversation turn N: the utterance the VAD cut (with the audio object that completed it), each stage's start and duration, the transcript and the reply, and the packet count of the synthesized audio. The reply audio itself rides in group N of <namespace>/reply, so a turn is one group id on both tracks. VoicePipeline now owns the turn numbering and the stage timing; feed() takes an opaque marker so the transport can name the audio it consumed without the pipeline knowing about MoQT.
…turn The example becomes a React page. It reads the topology object from the pipeline track and draws it with React Flow, lighting each node as the stage events of the current turn arrive, and lists every turn with the server's stage durations. The turn is timed in this browser's clock: the vad event names the audio object that completed the utterance, the publisher remembers when it sent that object, and the reply player reports when the last sample of the turn's audio has played, so the table shows the delay from the start of speech to the end of playback.
yuki-uchida
force-pushed
the
feat/python-voice-pipeline
branch
from
September 8, 2026 14:10
9776230 to
dcf85ee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
音声 track ごとに
VAD → STT → LLM → TTSのパイプラインを動かし、各 stage を環境変数で選べるようにしました。返答テキストは transcript track に、合成音声は Opus で
<ns>/replytrack に配信します。#353 の上に積んでいます。やったこと
<ns>/pipelineは group 0 にパイプラインの構造(nodes / edges)、group N にターン N の各ステージの start / done と所要時間・書き起こし・返答・音声 packet 数。<ns>/replyは同じ group N に返答音声(Opus)。つまり 1 会話ターン = 1 GroupIDvadイベントが名指しする audio object の送信時刻から求めるmoqt::TrackWriter/ Python binding のstart_group/write_groupに group id を指定できるようにした(逆行は拒否)pipeline/base.pyの 4 プロトコル(VoiceActivityDetector/SpeechToText/LanguageModel/TextToSpeech)と、PIPELINE_VAD=silero|energy、PIPELINE_STT=whisper|deepgram|openai|wav、PIPELINE_LLM=gemini|echo|none、PIPELINE_TTS=gemini|noneで選ぶ factory 表。noneはその stage 以降を止めるgoogle-genaiの chat、Gemini TTS の 24 kHz PCM は PyAV で Opus に再エンコードして<ns>/replyに 1 返答 = 1 group で配信replytrack も購読し、WebCodecs のAudioDecoderで復号してAudioContextで再生。transcript の JSON にtype(transcript/reply)を追加し、you:/assistant:として表示やらないこと
影響範囲
faster-whisper/onnxruntimeが example の必須依存になり(Silero が既定 VAD)、google-genai/httpxを追加、websocketsを削除テスト
echoLLM でyou:/assistant:の 2 行が表示されることを確認gemini-3.6-flashの返答 →gemini-3.1-flash-tts-previewの音声(24 kHz)→ Opus 301 packet を reply track へ配信、までブラウザ example 経由で通過備考
PcmFormat/OpusEncoder/TrackFanout/ イベントログとGET /transcripts/ 誰も設定しない env つまみを削除した(3 コミット)。途中で見つかった不具合(track ごとに Whisper モデルを重複ロード、終了した会話が hub に残る、購読者がいなくても TTS を Opus 化、Silero の pre-roll が 2 発話目以降 32 ms に縮む)は fix コミットに分けている