feat(examples): MoQT の audio track を STT へ流す FastAPI サーバーとブラウザ example を追加する - #353
Open
yuki-uchida wants to merge 5 commits into
Open
feat(examples): MoQT の audio track を STT へ流す FastAPI サーバーとブラウザ example を追加する#353yuki-uchida wants to merge 5 commits into
yuki-uchida wants to merge 5 commits into
Conversation
yuki-uchida
changed the title
… tracks examples/python/stt-server accepts MoQT sessions inside a FastAPI process, decodes the audio tracks it receives to PCM16 and streams them into a speech-to-text backend. Audio arrives either as PUBLISH (every track is accepted, the catalog names the audio tracks) or after PUBLISH_NAMESPACE (the server subscribes to the catalog and to the audio tracks it lists). Object payloads may be bare EncodedAudioChunk bytes (browser, LOC) or the bridges/live-ingest length-prefixed JSON framing; Opus and AAC are decoded with PyAV and resampled to the backend's PCM format. Backends share one SpeechToText protocol: a WAV recorder for local runs, Deepgram live (linear16 16 kHz) and OpenAI Realtime transcription (pcm16 24 kHz). Transcripts are logged and served at GET /transcripts. Verified with the live-ingest bridge: an ffmpeg RTMP push with AAC audio produced 282 objects that decoded to a 6.0 s 16 kHz WAV.
faster-whisper runs on this machine, so no service credentials are needed. Whisper is a batch model: SpeechSegmenter cuts the PCM stream into utterances at pauses (or at WHISPER_MAX_SEGMENT_SEC) and a worker task transcribes each one on a thread, so the track reader never waits for the model. The model dependencies live in the optional `whisper` dependency group; onnxruntime is pinned below 1.24 because newer wheels drop Python 3.10. Verified with macOS `say` output: 7.8 s of speech transcribed in 2.0 s with the base model, split into two utterances at the pause.
The STT server accepted audio but only exposed results over HTTP. It now
accepts SUBSCRIBE for <namespace>/transcript on any session and writes
every transcript there as its own group with a JSON payload
{track, text, final, at}, so a client that publishes audio can receive
the recognition result on the same MoQT session. Other SUBSCRIBEs are
still rejected.
…hone examples/browser/examples/stt captures the microphone with getUserMedia, encodes it to Opus with the shared WebCodecs audio encoder worker and publishes it to examples/python/stt-server over WebTransport: it answers the server's SUBSCRIBE for catalog and audio after PUBLISH_NAMESPACE, starts a new group every 50 chunks, and subscribes to the transcript track to render the results. Each group is closed with an EndOfGroup status object before the next one starts because the wasm client only closes the subgroup stream on EndOfGroup and sequential readers such as moqt's TrackReader wait for the stream to end before moving to the next group. Verified headless with Playwright: Chrome's fake microphone fed a WAV of synthesized speech; the local Whisper backend returned two transcripts that appeared on the page.
With a real microphone the segmenter emitted a silence-only buffer every
second and Whisper answered each one with a stock phrase
("ご視聴ありがとうございました"). Three gates now suppress it: buffers whose
frames never exceed WHISPER_SILENCE_RMS are discarded instead of emitted,
faster-whisper's VAD filter is enabled (WHISPER_VAD_FILTER), and result
segments whose no_speech_prob exceeds WHISPER_NO_SPEECH_THRESHOLD
(default 0.6) are dropped. Verified with the base model: six seconds of
microphone noise produce no transcript, speech surrounded by noise is
still transcribed.
yuki-uchida
force-pushed
the
feat/python-stt-example
branch
from
September 8, 2026 14:02
302a367 to
a057181
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
MoQT で届く audio track を音声認識(STT)へ流し、結果を transcript track として返す FastAPI サーバーと、マイク音声を publish して結果を表示するブラウザ example を追加しました。
やったこと
examples/python/stt-server: PUBLISH で届いた track は全て accept し catalog で audio track を判定。PUBLISH_NAMESPACE では<ns>/catalogを subscribe し、列挙された audio track を subscribe。payload は生EncodedAudioChunk(ブラウザ)と長さ付き JSON メタ(live-ingest)の両対応で、Opus / AAC を PyAV でデコードSpeechToTextプロトコルで差し替え:wav(既定)、deepgram、openai、whisper(faster-whisper をローカル実行、無音で区切った発話単位で文字起こし。無音のみの区間は渡さず、VAD とno_speech_probで幻覚を抑制)。結果は<ns>/transcriptへの SUBSCRIBE を accept して 1 transcript = 1 group の JSON{track, text, final, at}で配信(他の SUBSCRIBE は reject)examples/browser/examples/stt: getUserMedia → WebCodecs(Opus)→ PUBLISH_NAMESPACE で publish し、transcript track を subscribe して表示。50 chunk ごとに group を進める前に EndOfGroup を送る(wasm クライアントは EndOfGroup でのみストリームを閉じ、TrackReaderは現在のストリームが終わるまで次の group を読まないため)やらないこと
テスト
wavバックエンドに PCM として届くこと、偽バックエンドの結果が transcript track で届くこと、transcript 以外の SUBSCRIBE が reject されること、whisper のセグメンタとバックエンドsayの音声 →whisper(base)で 2 発話。Playwright の Chrome に擬似マイクとして音声 WAV を与え、whisper経由でページに 2 件の書き起こしが表示されることを確認備考
relay/keysの証明書を流用してmake chromeで開く手順を記載