Found while evaluating FlowCat (pinned at 37b09ba) for a self-hosted full-duplex voice assistant, running the cascaded pipeline live end-to-end: str0m WebRTC transport (aiortc client), whisper.cpp STT, OpenRouter LLM with tool calls, Kokoro TTS. Three related findings:
1. Frame::Interruption never reaches process_frame
run_processor (processor/runtime.rs) intercepts Frame::Interruption: it drains the interruptible queue and forwards the frame, but never calls process_frame with it. As a result the Frame::Interruption arms in e.g. CascadedTransportOutput/TransportOutput (send_clear, playout-estimate reset) are unreachable dead code. Frame-level barge-in therefore cannot work on the cascaded path today (the Gemini Live path is unaffected because the model handles barge-in service-side, which is presumably why this hasn't surfaced).
2. Nothing arms the VAD barge-in gate on the cascaded path
VadProcessor gates its barge-in broadcast on bot_speaking, learned from Frame::BotStartedSpeaking/BotStoppedSpeaking — but nothing in the cascaded chain emits those frames, so even with a VAD inserted, Interruption is never broadcast.
3. Even when delivered, the frame path is too slow for audible stop
With 1+2 fixed, we measured detection→sink delivery of the Interruption frame at 14 ms to 2.1 s, depending on whether an intermediate hop was mid-await (TTS synthesis, LLM stream). The biased/priority channel only helps while a task is inside its select; a busy process_frame cannot be preempted. An interrupted reply also cannot be cancelled: LlmProcessor awaits the whole stream, and AssistantContextAggregator will assemble and speak the full reply after the barge-in.
What we did about it
We have a working patch (PR to follow) validated live — barge-in stop latency ~110 µs from VAD detection, existing 302 unit tests + clippy -D warnings clean:
FrameProcessor::on_interruption() default-no-op hook, called by the runtime in the Interruption arm (delivery path for sinks/aggregators);- optional barge-in generation counter +
tokio::sync::Notify on VadProcessor, bumped synchronously at detection → cooperative LLM stream cancel + an out-of-band flush reactor;
BotSpeakingNotifier in the cascaded sink emitting the bot-speaking edges;- a
SpeechGate (VAD-edged segmentation with pre-roll + flush marker), because fixed-window batch STT with the turn lock removed hallucinates turns on silence and splits utterances;
- an opt-in
build_cascaded_call_duplex builder; the stock half-duplex builder is untouched.
Happy to split the PR if you'd prefer the hook/runtime fix separately from the duplex builder.
Two smaller items we hit on the same journey, mention them here rather than spamming issues — say the word and I'll file them separately:
factory::tts drops options.base_url for kokoro (must construct KokoroTts::with_base_url manually), and require_key rejects keyless local providers (whisper_local, kokoro) unless given a dummy key;- binding the media socket to
127.0.0.1 (the FLOWCAT_WEBRTC_BIND_IP default) breaks same-host clients that send ICE checks from per-interface sockets (aiortc, Chrome): replies fail with EINVAL (os error 22) and the connection never establishes. Bind wildcard + advertise loopback works.
Found while evaluating FlowCat (pinned at 37b09ba) for a self-hosted full-duplex voice assistant, running the cascaded pipeline live end-to-end: str0m WebRTC transport (aiortc client), whisper.cpp STT, OpenRouter LLM with tool calls, Kokoro TTS. Three related findings:
1.
Frame::Interruptionnever reachesprocess_framerun_processor(processor/runtime.rs) interceptsFrame::Interruption: it drains the interruptible queue and forwards the frame, but never callsprocess_framewith it. As a result theFrame::Interruptionarms in e.g.CascadedTransportOutput/TransportOutput(send_clear, playout-estimate reset) are unreachable dead code. Frame-level barge-in therefore cannot work on the cascaded path today (the Gemini Live path is unaffected because the model handles barge-in service-side, which is presumably why this hasn't surfaced).2. Nothing arms the VAD barge-in gate on the cascaded path
VadProcessorgates its barge-in broadcast onbot_speaking, learned fromFrame::BotStartedSpeaking/BotStoppedSpeaking— but nothing in the cascaded chain emits those frames, so even with a VAD inserted,Interruptionis never broadcast.3. Even when delivered, the frame path is too slow for audible stop
With 1+2 fixed, we measured detection→sink delivery of the
Interruptionframe at 14 ms to 2.1 s, depending on whether an intermediate hop was mid-await(TTS synthesis, LLM stream). The biased/priority channel only helps while a task is inside itsselect; a busyprocess_framecannot be preempted. An interrupted reply also cannot be cancelled:LlmProcessorawaits the whole stream, andAssistantContextAggregatorwill assemble and speak the full reply after the barge-in.What we did about it
We have a working patch (PR to follow) validated live — barge-in stop latency ~110 µs from VAD detection, existing 302 unit tests + clippy -D warnings clean:
FrameProcessor::on_interruption()default-no-op hook, called by the runtime in the Interruption arm (delivery path for sinks/aggregators);tokio::sync::NotifyonVadProcessor, bumped synchronously at detection → cooperative LLM stream cancel + an out-of-band flush reactor;BotSpeakingNotifierin the cascaded sink emitting the bot-speaking edges;SpeechGate(VAD-edged segmentation with pre-roll + flush marker), because fixed-window batch STT with the turn lock removed hallucinates turns on silence and splits utterances;build_cascaded_call_duplexbuilder; the stock half-duplex builder is untouched.Happy to split the PR if you'd prefer the hook/runtime fix separately from the duplex builder.
Two smaller items we hit on the same journey, mention them here rather than spamming issues — say the word and I'll file them separately:
factory::ttsdropsoptions.base_urlforkokoro(must constructKokoroTts::with_base_urlmanually), andrequire_keyrejects keyless local providers (whisper_local,kokoro) unless given a dummy key;127.0.0.1(theFLOWCAT_WEBRTC_BIND_IPdefault) breaks same-host clients that send ICE checks from per-interface sockets (aiortc, Chrome): replies fail with EINVAL (os error 22) and the connection never establishes. Bind wildcard + advertise loopback works.