feat/moonshine streaming breeze - #3
Merged
Merged
Conversation
sleep3r
commented
Sep 1, 2026
sleep3r
commented
Owner
- feat: Moonshine V2 streaming models + Breeze ASR
- feat: transcribe-cpp engine + GigaAM RNN-T and other GGUF models
- chore: release v0.8.0
- fix: stop NME-SC reading the speaker count off the neighbour graph
Adds the four transcription models Handy hosts that we did not, all now
mirrored into our own bucket (sha256-verified against Handy's published
values):
breeze-asr-q5_k.bin Taiwanese Mandarin, code-switching
moonshine-{tiny,small,medium}-streaming-en Moonshine V2
Breeze ASR is a whisper-large-v2 fine-tune, so it loads on the existing
Whisper engine unchanged. Moonshine V2 needs a new EngineType because it
is a different transcribe-rs type (StreamingModel, not MoonshineModel);
we drive it one-shot, exactly as Handy does.
Guard against an upstream transcribe-rs 0.3.11 crash while wiring it up.
Both Moonshine engines reject certain input lengths instead of padding
internally: moonshine_streaming feeds samples.chunks(1280) and its first
convolution aborts on a final chunk under 80 samples, so any recording
whose length mod 1280 lands in 1..=79 fails outright. Our tail chunk is
arbitrary-length and propagates with `?`, so a fully transcribed
recording would be thrown away. Verified against the real model: lengths
with remainder 1/7/40/79 error, 80+ succeed, padding clears all of them.
The same guard fixes a pre-existing batch-Moonshine crash under 1600
samples.
Also:
- snapshot audio duration before padding, or the fallback segment
stretches by up to 80 ms and drags diarization word alignment with it
- drop the outgoing engine before loading the next one; a
large -> breeze-asr switch transiently held both (~2.1 GB resident)
- correct size_mb on 7 models; the UI renders these as MiB and several
were off by up to 51 MB
- list every shipped model in MODEL_ORDER, so the half that was sorting
alphabetically stops looking arbitrary
Track scripts/ tooling in git (it was entirely gitignored) and replace
the one-shot migrate script with scripts/sync_models.sh, which reads
credentials from .env, verifies sha256 before upload, and re-checks each
object over the public URL. Documented in scripts/MODELS.md, including
the Selectel quirk where a denied GetBucketLocation makes mc sign with an
empty region and every request returns AccessDenied.
Replaces the whisper-only backend with transcribe-cpp, which reads the architecture out of a GGML/GGUF file and covers 18 model families. That unlocks models transcribe-rs structurally cannot run — notably GigaAM's RNN-T decoder heads, since transcribe-rs implements GigaAM as CTC-only. New models (single-file GGUF, mirrored to our bucket, all permissive licences — MIT / Apache-2.0 / CC-BY-4.0): gigaam-v3-e2e-rnnt Russian, most accurate, punctuation + casing gigaam-v3-rnnt Russian, lowercase, unpunctuated gigaam-v3-ctc Russian, fastest parakeet-unified-en English, 0.2 s for 6 s of audio qwen3-asr-0.6b 30 languages with language detection canary-1b-flash en/de/es/fr with translation Verified end to end on synthesized speech rather than assumed: GigaAM e2e-rnnt returns "Привет. Это проверка распознавания русской речи..." in 0.3 s for 6.9 s of audio, and Qwen3-ASR handles both Russian and English with correct language detection. MUST-KNOW: transcribe-rs's "whisper-cpp" feature is removed in this same commit and must never come back alongside transcribe-cpp. Both vendor their own ggml (0.9.5 vs 0.20.2) and export the same symbols; their cargo `links` keys differ, so cargo accepts both, the C++ compiles, the binary links, and the app launches and enumerates devices — but only one ggml survives, and ~46 of ~100 GGML_OP ordinals then mean different operations because GGML_OP_COL2IM_1D is inserted mid-enum. The failure is completely silent and shows up as garbage transcripts. The new `no-duplicate-ggml` CI job is the only automated defence. Existing users keep their downloaded models: transcribe-cpp loads our shipped ggml .bin files unchanged, confirmed by loading and transcribing with ggml-small.bin before making the swap. Fixes found while integrating: - chunk size now follows the engine. GigaAM reports max_audio_ms 25000, under our hardcoded 30 s chunk, so every recording longer than one chunk would have failed. The limit is read at load time, so a future model with a tighter cap needs no code change. - GGML_NATIVE=OFF via the workspace-root .cargo/config.toml. Apple clang rejects `-mcpu=native` on arm64 so the build failed outright, and where it does work it tunes the binary to the build machine's CPU — wrong for a shipped artifact. Root-level so it applies from any cargo invocation directory. - CI looked for bundles under src-tauri/target, but the [workspace] table puts them at ./target. The AppImage step that strips libwayland-client.so has therefore been silently no-opping, shipping unfixed AppImages, and the test workflows uploaded empty artifacts. - rename "GigaAM v3" to "GigaAM v3 E2E CTC"; with four variants listed the bare name no longer identifies which decoder head it is. Whisper also gains word-level timestamps, which whisper-rs could not produce at all, so diarization word alignment improves for free.
Three shipped tests fail on a clean tree, and diarization is on by
default, so this is live: a solo dictation of 10-30 speech chunks (~40 s
to 2 min) is reported as roughly four speakers. Measured on 192-dim
Gaussian embeddings at the shipped max_speakers=6, the single-speaker
false-split rate was 86.8% overall and 100% at n=8..20.
Cause: the p sweep starts at 1. At p <= 2 each speaker's pruned kNN graph
is a path — every segment keeps only its one or two nearest neighbours,
so one person's segments end up chained rather than mutually connected.
A path's normalized Laplacian carries a large *internal* spectral gap
(0.293 -> 1.0 for a 5-node path) that lands inside the 1..=kmax eigengap
window and outranks the gap that actually separates speakers, so k is
read off the shape of the neighbour graph instead of the speakers. For
the two-speaker fixture the eigenvalues at p=1 are
[0, 0, .2929, .2929, 1, 1, 1.7071, 1.7071, 2, 2]: the true gap sits at
i=2 (0.2929) but the path's interior gap at i=4 (0.7071) wins.
Nothing else rejects such a p. The NME numerator p/n grows x4 over
p=1..4 while the gap grows only x1.77, so the argmin is pinned to the
most degenerate p on offer.
Fix: floor the sweep at p >= 3, the smallest p at which no node keeps
degree <= 2 and therefore no cluster can still be a path. This is not a
departure from the reference — NeMo's p counts the self-loop it fills
into the diagonal, so their p equals our p + 1, and our p=1 built a graph
strictly more degenerate than the reference ever constructs.
Measured against the unpatched code:
single speaker, n=4..50, 25 seeds x 3 spreads 99/750 -> 743/750
false-split rate 86.8% -> 0.9%
balanced k in {1,2,3,4,6}, n=20 11/45 -> 45/45
error direction 26 over / 0 under -> 0 / 0
Behaviourally a no-op at n >= 60 and cost-neutral (it removes two of
p_max eigendecompositions); all the value is in recordings under two
minutes. Timings at n=200/500/2000 are unchanged.
Two larger candidates were measured and rejected. A connectivity guard
was vacuous on exactly the graphs it targeted — an articulation-point
test passes a perfect matching, so it admitted p=1 on paired chunks in
40/40 seeds and returned results bit-identical to the unfixed code.
Switching to an unnormalized Laplacian produced k = 1 + (number of
outlier chunks), making it worse than the shipped code on corpora with
realistic noise.
The three failing tests now pass with their fixtures unchanged. Adds two
regression tests, both verified to fail without the floor: one pins the
degenerate-p mechanism directly (asserting the NME ratio really does
prefer p=1, so only the sweep bound keeps it out), the other checks that
one speaker stays one speaker across n=5..30 on noisy 192-dim embeddings
built from a seeded xorshift, so no rand crate and no clock.
Also wires cargo test into CI, which was running no Rust tests at all.
It needs default features — `--no-default-features` drops diarization
and silently skips exactly these tests.
Not fixed, and still open: the eigengap window can still reach into the
spectral bulk (the floor masks it rather than repairing it), a
signal-free p is still made maximally attractive by gap.max(1e-6), and a
single NaN embedding still collapses a whole recording to one speaker
because f32::max turns a NaN distance into similarity 1.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.