INT8 ONNX export can insert severe word-localized noise while FP32 is clean

[4piu]

Summary

At PocketTTS.cpp commit e801e7d6c2692121a39e80ae525cb5265174a495,
the INT8 graphs produced by the included export_onnx.py can insert loud,
localized noise over otherwise intelligible words. The exporter's numerical
validation passes. The same voice, text, seed, and runtime are clean when all
three C++ graphs use FP32. Official PyTorch FP32 and dynamic-INT8 controls are
also clean on the affected test sentences.

This is concerning because short clips can sound normal and the corruption can
affect only isolated words in a longer message.

Relationship to existing issues

• This sounds adjacent to Fix for the robotic voice #10, but it appears to be a different failure mode.
  Fix for the robotic voice #10 reports robotic speech for both FP32 and INT8 after some generation time.
  Here the all-FP32 C++ control is clean, and the corruption remains when only
  flow_lm_main_int8.onnx is INT8 while flow_lm_flow.onnx and
  mimi_decoder.onnx remain FP32. I have not applied the queue-drain workaround
  proposed in Fix for the robotic voice #10.
• python export_onnx.py no longer works #12 documents that the exporter has drifted from current Pocket TTS models.
  This reproduction pins the older b6369a24 source/configuration and matching
  weights expected by this exporter, so it does not use the newer 2026-04 model
  or its BOS-token change.

Environment and pinned inputs

• macOS 26.6 (25G72), Apple M4 / arm64
• Apple Clang 21.0.0; CMake 4.2.1
• PocketTTS.cpp: e801e7d6c2692121a39e80ae525cb5265174a495
• original pocket_tts.cpp SHA-256:
  c9a844bd66c85d1a2831deb63b1ee61d0ab1c1a4810ce492378cb1c316fa289d
• evaluation source SHA-256 after adding only a mixed graph selector:
  8c7cc2207d1147dcea1cc42a69a90bfa7830952f295194e7186d63dbcf0a6445;
  the existing int8 and fp32 graph selections are unchanged
• original export_onnx.py SHA-256:
  a395b3427c555ccef260dba924ee1383f7d0890f93576985c772d5585723dc56
• Python 3.12.13; PyTorch 2.10.0; ONNX 1.19.1; ONNX Runtime 1.23.2
• Kyutai Pocket TTS source:
  ef69ab86521d4bedcd1fda861d70d5c05e3a939a
• pocket_tts/config/b6369a24.yaml SHA-256:
  8366d1be65dfbea6b9ff439ac7830ac295ee4d85e71d52555689ded1a6cc94a5
• weights: Verylicious/pocket-tts-ungated revision
  fa5a0a3a8c5f8abb79ad6706cf5744fc370ee32c
• tts_b6369a24.safetensors SHA-256:
  a4246e239af0f35a1c495b6d180961a6f10b379dc24dd537f64c695c08e4e216
• tokenizer.model SHA-256:
  d461765ae179566678c93091c5fa6f2984c31bbe990bf1aa62d92c64d91bc3f6
• natural English reference: Kyutai tts-voices, Voice-Zero Caro Davy,
  revision 323332d384469494c5de3bc73fa007218d15b462; normalized WAV SHA-256:
  2bf7a615b5a7cc4056e405dfb576ded0764276ce82f23bfb4d9eae11e862cb82

Reproduction

1. Install Pocket TTS from the pinned Kyutai commit above and stage the two
   verified model files for export_onnx.py.
2. Run the unmodified exporter. Its FP32-vs-ONNX numerical validation passes.
3. Build PocketTTS.cpp in Release mode.
4. Generate each case once with --precision int8 and once with
   --precision fp32, one thread, default temperature 0.7, one LSD step, and
   noise clamp disabled.

For deterministic comparison, the attached small harness calls the existing
pocket_tts::rng::seed(42) immediately before PocketTTS::stream. It otherwise
uses the same engine surface and does not change inference math. The attached
source patch adds a mixed precision selector for diagnostic isolation; stock
int8 and fp32 still select the original graph combinations. Each retained
WAV was regenerated in a second fresh process and required to be byte-identical.

Example commands with the attached harness:

pockettts-cpp-evaluation \
  --models /path/to/exported/models \
  --voice /path/to/reference.wav \
  --text-file numbers.txt \
  --precision int8 --seed 42 --threads 1 \
  --warmups 1 --iterations 1 --output int8.wav

pockettts-cpp-evaluation \
  --models /path/to/exported/models \
  --voice /path/to/reference.wav \
  --text-file numbers.txt \
  --precision fp32 --seed 42 --threads 1 \
  --warmups 1 --iterations 1 --output fp32.wav

Two affected inputs are:

Queue items 7, 18, and 42; then pause, cancel, and continue.

The local speech provider finished initialization and can now report progress without interrupting the audio already playing.

The ordinary upstream CLI also exposes the difference, although its
clock-derived RNG seed means the exact affected word can change between runs.

Results

With the stock exporter and one thread, all six corpus cases were byte-identical
across two fresh processes. Human listening had already identified the same
kind of inserted/overlaid noise in the stock INT8 output, while C++ FP32 was
clean.

As a reproducible machine check, I peak-normalized each stock-INT8 WAV and its
FP32 control, used a 20 ms STFT with 10 ms hop, retained loud frames above
-35 dB, and compared the 95th percentile fraction of energy at or above
8 kHz. This is a regression signature, not a general speech-quality metric:

Case	Stock INT8 / FP32 high-frequency ratio
plosives	7,193x
numbers and punctuation	5,694x
longer message	2,777x

The corresponding stock-INT8 fractions were 0.994, 0.980, and 0.929,
versus 0.000138, 0.000172, and 0.000335 for FP32. The attached WAV pairs
make the failure much clearer than these numbers.

With two threads and seed 42, the longest stock-INT8 case was not byte-identical
across fresh processes, even though the first five cases were. I have kept that
as a secondary observation because it may be independent execution-order
nondeterminism.

Quantization isolation

The included exporter dynamically quantizes every eligible ONNX MatMul in
flow_lm_main, flow_lm_flow, and mimi_decoder. Kyutai's documented dynamic
INT8 path instead quantizes PyTorch FlowLM attention and FFN modules while
leaving the flow-matching network and Mimi decoder FP32:

https://kyutai-labs.github.io/pocket-tts/quantization/

Using FP32 flow matching and Mimi with the stock INT8 main graph did not remove
the corruption. Projection-level experiments also did not establish a safe
INT8 subset: leaving every FFN linear2 contraction FP32 fixed one short word
but still inserted noise in the longer cases above. These experiments are only
diagnostic; I am not asking the project to support that selective graph.

Questions

1. Is the current generic ONNX Runtime quantize_dynamic(..., op_types_to_quantize=["MatMul"]) policy expected to be acoustically
   equivalent to the official PyTorch quantization path?
2. Are there intended operator exclusions, quantization parameters, or an
   acoustic regression corpus for the exported INT8 models?
3. Would you expect the queue-drain change proposed in Fix for the robotic voice #10 to affect a failure
   that remains with FP32 flow matching and decoding but disappears when
   flow_lm_main is FP32?

Artifacts

pockettts-cpp-int8-artifact-report.zip