Fix Whisper ASR timestamp regression

[anshusaurav]

Summary

Fixes #1684 — Whisper ASR timestamp regression between v3.8.1 and v4.2.0.

Three fixes targeting the seek loop and logits processor introduced in commit 43b2662 ("Overdue Whisper fixes #1594"):

• Use actual audio length for seek loop bounds (modeling_whisper.js): The seek loop used input_features.dims[2] (always 3000 = padded 30s) as total_frames instead of generation_config.num_frames (set by the ASR pipeline to the real mel frame count). For clips shorter than 30 seconds, this caused the loop to process silence/padding as real audio, producing hallucinated text and incorrect timestamps.

• Apply max_initial_timestamp_index constraint (logits_process.js): A continue statement in WhisperTimeStampLogitsProcessor._call() skipped the max_initial_timestamp_index check at line 326, making it dead code. This allowed the model to generate any timestamp as its first token instead of being constrained to the allowed range. The Python reference implementation has no such early exit — both blocks execute sequentially.

• Guard against infinite seek loop (modeling_whisper.js): If segment_offset computes to zero (e.g., from a degenerate timestamp pair), the seek loop would run forever. Added a break when segment_offset <= 0.

Test plan

• Verify with the reproduction case from ASR (Whisper) regression for detected timestamps #1684 (Steve Jobs keynote audio) — timestamps should match v3.8.1 output
• Verify [applause] segment is no longer dropped
• Existing Whisper prefix token tests (test_modeling_whisper.js) still pass — they use max_new_tokens which bypasses the seek loop
• Test with audio shorter than 30s — seek loop should terminate after one iteration
• Test with audio longer than 30s — multi-segment seek should still work correctly

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[nico-martin]

Hi @anshusaurav, thank you so much for looking into this! The diagnosis and overall direction make sense. One edge case remains: num_frames can be 0, so the truthy check falls back to the padded tensor. Could you use a nullish check and add direct tests for short audio, timestamp masking, and the non-progress case? I'm holding off on merging until the zero-frame bug and untested seek-loop changes are covered.

[nico-martin]

num_frames can legitimately be 0. This truthiness check then falls back to the padded 3,000-frame tensor and reintroduces padding processing. Please distinguish an omitted value from zero, for example with generation_config.num_frames != null, and add a zero/short-frame regression test.

[anshusaurav]

Thanks for the thorough review @nico-martin!

Just pushed the requested changes:

1. Nullish check — changed generation_config.num_frames (truthy) to generation_config.num_frames != null so that num_frames=0 is correctly treated as zero frames rather than falling back to the padded tensor length.

2. Tests added:

   • num_frames=0 — verifies the seek loop does not execute on zero frames
   • Short audio (500 frames < 3000 segment) with timestamps — verifies no crash on sub-segment input
   • Timestamp token presence — verifies return_timestamps=true produces timestamp tokens in the output

[nico-martin]

Thanks, the nullish check correctly fixes the num_frames=0 case. The added tests still do not exercise the changed seek-loop behavior, however: each passes max_new_tokens, and generate() explicitly bypasses _generate_with_seek() whenever that option is present.

Could you adjust the coverage so that:

• zero frames and padded short audio run through the seek path without max_new_tokens
• the short-audio case uses a padded 3000-frame tensor with num_frames: 500
• timestamp masking asserts that initial timestamp IDs above timestamp_begin + max_initial_timestamp_index are masked, preferably with a direct WhisperTimeStampLogitsProcessor unit test
• a mocked degenerate timestamp pair produces segment_offset === 0 and terminates after one pass
• normal multi-segment output advances through the expected passes?

The current timestamp-presence assertion and short-input smoke test would pass with the corresponding fixes reverted. Also, the added Tensor import is currently unused.