codec hardening

[int08h]

Summary

Improve encoder/decoder validation and fixe several bugs found through corpus testing, structured fuzzing, round-trip fuzzing, and differential testing against libFLAC.

Full disclosure: I used AI (OpenAI daybreak-blue) to find and validate a lot of these bugs. I (Stuart, a human) have reviewed the PR and proposed changes and wrote this PR description.

Bugs fixed

Decoder

• Excessive Rice partition orders could produce a zero-sized chunk and panic.
• LPC prediction could overflow during coefficient multiplication or residual reconstruction.
• Left-side, side-right, and mid-side reconstruction could overflow, panic on i32::MIN, or silently wrap invalid samples.
• Invariants on STREAMINFO frame-size, block-size, total-sample, frame-number, and blocking-strategy were not consistently enforced.
• Seeking past the declared sample count could underflow and mutate decoder state before returning an error.
• Byte-reader SeekFrom::End treated the total sample count as a byte length.

Encoder

• Finalizing partial interleaved input could silently discard samples.
• Finalizing partial PCM byte input could truncate to an empty frame and panic.
• Samples outside the declared bit depth could overflow internal decorrelation arithmetic.
• Encoding 1-bit STREAMINFO values could panic.
• Rice encoding of i32::MIN could overflow in debug builds.
• Very short fixed-predictor frames could select an invalid residual partition order, producing files rejected by both this crate and libFLAC.
• Zero-channel parameters could reach channel-dependent division before validation.

How these were found

• Ran raw and structure-aware decoder fuzzing with the bundled fixtures and flac-test-files corpus.
• Added encoder round-trip fuzzing across bit depths, channel counts, block sizes, and encoder presets.
• Added seek fuzzing covering start, middle, end, out-of-range, and repeated post-seek reads.
• Differentially decoded generated files with system libFLAC (I'm on debian 13, so libflac14 1.5.0+ds-2).
• Exercised every bit depth from 1 through 32, including non-byte-aligned PCM and channel extremes.

Tests added

• External subset, uncommon, and faulty corpus integration tests.
• Deterministic corruption testing plus exhaustive single-bit frame corruption.
• Malformed STREAMINFO, frame sequencing, partition-order, LPC, and channel-reconstruction regressions.
• Encoder edge matrices covering every supported bit depth and 1, 2, and 8 channels.
• Partial-input, invalid-parameter, sample-range, and short-frame regressions.
• Raw decoder, structured decoder, encoder round-trip, and seek fuzz targets.