fix(player): harden animated AVIF capture and player lifecycle

[bee-san]

Important

DRAFT — ACTIVELY BEING TESTED. This is not a merge-ready claim.

The Kotlin/JVM, native FFmpeg, and CI paths have substantial automated coverage, but the exact upstream branch still needs a real-device matrix across Qualcomm, MediaTek, Exynos, and Tensor MediaCodec implementations. Frame-zero decode, looping, cancellation, process death, SAF providers, and player lifecycle behavior are still being tested. Devices without a qualifying hardware AV1 encoder intentionally use the existing still-image fallback.

What this PR does

This upstream-focused branch contains only the three animated-scene/player-hardening commits from the fork. It deliberately excludes the unrelated immersion-statistics work.

It hardens the complete player → capture → FFmpegKit → MediaCodec → animated-AVIF path rather than changing only the AVIF container writer:

• encodes animated scenes directly with Android's hardware av1_mediacodec encoder;
• muxes AV1 directly into looping AVIF in one FFmpeg command;
• selects hardware-compatible, aspect-preserving geometry instead of assuming a fixed encoder size;
• moves scene FFmpeg/ffprobe execution into a private worker process;
• fixes SAF input, cancellation, resource ownership, output cleanup, bounded diagnostics, and remote-input handling;
• validates generated AVIF structure and first-frame random access before returning it;
• fixes related player surface, startup, PiP, config-directory, threading, and UI lifecycle failures.

Unsupported, unsafe, or failed animated captures still degrade to the already-captured still-image fallback.

Pipeline and architecture

1. Freeze capture state

   • Read mpv state before and after capturing the still fallback.
   • Reject the animated request if playback position, selected tracks, source, timing, or frozen video state changes during capture.
   • Resolve scene timing and selected FFmpeg stream indexes from that stable snapshot.

2. Resolve and constrain input

   • Accept seekable local files, SAF content:// documents, and public HTTP(S)/HLS inputs.
   • Reject DASH, transient/torrent inputs, extension-provided FFmpeg arguments, DRM/protected media, credential-bearing URLs, sensitive signed-query parameters, non-allowlisted headers, and nonseekable sources.
   • Bound decoders, protocols, TLS verification, and remote I/O timeouts.

3. Probe and encode in a private worker process

   • The main process communicates through AIDL with a non-exported :scene_processing service.
   • The worker owns scene FFmpegKit/ffprobe sessions and performs only lightweight app initialization.
   • SAF URIs cross Binder as encoded tokens and are registered with FFmpegKit's native saf: protocol inside the worker.
   • Callback output is capped to remain below Binder transaction limits.

4. Select hardware AV1 geometry

   • Probe coded dimensions, sample aspect ratio, display rotation, color primaries, transfer function, and bit depth.
   • Evaluate all MediaCodec AV1 candidates for planar YUV420, constant-quality mode, target quality, alignment, size/rate ranges, conditional width ranges, and 8 fps support.
   • Scale visible content without intentional distortion, then black-pad it into a codec-compatible canvas bounded to 640×640 and aligned to 16 pixels.

5. Encode once and validate

   • One FFmpeg command decodes the selected stream, samples at 8 fps, scales/pads, encodes with the selected MediaCodec encoder, and muxes directly to animated AVIF.
   • Accept output only when brands, AV1 configuration, dimensions, frame count, duration, sample tables/sizes, media bounds, and first-frame sync evidence are valid.
   • Keep descriptors and output files alive until native execution actually returns; delete invalid, partial, cancelled, or undelivered output.

Bugs found and fixes

Native AV1 and animated AVIF

• FFmpeg could hang while constructing av1C from malformed or truncated AV1 sequence-header data.

  • Uses com.github.bee-san:ffmpeg-kit:1.17.8.1, which includes the AV1 UVLC bounds fix.

• MediaCodec AV1 configuration was not consistently usable by the AVIF muxer.

  • The native fork initializes MediaCodec bitstream-filter options and extracts AV1 config OBUs from AV1CodecConfigurationRecord.

• Dummy-frame extradata generation could leave encoder GOP state referring to the discarded frame.

  • Android documents encoder behavior after flush() as undefined. The fork now performs stop → configure → start after extradata generation.

• A vendor could emit a non-key first packet and FFmpeg could omit stss both for all-key and zero-key streams.

  • The patched AVIF mux path rejects the first accepted video packet unless it is flagged as a keyframe.
  • App validation also rejects a present stss table unless sample 1 is a sync sample.

• The earlier raw-OBU normalization/remux design required an intermediate file and a second FFmpeg pass.

  • The final path muxes MediaCodec AV1 directly into AVIF in one native command. The Kotlin OBU normalizer and intermediate file are removed.

• Ten-bit SDR input was rejected solely because it was ten-bit.

  • 8-bit and 10-bit SDR input are accepted; HDR transfer functions and BT.2020 primaries remain rejected. Ten-bit input is converted to yuv420p; this does not claim ten-bit output preservation.

Device portability

• A fixed 640×640 capability check did not represent portrait, landscape, square, odd-sized, minimum-size, alignment, or block-budget-constrained encoders.

  • Encoder selection derives display geometry from width, height, SAR, and rotation, then searches each codec's actual size/rate constraints.

• Codec alignment could squash visible content or request unsupported dimensions.

  • Visible-content dimensions and encoder-canvas dimensions are separate. Aspect-preserving content is padded into a supported 16-pixel-aligned canvas.

• Selecting the first advertised AV1 encoder could pick a lower-quality or unusable implementation.

  • All compatible candidates are evaluated; retained content resolution is preferred, followed by the smaller required canvas.

• Embedded MP4 text subtitles could fail under the decoder whitelist even though subtitles are not output.

  • Adds mov_text to the bounded decoder whitelist.

Input, security, and diagnostics

• Reopening a SAF descriptor via /proc/self/fd/N could lose the grant and fail with EACCES.

  • Uses FFmpegKit's saf: protocol in the same worker process that executes FFmpeg.

• Remote scene processing could forward credentials or unconstrained extension arguments into FFmpeg.

  • Rejects user-info URLs, sensitive signed/auth query keys, non-allowlisted headers, extension FFmpeg arguments, transient sources, DRM, and nonseekable media.
  • Remote commands require certificate verification, a CA bundle, restricted protocols/decoders, and a 15-second I/O timeout.

• Scene failures silently fell back to a still, making vendor-specific failures difficult to diagnose.

  • Adds tagged SceneMining traces for request rejection, probe results, encoder selection, native failure, validation, timeout, cancellation, and cleanup.
  • HTTP(S) paths and embedded URLs are reduced to scheme/host before logging; FFmpegKit's own unredacted logcat output is suppressed.

• Large ffprobe callbacks could exceed Binder transaction limits.

  • Callback output is capped at 128 Ki characters and oversized output fails with a distinct diagnostic.

Cancellation and resource ownership

• Coroutine cancellation could close an input descriptor or delete output while native FFmpeg still used it.

  • Native-use leases are tracked separately from Kotlin ownership. Queued cancellation avoids starting work; running cancellation invokes FFmpegKit cancellation and waits for native return before cleanup.

• Cancellation after successful validation but before result delivery could orphan a file.

  • Undelivered output remains owned by the capture operation and is deleted unless ownership reaches the caller.

• Argument construction, probing, validation, partial output, binding failure, or native exceptions could leak descriptors/files or bypass the still fallback.

  • These paths now fail closed, release every acquired lease, remove partial files, and preserve coroutine-cancellation semantics.

Broader player and app lifecycle fixes

These are included because scene capture exposed races and framework failures outside the AVIF parser itself:

• Playback could call loadfile before the MPV surface existed or while it was being torn down.

  • Adds a surface-aware load gate that retains only the latest pending URL, retries after surface creation/resume, and closes before player destruction.

• Video changes could arrive off the main thread.

  • setVideo dispatches to the UI thread before touching player/activity state.

• External MPV config resolution could crash after a missing path or lost storage permission.

  • Null, blank, missing, and exceptional external lookups fall back to the internal config directory and record the cause.

• PiP framework calls could throw IllegalStateException even after feature detection.

  • PiP calls are centralized behind a guard. Framework rejection disables later attempts and follows the normal finish/back path; unexpected exceptions are not hidden.

• The FFmpeg-only worker process repeated full DI, Conscrypt, and WebView initialization.

  • The scene worker performs only minimal logging initialization.

• A modal scene dialog obscured playback and made progress/cancellation state unclear.

  • Replaces it with an accessible top overlay; cancellation is shown only before the operation's commit boundary.

Dependency and provenance

The app pins com.github.bee-san:ffmpeg-kit:1.17.8.1.

• Native/JitPack AAR SHA-256: fe66560bd45e28f1a80449568d31b54986601d8836d0c4b3d2158cde68caf4b2
• Native release: https://github.com/bee-san/ffmpeg-kit/releases/tag/1.17.8-native
• JitPack release: https://github.com/bee-san/ffmpeg-kit/releases/tag/1.17.8.1
• Both releases are immutable, and the served JitPack AAR was verified byte-for-byte against the native release AAR.
• The native tag is annotated but unsigned; this remains a provenance limitation.

The fork also pins and checksum-verifies the native inputs, including gas-preprocessor, and preserves the required smart-exception-java:0.2.1 POM dependency.

Automated coverage

The affected test files contain 70 tests, 52 more than upstream, covering:

• encoder capability filtering and geometry across landscape, portrait, square, narrow, minimum-size, conditional-range, alignment, and competing-encoder cases;
• one-command AVIF arguments, exact scale/pad filters, decoder/protocol restrictions, SAF input, and embedded mov_text;
• 8/10-bit SDR acceptance, HDR/protected-media rejection, SAR, rotation, and invalid dimensions;
• AVIF bounds, brands, timing, sample sizes, payload, and first-sync validation;
• successful capture, unavailable encoders, output-dimension mismatch, argument failures, partial-output deletion, cancellation during native work, and undelivered-output cleanup;
• tagged/redacted diagnostics;
• surface load deferral/retry/recreation/destruction;
• external MPV config fallback and PiP rejection behavior.

Native FFmpeg regression coverage independently verifies the full patch stack against pinned FFmpeg n7.1, including RED/GREEN checks for MediaCodec restart behavior and rejection of a non-key first AVIF packet.

Current verification status

• git diff --check upstream/main...HEAD
• ./gradlew :app:assembleDebug --no-build-cache
• The same final implementation passed the fork's app CI and signed release build.
• Native patch stack and fate-movenc regression review passed independently.
• Upstream-branch unit-test execution: currently blocked before tests run by an unrelated error already present on upstream/main:
  • ReaderOcrSourceTest.kt:40-41: missing mokuroAvailable argument.
• Repository-wide spotlessCheck: currently blocked by unrelated formatting violations already present on upstream/main, including untouched files under domain and presentation-widget (for example HideAnimeCategory.kt, RenameAnimeCategory.kt, ReorderAnimeCategory.kt, UpdatesWidget.kt, and GlanceUtils.kt).
• Upstream PR CI run 30687964886 stopped at that baseline format check before build/test execution. None of the files reported by the failing domain:spotlessKotlinCheck task are changed by this PR.

The unit-compile and formatting blockers were reproduced in a clean upstream/main worktree. They are reported here so this draft does not imply that every upstream check currently passes.

Active testing before marking ready

• Run the full unit suite after the upstream baseline compile error is resolved.
• Install release-derived APKs across supported ABIs.
• Exercise Qualcomm, MediaTek, Exynos, and Tensor devices where available, including devices with no AV1 encoder.
• Test landscape, portrait, square, rotated, anamorphic/SAR, odd-sized, 8-bit SDR, and 10-bit SDR sources.
• Confirm HDR/BT.2020 and DRM sources cleanly use the still fallback.
• Test local files, several SAF providers, public HTTPS MP4, and public HLS.
• Confirm signed URLs, auth headers, DASH, torrent/transient, nonseekable, and extension-argument sources fail closed without credential leakage.
• Inspect AVIF playback and Anki import for looping, frame-zero correctness, timing, aspect ratio, padding, and visual quality.
• Test cancellation during probe/encode, activity finish, rotation/backgrounding, worker-process death, and repeated captures.
• Test cold player startup, episode changes during surface recreation, resume, and destroy with pending loads.
• Test PiP on supported, unsupported, disabled, and framework-rejecting devices.
• Test revoked/missing external MPV config permissions.
• Verify adb logcat -s SceneMining remains useful without exposing signed paths or credentials.
• Stress oversized ffprobe output and verify bounded Binder failure/recovery.
• Verify still-image and sentence-audio fallback behavior remains unchanged.

Known limitations / non-claims

• This does not claim support on every device. Hardware AV1 support and vendor MediaCodec behavior still require the real-device matrix above.
• Devices without a qualifying AV1 encoder use the still fallback; there is no software AV1 fallback.
• HDR/BT.2020, DRM, DASH, torrents/transient sources, nonseekable media, extension FFmpeg arguments, signed URLs, auth headers, and credentialed streams intentionally fall back.
• Output is bounded to 8 fps, 640 pixels per dimension, 80 frames, 10 seconds, and 10 MiB. Alignment may add narrow black borders.
• JVM tests mock the native executor; they do not prove hardware codec behavior, Binder/service death, native cancellation, AVIF playback, or vendor interoperability.
• Process isolation is a lifecycle/global-state boundary and may limit the effect of a worker crash, but this PR does not claim that it is crash-proof. It does not fix a duplicate-SONAME conflict; there is no second competing FFmpeg implementation in aniyomi-mpv-lib.
• The custom FFmpeg tag is unsigned even though its GitHub release and served artifact are immutable and checksum-verified.

Review history

This work was developed and reviewed in the fork through:

• fix(player): make animated scene mining portable across devices bee-san/chimahon#19
• fix(player): use patched ffmpeg-kit for portable animated AVIF bee-san/chimahon#20

This upstream PR remains a draft specifically so the active device/integration test matrix can be completed and results added without presenting the current evidence as universal portability proof.