VC-1 (software path) playback judder due to missing frame PTS on I/P pictures

[classicjazz]

What happened?

VC-1 Advanced Profile content played through AetherEngine's software decode path judders, reproducibly, on Apple TV 4K (via Sodalite/Jellyfin or Moonfin/Jellyfin) and does not judder in mpv or other FFmpeg-based players on the same files on AppleTV. The interlace/deinterlace subsystem, container transport, and file bitrate have all been ruled out. The most likely cause is that AetherEngine's software decoder reads AVFrame.pts directly rather than FFmpeg's best_effort_timestamp, and AVFrame.pts is unset (AV_NOPTS_VALUE) on every I and P frame in affected files — only B-frames carry a valid raw pts. AVSampleBufferDisplayLayer, which the software path uses for presentation, has no independent pacing/cadence model, so it presents strictly on whatever CMTime it's given — any inaccurate fallback timestamp for the 60% of frames lacking raw PTS becomes visible, periodic judder.

Steps to reproduce

Play a VC-1 file in Sodalite
It will exhibit significant (unwatchable) video judder

AetherEngine version or commit SHA

as embedded in Sodalite 1.0.0 (20)

Host app

Sodalite

Platform

tvOS

OS version

tvos 26

Device / chip

Apple TV 4k 3g

Playback path

Software (dav1d) host

Source media (for playback bugs)

vc1 (Advanced) (WVC1 / 0x31435657), yuv420p(progressive), 1920x1080, 23.98 fps, 23.98 tbr, 1k tbn, Matroska container, TimecodeScale default (ms granularity)

ffprobe.txt

Error codes / log lines

Anything else

From Claude based on extensive analysis of video files, logs, and testing across devices and media types. I also have separate analysis of why this is an issue with AetherEngine and not mpv and other ffmpeg based media players.

---

Environment

• Repo: github.com/superuser404notfound/AetherEngine, HEAD examined 4219e140 (2026-08-22), release tag 6.34.1
• Clients: Sodalite and AetherPlayer both pin f4bebc0fd3c66f656ef18d2c7c0b3243df3ac86d = tag 6.34.1 (current release, not stale)
• Playback target: Apple TV 4K
• Reproduces via: Jellyfin/Sodalite over 1 Gbps, and direct SMB mount over 10 Gbps — identical judder both ways
• Sample file: 1920x1080 VC-1 (Advanced), progressive, 23.98 fps, ~19.7 Mb/s, Matroska container, B-frames present

What's been ruled out

• Profile-conditional interlace assumption. No code path branches on VC-1 profile; VideoRoutingPolicy.requiresSoftwarePath routes all VC-1 to software unconditionally regardless of field order (VideoRoutingPolicy.swift:29-47), and routesSoftwareForDeclaredInterlace is hard-gated to H.264 only (VideoRoutingPolicy.swift:58).
• Deinterlace filter engaging on progressive content. DeinterlaceFilter only activates on the decoded frame's actual AV_FRAME_FLAG_INTERLACED flag, checked per-frame (SoftwareVideoDecoder.swift:235-269). Confirmed via direct ffprobe -show_frames against the reproducing file: repeat_pict=0 on every frame in a 37.5s sample — no pulldown/interlace signaling present in the bitstream at all.
• Historical PTS-doubling bug (eedaa5b5). Fixed since 3.11.2, and in any case only fires when DeinterlaceFilter is active — moot here since the deinterlacer never engages for progressive content.
• Transport/bitrate. Identical judder over both a 1 Gbps Jellyfin stream and a 10 Gbps local SMB mount rules out network throughput and IO buffering as a cause.
• Container/demuxer-level timestamp irregularity. ffprobe -show_entries frame=pts_time,best_effort_timestamp_time,pkt_dts_time,pict_type,repeat_pict against the actual reproducing file shows best_effort_timestamp_time populated on 100% of frames, evenly spaced at the expected ~41.7/41.6/41.7ms cadence for 23.976fps, with no gaps or discontinuities across the full sample. The source data and FFmpeg's own reconstruction of it are both clean.

The finding

In the same ffprobe output, raw pts_time (i.e., AVFrame.pts as the VC-1 decoder set it, before any FFmpeg-side reconstruction) is N/A on every I-frame and every P-frame in the sample — only B-frames carry a valid raw PTS. This split is total and consistent across the entire 37.5s capture (GOP pattern I B P B P; 3 of every 5 frames have no raw PTS). best_effort_timestamp, which FFmpeg computes specifically to paper over exactly this kind of gap using DTS/reorder-depth reconstruction, is complete and correct for the same frames.

This means: whatever timestamp AetherEngine's software decoder hands to the renderer for 60% of frames in this file depends entirely on which field it reads. If it reads raw pts (matching SoftwareVideoDecoder.swift's apparent per-frame handling in drainDecodedFrames, SoftwareVideoDecoder.swift:235-269) rather than best_effort_timestamp, most frames arrive with no usable timestamp and whatever fallback is used — extrapolation, last-known-PTS reuse, arrival-time estimation — is a strong candidate for the judder's actual source.

This is also why the symptom is AetherEngine-specific despite shared FFmpeg internals: mpv and other FFmpeg-based players never consume raw frame->pts for exactly this reason, always using best_effort_timestamp instead. And it's why the software-decode presentation path is unusually sensitive to any timestamp error in the first place — AVSampleBufferDisplayLayer presents each sample strictly at the CMTime it's given, with no vsync-aware cadence model or interpolation to absorb an inaccurate value the way mpv's own pacing loop would.

(Why VC-1's decoder produces this split — B-frames tagged, I/P frames losing their association during internal reference-picture buffering — is a plausible mechanism based on the observed pattern, not something confirmable without libavcodec source access. It doesn't change the recommended fix.)

Recommended fix

1. In SoftwareVideoDecoder.swift's frame emission path, switch from reading the decoded frame's raw pts to its best_effort_timestamp (bridged via av_frame_get_best_effort_timestamp() if not already exposed as a Swift-accessible property). This is confirmed complete and correctly-paced for the reproducing file and should need no additional reconstruction logic.
2. Audit whatever fallback currently runs when pts == AV_NOPTS_VALUE — that code path is exercised on the majority of frames in files like this one and is the immediate suspect for the visible judder.
3. Once switched, set an explicit duration on each CMSampleTimingInfo (not just presentationTimeStamp), so CMSampleBufferMake/AVSampleBufferDisplayLayer has the intended per-frame duration rather than inferring it from enqueue timing.

Supporting data

• ffprobe -show_frames output against the reproducing file (pts_time, pkt_dts_time, best_effort_timestamp_time, pict_type, repeat_pict), ~900-frame / 37.5s sample — available on request, confirms the 100%-consistent I/P pts_time=N/A pattern and fully clean best_effort_timestamp_time sequence.
• Stream metadata: vc1 (Advanced) (WVC1 / 0x31435657), yuv420p(progressive), 1920x1080, 23.98 fps, 23.98 tbr, 1k tbn, Matroska container, TimecodeScale default (ms granularity).

[superuser404notfound]

Thanks for the depth here. The ffprobe capture is real and I reproduced the exact split on my own fixtures, but it does not describe what the engine sees, and I would rather say that now than have you spend another session on the timestamp path.

Fixtures I measured on, both public, both V_MS/VFW/FOURCC VC-1 Advanced in Matroska like yours: fate-suite.ffmpeg.org/vc1/SA10143.vc1 remuxed with -c copy, and samples.ffmpeg.org/V-codecs/WVC1/Test_1440x576_WVC1_6Mbps.wmv (progressive, I B P B P, the same GOP shape as your file).

The split is at the container, not at the decoder

Your output shows raw pts_time=N/A on I and P pictures. That pattern is already there one layer lower, at the packet level, before any decoder touches it:

$ ffprobe -show_entries packet=pts_time,dts_time,flags sa10143.mkv
pts_time=N/A     |dts_time=0.000000|flags=K__
pts_time=N/A     |dts_time=0.040000|flags=___
pts_time=0.080000|dts_time=0.080000|flags=___
pts_time=N/A     |dts_time=0.120000|flags=___
pts_time=0.160000|dts_time=0.160000|flags=___

The reason is in libavformat/matroskadec.c:4103 of n8.1.2, the FFmpeg AetherEngine links:

if (track->ms_compat)
    pkt->dts = timecode;
else
    pkt->pts = timecode;

ms_compat is set for V_MS/VFW/FOURCC tracks (matroskadec.c:2897), which is exactly how WVC1 is stored, and it is why ffprobe prints a fourcc for your stream at all (vc1 (Advanced) (WVC1 / 0x31435657)). The Matroska block time becomes DTS, pkt->pts is never written, and libavformat's generic reorder then recovers a PTS for the non-reference pictures and not for the rest. So the mechanism is not the VC-1 decoder losing the association during reference-picture buffering; it is a container rule that applies to every VFW-tagged Matroska track regardless of codec.

The engine never sees it

Demuxer.applyDemuxerOptions opens every source with fflags=+genpts, and filling exactly these gaps is that flag's entire job. Both avformat_open_input sites in Demuxer.swift go through it (lines 392 and 489).

Same file, same ffprobe, one flag apart:

without +genpts:  pts_time=N/A      best_effort=0.040  I
                  pts_time=0.080000 best_effort=0.080  B
                  pts_time=N/A      best_effort=0.120  P

with +genpts:     pts_time=0.040000 best_effort=0.040  I
                  pts_time=0.080000 best_effort=0.080  B
                  pts_time=0.120000 best_effort=0.120  P

And through the engine's own demuxer, under the playback open profile:

$ aetherctl pktdump --count 300 --profile playback vc1.mkv
[PktDump] stream[0] tb=1/1000 avg_frame_rate=25/1 codecpar.video_delay=1
[PktDump] video packets=300 keyframes=300 NOPTS_dts=0 NOPTS_pts=0 nonMonotonicDts=0
[PktDump] dts-delta histogram (delta x count): 40x299

NOPTS_pts=0. No packet reaches the decoder untimed, so no frame reaches the renderer untimed, and the fallback you identified is never entered on this path.

What that failure would actually look like

This is the part worth having, because it rules the hypothesis out by shape rather than by argument. I suppressed +genpts in a local build to force exactly the state your analysis describes, and played the same VC-1 file:

[Renderer] dropped 1 frame(s) with no usable timestamp (unschedulable)
[Renderer] dropped 250 frame(s) with no usable timestamp (unschedulable)
[Renderer] dropped 1250 frame(s) with no usable timestamp (unschedulable)
[SWDiag] clk=2.53 eof=y enq=+214 status=unknown r4d=n
VERDICT: clock did not advance (t=2.53s); transport stalled after load

An untimed frame is not presented at a slightly wrong moment, it is refused. The render synchronizer cannot pace a sample whose PTS reads back as NaN, and a NaN also breaks the reorder buffer's ordering (every comparison against NaN is false), so the gate drops it outright rather than let it reorder its neighbours. The result is no picture at all and a 58 s file drained in 2.5 seconds, because nothing is left to pace the demux loop. Your report is judder on a picture that is there, which is a different failure.

Sanity check that macOS is not papering over it: a normal run of the same file reads enq=+25 per second on a 25 fps source with delay=0.00, and measures identically to an MPEG-2 control on the same software path.

Shipped anyway, but not as the fix

Your point 1 is right as hardening even though it is not the cause here, so I took it. Branch fix/ae407-best-effort-pts reads best_effort_timestamp when the decoder set no PTS, placed directly after avcodec_receive_frame so captions, the deinterlace graph and the emit path all see one repaired timestamp rather than each reading the raw field separately. With +genpts suppressed, the session above now runs at 25 enq/s with an advancing clock instead of dying; with +genpts on it never fires. The value is that the drop becomes a genuine last resort instead of the only behaviour, which matters for live MPEG-TS, where untimed pictures do arrive on their own.

I did not take point 3 (an explicit CMSampleTimingInfo.duration). Not refusing it, but I have no measurement that it changes anything, and I would rather not ship a second unmeasured change into an open judder report.

The lead I would chase instead

libavcodec grants the VC-1 decoder no decode parallelism at all. Same machine, same 8-core request, one codec apart:

[SWDecoder] Opened: 1440x576, codec=vc1,        threads=1, 8-bit
[SWDecoder] Opened: 1440x576, codec=mpeg2video, threads=8, 8-bit

Confirmed independently with ffmpeg's own benchmark on the VC-1 file: utime=3.352s / rtime=3.336s, one core saturated, and -threads 0 produces the same numbers. The vc1 decoder supports neither frame threading nor, on a single-slice stream, slice threading.

Your source is 1920x1080 at 19.7 Mb/s, which is 2.5x the pixels of my sample at 3.3x the bitrate, decoding on one Apple TV core, with a CPU YUV to NV12 conversion per frame on top, next to FLAC 7.1 24-bit also decoded in software on the same box. If that does not sustain 23.976 fps, the picture judders while every other indicator looks healthy, and it judders in the reproducible periodic way you describe.

What would settle it

Sodalite, Settings > Diagnostic Log. It is not gated on a diagnostic build, so it is there in your 1.0.0 (20). Play until it judders, then send these three lines:

1. [SWDecoder] Opened: 1920x1080, codec=vc1, threads=N, ... and N is the parallelism your stream actually got.
2. [SWDiag] clk=... enq=+N layerDrop=...(+N) delay=...(+...) status=..., one per second, ideally a run of ten or so taken while it is juddering. This is the line that decides it. enq below 24 means the decoder is not keeping up and the cause is upstream of presentation; enq at 24 with layerDrop climbing or delay growing means the frames exist and the problem is where they are put on screen.
3. [DisplayCriteria] mode check (software): content=... requested=... panel=...Hz ..., which says whether Match Frame Rate landed on 23.976. A panel sitting at 60.000 under 23.976 content is 3:2 pulldown, periodic judder on its own, and nothing downstream can hide it.

And two A/B questions that split the space faster than any log:

• Does an MPEG-2 or VP9 file judder the same way? Both take the identical software path. If they do, this is not about VC-1. If only VC-1 does, the single-threaded decoder is the prime suspect.
• Does the same file judder with a different audio track selected, stereo instead of the FLAC 7.1? The software path's master clock is driven by audio, decoded on the same CPU.

Separately, a Sodalite change is queued so the in-player overlay carries this vocabulary too: [SWDiag], [SWDecoder], [SWHost] and [Renderer] were all filtered out of its focused view, which is why a software-path report had nothing screenshottable there. Settings > Diagnostic Log is unfiltered and works today.

One note on the analysis you attached, offered as a pointer rather than a complaint: it reads SoftwareVideoDecoder.swift accurately, but every claim about what the engine receives was derived from a default ffprobe open rather than from the engine's. That single substitution is what inverted the conclusion, and it is cheap to guard against next time, since the same reasoning also predicts a black screen where you are seeing judder.

[superuser404notfound]

Shipped in 6.38.0, and both consumers are pinned to it.

To be clear about what that does and does not cover: the frame-level best_effort_timestamp repair is in, so an untimed picture is recovered instead of refused. Your judder is still open, because nothing measured so far says the timestamp path was ever the one it took.

The three log lines and the two A/B questions above are what moves it next, and the [SWDiag] enqueue rate is the one that decides between a decoder that cannot keep up and a presentation problem. Take them while it is juddering, from Settings > Diagnostic Log.

[superuser404notfound]

Correction to my own lead, before it costs you a measurement session.

I pointed at the single-threaded VC-1 decoder as the likely cause. I have now measured the throughput instead of reasoning about it, and it does not hold up.

Full engine software path (release build, aetherctl swdecode, so decode plus the YUV to NV12 conversion plus the pixel-buffer allocation, all of it), on a 1280x720 VC-1 Advanced stream at 13.4 Mb/s:

2999 frames in 4.54 s  =  660 fps  =  608 Mpixel/s

Decode-only cross-check with ffmpeg on the same file: 5799 frames in 8.95 s, 648 fps, single core.

Your source needs 1920x1080 at 23.976, which is 49.7 Mpixel/s. That is about 8 percent of one core on this machine, roughly 12x headroom. This is an M1 rather than the A15 in the Apple TV, but those two cores are in the same class, so even halving the number leaves about 6x. Single-threaded or not, the decoder is not the thing that is late.

So both hypotheses on the table are now measured and dead: the timestamp path (never entered, the engine opens with +genpts) and decode starvation (12x headroom). What is left is all on the presentation side, and none of it is testable off tvOS, which is why the log lines still matter.

The three lines from my earlier comment are unchanged, but read them differently now:

• [SWDiag] ... enq=+N ... should read at the content rate (24). If it does, that is the confirmation that the frames exist and arrive on time, and the answer is downstream of the decoder. If it reads below 24, that contradicts the measurement above and is by itself the most interesting result in this thread.
• With enq at rate, layerDrop and delay are the discriminators: drops climbing means frames are being refused at the display queue, delay growing means they are being shown late.
• [DisplayCriteria] mode check (software): content=... panel=...Hz is now the first thing I would look at. A panel at 60.000 under 23.976 content is 3:2 pulldown, which is periodic judder on its own and nothing in the engine can hide it.

And the A/B that splits this fastest is still the one worth doing first, because it needs no logs at all: play a 23.976 MPEG-2 or VP9 file. Both take the identical software path with the identical presentation code. If that judders too, this is not about VC-1 at all, and the search narrows to what the software path does with 23.976 content. If only VC-1 judders, I am wrong about where to look and the decoder is back in scope despite the numbers above.