Bound the source-side frame backlog: encode in executor + frame skip

[queueeee]

Symptom

Live on the upgraded host (6 vCPU, 8 GB) at 1080p / one viewer trying to connect:

23:10:51  rss_mb=1163
23:11:07  rss_mb=2012   (+849 MB in 16 s, then OOM)

This isn't encoder overhead — it's a backlog accumulating somewhere upstream.

Root cause

aiortc's MediaPlayer exposes the decoded frames via a private asyncio.Queue (contrib/media.py:229):

self._queue: asyncio.Queue[Union[Frame, Packet]] = asyncio.Queue()

No maxsize. ffmpeg's worker thread pumps decoded VideoFrames in continuously; our broadcaster pump consumes one per loop iteration. At 1080p the raw frames are ~8 MB each (bgr0 from x11grab); even slightly slower-than-realtime consumption piles up megabytes per second. The math checks out: 850 MB / 16 s ≈ 53 MB/s ≈ ~7 frames/s of unconsumed 8-MB frames.

Why the consumer fell behind: codec.encode() is synchronous and ran on the event loop thread. Each call (10–30 ms at 1080p) blocked the entire loop, including the next await self._source.recv(). So even with libvpx running fine, the consumer was capped by event-loop occupancy.

Fix

Two changes that together cap the backlog at a single frame:

1. codec.encode() now runs in loop.run_in_executor. While libvpx works in a worker thread, the loop can keep draining the source queue. This mirrors what aiortc itself does for per-sender encoders in RTCRtpSender._next_encoded_frame.

2. _drain_to_latest peeks at the MediaPlayer's _queue (stable in aiortc 1.x) and pulls every already-buffered frame, keeping only the newest. Older frames are counted into stale_frames_dropped and discarded — far better than the alternative of a slowly-growing queue that ends in OOM. The counter shows up in video_broadcaster.stopped so an operator can tell whether the host is keeping up.

Test plan

• pytest — 252 passed.
  • New: test_drain_to_latest_skips_backlog_keeps_newest seeds the queue with two stale frames + one current and asserts only the newest is returned with the correct drop count.
  • New: test_drain_to_latest_no_backlog_returns_current confirms the empty-queue fast path is a no-op.
• mypy --strict clean.
• ruff check clean.
• Live: at 1080p30 with one viewer, RSS should plateau (no longer climb 50 MB/s); stale_frames_dropped should be small (a handful, not hundreds per second).

Operator notes

After pulling and rebuilding, watch the ts3.heartbeat rss_mb= line in the bot logs while one viewer connects. Expected behaviour:

• RSS climbs briefly during the encoder warmup + DTLS handshake (~300 → 500 MB)
• Then plateaus
• video_broadcaster.stopped stale_frames_dropped=N (when you down the stack) tells you whether you were keeping up:
  • N near zero → host has plenty of headroom, you can push to 60 fps or higher resolution
  • N in the thousands → host is at its limit, lower fps or resolution

cd /opt/ts6-stream-bot
git pull
docker compose -f docker-compose.yml -f docker-compose.host.yml down
docker compose -f docker-compose.yml -f docker-compose.host.yml up -d --build
docker compose -f docker-compose.yml -f docker-compose.host.yml logs -f

https://claude.ai/code/session_016DuCjRJK995Tj9aDhhB9at

---

Generated by Claude Code