Address bot crash on viewer kick: STUN + 720p defaults + robust SIGKILL recovery - #13
Merged
Merged
Conversation
The previous live deploy showed the bot exit 137 (SIGKILL) ~6 seconds into a successful viewer connection. Operator confirmed: the kick is the SYMPTOM of the crash, not the cause - bot dies, viewer's session follows. Likely root cause is OOM from the host: aiortc + PyAV doing 1080p30 VP8 encoding for even one peer is heavyweight, and the container had no memory budget set. Four changes: 1. Add STUN to RTCPeerConnection. The first-attempt ICE timeout we saw in the trace (`ice=checking` for 20 s before the client gave up and sent cmd=reconnect) is exactly the symptom of the bot lacking a server-reflexive candidate when both peers are behind NAT. Default to stun:stun.l.google.com:19302 - same conventional value the browser-side WebRTC stacks use out of the box. 2. Lower SCREEN_WIDTH/SCREEN_HEIGHT defaults from 1920x1080 to 1280x720. 720p30 keeps the per-viewer encode comfortably under the OOM line on a 4 GB host. Bump in .env if your host has the budget; remember to bump STREAM_BITRATE alongside. 3. Robust entrypoint cleanup against SIGKILL. The trap doesn't run on SIGKILL so the pid file persists, AND a stray pulseaudio child can linger inside the same restarted container. Now we pkill -9 pulseaudio + Xvfb up front and `find ... -name pid -path '*pulse*' -delete` to catch XDG_RUNTIME_DIR variants we missed in the explicit path list. Same on shutdown. 4. Memory in heartbeat: `ts3.heartbeat ... rss_mb=…` so the next exit-137 trace shows the trajectory before the kill. ruff + mypy strict + 218 tests pass. https://claude.ai/code/session_016DuCjRJK995Tj9aDhhB9at
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live trace summary
Operator clarified that the bot crashes FIRST → the viewer kick is a
consequence. The trace shows:
ice=checkingfor ~20 s, then the client gave upand sent
cmd=reconnect— classic NAT-punch failure when one sidehas no STUN-derived srflx candidate.
ice=completed,state=connected,parec_audio.started.exit 137(SIGKILL). Almost certainlyOOM from the host — aiortc + PyAV doing 1080p30 VP8 encoding is
memory-hungry and the container had no
mem_limit.the SIGKILL skipped our cleanup trap and the pid file persisted in
a path our explicit list didn't cover.
Fixes
STUN in
RTCPeerConnectionconfig:stun:stun.l.google.com:19302. Same default the browser stacks use.Should make the first ICE attempt succeed without needing the
reconnect dance.
Default capture size 1280x720 (was 1920x1080). Cuts the per-peer
encoder budget by ~2.25×. Operators with bigger hosts can override
via
SCREEN_WIDTH/SCREEN_HEIGHTin.env(and bumpSTREAM_BITRATEto match).Robust entrypoint cleanup against SIGKILL:
pkill -9of any leftoverpulseaudio/Xvfbfrom aprior crashed run.
find /run /var/run /root /tmp /home -name pid -path '*pulse*' -deleteto catchXDG_RUNTIME_DIRvariants the explicit pathlist missed.
findin the shutdown trap so a normal stop is also clean.Memory in heartbeat:
ts3.heartbeat ... rss_mb=…lets us seethe memory trajectory leading up to the next OOM (if there still is
one).
Operator notes
If
exit 137still happens after this, on the host run:If
dmesgconfirms OOM and the host genuinely has no headroom, set acontainer limit in
docker-compose.yml:…and consider lowering further:
SCREEN_WIDTH=854 SCREEN_HEIGHT=480 SCREEN_FPS=20 STREAM_BITRATE=1500will encode comfortably in wellunder 1 GB.
Test plan
ruff check src tests— cleanmypy --strict— cleanpytest— 218 passedcmd=reconnectdetour) thanks to STUN.
ts3.heartbeat rss_mb=…shows memory pattern; if it stays under~600 MB the OOM should be gone.
Operator commands
cd /opt/ts6-stream-bot git pull docker compose down docker compose build --no-cache docker compose up -d docker compose logs -fhttps://claude.ai/code/session_016DuCjRJK995Tj9aDhhB9at
Generated by Claude Code