Log capture health so a video outage points at a layer - #108
Merged
Conversation
This was referenced Jul 29, 2026
leandropineda
force-pushed
the
feat/video-health-counters
branch
from
July 29, 2026 22:41
ef4dc55 to
1b830c5
Compare
leandropineda
force-pushed
the
fix/video-frame-staleness
branch
2 times, most recently
from
July 29, 2026 22:51
a489c23 to
ec99000
Compare
leandropineda
force-pushed
the
feat/video-health-counters
branch
2 times, most recently
from
July 29, 2026 22:55
42fe0e3 to
559e0cd
Compare
When video stops there is nothing in the logs to say whether the camera went away, the platform never asked for a stream, or frames were captured and lost downstream. Diagnosing it meant attaching to a running process. The capture thread now logs one line per minute -- grabbed, served, stale, reopens -- and the same signals are exposed as OTEL counters next to the existing publish instruments, so they surface through the Prometheus endpoint connector-python already ships. Tests: 1 new case in test_video.py.
The README never mentioned video: not the extra to install, not how to register a camera, and not the OPENCV_FFMPEG_CAPTURE_OPTIONS that decide how quickly a dead stream is noticed. Covers the settings the video train adds (stale_frame_seconds, api_preference, the reopen backoff and the health log) in one place, next to the metrics section that now carries the new counters.
Two limitations worth stating where the code is, not only in review threads: the platform's per-camera rate/quality arrives in cameras_config via modules/set_state and is still ignored, and the drain loop races through non-live sources because grab() does not block on them.
leandropineda
force-pushed
the
feat/video-health-counters
branch
from
July 29, 2026 23:02
559e0cd to
874fef7
Compare
The test set stale_frame_seconds to 0.0 and expected the next read to be withheld, but the check is age > window and the capture thread was still refreshing the frame: on a coarse clock the age landed at exactly 0.0, so the frame was served and the assertion saw 2 instead of 1 (Windows 3.12). The capture thread is stopped first, then the buffered frame is aged past the window explicitly. 30 consecutive runs pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #107. Last of the video train.
When video stops, nothing in the logs says whether the camera went away, the platform never asked for a stream, or frames were captured and lost downstream — diagnosing it meant attaching to a running process.
The capture thread logs one line per minute:
No line at all → the platform never loaded the camera module.
grabbed=0→ stream unreachable. Grabbed but not served → the streamer isn't consuming. Served with nothing visible → MQTT side. The same signals are exposed as OTEL counters next to the existing publish instruments, so they surface through the Prometheus endpoint connector-python already ships.Also adds the Camera streaming README section the train was missing: the
videoextra, how to register a camera, theOPENCV_FFMPEG_CAPTURE_OPTIONSline RTSP deployments need (rtsp_transport;tcp|timeout;3000000— without a timeout a dead stream waits on OpenCV's 30s watchdog), and a table of every setting the train introduced (stale_frame_seconds,api_preference,REOPEN_BACKOFF_SECONDS,HEALTH_LOG_SECONDS). Documented here rather than in #105 so all the knobs land in one place; happy to move it if you'd rather each PR carried its own.Tests: 1 new case. Suite green (151), flake8/black clean.