Counterpart to #136 — that one argues the unauthenticated /health exposes too much to anonymous callers. This one is that it exposes too little to a legitimate one.
The situation
A node's most operationally interesting signals — signing_failures_total, registry_outbox_stalled, ruleset_load_failures_total, and outbox depth — are on the Prometheus metrics endpoint, which binds to container loopback. A monitor outside the host cannot reach it, and at small scale there is no scraper: standing up Prometheus per operator to read three counters is not proportionate.
So an external probe sees status: ok for a node whose signing has been failing for a week, whose registry outbox has been stalled since Tuesday, or whose nightly backup has not completed in ten days. A green probe currently means "the node answers HTTP", not "the node is well" — and nothing communicates that distinction.
What would close it
An authenticated health detail route that reports the counters an operator would page on, alongside what /health already carries. Concretely: signing failures, ruleset load failures, registry outbox depth and stalled count, and a last-successful-backup timestamp if the node is in a position to know one.
Authenticated matters for the same reason #136 argues the anonymous surface should shrink: this is a precise description of where a node is weak.
Why not just scrape metrics
That is the right answer at scale and the wrong one at one operator per host. The gap is specifically the small deployment, where the monitor is an HTTPS probe from elsewhere and the metrics port is not reachable by design.
Related
Deployment-side monitors currently search the /health response for an outbox stalled key that does not appear to exist, which means the alarm silently never fires. That is being fixed on the deployment side by making a missing field loud instead of skipped — but the underlying reason the field was expected is this gap.
Counterpart to #136 — that one argues the unauthenticated
/healthexposes too much to anonymous callers. This one is that it exposes too little to a legitimate one.The situation
A node's most operationally interesting signals —
signing_failures_total,registry_outbox_stalled,ruleset_load_failures_total, and outbox depth — are on the Prometheus metrics endpoint, which binds to container loopback. A monitor outside the host cannot reach it, and at small scale there is no scraper: standing up Prometheus per operator to read three counters is not proportionate.So an external probe sees
status: okfor a node whose signing has been failing for a week, whose registry outbox has been stalled since Tuesday, or whose nightly backup has not completed in ten days. A green probe currently means "the node answers HTTP", not "the node is well" — and nothing communicates that distinction.What would close it
An authenticated health detail route that reports the counters an operator would page on, alongside what
/healthalready carries. Concretely: signing failures, ruleset load failures, registry outbox depth and stalled count, and a last-successful-backup timestamp if the node is in a position to know one.Authenticated matters for the same reason #136 argues the anonymous surface should shrink: this is a precise description of where a node is weak.
Why not just scrape metrics
That is the right answer at scale and the wrong one at one operator per host. The gap is specifically the small deployment, where the monitor is an HTTPS probe from elsewhere and the metrics port is not reachable by design.
Related
Deployment-side monitors currently search the
/healthresponse for an outboxstalledkey that does not appear to exist, which means the alarm silently never fires. That is being fixed on the deployment side by making a missing field loud instead of skipped — but the underlying reason the field was expected is this gap.