Scrape the UPS self-test schedule — nothing watches the control #93 leans on

[Gerrrt]

#93 closed with mjolnir's
management card set to test itself every fortnight. Nothing in this stack can
see that setting, so the control that keeps upsTestResultsSummary meaningful
is itself unmonitored.

The gap

Read off the card on 2026-09-03 with
scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2:

OID	Object	Reads
…7.2.1.0	upsAdvTestDiagnosticSchedule	8 — biweeklySinceLastTest
…7.2.3.0	upsAdvTestDiagnosticsResults	1 — ok
…7.2.4.0	upsAdvTestLastDiagnosticsDate	08/28/2026

All three are PowerNet (1.3.6.1.4.1.318). The apc_ups module walks
1.3.6.1.2.1.33.1 — the standard UPS-MIB — and nothing else.

So if the card drops back to never(5) — a config reset, a firmware update, a
restored configuration file, someone in the web UI — then:

• upsTestResultsSummary holds 1 (donePass) forever. It is the last result,
  and nothing refreshes it.
• Every rule in ups.rules.yaml stays quiet, correctly, because none of them
  can read a stale one.
• UpsBatteryUnproven cannot catch it: that rule matches 6 (noTestsInitiated)
  and this card reads 1.
• upsTestStartTime is sysUpTime-relative and does not survive an agent
  restart, so it is not a usable staleness signal either.

Note the shape of it. The missing battery pack was visible in a MIB already
walked — upsTestResultsSummary = 4 needed no new OIDs, which
docs/security.md makes a point of. The missing schedule would not be. This
is the same failure mode one level up: a healthy-looking metric with nothing
underneath it.

What to add

Three scalars, and only those three. Do not walk 1.3.6.1.4.1.318 — the
pfTablesAddrTable lesson in generator.yaml is what that costs. The subtree
1.3.6.1.4.1.318.1.1.1.7.2 is 7 rows and answered in 0.21s over two GETBULKs.

Two rules to go with them:

• UpsSelfTestScheduleOff — upsAdvTestDiagnosticSchedule in {1, 4, 5}
  (unknown, atTurnOn, never). Warning, category: power. This is the fast
  signal: the control is gone the moment it reads 5.
• UpsSelfTestStale — the last-test date has not moved in 21 days
  (14 + a week of grace). The backstop for a card that says it is scheduled and
  is not.

upsAdvTestDiagnosticSchedule is 1 unknown, 2 biweekly, 3 weekly, 4
atTurnOn, 5 never, 6 fourWeeks, 7 twelveWeeks, 8 biweeklySinceLastTest,
9 weeklySinceLastTest — from APC's own PowerNet-MIB, not the older
enumeration that stops at 7. Getting that wrong reads 8 as out-of-range.

The staleness rule needs no date parsing

upsAdvTestLastDiagnosticsDate is a DisplayString ("08/28/2026"), which
snmp_exporter renders as a value-1 gauge carrying the string as a label —
confirmed against what is already scraped:

upsIdentModel{device="mjolnir", upsIdentModel="Smart-UPS X 1500"} 1

Each distinct date is therefore a distinct series, so counting series over a
range counts tests:

count by (device) (
  last_over_time(upsAdvTestLastDiagnosticsDate[21d])
) == 1

One distinct date in 21 days means the date has not changed in 21 days. It
reads the stored series rather than counting pending time, so a Prometheus
restart does not reset it — the same reasoning UpsBatteryUnproven is built on,
and for the same reason a for: 21d would be useless here.

Two things to check when implementing: 21d must sit inside retention (30d /
12GB, and prometheus_tsdb_time_retentions_total was 0 last time it was
looked at), and the rule must be gated on the scrape being up so a dead exporter
does not read as a card that stopped testing.

The actual cost: a new vendor MIB source

scripts/snmp-mibs.sh fetches nothing from APC today, and adding it is a
pinning decision, not a URL. That file argues the case at length: the UPS-MIB
is pinned to a commit precisely because a moved MIB does not fail, it renders a
different snmp.yaml and the diff arrives looking like it came from nowhere.
The other three sources are first-party refs on projects that do not move
release tags.

PowerNet has no equivalent — Schneider's distribution is a versioned download,
not a git ref. Options, roughly in order of preference:

1. Vendor the file into the repository. It is one text file, unlike the 5.5 MB
   HPE bundle that motivated not vendoring; check the licence.
2. Pin the vendor download by version and verify a checksum in fetch().
3. Take it from snmp_exporter's own generator mibs, matching whatever upstream
   considers canonical, and pin the commit.

Whichever, make snmp-generate must be re-run and the metric-count guard will
compare before and after, as designed.

Done when

• scripts/snmp-mibs.sh fetches PowerNet, pinned, with the reasoning
  written down next to the existing pins
• apc_ups in generator.yaml walks the three OIDs and no more of 318
• snmp.yaml regenerated with make snmp-generate
• UpsSelfTestScheduleOff and UpsSelfTestStale in ups.rules.yaml, with
  a tests/ups.test.yaml — that file has no promtool tests today
• docs/runbooks/fit-the-ups-battery.md step 6 stops saying the check is by
  hand, and ups.rules.yaml's "WHAT THIS FILE STILL CANNOT SEE" header
  shrinks to what is still true
• docs/security.md — the paragraph that currently names this gap

Related

• #93 — where this came from
• docs/runbooks/fit-the-ups-battery.md
  step 6, which carries the by-hand check this replaces

One thing worth waiting for, or at least noting: the proof that the schedule
runs rather than merely being set is upsAdvTestLastDiagnosticsDate advancing
with nobody at the card, due around 2026-09-11. If it has not moved by then,
this issue stops being a monitoring gap and becomes a fault on the card.