You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Every host in this estate is monitored by pushing. prometheus, oracle
and Saruman all run Alloy agents that remote_write to 10.0.99.20:9090 and
push to :3100; the only scraped things are the six containers on the compose
network, plus SNMP devices and blackbox endpoints behind their exporters.
ADR-0016
breaks that for the NAS on CasaBonita (40), and had to: a host on a terminal
segment must not initiate upward, so Prometheus reaches in and scrapes node_exporter over 99→40:9100 instead. That makes zion the estate's first
scraped host — the first target that is a machine rather than a container or
a device behind an exporter.
That is more than a line in a YAML file, which is why it is not just a checkbox
on #102.
What it needs
A node job in prometheus.yaml with file_sd_configs, following the
shape the snmp and blackbox jobs already use — the comment above the
SNMP job explains why file_sd rather than static targets, and the same
reasoning applies.
prometheus/targets/node.yaml, with a header in the house style: what
each label means and who reads it. blackbox.yaml's header is the model.
Decide the label set. snmp.yaml carries device, role and vlan;
whatever this file carries has to work with the existing host-overview
dashboard, which is built on node-exporter metrics and currently sees
only what Alloy pushes.
Alert rules. A pushed host that dies stops writing and existing rules
notice; a scraped host that dies produces up == 0, which is a different
signal and may not be covered. Check whether anything currently fires on up == 0 for a non-container target — if it does not, this host would
go dark silently, which is the exact class of fault #63 and #62 were.
A promtool test rules case for whatever rule that turns out to be. The
repo has unit tests for exactly one rule today, and the reason it has them
is that syntax-checking cannot tell you an expression can never be true.
Consider whether docs/runbooks/add-monitored-device.md grows a second
path, or whether a scraped host is different enough to want its own.
The ordering trap
The scrape target cannot be written before the firewall rule exists, and the
firewall rule should not be created before the host answers on that address —
ADR-0016 declines to create it for exactly that reason: a pass to an address
with nothing behind it is a rule nobody can test. So the sequence is hardware →
rule → target → alert, and each step is testable only after the one before it.
Writing the YAML early gets a target that has never been scraped, which is how blackbox.yaml first shipped probing the wiki over the wrong scheme.
Blocked on the hardware in #102.
Found writing ADR-0016 for #95.
Every host in this estate is monitored by pushing.
prometheus,oracleand
Sarumanall run Alloy agents thatremote_writeto10.0.99.20:9090andpush to
:3100; the only scraped things are the six containers on the composenetwork, plus SNMP devices and blackbox endpoints behind their exporters.
ADR-0016
breaks that for the NAS on CasaBonita (40), and had to: a host on a terminal
segment must not initiate upward, so Prometheus reaches in and scrapes
node_exporterover 99→40:9100 instead. That makeszionthe estate's firstscraped host — the first target that is a machine rather than a container or
a device behind an exporter.
That is more than a line in a YAML file, which is why it is not just a checkbox
on #102.
What it needs
nodejob inprometheus.yamlwithfile_sd_configs, following theshape the
snmpandblackboxjobs already use — the comment above theSNMP job explains why file_sd rather than static targets, and the same
reasoning applies.
prometheus/targets/node.yaml, with a header in the house style: whateach label means and who reads it.
blackbox.yaml's header is the model.snmp.yamlcarriesdevice,roleandvlan;whatever this file carries has to work with the existing host-overview
dashboard, which is built on
node-exportermetrics and currently seesonly what Alloy pushes.
notice; a scraped host that dies produces
up == 0, which is a differentsignal and may not be covered. Check whether anything currently fires on
up == 0for a non-container target — if it does not, this host wouldgo dark silently, which is the exact class of fault
#63 and
#62 were.
promtool test rulescase for whatever rule that turns out to be. Therepo has unit tests for exactly one rule today, and the reason it has them
is that syntax-checking cannot tell you an expression can never be true.
docs/runbooks/add-monitored-device.mdgrows a secondpath, or whether a scraped host is different enough to want its own.
The ordering trap
The scrape target cannot be written before the firewall rule exists, and the
firewall rule should not be created before the host answers on that address —
ADR-0016 declines to create it for exactly that reason: a
passto an addresswith nothing behind it is a rule nobody can test. So the sequence is hardware →
rule → target → alert, and each step is testable only after the one before it.
Writing the YAML early gets a target that has never been scraped, which is how
blackbox.yamlfirst shipped probing the wiki over the wrong scheme.Blocked on the hardware in #102.
Found writing ADR-0016 for #95.