Skip to content

Add the scrape path for a host that pushes nothing #256

Description

@Gerrrt

Every host in this estate is monitored by pushing. prometheus, oracle
and Saruman all run Alloy agents that remote_write to 10.0.99.20:9090 and
push to :3100; the only scraped things are the six containers on the compose
network, plus SNMP devices and blackbox endpoints behind their exporters.

ADR-0016
breaks that for the NAS on CasaBonita (40), and had to: a host on a terminal
segment must not initiate upward, so Prometheus reaches in and scrapes
node_exporter over 99→40:9100 instead. That makes zion the estate's first
scraped host
— the first target that is a machine rather than a container or
a device behind an exporter.

That is more than a line in a YAML file, which is why it is not just a checkbox
on #102.

What it needs

  • A node job in prometheus.yaml with file_sd_configs, following the
    shape the snmp and blackbox jobs already use — the comment above the
    SNMP job explains why file_sd rather than static targets, and the same
    reasoning applies.
  • prometheus/targets/node.yaml, with a header in the house style: what
    each label means and who reads it. blackbox.yaml's header is the model.
  • Decide the label set. snmp.yaml carries device, role and vlan;
    whatever this file carries has to work with the existing host-overview
    dashboard, which is built on node-exporter metrics and currently sees
    only what Alloy pushes.
  • Alert rules. A pushed host that dies stops writing and existing rules
    notice; a scraped host that dies produces up == 0, which is a different
    signal and may not be covered. Check whether anything currently fires on
    up == 0 for a non-container target
    — if it does not, this host would
    go dark silently, which is the exact class of fault
    #63 and
    #62 were.
  • A promtool test rules case for whatever rule that turns out to be. The
    repo has unit tests for exactly one rule today, and the reason it has them
    is that syntax-checking cannot tell you an expression can never be true.
  • Consider whether docs/runbooks/add-monitored-device.md grows a second
    path, or whether a scraped host is different enough to want its own.

The ordering trap

The scrape target cannot be written before the firewall rule exists, and the
firewall rule should not be created before the host answers on that address —
ADR-0016 declines to create it for exactly that reason: a pass to an address
with nothing behind it is a rule nobody can test. So the sequence is hardware →
rule → target → alert, and each step is testable only after the one before it.
Writing the YAML early gets a target that has never been scraped, which is how
blackbox.yaml first shipped probing the wiki over the wrong scheme.

Blocked on the hardware in #102.
Found writing ADR-0016 for #95.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions