diff --git a/SECURITY.md b/SECURITY.md index 67a5c76..c6826da 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -47,6 +47,7 @@ is a very different thing from an overlooked one. Full detail in | Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. | | Prometheus and Loki published on `0.0.0.0` with no authentication | **Accepted residual, not a fix in progress.** Anything that can route to `10.0.99.20:9090` or `10.0.99.20:3100` can read every metric and log line, inject metrics through Prometheus' remote-write receiver, and delete log ranges through Loki's delete API. Both stay published because `oracle`'s Alloy agent remote-writes to 9090 and pushes to 3100 — it is not a scrape target, so those ports are its only path. Firewall default-deny is the whole control, and since 2026-09-02 it is narrower than it was: the ingest ports are reachable from Winterfell (99) itself and from `10.0.30.110` on ImaginationLAN, which has an explicit pass for `Saruman`'s Alloy agent. Hicks (50) reaches `10.0.99.20` on `3000` only — a logged *Block access to Winterfell* drops the rest — and no untrusted segment reaches it at all. `docs/network.md` lists what Hicks may reach. Closing it properly means authentication in front of the ingest ports and a credential on every agent, which is a separate piece of work — see below. | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | +| Loki's log volume has no size ceiling | **Accepted residual, not a fix in progress** — see below. Prometheus has a byte ceiling since [#184](https://github.com/Gerrrt/HomeLab/issues/184); Loki has no size-based retention to set, only time. `HostDiskWillFillIn24h` is the control. | The switch is the honest gap, and it is a deliberate one. `neo` (10.7.7.2) is rotated and polling, but it also still accepts the community it held before the @@ -114,6 +115,40 @@ was only serious *because* of this — an undo file at mode 664 on an encrypted disk is a much smaller problem, and the two facts are easy to lose track of separately. +### Loki has no size ceiling, and rate limits would not give it one + +`--storage.tsdb.retention.size=12GB` bounds Prometheus. Loki has +`retention_period: 720h` and no size equivalent, because Loki has no +size-based retention to configure — so it is the only store on this disk +bounded by time alone, fed by a push endpoint that is unauthenticated for the +reason the ingest-port row above gives +([#189](https://github.com/Gerrrt/HomeLab/issues/189)). + +**Rate limits are the obvious response and they do not solve it.** The lab +produces roughly 0.012 MB/s of logs, so even a deliberately tight 0.5 MB/s +`ingestion_rate_mb` is 40x headroom — and 0.5 MB/s still fills 59 GiB inside a +day and a half. Any limit loose enough not to drop real logs is loose enough to +fill the disk. There is no clean number, which is why one has not been picked. + +What actually bounds this is **who can write**, not how fast, and that is +authentication in front of the ingest ports — the work in +[#182](https://github.com/Gerrrt/HomeLab/issues/182), which this is a second +justification for rather than a separate task. Until then the volume is bounded +by trust in the segments that can reach `10.0.99.20:3100`, which is the same +control and the same firewall rules as the row above. + +The measured position, so a later reader can tell drift from noise: +`loki-data` was 461 MB on 2026-08-31 and 549 MB on 2026-09-04 — about 22 MB a +day against 53 GiB free, which is years rather than months. Nothing here is +urgent; it is unbounded, which is a different property from close. + +`HostDiskWillFillIn24h` is the whole control if that ever changes, so it is no +longer an untested rule: `prometheus/tests/host.test.yaml` asserts it fires on a +disk that is filling, and stays quiet both on one that is full but stable and on +one that is filling with room to spare. That file was added with this residual +and for it — a residual is only as good as the control it leans on, and that +control had no test at all. + ## What this repository will not contain Deliberate omissions, so their absence is not mistaken for an oversight: diff --git a/docs/observability.md b/docs/observability.md index c9daa3b..6500ea8 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -360,14 +360,14 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is twenty-eight rules of 56 so far — the five +against the broken rule too. Coverage is twenty-nine rules of 56 so far — the five in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from [#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, -and `SuricataStopped`. -The other 28 are still validated for syntax only, which is exactly the +`SuricataStopped`, and `HostDiskWillFillIn24h`. +The other 27 are still validated for syntax only, which is exactly the standing #63 had. Both numbers are checked by `scripts/check_docs.py` — the sentence they replaced claimed six and named two, and had been wrong for weeks. diff --git a/stacks/observability/prometheus/tests/host.test.yaml b/stacks/observability/prometheus/tests/host.test.yaml new file mode 100644 index 0000000..f00f7f2 --- /dev/null +++ b/stacks/observability/prometheus/tests/host.test.yaml @@ -0,0 +1,105 @@ +--- +# Unit tests for host.rules.yaml. +# +# Added for #189, which proposes accepting Loki's unbounded growth as a residual +# and relying on HostDiskWillFillIn24h to catch a disk actually filling. A +# residual is only as good as the control it leans on, and this file had no +# tests at all — so the control was as unverified as ContainerCpuThrottled was +# in #185, and for the same reason nobody noticed: `promtool check rules` parses +# PromQL and never asks whether an expression can be true. +# +# The rule is not in #185's position. node_filesystem_avail_bytes and +# node_filesystem_size_bytes are real series that every Alloy agent here +# produces — prometheus, oracle and Saruman all report them today — so the +# input exists and the question is only whether the expression handles it. That +# is what these cases answer. +# +# Every firing case is paired with a quiet one, per containers.test.yaml's +# header: a test that only ever expects silence would pass against a rule that +# can never fire. + +rule_files: + - ../rules/host.rules.yaml + +evaluation_interval: 5m + +tests: + # --- HostDiskWillFillIn24h: fires ----------------------------------------- + # A 100 GB filesystem starting at 29% free and losing 200 MB every 5 minutes + # (2.4 GB/h). Both halves of the `and` have to hold: + # + # predict_linear over 6h, extrapolated 24h, is negative — at 2.4 GB/h the + # projection is 57.6 GB below where it stands, which it is from early on; + # and free space is under 30%, which it is from the first sample. + # + # 29% and not 5%: the rule exists to catch a disk that is still comfortable + # and moving fast, which is exactly the shape #189 is about. A fixture already + # at 5% would pass against a rule that had lost its predictive half entirely. + - interval: 5m + input_series: + - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "29000000000-200000000x96" + - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "100000000000x96" + alert_rule_test: + - eval_time: 7h + alertname: HostDiskWillFillIn24h + exp_alerts: + - exp_labels: + alertname: HostDiskWillFillIn24h + instance: prometheus + device: /dev/mapper/ubuntu--vg-ubuntu--lv + mountpoint: / + fstype: ext4 + job: integrations/node_exporter + component: host + severity: warning + category: capacity + exp_annotations: + summary: "prometheus:/ fills within 24h" + description: >- + / on prometheus (/dev/mapper/ubuntu--vg-ubuntu--lv) is below 30% + free and, extrapolating the last 6 hours, reaches zero within a + day. + + # Not yet held long enough — for: 1h has not elapsed at 30m. + - eval_time: 30m + alertname: HostDiskWillFillIn24h + exp_alerts: [] + + # --- HostDiskWillFillIn24h: quiet on a full-but-stable disk --------------- + # 12% free and not moving. This is the case the rule was written to NOT fire + # on — "a disk sitting at 86% and stable is not an emergency" — and it is the + # half a static threshold would get wrong. predict_linear over a flat series + # is flat, so the extrapolation never reaches zero however low the level is. + - interval: 5m + input_series: + - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "12000000000x96" + - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "100000000000x96" + alert_rule_test: + - eval_time: 7h + alertname: HostDiskWillFillIn24h + exp_alerts: [] + + # --- HostDiskWillFillIn24h: quiet on a roomy disk that IS filling ---------- + # Losing 200 MB every 5 minutes from 60% free — the same slope as the firing + # case, and the guard is what keeps it silent. This is the pair that proves + # the 30% condition is load-bearing rather than decorative: without it, every + # ordinary week of log growth on a half-empty disk would page. + # + # Evaluated at 4h and not 2h. Without the guard the projection first goes + # negative at 1h and the alert would fire at 2h, so an eval there lands on the + # exact boundary and would report silence either way — a quiet case that + # cannot fail is the thing this file exists to avoid. + - interval: 5m + input_series: + - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "60000000000-200000000x96" + - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "100000000000x96" + alert_rule_test: + - eval_time: 4h + alertname: HostDiskWillFillIn24h + exp_alerts: []