From 3b20dc905468a259b98742085bacfeb596f3fee5 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 4 Sep 2026 23:33:38 +0000 Subject: [PATCH] docs(security): record Loki's unbounded volume, and test the control (#189) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #189 asks for one of three things chosen and written down. This is option 3, and it is the one the issue itself leans toward once its own reasoning is followed: the real answer is authentication in front of the ingest ports, which is #182's whole scope, so #189 is a second justification for that work rather than a separate piece of it. The argument worth preserving is that rate limits look like a fix and are not. The lab produces ~0.012 MB/s, so a deliberately tight 0.5 MB/s limit is 40x headroom and still fills 59 GiB in a day and a half. Any limit loose enough not to drop real logs is loose enough to fill the disk. That reasoning existed only in a GitHub issue; in SECURITY.md it stops the next person reaching for ingestion_rate_mb and calling it done. Measured position recorded with it: loki-data was 461 MB on 2026-08-31 and 549 MB on 2026-09-04, ~22 MB/day against 53 GiB free. Unbounded is a different property from close, and a later reader should be able to tell drift from noise without re-deriving the rate. HostDiskWillFillIn24h is the entire control under this option and had no test — host.rules.yaml had no test file at all. Accepting a residual on the strength of an unverified rule is #185's mistake with the stakes moved, so host.test.yaml is added with it: fires on a disk that is filling, quiet on one that is full but stable, quiet on one that is filling with room to spare. The inputs are real, unlike #185's CFS counters — node_filesystem_avail_bytes reports for prometheus, oracle and Saruman today — so the question was only whether the expression handles them, and all three mutations now fail the suite: dropping the predictive half, dropping the 30% guard, and inverting the prediction's sign. The roomy-but-filling case evaluates at 4h rather than 2h. Without the guard the projection first goes negative at 1h and the alert fires at 2h, so an eval there sits on the boundary and reports silence whether the guard exists or not — a quiet case that cannot fail is what this file exists to avoid. observability.md's coverage counts move again with the new tests, 28 to 29 tested and 28 to 27 not. Co-Authored-By: Claude Opus 5 --- SECURITY.md | 35 ++++++ docs/observability.md | 6 +- .../prometheus/tests/host.test.yaml | 105 ++++++++++++++++++ 3 files changed, 143 insertions(+), 3 deletions(-) create mode 100644 stacks/observability/prometheus/tests/host.test.yaml diff --git a/SECURITY.md b/SECURITY.md index 67a5c76..c6826da 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -47,6 +47,7 @@ is a very different thing from an overlooked one. Full detail in | Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. | | Prometheus and Loki published on `0.0.0.0` with no authentication | **Accepted residual, not a fix in progress.** Anything that can route to `10.0.99.20:9090` or `10.0.99.20:3100` can read every metric and log line, inject metrics through Prometheus' remote-write receiver, and delete log ranges through Loki's delete API. Both stay published because `oracle`'s Alloy agent remote-writes to 9090 and pushes to 3100 — it is not a scrape target, so those ports are its only path. Firewall default-deny is the whole control, and since 2026-09-02 it is narrower than it was: the ingest ports are reachable from Winterfell (99) itself and from `10.0.30.110` on ImaginationLAN, which has an explicit pass for `Saruman`'s Alloy agent. Hicks (50) reaches `10.0.99.20` on `3000` only — a logged *Block access to Winterfell* drops the rest — and no untrusted segment reaches it at all. `docs/network.md` lists what Hicks may reach. Closing it properly means authentication in front of the ingest ports and a credential on every agent, which is a separate piece of work — see below. | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | +| Loki's log volume has no size ceiling | **Accepted residual, not a fix in progress** — see below. Prometheus has a byte ceiling since [#184](https://github.com/Gerrrt/HomeLab/issues/184); Loki has no size-based retention to set, only time. `HostDiskWillFillIn24h` is the control. | The switch is the honest gap, and it is a deliberate one. `neo` (10.7.7.2) is rotated and polling, but it also still accepts the community it held before the @@ -114,6 +115,40 @@ was only serious *because* of this — an undo file at mode 664 on an encrypted disk is a much smaller problem, and the two facts are easy to lose track of separately. +### Loki has no size ceiling, and rate limits would not give it one + +`--storage.tsdb.retention.size=12GB` bounds Prometheus. Loki has +`retention_period: 720h` and no size equivalent, because Loki has no +size-based retention to configure — so it is the only store on this disk +bounded by time alone, fed by a push endpoint that is unauthenticated for the +reason the ingest-port row above gives +([#189](https://github.com/Gerrrt/HomeLab/issues/189)). + +**Rate limits are the obvious response and they do not solve it.** The lab +produces roughly 0.012 MB/s of logs, so even a deliberately tight 0.5 MB/s +`ingestion_rate_mb` is 40x headroom — and 0.5 MB/s still fills 59 GiB inside a +day and a half. Any limit loose enough not to drop real logs is loose enough to +fill the disk. There is no clean number, which is why one has not been picked. + +What actually bounds this is **who can write**, not how fast, and that is +authentication in front of the ingest ports — the work in +[#182](https://github.com/Gerrrt/HomeLab/issues/182), which this is a second +justification for rather than a separate task. Until then the volume is bounded +by trust in the segments that can reach `10.0.99.20:3100`, which is the same +control and the same firewall rules as the row above. + +The measured position, so a later reader can tell drift from noise: +`loki-data` was 461 MB on 2026-08-31 and 549 MB on 2026-09-04 — about 22 MB a +day against 53 GiB free, which is years rather than months. Nothing here is +urgent; it is unbounded, which is a different property from close. + +`HostDiskWillFillIn24h` is the whole control if that ever changes, so it is no +longer an untested rule: `prometheus/tests/host.test.yaml` asserts it fires on a +disk that is filling, and stays quiet both on one that is full but stable and on +one that is filling with room to spare. That file was added with this residual +and for it — a residual is only as good as the control it leans on, and that +control had no test at all. + ## What this repository will not contain Deliberate omissions, so their absence is not mistaken for an oversight: diff --git a/docs/observability.md b/docs/observability.md index c9daa3b..6500ea8 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -360,14 +360,14 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is twenty-eight rules of 56 so far — the five +against the broken rule too. Coverage is twenty-nine rules of 56 so far — the five in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from [#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, -and `SuricataStopped`. -The other 28 are still validated for syntax only, which is exactly the +`SuricataStopped`, and `HostDiskWillFillIn24h`. +The other 27 are still validated for syntax only, which is exactly the standing #63 had. Both numbers are checked by `scripts/check_docs.py` — the sentence they replaced claimed six and named two, and had been wrong for weeks. diff --git a/stacks/observability/prometheus/tests/host.test.yaml b/stacks/observability/prometheus/tests/host.test.yaml new file mode 100644 index 0000000..f00f7f2 --- /dev/null +++ b/stacks/observability/prometheus/tests/host.test.yaml @@ -0,0 +1,105 @@ +--- +# Unit tests for host.rules.yaml. +# +# Added for #189, which proposes accepting Loki's unbounded growth as a residual +# and relying on HostDiskWillFillIn24h to catch a disk actually filling. A +# residual is only as good as the control it leans on, and this file had no +# tests at all — so the control was as unverified as ContainerCpuThrottled was +# in #185, and for the same reason nobody noticed: `promtool check rules` parses +# PromQL and never asks whether an expression can be true. +# +# The rule is not in #185's position. node_filesystem_avail_bytes and +# node_filesystem_size_bytes are real series that every Alloy agent here +# produces — prometheus, oracle and Saruman all report them today — so the +# input exists and the question is only whether the expression handles it. That +# is what these cases answer. +# +# Every firing case is paired with a quiet one, per containers.test.yaml's +# header: a test that only ever expects silence would pass against a rule that +# can never fire. + +rule_files: + - ../rules/host.rules.yaml + +evaluation_interval: 5m + +tests: + # --- HostDiskWillFillIn24h: fires ----------------------------------------- + # A 100 GB filesystem starting at 29% free and losing 200 MB every 5 minutes + # (2.4 GB/h). Both halves of the `and` have to hold: + # + # predict_linear over 6h, extrapolated 24h, is negative — at 2.4 GB/h the + # projection is 57.6 GB below where it stands, which it is from early on; + # and free space is under 30%, which it is from the first sample. + # + # 29% and not 5%: the rule exists to catch a disk that is still comfortable + # and moving fast, which is exactly the shape #189 is about. A fixture already + # at 5% would pass against a rule that had lost its predictive half entirely. + - interval: 5m + input_series: + - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "29000000000-200000000x96" + - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "100000000000x96" + alert_rule_test: + - eval_time: 7h + alertname: HostDiskWillFillIn24h + exp_alerts: + - exp_labels: + alertname: HostDiskWillFillIn24h + instance: prometheus + device: /dev/mapper/ubuntu--vg-ubuntu--lv + mountpoint: / + fstype: ext4 + job: integrations/node_exporter + component: host + severity: warning + category: capacity + exp_annotations: + summary: "prometheus:/ fills within 24h" + description: >- + / on prometheus (/dev/mapper/ubuntu--vg-ubuntu--lv) is below 30% + free and, extrapolating the last 6 hours, reaches zero within a + day. + + # Not yet held long enough — for: 1h has not elapsed at 30m. + - eval_time: 30m + alertname: HostDiskWillFillIn24h + exp_alerts: [] + + # --- HostDiskWillFillIn24h: quiet on a full-but-stable disk --------------- + # 12% free and not moving. This is the case the rule was written to NOT fire + # on — "a disk sitting at 86% and stable is not an emergency" — and it is the + # half a static threshold would get wrong. predict_linear over a flat series + # is flat, so the extrapolation never reaches zero however low the level is. + - interval: 5m + input_series: + - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "12000000000x96" + - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "100000000000x96" + alert_rule_test: + - eval_time: 7h + alertname: HostDiskWillFillIn24h + exp_alerts: [] + + # --- HostDiskWillFillIn24h: quiet on a roomy disk that IS filling ---------- + # Losing 200 MB every 5 minutes from 60% free — the same slope as the firing + # case, and the guard is what keeps it silent. This is the pair that proves + # the 30% condition is load-bearing rather than decorative: without it, every + # ordinary week of log growth on a half-empty disk would page. + # + # Evaluated at 4h and not 2h. Without the guard the projection first goes + # negative at 1h and the alert would fire at 2h, so an eval there lands on the + # exact boundary and would report silence either way — a quiet case that + # cannot fail is the thing this file exists to avoid. + - interval: 5m + input_series: + - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "60000000000-200000000x96" + - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}' + values: "100000000000x96" + alert_rule_test: + - eval_time: 4h + alertname: HostDiskWillFillIn24h + exp_alerts: []