docs(security): record Loki's unbounded volume, and test the control (#189)

SECURITY.md

@@ -47,6 +47,7 @@ is a very different thing from an overlooked one. Full detail in
 | Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. |
 | Prometheus and Loki published on `0.0.0.0` with no authentication | **Accepted residual, not a fix in progress.** Anything that can route to `10.0.99.20:9090` or `10.0.99.20:3100` can read every metric and log line, inject metrics through Prometheus' remote-write receiver, and delete log ranges through Loki's delete API. Both stay published because `oracle`'s Alloy agent remote-writes to 9090 and pushes to 3100 — it is not a scrape target, so those ports are its only path. Firewall default-deny is the whole control, and since 2026-09-02 it is narrower than it was: the ingest ports are reachable from Winterfell (99) itself and from `10.0.30.110` on ImaginationLAN, which has an explicit pass for `Saruman`'s Alloy agent. Hicks (50) reaches `10.0.99.20` on `3000` only — a logged *Block access to Winterfell* drops the rest — and no untrusted segment reaches it at all. `docs/network.md` lists what Hicks may reach. Closing it properly means authentication in front of the ingest ports and a credential on every agent, which is a separate piece of work — see below. |
 | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. |
+| Loki's log volume has no size ceiling | **Accepted residual, not a fix in progress** — see below. Prometheus has a byte ceiling since [#184](https://github.com/Gerrrt/HomeLab/issues/184); Loki has no size-based retention to set, only time. `HostDiskWillFillIn24h` is the control. |
 
 The switch is the honest gap, and it is a deliberate one. `neo` (10.7.7.2) is
 rotated and polling, but it also still accepts the community it held before the
@@ -114,6 +115,40 @@ was only serious *because* of this — an undo file at mode 664 on an encrypted
 disk is a much smaller problem, and the two facts are easy to lose track of
 separately.
 
+### Loki has no size ceiling, and rate limits would not give it one
+
+`--storage.tsdb.retention.size=12GB` bounds Prometheus. Loki has
+`retention_period: 720h` and no size equivalent, because Loki has no
+size-based retention to configure — so it is the only store on this disk
+bounded by time alone, fed by a push endpoint that is unauthenticated for the
+reason the ingest-port row above gives
+([#189](https://github.com/Gerrrt/HomeLab/issues/189)).
+
+**Rate limits are the obvious response and they do not solve it.** The lab
+produces roughly 0.012 MB/s of logs, so even a deliberately tight 0.5 MB/s
+`ingestion_rate_mb` is 40x headroom — and 0.5 MB/s still fills 59 GiB inside a
+day and a half. Any limit loose enough not to drop real logs is loose enough to
+fill the disk. There is no clean number, which is why one has not been picked.
+
+What actually bounds this is **who can write**, not how fast, and that is
+authentication in front of the ingest ports — the work in
+[#182](https://github.com/Gerrrt/HomeLab/issues/182), which this is a second
+justification for rather than a separate task. Until then the volume is bounded
+by trust in the segments that can reach `10.0.99.20:3100`, which is the same
+control and the same firewall rules as the row above.
+
+The measured position, so a later reader can tell drift from noise:
+`loki-data` was 461 MB on 2026-08-31 and 549 MB on 2026-09-04 — about 22 MB a
+day against 53 GiB free, which is years rather than months. Nothing here is
+urgent; it is unbounded, which is a different property from close.
+
+`HostDiskWillFillIn24h` is the whole control if that ever changes, so it is no
+longer an untested rule: `prometheus/tests/host.test.yaml` asserts it fires on a
+disk that is filling, and stays quiet both on one that is full but stable and on
+one that is filling with room to spare. That file was added with this residual
+and for it — a residual is only as good as the control it leans on, and that
+control had no test at all.
+
 ## What this repository will not contain
 
 Deliberate omissions, so their absence is not mistaken for an oversight:

docs/observability.md

@@ -360,14 +360,14 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
 `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
 feed a rule synthetic series and assert it fires — paired with a case asserting
 it stays quiet, because a test that only ever expects silence would have passed
-against the broken rule too. Coverage is twenty-eight rules of 56 so far — the five
+against the broken rule too. Coverage is twenty-nine rules of 56 so far — the five
 in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
 `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and
 `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from
 [#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in
 `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`,
-and `SuricataStopped`.
-The other 28 are still validated for syntax only, which is exactly the
+`SuricataStopped`, and `HostDiskWillFillIn24h`.
+The other 27 are still validated for syntax only, which is exactly the
 standing #63 had. Both numbers are checked by `scripts/check_docs.py` — the
 sentence they replaced claimed six and named two, and had been wrong for
 weeks.

stacks/observability/prometheus/tests/host.test.yaml

@@ -0,0 +1,105 @@
+---
+# Unit tests for host.rules.yaml.
+#
+# Added for #189, which proposes accepting Loki's unbounded growth as a residual
+# and relying on HostDiskWillFillIn24h to catch a disk actually filling. A
+# residual is only as good as the control it leans on, and this file had no
+# tests at all — so the control was as unverified as ContainerCpuThrottled was
+# in #185, and for the same reason nobody noticed: `promtool check rules` parses
+# PromQL and never asks whether an expression can be true.
+#
+# The rule is not in #185's position. node_filesystem_avail_bytes and
+# node_filesystem_size_bytes are real series that every Alloy agent here
+# produces — prometheus, oracle and Saruman all report them today — so the
+# input exists and the question is only whether the expression handles it. That
+# is what these cases answer.
+#
+# Every firing case is paired with a quiet one, per containers.test.yaml's
+# header: a test that only ever expects silence would pass against a rule that
+# can never fire.
+
+rule_files:
+  - ../rules/host.rules.yaml
+
+evaluation_interval: 5m
+
+tests:
+  # --- HostDiskWillFillIn24h: fires -----------------------------------------
+  # A 100 GB filesystem starting at 29% free and losing 200 MB every 5 minutes
+  # (2.4 GB/h). Both halves of the `and` have to hold:
+  #
+  #   predict_linear over 6h, extrapolated 24h, is negative — at 2.4 GB/h the
+  #   projection is 57.6 GB below where it stands, which it is from early on;
+  #   and free space is under 30%, which it is from the first sample.
+  #
+  # 29% and not 5%: the rule exists to catch a disk that is still comfortable
+  # and moving fast, which is exactly the shape #189 is about. A fixture already
+  # at 5% would pass against a rule that had lost its predictive half entirely.
+  - interval: 5m
+    input_series:
+      - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}'
+        values: "29000000000-200000000x96"
+      - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}'
+        values: "100000000000x96"
+    alert_rule_test:
+      - eval_time: 7h
+        alertname: HostDiskWillFillIn24h
+        exp_alerts:
+          - exp_labels:
+              alertname: HostDiskWillFillIn24h
+              instance: prometheus
+              device: /dev/mapper/ubuntu--vg-ubuntu--lv
+              mountpoint: /
+              fstype: ext4
+              job: integrations/node_exporter
+              component: host
+              severity: warning
+              category: capacity
+            exp_annotations:
+              summary: "prometheus:/ fills within 24h"
+              description: >-
+                / on prometheus (/dev/mapper/ubuntu--vg-ubuntu--lv) is below 30%
+                free and, extrapolating the last 6 hours, reaches zero within a
+                day.
+
+      # Not yet held long enough — for: 1h has not elapsed at 30m.
+      - eval_time: 30m
+        alertname: HostDiskWillFillIn24h
+        exp_alerts: []
+
+  # --- HostDiskWillFillIn24h: quiet on a full-but-stable disk ---------------
+  # 12% free and not moving. This is the case the rule was written to NOT fire
+  # on — "a disk sitting at 86% and stable is not an emergency" — and it is the
+  # half a static threshold would get wrong. predict_linear over a flat series
+  # is flat, so the extrapolation never reaches zero however low the level is.
+  - interval: 5m
+    input_series:
+      - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}'
+        values: "12000000000x96"
+      - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}'
+        values: "100000000000x96"
+    alert_rule_test:
+      - eval_time: 7h
+        alertname: HostDiskWillFillIn24h
+        exp_alerts: []
+
+  # --- HostDiskWillFillIn24h: quiet on a roomy disk that IS filling ----------
+  # Losing 200 MB every 5 minutes from 60% free — the same slope as the firing
+  # case, and the guard is what keeps it silent. This is the pair that proves
+  # the 30% condition is load-bearing rather than decorative: without it, every
+  # ordinary week of log growth on a half-empty disk would page.
+  #
+  # Evaluated at 4h and not 2h. Without the guard the projection first goes
+  # negative at 1h and the alert would fire at 2h, so an eval there lands on the
+  # exact boundary and would report silence either way — a quiet case that
+  # cannot fail is the thing this file exists to avoid.
+  - interval: 5m
+    input_series:
+      - series: 'node_filesystem_avail_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}'
+        values: "60000000000-200000000x96"
+      - series: 'node_filesystem_size_bytes{instance="prometheus",device="/dev/mapper/ubuntu--vg-ubuntu--lv",mountpoint="/",fstype="ext4",job="integrations/node_exporter"}'
+        values: "100000000000x96"
+    alert_rule_test:
+      - eval_time: 4h
+        alertname: HostDiskWillFillIn24h
+        exp_alerts: []