docs(security): record Loki's unbounded volume, and test the control (#189) - #319
Merged
Conversation
…189) #189 asks for one of three things chosen and written down. This is option 3, and it is the one the issue itself leans toward once its own reasoning is followed: the real answer is authentication in front of the ingest ports, which is #182's whole scope, so #189 is a second justification for that work rather than a separate piece of it. The argument worth preserving is that rate limits look like a fix and are not. The lab produces ~0.012 MB/s, so a deliberately tight 0.5 MB/s limit is 40x headroom and still fills 59 GiB in a day and a half. Any limit loose enough not to drop real logs is loose enough to fill the disk. That reasoning existed only in a GitHub issue; in SECURITY.md it stops the next person reaching for ingestion_rate_mb and calling it done. Measured position recorded with it: loki-data was 461 MB on 2026-08-31 and 549 MB on 2026-09-04, ~22 MB/day against 53 GiB free. Unbounded is a different property from close, and a later reader should be able to tell drift from noise without re-deriving the rate. HostDiskWillFillIn24h is the entire control under this option and had no test — host.rules.yaml had no test file at all. Accepting a residual on the strength of an unverified rule is #185's mistake with the stakes moved, so host.test.yaml is added with it: fires on a disk that is filling, quiet on one that is full but stable, quiet on one that is filling with room to spare. The inputs are real, unlike #185's CFS counters — node_filesystem_avail_bytes reports for prometheus, oracle and Saruman today — so the question was only whether the expression handles them, and all three mutations now fail the suite: dropping the predictive half, dropping the 30% guard, and inverting the prediction's sign. The roomy-but-filling case evaluates at 4h rather than 2h. Without the guard the projection first goes negative at 1h and the alert fires at 2h, so an eval there sits on the boundary and reports silence whether the guard exists or not — a quiet case that cannot fail is what this file exists to avoid. observability.md's coverage counts move again with the new tests, 28 to 29 tested and 28 to 27 not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #189 via its option 3, and adds the test that option 3 turns out to depend on.
Why this option
The issue's own conclusion, followed through: option 1 (authenticate the ingest ports) is the real answer, and it is #182's entire scope — both ports, every pushing Alloy agent, a credential to distribute and rotate. Doing it here would close #182 by accident with #189's narrower framing as the record of why. Option 2 bounds the blast radius without bounding the volume, and it is a host/LVM change that buys more of the thing there is most of: 53 GiB free against ~22 MB/day.
So: record it, and make #182's second justification explicit.
What SECURITY.md gains
A row in the exposure table beside the existing Prometheus/Loki residual, and a section carrying the part that was only ever in the issue:
That is the durable half. Without it the next person notices the gap, reaches for
ingestion_rate_mb, and calls it fixed.The measured position is recorded too — 461 MB on 2026-08-31, 549 MB on 2026-09-04 — so drift is distinguishable from noise later without re-deriving the rate. Unbounded is a different property from close.
The control had no test
HostDiskWillFillIn24his the whole control under this option.host.rules.yamlhad no test file at all, so it was as unverified asContainerCpuThrottledwas in #185 — and for the same reason nobody noticed.It is not in #185's position, though:
node_filesystem_avail_bytesandnode_filesystem_size_bytesreport forprometheus,oracleandSarumanright now, so the input exists and the only question was whether the expression handles it.host.test.yamlanswers that with three cases and mutation testing:Two corrections from doing it
Checks
validate.sh— 48 PASS, 0 FAIL, 2 SKIP.promtool test rulesnow covers 10 files in observability, up from 9. yamllint caught a 162-char annotation line, andcheck_docs.pycaught the coverage counts moving (28→29 tested, 28→27 not) — the third time that sentence has shifted this week, and the third time the guard from #212 caught it rather than a person.🤖 Generated with Claude Code