feat(lab): build stacks/lab — Prometheus, Loki, Grafana, Alloy on a guest (#264)

[Gerrrt]

Builds #264. Shape settled by
ADR-0020
(numbered 0019 when it merged; main renumbered it).

Two commits, each independently green:

1. refactor(scripts) — four scripts made stack-aware. Nothing in it
   depends on the new stack existing.
2. feat(lab) — the stack, its docs rows, and the one checker change they
   need.

What the stack is

Four services on alexander (10.0.30.40), a guest on Saruman — not the
hypervisor, because a compose stack is Docker and Docker rewrites the iptables
of the box whose own firewall ADR-0014 relies on.

prometheus is the fourth service ADR-0007 did not name. That ADR listed three
while also saying config.alloy is reused with only the two *_URL variables
changed, and that file has two sinks — so without a lab Prometheus the second
points at 10.0.99.20, which the Decision forbids.

The agent config is mounted, not copied. compose.yaml bind-mounts
config.alloy and docker.alloy out of ../observability/alloy/. Two files
rather than the directory, so syslog.alloy — the firewall's log listener —
stays on the monitoring host. deploy-agent.sh exists because oracle drifted
four separate ways from a hand-copied agent setup, and its header states the
rule: "the fix is to not copy."

Absent on purpose, each with the reasoning in the file: no Alertmanager, no
snmp-exporter, no blackbox-exporter, no renderer, no dashboards. On the
last: copying the estate's seven would render four rows of empty panels and one
that works, and an empty panel is indistinguishable from a broken collector —
which is the failure this repo has been bitten by most (#62, #63, #71).

Prometheus and Loki are exposed, not published. ADR-0012 publishes a port
only when something off-host uses it, and today only Alloy talks to them;
compose.yaml marks the exact lines to uncomment when
#265 gives them real clients.

Retention is 15d/4GB and is a bound, not a measurement — the stack has
never run, so there is nothing to derive from. compose.yaml carries the
queries to re-derive it against the spindles, and PrometheusSizeRetentionActive
is what says the ceiling started binding.

The tooling had to move first

Four scripts assumed one stack. Each would have failed, or quietly done the
wrong thing, the first time a second existed:

render-config.sh	Demanded all ten estate keys from any stack, so make render STACK=lab died naming four SNMP communities and four Alertmanager URLs the lab has no use for. Now derived — the Grafana half from the stack's own ${VAR:?} guards, SNMP and Alertmanager conditional on their input files.
reload-config.sh	Reloaded a SERVICES array that is the union across stacks, dying on the first service a stack lacks — through the "is not running" path, which is right for a stopped service and wrong for an absent one. Now filters on compose config --services, keeping those two cases separate.
bootstrap.sh	Filled in "the first placeholder found" in .sops.yaml. Now finds this stack's, and refuses when the key it would write is already a recipient of another rule.
backup-firewall.sh	Read the recipient as the first age1 key in .sops.yaml. Correct with one rule; with the lab rule above it, head -1 would encrypt every firewall export to the lab guest the moment that placeholder became real. Now reads the key from the rule covering backups/firewall/.

.sops.yaml gets the lab rule ADR-0020 asked for, above the catch-all — the
catch-all matches all of secrets/, so a recipient added there could decrypt
the estate's SNMP communities and Grafana admin password.

The render-config.sh change touches the live estate deploy, so it was
verified rather than reasoned about: the derived list for
stacks/observability is byte-identical to the ten-name array on main,
checked by running the real code path from the file against both stacks.

One checker change, and it adds assertions

check_host_stack_table fails on any stacks/ directory no architecture.md
row names, and those rows must match a host in network.md. alexander is
#262 and is not on the wire, so
writing it into network.md would be exactly the false claim check_docs.py
exists to catch — and would mean inventing a MAC, a device and an OS for a
machine whose distribution is explicitly undecided.

So the row is marked **Not built yet**, and the marker inverts the check
rather than switching it off:

• a normal row's host must appear in network.md;
• a marked row's host must be absent from it;
• the VLAN must be one network.md has a table for;
• the address must not collide with a host already there.

That makes the marker self-clearing — rack the host, add its network.md row,
and the check fails saying the marker is stale. It also drops such a row from
count_alloy_agents, so removing the marker pushes that count to four and
fails hardware.md's "three Alloy agents" on the same commit, which is when
that sentence should be forced to change.

All three new branches were verified to fire by temporarily introducing each
fault, not just by reading the code.

Verification

• make validate — all checks passed, 2 skips (the usual "not the deployment
  checkout" pair).
• docker compose -f stacks/lab/compose.yaml config -q — valid; config --services returns exactly the four, which is what the new
  reload-config.sh filter reads.
• promtool check config, check rules, test rules against the lab's files
  in the pinned image — all SUCCESS. Four rules, each with a firing case and a
  quiet case, because a case that only expects silence passes against a rule
  that cannot fire (ContainerHighMemory cannot fire — no service sets a memory limit #63).
• Commit 1 verified green on its own, with commit 2 stashed.

Not done, and written down rather than left to be discovered

• No validator runs against this stack.
  #263 is untouched — this PR
  deliberately did not widen its scope into it.
• Nothing converges it. #99
  landed while this was in progress; converge.sh runs a bare make up, which
  is STACK=observability on the monitoring host. The lab is applied by hand.
• Nothing outside the lab knows if it dies —
  #257, unchanged. ADR-0020
  decided only that no Alertmanager goes inside the stack.
• It is not deployed. VLAN 30 is not reachable from the monitoring host,
  and the guest does not exist yet (Build the guest on Saruman that stacks/lab/ runs in #262). Nothing on Saruman was touched.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

It changes stack-critical operational scripts and SOPS key-selection behavior, which warrants a final human review of failure modes despite only minor requested adjustments.

Pull request overview

This PR introduces a new stacks/lab compose stack (Prometheus, Loki, Grafana, Alloy) intended to run on a VLAN 30 guest host, and refactors existing tooling/docs to support multiple stacks without leaking lab secrets or misapplying stack-specific operations.

Changes:

• Add stacks/lab/ with compose, configs, Prometheus rules + promtool unit tests, and stack-specific documentation.
• Make key operational scripts stack-aware (render-config.sh, reload-config.sh, bootstrap.sh, backup-firewall.sh) and update .sops.yaml to isolate lab secrets with a dedicated creation rule.
• Update documentation to describe the new stack and support “planned/not built yet” host rows with additional assertions in check_docs.py.

File summaries

File	Description
stacks/lab/README.md	Documents the lab stack purpose, layout, and manual validation steps.
stacks/lab/prometheus/tests/lab.test.yaml	Adds promtool unit tests covering all lab alert rules (firing + quiet cases).
stacks/lab/prometheus/rules/lab.rules.yaml	Introduces lab self-monitoring alert rules (no paging / no Alertmanager).
stacks/lab/prometheus/prometheus.yaml	Lab Prometheus scrape config and external labels (segment identification).
stacks/lab/loki/loki-config.yaml	Lab Loki single-binary filesystem config with 15d retention and no ruler.
stacks/lab/grafana/provisioning/datasources/datasources.yaml	Provisions Prometheus + Loki datasources for the lab Grafana.
stacks/lab/grafana/provisioning/dashboards/dashboards.yaml	Provisions an intentionally empty dashboards directory (future dashboards).
stacks/lab/grafana/dashboards/README.md	Explains why dashboards are intentionally absent and how to add them later.
stacks/lab/compose.yaml	Defines the lab compose stack (4 services) with retention bounds and isolation.
stacks/lab/.env.example	Adds non-sensitive tunables for the lab stack (ports/retention).
secrets/lab.example.yaml	Adds lab secrets template (Grafana admin password) and key-isolation guidance.
scripts/render-config.sh	Derives required keys per-stack (compose guards + conditional SNMP/AM).
scripts/reload-config.sh	Filters reload targets to only services declared by the selected stack.
scripts/check_docs.py	Adds “Not built yet” handling/inverted checks and excludes planned agents from counts.
scripts/bootstrap.sh	Makes SOPS placeholder selection stack-specific; refuses collapsing stack recipients.
scripts/backup-firewall.sh	Reads SOPS age recipient from the rule covering backups/firewall/ (not first key).
README.md	Updates repo tree overview to include stacks/lab.
docs/roadmap.md	Updates roadmap status to reflect lab stack now built and tooling updated.
docs/observability.md	Clarifies this doc is estate-only; notes existence and isolation of lab stack.
docs/network.md	Adds planned guest narrative while keeping inventory table accurate.
docs/architecture.md	Adds host/stack mapping row for alexander marked “Not built yet”.
.sops.yaml	Adds dedicated creation rule for lab secrets above catch-all to prevent trust inversion.
.github/dependabot.yml	Adds Dependabot updates for /stacks/lab docker-compose pins.

Review details

• Files reviewed: 23/23 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.