feat(lab): build stacks/lab — Prometheus, Loki, Grafana, Alloy on a guest (#264)

.github/dependabot.yml

@@ -19,6 +19,28 @@ updates:
       prometheus-stack:
         patterns: ["prom/*"]
 
+  # The lab stack. A second entry rather than a second directory on the one
+  # above, because Dependabot takes a single directory per entry — and without
+  # it the lab's pins would rot exactly the way the comment above says pinned
+  # tags do, while the estate's stayed current. The two run the same four
+  # images, so they are expected to move together; they get separate PRs
+  # because they are separate deploys to separate hosts, and the lab's is a
+  # host the operator has to be sitting in front of.
+  - package-ecosystem: docker-compose
+    directory: /stacks/lab
+    schedule:
+      interval: weekly
+      day: sunday
+    open-pull-requests-limit: 5
+    commit-message:
+      prefix: "chore(deps)"
+    labels: ["dependencies", "lab"]
+    groups:
+      grafana-stack:
+        patterns: ["grafana/*"]
+      prometheus-stack:
+        patterns: ["prom/*"]
+
   - package-ecosystem: github-actions
     directory: /
     schedule:

.sops.yaml

@@ -24,6 +24,31 @@
 # public half into this file.
 
 creation_rules:
+  # The lab stack, FIRST — SOPS takes the first rule whose path_regex matches,
+  # so this has to sit above the catch-all below or it would never apply.
+  #
+  # It exists because the rule below matches all of secrets/, which means any
+  # recipient added to it can decrypt every file there. The lab guest needs to
+  # decrypt secrets/lab.sops.yaml on `make render STACK=lab`; adding its key to
+  # the general rule to achieve that would also hand it the estate's SNMP
+  # communities and Grafana admin password — a lab host holding the credentials
+  # of the estate it is supposed to be isolated from, which inverts the trust
+  # direction ADR-0007 exists to protect. So it gets a rule, and a key, of its
+  # own. See docs/adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md.
+  #
+  # This is NOT the "a second rule would be a second copy of the key" case the
+  # note above warns about: that argument is against two rules sharing one
+  # recipient. These two hold different recipients on purpose, which is the
+  # whole point.
+  #
+  # Placeholder until the lab guest exists. `make secrets-init STACK=lab`, run
+  # ON that guest, fills it in — and refuses if the key it would write is
+  # already a recipient below, because that would collapse the two rules back
+  # into one.
+  - path_regex: secrets/lab\..*\.sops\.ya?ml$
+    age: >-
+      REPLACE_WITH_LAB_AGE_PUBLIC_KEY
+
   - path_regex: (secrets/.*|backups/firewall/.*)\.sops\.ya?ml$
     age: >-
       age1yrdu996u5mhdh0qf93l7s8zz8stneqnqxpncrcarrmgxvsy264rqmkcs6x

README.md

@@ -169,6 +169,9 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
 │   ├── alloy/                # the agent config directory, shipped to every host
 │   ├── snmp-exporter/        # generator.yaml is the source of truth
 │   └── grafana/              # provisioning + 7 dashboards
+├── stacks/lab/               # the lab's own stack — four services, not yet deployed
+│                             #   runs on a guest on Saruman, never remote-writes
+│                             #   to VLAN 99. See its README and ADR-0020
 ├── secrets/                  # SOPS-encrypted; see secrets/README.md
 ├── scripts/                  # bootstrap, render, validate, pin-digests, purge
 ├── SECURITY.md               # disclosure policy and known exposure

docs/architecture.md

@@ -181,7 +181,8 @@ hole from the monitoring VLAN into the monitored one.
 | Host | VLAN | Stack | Contents |
 | --- | --- | --- | --- |
 | `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, Alloy |
-| `Saruman` (10.0.30.110) | 🟢 30 | *(none yet)* | Proxmox VE 9, no guests — see [roadmap](roadmap.md); Alloy agent (native package) |
+| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, no guests yet — see [roadmap](roadmap.md); Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest |
+| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | **Not built yet** — the guest is [#262](https://github.com/Gerrrt/HomeLab/issues/262), the stack is committed and deployable. Prometheus, Loki, Grafana, Alloy: the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)) |
 | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
 
 One directory per stack, not one per service. A stack is the unit that gets

docs/network.md

@@ -278,7 +278,21 @@ Where things get broken on purpose.
   dedicated port, `Saruman` is the Proxmox install. They are separate addresses
   and separate names, and conflating them is a mistake this document previously
   made.
-- `Saruman` currently runs no guests.
+- `Saruman` currently runs no guests. The first will be `alexander`, at
+  `10.0.30.40` — a static below `.100` with a reservation, single-homed on this
+  segment like its host — which runs
+  [`stacks/lab`](../stacks/lab): the lab's own Prometheus, Loki, Grafana and
+  Alloy. The stack is built and committed; the guest is
+  [#262](https://github.com/Gerrrt/HomeLab/issues/262), which is why it is
+  described here and not in the table above. **It is a guest and not the
+  hypervisor for a reason**: a compose stack is Docker, and Docker would
+  rewrite the iptables of the box whose own firewall ADR-0014 relies on — the
+  same fact that put the native `.deb` agent on `Saruman` rather than a
+  container
+  ([ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)).
+  It gets **no** pass into Winterfell: the rule below is the hypervisor's, and
+  ADR-0007's "guests get no such rule" covers this one too. Nothing in that
+  stack remote-writes off the segment.
 - `Saruman` runs an Alloy agent and is the one host on this segment with a path
   into Winterfell: a single pass, `10.0.30.110 → 10.0.99.20` on 9090 and 3100
   TCP, unlogged and above the ADR-0014 tripwire. The hypervisor's own telemetry

docs/observability.md

@@ -2,6 +2,29 @@
 
 What is collected, where it goes, and how to change it.
 
+**This document describes the estate's stack, on `prometheus` (10.0.99.20).**
+There is a second one. [`stacks/lab`](../stacks/lab) is the lab's own
+Prometheus, Loki, Grafana and Alloy, and it is deliberately not part of any of
+what follows: no series it holds reaches this Prometheus, no log line reaches
+this Loki, and none of the alert rules or dashboards below can see it. That is
+ADR-0007's decision — lab telemetry stays in the lab, so that deliberately
+hostile data never lands in the store the estate is actually run from — and
+[ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)
+settles its shape. It is built but not yet deployed; the guest that runs it is
+[#262](https://github.com/Gerrrt/HomeLab/issues/262).
+
+The one path that does cross belongs to the hypervisor and not to any guest:
+`Saruman`'s own agent remote-writes here over a single unlogged pass
+([#88](https://github.com/Gerrrt/HomeLab/issues/88)). A DL360 with an ageing
+mirrored pair is estate hardware, and its health belongs with the rest of the
+estate's.
+
+The consequence worth carrying into everything below: **nothing here can tell a
+quiet lab from a dead one.** `RemoteWriteJobStale` keys on jobs that arrive on
+this Prometheus, so by construction it can never cover a stack that never
+arrives. That gap is [#257](https://github.com/Gerrrt/HomeLab/issues/257), and
+it is not closed by anything in this document.
+
 ## What is collected
 
 | Source | Via | Interval | Examples |

docs/roadmap.md

@@ -298,12 +298,19 @@ what left this one unfireable for months.
   `*_URL` variables changed, and that file has two sinks, so a lab without a
   Prometheus points the second one at `10.0.99.20` and inverts the isolation
   the ADR exists for.
-  In order — [#262](https://github.com/Gerrrt/HomeLab/issues/262) the guest,
-  then [#263](https://github.com/Gerrrt/HomeLab/issues/263) the validators,
-  because `STACK ?=` reaches the lifecycle targets and stops there and every
-  checker in this repository is pinned to `stacks/observability`, so a second
-  stack today would be one CI has never seen; then
-  [#264](https://github.com/Gerrrt/HomeLab/issues/264) the stack itself.
+  [#264](https://github.com/Gerrrt/HomeLab/issues/264) is built:
+  `stacks/lab/` holds the compose file, both configs, four alert rules and
+  their unit tests, and the secrets template. What is left of it is a deploy,
+  which needs [#262](https://github.com/Gerrrt/HomeLab/issues/262) — the guest
+  on `Saruman` — to exist first. Building it made the tooling stack-aware
+  (`render-config.sh` derives its required keys per stack rather than demanding
+  the estate's ten, `reload-config.sh` skips services a stack does not declare,
+  `bootstrap.sh` refuses to give one age key both stacks) and gave `.sops.yaml`
+  the lab rule ADR-0020 asked for. It did **not** touch
+  [#263](https://github.com/Gerrrt/HomeLab/issues/263): `STACK ?=` reaches the
+  lifecycle targets and stops there, every checker is still pinned to
+  `stacks/observability`, and so the new stack is one CI has never seen —
+  validated only by hand and by the checks that already follow `STACK`.
   [#265](https://github.com/Gerrrt/HomeLab/issues/265) the domain is what
   everything else is pointed at, and blocks both
   [#266](https://github.com/Gerrrt/HomeLab/issues/266) Wazuh — the heaviest

scripts/backup-firewall.sh

@@ -137,8 +137,26 @@ need() { command -v "$1" >/dev/null 2>&1 || { red "missing dependency: $1"; exit
 
 # The age recipient is read from .sops.yaml rather than duplicated here. One
 # source of truth for the key; rotating it in .sops.yaml rotates it here too.
+#
+# Read from the rule that actually covers backups/firewall/, not the first key
+# in the file. This was `grep ... | head -1` while .sops.yaml held exactly one
+# creation_rule, which made "first key" and "the right key" the same string.
+# ADR-0020 added a second rule above it for the lab stack, and the moment that
+# rule's placeholder is replaced with a real key, `head -1` would encrypt every
+# firewall backup to the LAB guest — silently, since sops would happily do it
+# and the file would still look like a backup. That is the same trust inversion
+# the second rule was created to prevent, arriving through the back door.
+#
+# Anchored on the path_regex naming backups/firewall, so it follows the rule
+# rather than the ordering. Exits at the first key after that line: `age:` uses
+# a folded scalar, so the key is on the line following the one that matches.
 recipient() {
-  grep -oE 'age1[0-9a-z]{50,}' "$SOPS_POLICY" | head -1
+  awk '
+    /path_regex:.*backups\/firewall/ { inrule = 1 }
+    inrule && match($0, /age1[0-9a-z]{50,}/) {
+      print substr($0, RSTART, RLENGTH); exit
+    }
+  ' "$SOPS_POLICY"
 }
 
 # Newest first, sorted by NAME and not by mtime. The stamp is UTC ISO-8601

scripts/bootstrap.sh

@@ -51,16 +51,48 @@ info "public key: ${PUBLIC_KEY}"
 # ---------------------------------------------------------------------------
 # 2. register it in .sops.yaml
 # ---------------------------------------------------------------------------
-if grep -q "REPLACE_WITH_YOUR_AGE_PUBLIC_KEY" "${SOPS_CONFIG}"; then
-  info "writing public key into .sops.yaml"
-  sed -i.bak "s|REPLACE_WITH_YOUR_AGE_PUBLIC_KEY|${PUBLIC_KEY}|" "${SOPS_CONFIG}"
+# Which placeholder belongs to THIS stack.
+#
+# There is no longer one placeholder to fill. .sops.yaml carries a creation_rule
+# per stack that needs its own recipient — ADR-0020 gives `lab` one, because the
+# single rule that used to match all of secrets/ meant any recipient added to it
+# could decrypt every other stack's credentials too. Filling in "the first
+# placeholder found" would write the lab guest's key into the estate's rule, or
+# the estate's into the lab's, which is the failure that decision exists to
+# prevent.
+#
+# The generic name is kept as the fallback so a fresh clone bootstrapping
+# `observability` behaves exactly as it always has.
+PLACEHOLDER="REPLACE_WITH_${STACK^^}_AGE_PUBLIC_KEY"
+grep -q "${PLACEHOLDER}" "${SOPS_CONFIG}" 2>/dev/null \
+  || PLACEHOLDER="REPLACE_WITH_YOUR_AGE_PUBLIC_KEY"
+
+if grep -q "${PLACEHOLDER}" "${SOPS_CONFIG}"; then
+  # Refused rather than warned about. Reaching here means this host's key is
+  # already a recipient of some other rule in this file, and is now being asked
+  # to become ${STACK}'s as well — one key that decrypts both stacks, which is
+  # the exact collapse the separate rules exist to stop. It is also the easy
+  # mistake: `make secrets-init STACK=lab` typed on the monitoring host rather
+  # than on the lab guest does precisely this, and the result would look like a
+  # successful bootstrap.
+  if grep -q "${PUBLIC_KEY}" "${SOPS_CONFIG}"; then
+    die "this host's key is already a recipient in ${SOPS_CONFIG##*/}, and
+making it ${STACK}'s recipient as well would give one key both stacks.
+
+Run this on the host that will run ${STACK}, so that stack gets a key of its
+own. If one key for both is genuinely what you want, edit .sops.yaml by hand —
+it should be a decision, not a side effect of where you happened to type this."
+  fi
+  info "writing public key into .sops.yaml (${PLACEHOLDER})"
+  sed -i.bak "s|${PLACEHOLDER}|${PUBLIC_KEY}|" "${SOPS_CONFIG}"
   rm -f "${SOPS_CONFIG}.bak"
 elif grep -q "${PUBLIC_KEY}" "${SOPS_CONFIG}"; then
   info ".sops.yaml already lists this key"
 else
   warn ".sops.yaml lists a different age recipient."
   warn "Add this key as an additional recipient by hand, then run:"
   warn "  sops updatekeys ${SECRETS_FILE}"
+  warn "Add it to the rule matching secrets/${STACK}. — NOT to another stack's."
 fi
 
 # ---------------------------------------------------------------------------

scripts/check_docs.py

@@ -178,6 +178,16 @@ def strip_md(cell: str) -> str:
     return cell.strip()
 
 
+# A host-and-stack row for something that does not exist yet. See the block
+# above check_host_stack_table() for what it does there; count_alloy_agents()
+# below reads it too, because a row describing an undeployed host describes an
+# undeployed agent.
+#
+# Matched against the RAW cell, never strip_md()'s output: that helper removes
+# every `*`, which takes the emphasis with it and leaves the marker unfindable.
+NOT_BUILT = re.compile(r"\*\*not built yet\*\*", re.I)
+
+
 # ---------------------------------------------------------------------------
 # Facts, computed from the configs
 # ---------------------------------------------------------------------------
@@ -253,6 +263,13 @@ def count_alloy_agents() -> int:
     is the machine-readable side. A row whose Contents cell names Alloy is an
     agent; "two Alloy agents" in hardware.md was unguarded and stale for as
     long as it took to deploy a third (#88).
+
+    A row marked NOT_BUILT is not counted, because an agent on a host that does
+    not exist is not an agent. `stacks/lab` declares one for `alexander`, and it
+    collects nothing until that guest is racked (#262). The exclusion is not a
+    convenience: dropping the marker on the commit that builds the host pushes
+    this count to four and fails hardware.md's "three Alloy agents" in the same
+    run, which is exactly when that sentence should be forced to change.
     """
     tables = tables_under(
         ARCH_MD.read_text(encoding="utf-8"),
@@ -262,7 +279,9 @@ def count_alloy_agents() -> int:
         return 0
     return sum(
         1 for row in tables[0][1:]
-        if len(row) > 3 and "alloy" in strip_md(row[3]).lower()
+        if len(row) > 3
+        and "alloy" in strip_md(row[3]).lower()
+        and not NOT_BUILT.search(row[3])
     )
 
 
@@ -407,6 +426,27 @@ def check_snmp_targets() -> list[str]:
 # ---------------------------------------------------------------------------
 # 3. Host and stack mapping
 # ---------------------------------------------------------------------------
+# A stack directory can legitimately exist before the host that runs it does.
+# `stacks/lab` was committed complete — compose file, configs, rules, unit
+# tests — while the guest that will run it, `alexander`, was still an issue
+# (#262, #264). ADR-0004 puts the host-to-stack mapping in this document, so the
+# row has to exist; but docs/network.md is the inventory of what is actually on
+# the wire, and writing an unbuilt guest into it would be the precise kind of
+# false claim this file exists to catch. It would also mean inventing a MAC, a
+# device and an OS for a machine whose distribution is explicitly undecided.
+#
+# So the row is marked, and the marker INVERTS the check rather than switching
+# it off. A normal row's host must APPEAR in network.md; a row marked "not built
+# yet" must be ABSENT from it. That is what makes the marker self-clearing —
+# rack the host, add its network.md row, and this fails saying the marker is
+# stale, instead of quietly tolerating a row that claims both things at once.
+# The address is still required and still checked for collisions, so a plan is
+# held to the same standard as a deployment.
+#
+# The marker itself is defined next to strip_md(), because count_alloy_agents()
+# reads it too.
+
+
 def check_host_stack_table() -> list[str]:
     text = ARCH_MD.read_text(encoding="utf-8")
     tables = tables_under(text, re.compile(r"^##\s+Host and stack mapping"))
@@ -424,6 +464,7 @@ def check_host_stack_table() -> list[str]:
         ip_match = re.search(r"(\d+\.\d+\.\d+\.\d+)", host_cell)
         vlan_match = re.search(r"(\d+)", strip_md(row[1]))
         named_stacks.update(re.findall(r"stacks/([a-z0-9-]+)", row[2]))
+        planned = bool(len(row) > 3 and NOT_BUILT.search(row[3]))
 
         if not (ip_match and vlan_match):
             problems.append(
@@ -438,7 +479,35 @@ def check_host_stack_table() -> list[str]:
             r for r in rows_for_vlan
             if strip_md(r[0]).lower() == host.lower() and ip in strip_md(r[1])
         ]
-        if not hit:
+
+        if planned:
+            # The VLAN must be one network.md actually describes, or a typo'd
+            # segment would make every assertion below vacuously true.
+            if not rows_for_vlan:
+                problems.append(
+                    f"docs/architecture.md plans {host} on VLAN {vlan}, which "
+                    f"docs/network.md has no table for"
+                )
+            if hit:
+                problems.append(
+                    f"docs/architecture.md still marks {host} 'not built yet', "
+                    f"and docs/network.md now lists it at {ip} on VLAN {vlan} — "
+                    f"it has been built, so drop the marker"
+                )
+            # An unbuilt host planned onto an address something else already
+            # holds is a real conflict, and the cheapest possible moment to
+            # find it is before anyone racks it.
+            clash = [
+                r for r in rows_for_vlan
+                if ip in strip_md(r[1]) and strip_md(r[0]).lower() != host.lower()
+            ]
+            if clash:
+                problems.append(
+                    f"docs/architecture.md plans {host} at {ip}, which "
+                    f"docs/network.md already gives to "
+                    f"{strip_md(clash[0][0])} on VLAN {vlan}"
+                )
+        elif not hit:
             problems.append(
                 f"docs/architecture.md places {host} at {ip} on VLAN {vlan}; "
                 f"docs/network.md does not list it there"