diff --git a/docs/architecture.md b/docs/architecture.md index 08d1334..32e2354 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -182,7 +182,7 @@ hole from the monitoring VLAN into the monitored one. | --- | --- | --- | --- | | `prometheus` (10.0.99.20) | ๐Ÿ”ด 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, Alloy | | `Saruman` (10.0.30.110) | ๐ŸŸข 30 | *(none โ€” and none intended)* | Proxmox VE 9, no guests yet โ€” see [roadmap](roadmap.md); Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | -| `alexander` (10.0.30.40) | ๐ŸŸข 30 | [`stacks/lab`](../stacks/lab) | **Not built yet** โ€” the guest is [#262](https://github.com/Gerrrt/HomeLab/issues/262), the stack is committed and deployable. Prometheus, Loki, Grafana, Alloy: the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)) | +| `alexander` (10.0.30.40) | ๐ŸŸข 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy โ€” the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `oracle` (10.0.99.30) | ๐Ÿ”ด 99 | *(none โ€” hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs โ€” [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | One directory per stack, not one per service. A stack is the unit that gets diff --git a/docs/hardware.md b/docs/hardware.md index ba93d6c..09f1d0b 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -36,7 +36,7 @@ quietly swapped. | `oracle` | Dell Inspiron 15-3565 | AMD A6-9200 (2 cores) | 4 GB | 500 GB HDD | Ubuntu Server 24.04 LTS | The observability stack runs on a thirteen-year-old MacBook. It handles four -SNMP devices at a 60-second interval, three Alloy agents, and 30 days of metric +SNMP devices at a 60-second interval, four Alloy agents, and 30 days of metric retention without complaint โ€” which is a useful thing to know before spending money on a monitoring host. Its RAM is soldered at 8 GB and it has no built-in Ethernet, so it reaches the network over a USB NIC. diff --git a/docs/network.md b/docs/network.md index d9dae77..466e9de 100644 --- a/docs/network.md +++ b/docs/network.md @@ -272,6 +272,7 @@ Where things get broken on purpose. | morpheus | `10.0.30.1` | `02:26:26:xx:xx:xx` | HP ProDesk 600 G4 Mini | FreeBSD 16.0 | Rack U5 | Firewall | | shiva | `10.0.30.10` | `94:57:a5:xx:xx:xx` | HPE iLO 4 (DL360 Gen9 BMC)[^Shiva] | iLO 2.82 | Rack U3 | Out-of-band management | | Saruman | `10.0.30.110` | `14:02:ec:xx:xx:xx` | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox VE 9 | Rack U3 | Hypervisor | +| alexander | `10.0.30.40` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Lab observability | ### Notes @@ -280,21 +281,19 @@ Where things get broken on purpose. dedicated port, `Saruman` is the Proxmox install. They are separate addresses and separate names, and conflating them is a mistake this document previously made. -- `Saruman` currently runs no guests. The first will be `alexander`, at - `10.0.30.40` โ€” a static below `.100` with a reservation, single-homed on this - segment like its host โ€” which runs - [`stacks/lab`](../stacks/lab): the lab's own Prometheus, Loki, Grafana and - Alloy. The stack is built and committed; the guest is - [#262](https://github.com/Gerrrt/HomeLab/issues/262), which is why it is - described here and not in the table above. **It is a guest and not the - hypervisor for a reason**: a compose stack is Docker, and Docker would - rewrite the iptables of the box whose own firewall ADR-0014 relies on โ€” the - same fact that put the native `.deb` agent on `Saruman` rather than a - container +- `Saruman` runs one guest, `alexander`, built 2026-09-05 + ([#262](https://github.com/Gerrrt/HomeLab/issues/262)). It runs + [`stacks/lab`](../stacks/lab) โ€” the lab's own Prometheus, Loki, Grafana and + Alloy. **It is a guest and not the hypervisor for a reason**: a compose stack + is Docker, and Docker would rewrite the iptables of the box whose own + firewall ADR-0014 relies on โ€” the same fact that put the native `.deb` agent + on `Saruman` rather than a container ([ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). It gets **no** pass into Winterfell: the rule below is the hypervisor's, and ADR-0007's "guests get no such rule" covers this one too. Nothing in that - stack remote-writes off the segment. + stack remote-writes off the segment, so nothing outside the lab sees it โ€” and + nothing outside the lab can tell it apart from a lab nobody is using + ([#257](https://github.com/Gerrrt/HomeLab/issues/257)). - `Saruman` runs an Alloy agent and is the one host on this segment with a path into Winterfell: a single pass, `10.0.30.110 โ†’ 10.0.99.20` on 9090 and 3100 TCP, unlogged and above the ADR-0014 tripwire. The hypervisor's own telemetry diff --git a/docs/runbooks/build-the-lab-guest.md b/docs/runbooks/build-the-lab-guest.md index c4425d5..5829fcd 100644 --- a/docs/runbooks/build-the-lab-guest.md +++ b/docs/runbooks/build-the-lab-guest.md @@ -24,7 +24,7 @@ is already built and validated; what is missing is somewhere to run it. | Name | `alexander` | A Final Fantasy summon, like `shiva` and `ifrit` already on this segment. A fortress, which is what the defended estate's own observer is | | Address | `10.0.30.40/24` | Statics on this segment live **below `.100`**; the pool is `.100โ€“.200`. Continues the decade spacing โ€” `shiva` .10, `Saruman` .20 (after #96), `ifrit` .30 | | Kind | **VM, not LXC** | `stacks/lab` uses `cgroup: host`, `cap_drop: [ALL]` and a Docker socket mount. Docker in an LXC needs nesting and keyctl workarounds, and those settings behave differently under one. A VM has no such asterisks | -| OS | **Ubuntu Server 24.04 LTS** | See below. This is the one that would have bitten quietly | +| OS | **Ubuntu Server LTS** โ€” `alexander` runs 26.04 | See below. This is the one that would have bitten quietly | | Disk | 64 GB | Prometheus is capped at 4 GB and Loki keeps 15 days of a small estate. 64 GB leaves room without pretending the spindles are free | | RAM | 8 GB | The estate's whole stack runs on a 2012 MacBook with 8 GB. This one is smaller and has headroom for the domain arriving | @@ -38,12 +38,15 @@ is already built and validated; what is missing is somewhere to run it. > two of its four log sources. > > That is the exact failure this repository keeps paying for โ€” #62 and #63 were -> both collectors that ran healthy and produced nothing. Ubuntu Server 24.04 -> ships rsyslog, has both files, and is what `prometheus` and `oracle` already -> run, so the agent behaves identically on every Linux host in the estate. +> both collectors that ran healthy and produced nothing. Ubuntu is what +> `prometheus` and `oracle` already run, so the agent behaves identically on +> every Linux host in the estate. > -> If you ever do put this on Debian, install `rsyslog` in the same breath and -> check ยง7 rather than assuming. +> **Verify it rather than trusting the distribution.** 24.04 ships rsyslog; +> Ubuntu has been steadily narrowing what it installs by default, and this +> runbook does not know what the LTS you are installing does. ยง7 is the check, +> and it is the reason ยง7 exists. If either file is missing โ€” on Ubuntu or on +> Debian โ€” `sudo apt-get install -y rsyslog` restores both, and ยง7 confirms it. ## 1. Create the VM @@ -337,8 +340,13 @@ if you forget: That marker is load-bearing: while it is there, `check_docs.py` requires the host to be **absent** from `network.md`, so adding the row without removing the marker fails and says the marker is stale. -- `docs/hardware.md` โ€” removing the marker also pushes the Alloy agent count to - four, which fails the "three Alloy agents" sentence in the same run. That is - deliberate, and it is the moment that sentence should change. +- `docs/hardware.md` โ€” removing the marker also raises the Alloy agent count, + which fails the sentence there that states it. That is deliberate, and it is + the moment that sentence should change. + + This bullet used to quote that sentence verbatim, and `check_docs.py` counted + the quotation as a second claim of the same fact โ€” so the runbook describing + the failure became one more place to fix when it fired. Correctly: the check + does not care whether prose is describing a number or asserting one. `make check-docs` walks you through all three.