Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ hole from the monitoring VLAN into the monitored one.
| --- | --- | --- | --- |
| `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, Alloy |
| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, no guests yet — see [roadmap](roadmap.md); Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest |
| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | **Not built yet** — the guest is [#262](https://github.com/Gerrrt/HomeLab/issues/262), the stack is committed and deployable. Prometheus, Loki, Grafana, Alloy: the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)) |
| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloythe lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) |
| `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |

One directory per stack, not one per service. A stack is the unit that gets
Expand Down
2 changes: 1 addition & 1 deletion docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ quietly swapped.
| `oracle` | Dell Inspiron 15-3565 | AMD A6-9200 (2 cores) | 4 GB | 500 GB HDD | Ubuntu Server 24.04 LTS |

The observability stack runs on a thirteen-year-old MacBook. It handles four
SNMP devices at a 60-second interval, three Alloy agents, and 30 days of metric
SNMP devices at a 60-second interval, four Alloy agents, and 30 days of metric
retention without complaint — which is a useful thing to know before spending
money on a monitoring host. Its RAM is soldered at 8 GB and it has no built-in
Ethernet, so it reaches the network over a USB NIC.
Expand Down
23 changes: 11 additions & 12 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,7 @@ Where things get broken on purpose.
| morpheus | `10.0.30.1` | `02:26:26:xx:xx:xx` | HP ProDesk 600 G4 Mini | FreeBSD 16.0 | Rack U5 | Firewall |
| shiva | `10.0.30.10` | `94:57:a5:xx:xx:xx` | HPE iLO 4 (DL360 Gen9 BMC)[^Shiva] | iLO 2.82 | Rack U3 | Out-of-band management |
| Saruman | `10.0.30.110` | `14:02:ec:xx:xx:xx` | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox VE 9 | Rack U3 | Hypervisor |
| alexander | `10.0.30.40` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Lab observability |

### Notes

Expand All @@ -280,21 +281,19 @@ Where things get broken on purpose.
dedicated port, `Saruman` is the Proxmox install. They are separate addresses
and separate names, and conflating them is a mistake this document previously
made.
- `Saruman` currently runs no guests. The first will be `alexander`, at
`10.0.30.40` — a static below `.100` with a reservation, single-homed on this
segment like its host — which runs
[`stacks/lab`](../stacks/lab): the lab's own Prometheus, Loki, Grafana and
Alloy. The stack is built and committed; the guest is
[#262](https://github.com/Gerrrt/HomeLab/issues/262), which is why it is
described here and not in the table above. **It is a guest and not the
hypervisor for a reason**: a compose stack is Docker, and Docker would
rewrite the iptables of the box whose own firewall ADR-0014 relies on — the
same fact that put the native `.deb` agent on `Saruman` rather than a
container
- `Saruman` runs one guest, `alexander`, built 2026-09-05
([#262](https://github.com/Gerrrt/HomeLab/issues/262)). It runs
[`stacks/lab`](../stacks/lab) — the lab's own Prometheus, Loki, Grafana and
Alloy. **It is a guest and not the hypervisor for a reason**: a compose stack
is Docker, and Docker would rewrite the iptables of the box whose own
firewall ADR-0014 relies on — the same fact that put the native `.deb` agent
on `Saruman` rather than a container
([ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)).
It gets **no** pass into Winterfell: the rule below is the hypervisor's, and
ADR-0007's "guests get no such rule" covers this one too. Nothing in that
stack remote-writes off the segment.
stack remote-writes off the segment, so nothing outside the lab sees it — and
nothing outside the lab can tell it apart from a lab nobody is using
([#257](https://github.com/Gerrrt/HomeLab/issues/257)).
- `Saruman` runs an Alloy agent and is the one host on this segment with a path
into Winterfell: a single pass, `10.0.30.110 → 10.0.99.20` on 9090 and 3100
TCP, unlogged and above the ADR-0014 tripwire. The hypervisor's own telemetry
Expand Down
26 changes: 17 additions & 9 deletions docs/runbooks/build-the-lab-guest.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ is already built and validated; what is missing is somewhere to run it.
| Name | `alexander` | A Final Fantasy summon, like `shiva` and `ifrit` already on this segment. A fortress, which is what the defended estate's own observer is |
| Address | `10.0.30.40/24` | Statics on this segment live **below `.100`**; the pool is `.100–.200`. Continues the decade spacing — `shiva` .10, `Saruman` .20 (after #96), `ifrit` .30 |
| Kind | **VM, not LXC** | `stacks/lab` uses `cgroup: host`, `cap_drop: [ALL]` and a Docker socket mount. Docker in an LXC needs nesting and keyctl workarounds, and those settings behave differently under one. A VM has no such asterisks |
| OS | **Ubuntu Server 24.04 LTS** | See below. This is the one that would have bitten quietly |
| OS | **Ubuntu Server LTS** — `alexander` runs 26.04 | See below. This is the one that would have bitten quietly |
| Disk | 64 GB | Prometheus is capped at 4 GB and Loki keeps 15 days of a small estate. 64 GB leaves room without pretending the spindles are free |
| RAM | 8 GB | The estate's whole stack runs on a 2012 MacBook with 8 GB. This one is smaller and has headroom for the domain arriving |

Expand All @@ -38,12 +38,15 @@ is already built and validated; what is missing is somewhere to run it.
> two of its four log sources.
>
> That is the exact failure this repository keeps paying for — #62 and #63 were
> both collectors that ran healthy and produced nothing. Ubuntu Server 24.04
> ships rsyslog, has both files, and is what `prometheus` and `oracle` already
> run, so the agent behaves identically on every Linux host in the estate.
> both collectors that ran healthy and produced nothing. Ubuntu is what
> `prometheus` and `oracle` already run, so the agent behaves identically on
> every Linux host in the estate.
>
> If you ever do put this on Debian, install `rsyslog` in the same breath and
> check §7 rather than assuming.
> **Verify it rather than trusting the distribution.** 24.04 ships rsyslog;
> Ubuntu has been steadily narrowing what it installs by default, and this
> runbook does not know what the LTS you are installing does. §7 is the check,
> and it is the reason §7 exists. If either file is missing — on Ubuntu or on
> Debian — `sudo apt-get install -y rsyslog` restores both, and §7 confirms it.

## 1. Create the VM

Expand Down Expand Up @@ -337,8 +340,13 @@ if you forget:
That marker is load-bearing: while it is there, `check_docs.py` requires the
host to be **absent** from `network.md`, so adding the row without removing
the marker fails and says the marker is stale.
- `docs/hardware.md` — removing the marker also pushes the Alloy agent count to
four, which fails the "three Alloy agents" sentence in the same run. That is
deliberate, and it is the moment that sentence should change.
- `docs/hardware.md` — removing the marker also raises the Alloy agent count,
which fails the sentence there that states it. That is deliberate, and it is
the moment that sentence should change.

This bullet used to quote that sentence verbatim, and `check_docs.py` counted
the quotation as a second claim of the same fact — so the runbook describing
the failure became one more place to fix when it fired. Correctly: the check
does not care whether prose is describing a number or asserting one.

`make check-docs` walks you through all three.