From 2936136ddba7d3c64ddf207d62c38a98fcfb23f7 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 5 Sep 2026 04:16:53 +0000 Subject: [PATCH] docs(runbook): say where the lab's certificates are and how to move them (#262) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §5 said "copy three files into ~/HomeLab/certificates/ on alexander" and left every part of that unanswered: which directory they come from, that the destination does not exist, and that the obvious way to move them cannot work. **99 → 30 is closed.** An scp from `prometheus` to `alexander` has no rule to carry it — the one exception on that interface is the iLO's SNMP return path. Hicks reaches both segments, so the Mac is the only machine that can see both ends, and the copy has to be driven from there. Nothing said so, and the failure is a hang rather than a refusal. `certificates/` is gitignored, so a fresh clone on the guest does not have the directory at all and scp fails into a path that is not there. The mkdir is now the first command rather than something to work out from an error. `scp -3 -p`, so the private key is routed through the Mac without being written to its disk, and so the mode survives the trip — a default umask would leave a TLS private key world-readable, and the runbook now says to check rather than assume. 0640 owned by the deploying user is what compose.yaml's `group_add: ${RENDER_GID}` on grafana is built around, so the reason the mode matters is stated where the mode is set. A CAUTION on `ca-key.pem`: it is deliberately absent from the list, and `scp certificates/*` would put the key that signs every certificate in the estate onto the segment built to hold attackers. Naming the three files is the control, so the reason for naming them is written down. The files are listed in a table with what each one is for, because "copy three files" reads as three interchangeable blobs and one of them is a private key. Co-Authored-By: Claude Opus 5 --- docs/runbooks/build-the-lab-guest.md | 58 +++++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 2 deletions(-) diff --git a/docs/runbooks/build-the-lab-guest.md b/docs/runbooks/build-the-lab-guest.md index eae9b76..c4425d5 100644 --- a/docs/runbooks/build-the-lab-guest.md +++ b/docs/runbooks/build-the-lab-guest.md @@ -212,8 +212,62 @@ ciphertext belongs in the repository, the private key never does. make certs ARGS="--host grafana-lab.matrix.elysium --ip 10.0.30.40 --dns grafana" ``` -Copy three files into `~/HomeLab/certificates/` on `alexander`: -`ca.pem`, `grafana-lab.matrix.elysium.pem`, `grafana-lab.matrix.elysium-key.pem`. +That writes into `certificates/` at the root of the checkout the stack is +deployed from — `/home/robo/code/Gerrrt/HomeLab/certificates/` on `prometheus`. +Three of the files there go to `alexander`: + +| File | What it is | +| --- | --- | +| `ca.pem` | The lab CA. Prometheus verifies Grafana's leaf against it | +| `grafana-lab.matrix.elysium.pem` | The leaf you just issued | +| `grafana-lab.matrix.elysium-key.pem` | Its private key | + +> [!CAUTION] +> **`ca-key.pem` is not on that list and must never leave the monitoring +> host.** It is the key that signs every certificate in the estate; a copy of +> it on a machine that sits on the segment built to hold attackers is a +> different class of problem from a leaked leaf. Copy the three files by name. +> Do not `scp certificates/*`. + +### Do it from Hicks, because the two hosts cannot reach each other + +**`99 → 30` is closed.** An `scp` from `prometheus` to `alexander` does not +work — there is no rule that would carry it, and the one exception on that +interface is the iLO's SNMP return path. Hicks reaches both segments, so the +Mac is the only machine that can see both ends. + +From the Mac. The first command matters because `certificates/` is gitignored, +so a fresh clone on `alexander` does not have it and `scp` would fail into a +directory that is not there: + +```bash +ssh you@10.0.30.40 'mkdir -p HomeLab/certificates && chmod 700 HomeLab/certificates' + +CERTS=/home/robo/code/Gerrrt/HomeLab/certificates +scp -3 -p \ + robo@10.0.99.20:$CERTS/ca.pem \ + robo@10.0.99.20:$CERTS/grafana-lab.matrix.elysium.pem \ + robo@10.0.99.20:$CERTS/grafana-lab.matrix.elysium-key.pem \ + you@10.0.30.40:HomeLab/certificates/ +``` + +`-3` routes the copy through the Mac without writing either file to its disk, +which is worth having for a private key. If your `scp` does not support it, do +it in two steps and `shred -u` the local copies afterwards — the key is the +whole point of the care. + +`-p` preserves the mode. Check that it did, on `alexander`, because a default +umask would otherwise leave a private key world-readable: + +```bash +ls -l ~/HomeLab/certificates/ +chmod 640 ~/HomeLab/certificates/grafana-lab.matrix.elysium-key.pem +``` + +`0640` and owned by you is what the stack expects: Grafana runs as uid 472 and +gets your gid as a supplementary group (`group_add: ${RENDER_GID}` in +`compose.yaml`), which is how it reads the key without the key being readable +by everything on the host. > [!WARNING] > **Do not run `make certs ARGS=--ca` on `alexander`.** If the files are