diff --git a/Makefile b/Makefile index 40e3eca..c1bbc2b 100644 --- a/Makefile +++ b/Makefile @@ -46,6 +46,18 @@ down: ## Stop the stack (volumes are preserved) .PHONY: restart restart: down up ## Restart the stack +.PHONY: converge +converge: ## Fetch main, verify it, fast-forward and deploy (ARGS=--dry-run) + @# What the hourly timer runs, and what a human runs to deploy on purpose + @# without waiting for it. It ends in `make up` rather than replacing it, so + @# there is exactly one deployment path and both callers exercise it. + @# + @# It refuses to run anywhere but /home/robo/code/Gerrrt/HomeLab, for the + @# reason `make up` cares about and `make deploy-agent` does not: render + @# writes into the .rendered/ of the tree it is run from, and no container + @# mounts a worktree's copy. ARGS=--dry-run says what it would do. + ./scripts/converge.sh $(ARGS) + .PHONY: pull pull: ## Pull the pinned images $(COMPOSE) pull diff --git a/README.md b/README.md index 053c775..b288a61 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ incident. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 140 panels, and - 64 alert rules — 48 metric-based in Prometheus, 16 log-based in Loki — sharing + 69 alert rules — 53 metric-based in Prometheus, 16 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -160,7 +160,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 48 alert rules +│ ├── prometheus/ # config, file_sd targets, 53 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 16 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host @@ -172,12 +172,12 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). ├── docs/ │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md -│ ├── adr/ # 19 architecture decision records -│ └── runbooks/ # deploy, add device, rotate creds, certs, key backup, -│ # purge, restore the firewall, restore the stack, -│ # ship firewall logs, verify the alert path, -│ # enable suricata, fit the UPS battery, -│ # add a host override +│ ├── adr/ # 21 architecture decision records +│ └── runbooks/ # deploy, converge, add device, rotate creds, certs, +│ # key backup, purge, restore the firewall, +│ # restore the stack, ship firewall logs, +│ # verify the alert path, enable suricata, +│ # fit the UPS battery, add a host override └── Makefile # make help ``` @@ -210,9 +210,18 @@ trust `certificates/ca.pem` — step 4 of that runbook. Full procedure, verification steps and troubleshooting in [`docs/runbooks/deploy-stack.md`](docs/runbooks/deploy-stack.md). +That is the first deploy. After it, the monitoring host deploys itself: a timer +runs `scripts/converge.sh` hourly, which fetches `main`, refuses it unless the +tip carries GitHub's signature, fast-forwards and runs the same `make up` — +recording what it deployed and refusing to overwrite anything edited on the host +([#99](https://github.com/Gerrrt/HomeLab/issues/99), +[ADR-0021](docs/adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md), +[`docs/runbooks/converge-the-host.md`](docs/runbooks/converge-the-host.md)). + ```console $ make help up Render config and start the stack + converge Fetch main, verify it, fast-forward and deploy down Stop the stack (volumes are preserved) reload Hot-reload Prometheus, Alertmanager and snmp-exporter (no restart) secrets-init Generate an age keypair and create the encrypted secrets file @@ -225,8 +234,8 @@ $ make help ... ``` -The timers are what stop `backup`, `backup-firewall` and `snmp-verify` being -things someone has to remember, and the alert rules that come with them fire on a +The timers are what stop `backup`, `backup-firewall`, `snmp-verify` and now +deployment itself being things someone has to remember, and the alert rules that come with them fire on a job having *stopped being run* rather than only on one that failed ([#77](https://github.com/Gerrrt/HomeLab/issues/77)). One job deliberately has no timer: `secrets-verify-backup` needs a human to mount removable media, so it gets diff --git a/docs/adr/0007-defensive-estate-and-offensive-range.md b/docs/adr/0007-defensive-estate-and-offensive-range.md index 90411a3..81ac3aa 100644 --- a/docs/adr/0007-defensive-estate-and-offensive-range.md +++ b/docs/adr/0007-defensive-estate-and-offensive-range.md @@ -23,7 +23,7 @@ > a compose stack runs on a hypervisor that must not run Docker, and where > `config.alloy`'s second `*_URL` points when no Prometheus is named — are > settled by -> [ADR-0019](0019-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md), +> [ADR-0020](0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md), > 2026-09: the stack runs in a guest on `Saruman`, single-homed on VLAN 30, and > carries its own Prometheus alongside Loki, Grafana and Alloy. Nothing here is > amended — "lab telemetry stays in the lab" is what makes both answers follow. diff --git a/docs/adr/0019-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md b/docs/adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md similarity index 99% rename from docs/adr/0019-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md rename to docs/adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md index f11acd2..e3bcb61 100644 --- a/docs/adr/0019-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md +++ b/docs/adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md @@ -1,4 +1,4 @@ -# ADR-0019: Run the lab stack in a guest, and give it its own Prometheus +# ADR-0020: Run the lab stack in a guest, and give it its own Prometheus **Status:** Accepted · 2026-09 diff --git a/docs/adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md b/docs/adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md new file mode 100644 index 0000000..8af524e --- /dev/null +++ b/docs/adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md @@ -0,0 +1,254 @@ +# ADR-0021: Converge on a timer instead of deploying over SSH + +**Status:** Accepted · 2026-09 + +## Context + +Deployment is `make up`, typed into an SSH session on `prometheus` +(`10.0.99.20`). [#99](https://github.com/Gerrrt/HomeLab/issues/99) asks for +something pull-based, so the host converges on the repository rather than being +pushed to, and names three reasons. + +**Nothing records what is deployed.** `make up` on an uncommitted working tree +and `make up` on `main` produce the same output and the same exit code. The +difference surfaces weeks later, as a configuration nobody can account for. +This is not hypothetical here: the header of +[`scripts/deploy-agent.sh`](../../scripts/deploy-agent.sh) is four paragraphs +about `oracle` running a different Alloy version, a different container shape +and a config three weeks stale, for two days, because nothing compared the host +to the repository. + +**Nothing detects drift.** A configuration edited on the host stays edited until +the next deploy overwrites it silently. Both halves are bad, and the second is +worse: the edit is destroyed *and* the destruction is invisible. + +**It is [#77](https://github.com/Gerrrt/HomeLab/issues/77) again.** "Nothing +schedules anything" and "nothing deploys anything" want the same three parts — +something on a timer, a wrapper that records what it did, and alert rules that +read the record. #77 already built all three. + +### The secrets constraint, re-examined + +The issue names the awkward part: `render-config.sh` needs the age key, so a +pull-based agent needs it too, "and that is a key sitting on a host that pulls +from a public repository." + +Checked rather than accepted, and it dissolves. The key is already there. It has +been since the stack was first deployed — [`.sops.yaml`](../../.sops.yaml) says +so in as many words, `~/.config/sops/age/keys.txt` is where it lives, and every +`make up` over SSH has decrypted with it on that host. A pull-based agent +running as the same user, on the same machine, reading the same file, adds no +exposure at all. + +The word doing the damage in that sentence is *public*, and public means +**readable**. A key that decrypts local files is not endangered by strangers +being able to read the repository; it would be endangered by strangers being +able to *write* it. So the constraint the issue was reaching for is real but +differently shaped: + +> A host that executes whatever a branch says, unattended, has replaced "do I +> trust this code" with "do I trust whoever can move that branch." + +That is the question this ADR has to answer, and it is a question about write +access, not about visibility. + +### What actually protects `main` today + +Measured on 2026-09-04, against this repository: + +| Property of `main` | Result | +| --- | --- | +| First-parent commits | 177 | +| Consecutive GitHub-signed merge commits from the tip | 110 | +| Oldest of that run | `2ea4cb4`, PR #33, 2026-08-19 | +| What is at 2026-08-19 | The git-history secret purge — the last force-push | +| Signing key on all 110 | `968479A1AFF927E37D1A566BB5690EEEBB952194` | +| `%G?` on all 110, with that key imported | `U` — good signature, untrusted keyring | + +Every advance of `main` for the last three weeks and 110 merges has been a merge +commit GitHub made and signed, because every change went through a pull request. +Nothing enforced that; it is simply how this repository has been worked. But it +is a property that can be *checked*, and checking it converts "trust the remote" +into "trust one fingerprint pinned on the host." + +### The options + +| Option | Why not | +| --- | --- | +| Flux, Argo CD | Both reconcile Kubernetes objects. There is no Kubernetes here and [ADR-0004](0004-one-compose-stack-per-host.md) is a decision not to have any. Adopting one to deploy a Compose file means adopting a cluster to run the operator that deploys the Compose file | +| `ansible-pull` | The closest fit, and still the wrong size. It brings a second configuration language, a second templating system and a second secrets story alongside SOPS, to schedule a `make` target that already exists. The playbook would be a wrapper around `make up` | +| A GitHub Actions self-hosted runner | Inverts the security story rather than improving it. A runner holds a registration token, keeps an outbound connection to GitHub, and runs whatever a workflow file says — and workflow files are in the repository being deployed. It is a push model with extra steps and a credential | +| A webhook receiver | A port to publish, a service to pin and back up, and a secret to rotate, so that deploys are prompt. [ADR-0012](0012-publish-only-ports-with-an-off-host-consumer.md) is the standing decision against publishing a port with no off-host consumer, and "prompt" is worth an hour at most here | +| `git pull` in cron | This, minus the record, minus the drift check, minus the refusals, and minus the alerting. The distance between that one line and what is decided below *is* the issue | + +## Decision + +**A script on an hourly systemd timer fetches `main`, verifies its signature, +fast-forwards, and runs `make up` — reusing #77's timer, wrapper and alert +machinery rather than introducing a second way to run things on a schedule.** + +[`scripts/converge.sh`](../../scripts/converge.sh), `homelab-converge.timer`, +one row in the `JOBS` table in +[`scripts/install-timers.sh`](../../scripts/install-timers.sh), and +[`prometheus/rules/deploy.rules.yaml`](../../stacks/observability/prometheus/rules/deploy.rules.yaml). + +### It decides when to deploy; it does not reimplement deploying + +The last thing `converge.sh` does is `make up`. It does not learn to render +config, start containers or reload Prometheus — `make up` already does all +three, and it is what every runbook tells a human to type. + +This keeps the blast radius of this ADR on the *decision* to deploy rather than +on deployment itself. Every existing runbook stays true, there is exactly one +deployment path, and both callers — the timer and the human — exercise it. + +### The gate is a fingerprint, not a remote + +Convergence refuses to move unless the fetched tip verifies against +`968479A1AFF927E37D1A566BB5690EEEBB952194`, pinned in the script. A full +fingerprint and not the 16-hex key id, because a key id is claimed by the +signature itself and a fingerprint is not. + +It also fetches an explicit `https://` URL rather than `origin`. `origin` is +SSH, and an unattended process using it would need a passphraseless key that can +also *push* to the repository this host executes. The repository is public, so +the agent needs no credential whatsoever — and a read-only URL that cannot push +is a better thing for a deployment host to hold than a key that can. + +**What the gate buys:** a commit pushed straight to `main` past the pull request +does not deploy, and neither does a tip served by anything that is not GitHub. + +**What it does not buy, stated plainly:** it does not stop a compromised GitHub +account. Someone who can open and merge a pull request gets a signature like +everybody else, and this host deploys it within the hour. That exposure is not +new — the operator ran `make up` from this checkout after pulling, which +executed exactly the same code — but the window changes, from "whenever someone +next deploys, having probably glanced at the diff" to "at most an hour, with +nobody looking." **The compensating control is the record, not the gate.** + +### Drift is refused, never overwritten + +An uncommitted change in the deployment checkout stops the run. It is not +overwritten, not stashed and not forced past; the run exits non-zero and keeps +doing so every hour, with `DeployDrifted` and `ScheduledJobFailed` both firing, +until a human commits the change or throws it away. + +Overwriting is what the old model did. Refusing is the entire point, so there is +no `--force`: `git checkout -- .` is one command and it belongs to the human. + +### What gets recorded + +`converge.sh` writes five gauges into the textfile directory +[`run-scheduled.sh`](../../scripts/run-scheduled.sh) already writes to, in its +own file, on every exit path — **including the refusals**, because a convergence +that declined to move still knows what the host is running. + +| Metric | Answers | +| --- | --- | +| `homelab_deploy_revision_info{revision}` | What is deployed | +| `homelab_deploy_commit_timestamp_seconds` | How old the running configuration is | +| `homelab_deploy_behind_commits` | How far behind `main` the host is; `-1` for "the fetch failed", so not-knowing cannot read as being-behind | +| `homelab_deploy_tree_dirty` | Whether anyone edited the host | +| `homelab_deploy_verified` | Whether the deployed revision carries a good signature | +| `homelab_deploy_apply_enabled` | Whether the host applies what it fetches, or is in report-only mode | + +Because `converge` is an ordinary row in the `JOBS` table, `ScheduledJobStale`, +`ScheduledJobFailed` and `ScheduledJobNeverRan` cover "the convergence stopped +running" and "it ran and failed" with no new rule. `deploy.rules.yaml` adds only +what those cannot say — which *refusal*, and how far behind the host is as a +result. + +### Hourly, and the no-op path is free + +The cadence is a claim about how long a merged change may take to reach the +host, and an hour makes "merged" and "deployed" nearly the same word without +anything chasing a webhook. + +It is affordable because it costs one fetch, not one deploy: when the checkout +is already at the fetched tip and the tree is clean, `converge.sh` exits before +rendering anything and never calls Docker. Almost all 24 daily runs touch no +container. + +### Three things it deliberately does not do + +**It does not roll back.** Fast-forward only. A non-fast-forward means `main` +was rewritten or the checkout has local commits, and a timer that resolves +either one is a timer that can roll the host onto a revision somebody +deliberately replaced. + +**It does not converge anything but this host's stack.** `oracle` and `saruman` +run Alloy, which [`scripts/deploy-agent.sh`](../../scripts/deploy-agent.sh) +still pushes over SSH. Those hosts have no age key, no repository checkout and +no reason to grow either — and giving three machines a copy of this loop is a +larger decision than #99 asked for. + +**It does not check that running containers match their pins.** `make up` +converges the *configuration*; whether a running image still matches the digest +`compose.yaml` names is `make check-digests`'s question, and answering it here +would be two checks with one name. + +### Report-only exists, it is one variable, and it is how this ships + +`HOMELAB_CONVERGE_APPLY=0` in `/etc/default/homelab-timers` — the file every +unit here already reads — makes every run fetch, verify and record while +applying nothing. + +**That is the chosen rollout.** The timer goes on in report-only, and applying +is switched on once it has been watched deciding correctly for a while. "Let a +timer restart my monitoring stack unattended" is a reasonable thing to want to +see working first, and the alternative to a switch is not installing the timer +at all, which is the state that produced #99. + +The mode is a gauge, not just a file. `homelab_deploy_apply_enabled` is written +with the rest of the record, because a mode that lives only in `/etc` on one +host is unrecorded state — the exact thing this ADR exists to stop — and +because without it `DeployBehind` could only say "it is refusing, OR +report-only is set" and leave the operator to go and read a file. + +`DeployApplyDisabled` (info) reports the mode, and it is deliberately an alert +rather than an issue on the roadmap: it fires six hours in, and it **resolves by +itself** on the first run after the line is removed. There is nothing to close +and no way to leave it stale, which is the failure mode a tracked task for +"remember to turn this on" would have. Being temporary is enforced by the thing +itself rather than by anyone remembering. + +## Consequences + +- **`make up` over SSH still works and is still correct.** It is the escape + hatch, and it remains the documented way to deploy something urgently. What + changes is that the host no longer waits to be told. + +- **A merged pull request reaches the monitoring host within an hour, with no + human in the loop — once report-only is switched off.** That is the point, and + it is also the cost. Merging becomes deploying, so a change that would break + the stack breaks it at 03:25 rather than when someone was watching. The + mitigations are the ones this repository already leans on — `make validate` in + CI on every pull request, and a stack whose failure is loud. Until the switch + is flipped, `DeployBehind` and `DeployApplyDisabled` fire as a pair and + deployment stays manual. + +- **One new setup step, and it fails loudly rather than silently.** GitHub's + web-flow key has to be imported into `robo`'s keyring once. Until it is, + nothing on the host can verify anything, convergence refuses every run, and + `DeployUnverified` says so. That is deliberate: the alternative — treating an + unverifiable commit as fine — would make the gate decorative. + +- **Editing a file on the monitoring host now stops deployment.** Previously it + was silently destroyed at the next deploy. This is strictly better and will + still be annoying the first time it happens at an inconvenient moment; the + runbook's answer is two commands. + +- **`homelab_deploy_revision_info` adds one series per deployed revision.** At + 177 first-parent commits in roughly six months, that is a rounding error + against a stack already holding cAdvisor's per-container series, and it buys + the ability to read what was deployed and when off a Grafana panel. + +- **The estate is now split two ways on deployment, on purpose.** This host + pulls; `oracle` and `saruman` are pushed to by `deploy-agent.sh`. That is not + a transitional state and no issue tracks unifying it — the hosts that hold no + key and no checkout are better served by a script that ships them files. + +- **`ScheduledJobFailed` becomes a routine sight during the first week.** Every + refusal is an exit code, so a missing key or a stray edited file will fire it. + That is the design, and it is the reason `deploy.rules.yaml` names which + refusal rather than leaving a bare non-zero exit to be interpreted. diff --git a/docs/observability.md b/docs/observability.md index f715613..4275d80 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -275,7 +275,7 @@ separates a quiet stream from a stopped one. ## Alerting -64 rules in total: 48 metric-based in `prometheus/rules/`, and 16 log-based in +69 rules in total: 53 metric-based in `prometheus/rules/`, and 16 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -314,7 +314,7 @@ boot check. ### Metric-based (Prometheus) -48 rules across nine files in `prometheus/rules/`: +53 rules across ten files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -326,6 +326,7 @@ boot check. | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, and never-ran | +| `deploy.rules.yaml` | Whether this host is running what the repository says — an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` the host is. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)) | | `ids.rules.yaml` | Whether Suricata is running on each interface it is declared for, read from the firewall's process table over SNMP — the process metric `security.rules.yaml` says a log rule cannot be ([#90](https://github.com/Gerrrt/HomeLab/issues/90)) | `promtool check rules` validates that these parse. It does not — and cannot — @@ -335,11 +336,12 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is eighteen rules of 48 so far — the five +against the broken rule too. Coverage is twenty-three rules of 53 so far — the five in `blackbox.rules.yaml`, `ContainerHighMemory` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from [#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in -`backup.test.yaml`, `RemoteWriteJobStale`, and `SuricataStopped`. +`backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, +and `SuricataStopped`. The other 30 are still validated for syntax only, which is exactly the standing #63 had. Both numbers are checked by `scripts/check_docs.py` — the sentence they replaced claimed six and named two, and had been wrong for diff --git a/docs/roadmap.md b/docs/roadmap.md index 624835e..c961ead 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -286,7 +286,7 @@ what left this one unfireable for months. and a second observability stack. The umbrella. Nothing of it is built, but the shape is settled and the work is split seven ways, which is what moved it out of *Decided but not built* below. - [ADR-0019](adr/0019-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) + [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) answered the two questions ADR-0007 left open, and both of them blocked the first line of work. `stacks/lab/` runs in a **guest**, not on the hypervisor: `Saruman` is the one host in the estate that must not run Docker, because @@ -311,7 +311,7 @@ what left this one unfireable for months. [#268](https://github.com/Gerrrt/HomeLab/issues/268) PBS decides what it is for before it is installed, since a hypervisor backing up its own guests to itself is not a backup. Liveness stays where it already was, with - [#257](https://github.com/Gerrrt/HomeLab/issues/257): ADR-0019 decides only + [#257](https://github.com/Gerrrt/HomeLab/issues/257): ADR-0020 decides only that no Alertmanager goes *inside* the stack, and the lab is otherwise being built to go quiet. - **[#96](https://github.com/Gerrrt/HomeLab/issues/96) Procure `ifrit` and build @@ -372,7 +372,17 @@ what left this one unfireable for months. → [runbook](runbooks/ship-firewall-logs.md) - **[#99](https://github.com/Gerrrt/HomeLab/issues/99) Move deployment from `make up` over SSH to something pull-based**, so the host converges on the repo - rather than being pushed to. + rather than being pushed to. Answered by + [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md): an + hourly timer running `scripts/converge.sh`, on #77's existing wrapper and + alert machinery. The issue's stated blocker — an age key on a host that pulls + from a public repository — turned out not to be one, because the key was + already on that host and *public* means readable. The real question was + unattended execution, and the answer is a pinned signing fingerprint plus a + record of every revision deployed. What is deliberately left out: `oracle` and + `saruman` are still pushed to with `deploy-agent.sh`, and nothing tracks + unifying that. +- **[#100](https://github.com/Gerrrt/HomeLab/issues/100) Automate the Grafana ## Decided but not built diff --git a/docs/runbooks/converge-the-host.md b/docs/runbooks/converge-the-host.md new file mode 100644 index 0000000..ed9a405 --- /dev/null +++ b/docs/runbooks/converge-the-host.md @@ -0,0 +1,256 @@ +# Runbook: Let the host deploy itself + +How the monitoring host converges on `main`, how to set it up, and what to do +when it refuses. + +The decision and its reasoning are in +[ADR-0021](../adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md). +This is the operating half. The schedule this joins is +[`schedule-maintenance.md`](schedule-maintenance.md), and the manual deployment +it does not replace is [`deploy-stack.md`](deploy-stack.md). + +## Read this part first + +`scripts/converge.sh` runs hourly on `prometheus` and does five things: + +1. Fetches `main` from `https://github.com/Gerrrt/HomeLab.git` — anonymously, + with no credential, over a URL that cannot push. +2. Refuses to continue if the checkout has uncommitted changes. +3. Refuses to continue unless the fetched tip carries a good GPG signature from + GitHub's web-flow key. +4. Fast-forwards `main` — never a merge, never a rebase, never a rollback. +5. Runs `make up`, which is the same command a human runs. + +**A merged pull request is deployed within the hour.** That is the point of it. +If you need it sooner, `make converge` on the host does it now. + +**Every refusal is loud and leaves the host where it was.** Nothing here ever +overwrites a local edit, and nothing rolls the host backwards. + +> **This is not how `oracle` and `saruman` are deployed.** They run Alloy and +> are pushed to with [`make deploy-agent`](../../scripts/deploy-agent.sh). They +> have no repository checkout and no age key, and ADR-0021 §"Three things it +> deliberately does not do" is why that is not changing. + +## Set it up + +The timer is installed by `make install-timers` along with every other job — +see [`schedule-maintenance.md`](schedule-maintenance.md) §Install. There is one +step specific to this job, and it belongs **before** that: `make install-timers` +primes every job by running it once, so an unimported key means convergence +fails its very first run. + +### Import GitHub's signing key + +Convergence verifies every commit against one fingerprint. Until the key is in +`robo`'s keyring, nothing on this host can verify anything, so every run refuses +and `DeployUnverified` fires. Do this once, on the monitoring host, as `robo`: + +```bash +curl -fsSL https://github.com/web-flow.gpg | gpg --import +``` + +Then confirm you imported what you meant to. The fingerprint must be +`968479A1AFF927E37D1A566BB5690EEEBB952194`: + +```bash +gpg --fingerprint 968479A1AFF927E37D1A566BB5690EEEBB952194 +``` + +That file also contains `4AEE18F83AFDEB23`, which **expired on 2024-01-16** and +is not the key in use. Importing both is harmless — `converge.sh` accepts only +the fingerprint above — but do not confuse them if you are checking by eye. + +### Prove it before trusting it + +`--dry-run` does everything except the fast-forward and the `make up`: + +```bash +make converge ARGS=--dry-run +``` + +A converged host prints `converged — is main` and nothing else. A +host that is behind lists the commits it would apply and stops. + +### Start in report-only mode + +**This is how it is being rolled out.** The timer is installed and watched for a +while before it is allowed to change anything. Do this before +`make install-timers`, as root: + +```bash +printf 'HOMELAB_CONVERGE_APPLY=0\n' | sudo tee -a /etc/default/homelab-timers +``` + +Every run then fetches, verifies and records, and applies nothing — deployment +stays a thing a human does with `make up`. + +Two alerts describe that state together, and reading them as a pair is the +point: + +- `DeployApplyDisabled` (info) says the host is deliberately not deploying. It + fires six hours in and **resolves by itself** on the first run after the line + is removed. +- `DeployBehind` (warning) says how far behind it has got. In this mode that + alert *is* the report and firing is expected, not a fault. + +When only `DeployBehind` is firing, report-only is **not** the explanation and +something is genuinely refusing — see §When it refuses. + +### Letting it act + +Remove the line, then restart the timer: + +```bash +sudo sed -i '/^HOMELAB_CONVERGE_APPLY=0$/d' /etc/default/homelab-timers +``` + +```bash +sudo systemctl restart homelab-converge.timer && sudo systemctl start homelab-converge.service +``` + +`DeployApplyDisabled` resolves on that run. Nothing else needs doing: the first +convergence catches up however many commits have accumulated, in one +fast-forward. + +## What it records + +Six gauges in `/var/lib/node_exporter/textfile_collector/homelab-deploy.prom`, +written on every exit path including the refusals, so a run that declined to +move still reports what the host is on. + +| Metric | Question it answers | +| --- | --- | +| `homelab_deploy_revision_info{revision}` | What is deployed | +| `homelab_deploy_commit_timestamp_seconds` | How old the running configuration is | +| `homelab_deploy_behind_commits` | How far behind `main`; `-1` means the fetch failed | +| `homelab_deploy_tree_dirty` | Whether someone edited a file on the host | +| `homelab_deploy_verified` | Whether the deployed revision has a valid signature | +| `homelab_deploy_apply_enabled` | Whether this host applies what it fetches, or is in report-only mode | + +The quickest read of "what is this host running" is the journal, which Alloy +already ships to Loki: + +```bash +journalctl -u homelab-converge.service -n 20 --no-pager | grep homelab-deploy +``` + +`converge` is also an ordinary row in the `JOBS` table, so `ScheduledJobStale`, +`ScheduledJobFailed` and `ScheduledJobNeverRan` cover it exactly as they cover +the backups. + +## When it refuses + +Every one of these leaves the host running what it was already running. None of +them is an emergency, and none of them is fixed by re-running the timer. + +### The tree is dirty + +```text +error: the deployment checkout has uncommitted changes (above). +``` + +Someone edited a file on the monitoring host instead of in the repository. +Convergence has stopped and will stay stopped — this is the drift that used to +be destroyed silently at the next deploy. + +Look at it first, then choose. There is no third option and no `--force`: + +```bash +git -C /home/robo/code/Gerrrt/HomeLab status +``` + +```bash +git -C /home/robo/code/Gerrrt/HomeLab diff +``` + +To keep it, get it into the repository the normal way — a branch, a pull +request, CI. To discard it, `git checkout -- .` in that directory, and delete +any untracked files the status listed. + +### The tip did not verify + +```text +error: did not verify (signature: E, key: B5690EEEBB952194). +``` + +`signature: E` means gpg could not check it at all, which almost always means +the key was never imported — do the import above. + +```text +error: did not verify (signature: N, key: ). +``` + +`signature: N` means the commit carries no signature. `main` moved by something +other than a GitHub merge: a direct push past the pull request, or a tip served +by something that is not GitHub. **Look at it before you deploy it.** + +```bash +git -C /home/robo/code/Gerrrt/HomeLab log --show-signature -1 FETCH_HEAD +``` + +If it is genuinely yours and you want it anyway, deploy it deliberately and by +hand rather than teaching the timer to ignore signatures: + +```bash +/home/robo/code/Gerrrt/HomeLab/scripts/converge.sh --allow-unsigned +``` + +### It is not a fast-forward + +```text +error: is not a fast-forward from . +``` + +Either `main` was rewritten, or somebody committed on the deployment host. The +two commands the error prints tell you which — the second one lists commits the +host has that `main` does not. + +A rewritten `main` is a human decision to re-point the host at, and local +commits on the deployment checkout want rescuing to a branch before anything +else happens. Convergence deliberately resolves neither, because both +resolutions can roll the host onto a revision somebody replaced on purpose. + +### It refuses the directory + +```text +error: refusing to converge /home/robo/code/Gerrrt/HomeLab/.claude/worktrees/... +``` + +You ran it from a worktree or a second clone. `make render` writes into the +`.rendered/` of the tree it runs from, and no container mounts a worktree's +copy — so converging there would report success and change nothing. Run it from +`/home/robo/code/Gerrrt/HomeLab`. + +## If something goes wrong + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `DeployUnverified` right after install | GitHub's key is not in `robo`'s keyring, so nothing can verify | The import above. This is the expected state between installing the timer and doing it | +| `DeployUnverified` with the key present | `HEAD` is a commit that did not come through a pull request — usually someone committing on the host | `git log --show-signature -1` on the host. Get the commit onto a branch and merge it properly | +| `DeployDrifted` | A file was edited on the monitoring host | §"The tree is dirty". The edit is still there — this alert exists because it used to not be | +| `DeployBehind` **with** `DeployApplyDisabled` | Report-only mode — the host is fetching and recording but not applying | Working as intended. §Letting it act when you want it to deploy | +| `DeployBehind` **without** `DeployApplyDisabled` | Convergence is genuinely refusing | `journalctl -u homelab-converge.service -n 50` names the refusal; every case is in §When it refuses | +| `DeployApplyDisabled` you did not expect | Somebody set `HOMELAB_CONVERGE_APPLY=0` and it was forgotten | That is what this alert is for. `grep CONVERGE /etc/default/homelab-timers` | +| `DeployBehind` with `ScheduledJobFailed` | The refusal is real and recurring | The journal names it; every case is in §"When it refuses" | +| `DeployMetricsAbsent` | `homelab-deploy.prom` stopped arriving, while the backup metrics still do | Two separate files fail independently. Check `node_textfile_scrape_error`, then the file itself. If the timer was never installed, `make install-timers` | +| `homelab_job_last_exit_code{homelab_job="converge"}` is 75 | It never started — the weekly backup held the `backups` lock for the full 900s | Expected at most once a week, on Sunday. Persistent means a backup is hanging: `systemctl list-units 'homelab-*'` | +| The stack restarted at 03:25 and nobody deployed | Somebody merged a pull request | Working as designed — ADR-0021 §Consequences. `journalctl -u homelab-converge.service` names the revision | +| Convergence succeeds but a config change did not take | The container reads its config once at startup and nothing reloaded it | `make up` runs `reload-config.sh`, so this should not happen. If it does, it is a bug in that script's service list, not in convergence | +| `git fetch` fails every hour | No egress to github.com, or DNS | The host stays where it is, which is correct. `homelab_deploy_behind_commits` goes to `-1` rather than lying about the lag | + +## Turning it off + +The timer is one of several installed together; removing just this one: + +```bash +sudo systemctl disable --now homelab-converge.timer +``` + +The host then stays on whatever revision it is on until someone runs `make up` +or `make converge`, which is exactly the pre-#99 model. `homelab-deploy.prom` is +deliberately left in place — deleting it would make the host look like it had +never deployed rather than like it had stopped converging, and those are +different things. + +`make install-timers` puts it back. diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index bff123f..0995658 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -104,7 +104,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 48 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 53 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Seven dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken @@ -148,14 +148,36 @@ systemctl list-timers 'homelab-*' Full detail, including the one job that has no timer and never will, is in [`schedule-maintenance.md`](schedule-maintenance.md). +One of those timers deploys this host. `converge` runs hourly, fetches `main`, +verifies its signature and runs `make up` — so after this runbook, the section +below stops being something anybody has to do. It needs one setup step of its +own, a single GPG import: +[`converge-the-host.md`](converge-the-host.md). + ## Updating +**Normally you do not.** A merged pull request reaches this host within the hour +on its own ([ADR-0021](../adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)). +What follows is how to deploy something *now* rather than waiting, and what the +timer is doing on your behalf. + +```bash +make converge # fetch main, verify it, fast-forward, make up — the timer's job, now +``` + +Or by hand, which is the escape hatch when the change is not on `main` yet: + ```bash git pull make validate make up # recreates changed services, then reloads config on the rest ``` +Note that a hand-deploy of something uncommitted leaves the tree dirty, which +**stops convergence** until it is committed or discarded — deliberately, because +the alternative is the timer destroying your edit at :25. `DeployDrifted` says so +within two hours. + `make up` recreates a container only when its *service definition* changes — a changed bind-mounted config file is invisible to `docker compose up -d`. So `make up` finishes by reloading Prometheus, Alertmanager and snmp-exporter from @@ -177,6 +199,12 @@ git revert make up ``` +Revert on a branch and merge it if you can: a revert committed straight to the +deployment checkout is an unsigned local commit, which is both a non-fast-forward +against `main` and an unverified `HEAD`, so convergence stops until `main` +catches up. Fine as an emergency measure — that is what the escape hatch is for — +but it is a state to leave, not to stay in. + Data volumes survive `make down` and `make up`. Only `make nuke` destroys them, it prompts, and it is recoverable from a backup set — see [`restore-the-stack.md`](restore-the-stack.md). diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 03f75e8..4d45277 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -35,9 +35,13 @@ run, not a partial success. The same run bounds what accumulates on each side to change it. The copy needs a one-time key exchange between the two laptops, in [`restore-the-firewall.md`](restore-the-firewall.md) §0, and fails on purpose until that is done. The volume sets still do not leave -([#92](https://github.com/Gerrrt/HomeLab/issues/92)), and moving deployment to -something pull-based is [#99](https://github.com/Gerrrt/HomeLab/issues/99). -Until then the honest summary is that this host watches its own chores, and the +([#92](https://github.com/Gerrrt/HomeLab/issues/92)). Deployment itself is now +one of these jobs rather than something a human remembers to do — +[#99](https://github.com/Gerrrt/HomeLab/issues/99), +[ADR-0021](../adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md), and +[`converge-the-host.md`](converge-the-host.md) for the one setup step it needs +beyond this runbook. +The honest summary is still that this host watches its own chores, and the *external* cron-monitor described in [`verify-the-alert-path.md`](verify-the-alert-path.md) is the only thing watching the host. @@ -48,6 +52,7 @@ the host. | Job | Command | When | Alerts if not seen in | | --- | --- | --- | --- | +| `converge` | `make converge` | hourly, :25 | 3 hours | | `backup-volumes` | `make backup` | Sundays 03:30 | 14 days | | `verify-backups` | `make backup ARGS='--verify-only --all'` | daily 05:30 | 3 days | | `backup-firewall` | `make backup-firewall` | daily 04:30 | 3 days | @@ -55,9 +60,12 @@ the host. | `dashboards-drift` | `make dashboards-export ARGS=--check` | daily 07:30 | 2 days | | `verify-key-backup` | **you**, `make secrets-verify-backup KEY=…` | no timer | 90 days | -Thresholds are roughly twice the period in every case, never once: a threshold -equal to the period fires on every run that slips past its jitter window, whereas -twice tolerates one missed run and not two. +Thresholds are roughly twice the period, never once: a threshold equal to the +period fires on every run that slips past its jitter window, whereas twice +tolerates one missed run and not two. `converge` is the one exception at three +times, because it shares the `backups` lock and a run that queues behind the +weekly archive can legitimately spend its full 900-second wait and then be an +hour late. Being late for a reason is not the finding. `dashboards-drift` is the odd one out, and worth reading as a different kind of job. Every other row here proves that something *happened* — an archive was @@ -245,6 +253,7 @@ expected rather than a second fault. | `docker info` fails only under systemd | The unit is missing `SupplementaryGroups=docker` | A login shell picks the group up from `/etc/group` and a unit does not, which is why this never reproduces by hand | | Timers exist but never fire | `WantedBy=timers.target` missing, or the timers were never enabled | `systemctl list-timers 'homelab-*'` shows nothing; re-run `make install-timers` | | `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The four other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` | +| `converge` fails every hour with a signature error | GitHub's signing key was never imported into `robo`'s keyring, so nothing on this host can verify | The one-time import in [`converge-the-host.md`](converge-the-host.md) §Set it up. Every other job here is unaffected | | `refusing to install from …` | You are in a worktree or a second clone | The units hardcode the deployment path. Install from `/home/robo/code/Gerrrt/HomeLab` | | `make validate` fails on the schedule | A cadence and its threshold disagree | `make check-timers` names the job and both numbers. Fix the `JOBS` table or the `.timer`, not the alert | | `make validate` fails with *no `homelab-*` units are installed* | The stack is running on this host but the schedule was never installed | Exactly the condition above, caught before an alert has to. Only a host running the stack is asked; a laptop with the repository checked out skips it | diff --git a/docs/security.md b/docs/security.md index 7c28ff8..afc237c 100644 --- a/docs/security.md +++ b/docs/security.md @@ -16,7 +16,7 @@ What this network is actually built to survive: | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 64 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 69 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | diff --git a/scripts/converge.sh b/scripts/converge.sh new file mode 100755 index 0000000..a6288a3 --- /dev/null +++ b/scripts/converge.sh @@ -0,0 +1,454 @@ +#!/usr/bin/env bash +# +# Bring the deployment checkout to what `main` says, and record what is +# deployed (#99). +# +# scripts/converge.sh [--dry-run] [--allow-unsigned] +# +# WHY THIS EXISTS +# +# Deployment was `make up` typed into an SSH session, which has three problems +# and only the first one is obvious. +# +# 1. Nothing recorded what was deployed. `make up` on an uncommitted working +# tree and `make up` on `main` produce the same output and the same exit +# code, and the difference surfaces weeks later as a config nobody can +# account for. `oracle` is the worked example — scripts/deploy-agent.sh's +# header is four paragraphs of one host quietly running something other +# than what the repository said, for two days, because nothing compared +# the two. +# +# 2. Nothing detected drift. A config edited on the host stayed edited until +# the next deploy overwrote it silently, so the edit was lost AND never +# seen. +# +# 3. It is the "nothing schedules anything" problem (#77) wearing a different +# hat, and #77 already built the answer: a timer, a wrapper that records +# the outcome, and alert rules that read the record. This reuses all +# three rather than introducing a second way to run things on a schedule. +# +# WHAT CONVERGENCE MEANS HERE, EXACTLY +# +# Fetch `main` from the canonical URL, refuse to move unless the tip carries a +# good signature from the pinned key, fast-forward, and run `make up`. That is +# the whole loop. It deliberately does NOT reimplement deployment: `make up` is +# still what renders the config and starts the stack, so every runbook that +# says `make up` stays true and this script's blast radius is the DECISION to +# deploy, not the deployment. +# +# The no-op path costs one fetch. When the checkout is already at the fetched +# tip and the tree is clean, nothing is rendered, no container is touched and +# docker is never called — which is what makes an hourly cadence reasonable. +# +# WHY IT FETCHES A URL AND NOT `origin` +# +# `origin` is git@github.com:Gerrrt/HomeLab.git — SSH, with a key that can also +# push. A systemd unit has no ssh-agent, so that path would need a +# passphraseless key readable by an unattended process, and that key would +# carry write access to the repository this host executes. +# +# The repository is public, so the agent needs no credential at all. It fetches +# an explicit https:// URL, which cannot push and cannot be redirected by a +# rewritten `remote.origin.url` in a checkout someone has already edited. The +# URL is pinned below and asserted against `origin` only as a sanity check, not +# trusted from it. +# +# WHY THE SIGNATURE GATE, AND WHAT IT DOES NOT BUY +# +# A host that executes whatever a branch says, unattended, has moved the +# question from "do I trust this code" to "do I trust whoever can move that +# branch". The gate narrows it back: every commit is checked against ONE +# fingerprint pinned in this file, and `main` only moves if the tip verifies. +# +# That is not a guess about how this repository works, it is a measured +# property of it. Every one of the last 110 first-parent commits on `main` — +# unbroken back to PR #33 on 2026-08-19, which is the history purge and the +# last time anything reached `main` other than through a pull request — is a +# merge commit GitHub made and signed. All 110 verify against the fingerprint +# below, with `%G?` of `U` and `%GF` equal to the pin. +# +# So the gate costs nothing today and refuses two things it should refuse: a +# commit pushed straight to `main` past the pull request, and a tip served by +# anything that is not GitHub. +# +# It does NOT stop a compromised GitHub account. An attacker who can open and +# merge a pull request gets a signature like anyone else, and this host will +# deploy it within the hour. That risk is real, it is not new — the operator +# ran `make up` from this checkout after pulling, which executed the same code +# — and what this change alters is the window: from "whenever someone next +# deploys" to "at most an hour", with no human glancing at the diff. The +# compensating control is the record, not the gate. Every convergence writes +# the revision it deployed, and DeployBehind / DeployUnverified / DeployDrifted +# in prometheus/rules/deploy.rules.yaml make an unexpected one visible. +# +# WHY A DIRTY TREE IS A HARD STOP +# +# Refusing is the point. Overwriting is what the old model did, and losing the +# edit while never reporting it is problem 2 above. So an uncommitted change in +# the deployment checkout stops the run, exits non-zero, and shows up as +# ScheduledJobFailed and DeployDrifted — loudly, every hour, until a human +# either commits it or throws it away. There is no --force. `git checkout -- .` +# is one command and it is the human's to type. +# +# Usage: +# scripts/converge.sh fetch, verify, fast-forward, make up +# scripts/converge.sh --dry-run say what it would do, change nothing +# scripts/converge.sh --allow-unsigned fast-forward past a failed signature +# +# Environment: +# TEXTFILE_DIR where homelab-deploy.prom goes +# (default /var/lib/node_exporter/textfile_collector) +# HOMELAB_CONVERGE_APPLY 0 makes every run report-only, as though --dry-run +# had been passed. Set in /etc/default/homelab-timers +# to watch the agent decide for a while before +# letting it act. Recorded as +# homelab_deploy_apply_enabled, so the mode is +# visible from Prometheus rather than only from a +# file on the host. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +# The checkout the stack actually runs from. Same constant, same reasoning and +# the same refusal as scripts/install-timers.sh: `make render` writes into +# .rendered/ under the tree it is run from, and no container mounts a worktree's +# copy — so converging a second clone would report success while changing +# nothing the stack can see. +DEPLOY_ROOT="/home/robo/code/Gerrrt/HomeLab" + +# Read-only, credential-free, and not taken from the checkout's own config. +CANONICAL_URL="https://github.com/Gerrrt/HomeLab.git" +BRANCH="main" + +# GitHub's web-flow signing key, full fingerprint. Not the 16-hex key id: a key +# id is claimed by the signature itself and a fingerprint is not. The `%GF` +# placeholder is empty unless gpg actually verified, so comparing it to this is +# one comparison that asserts both "verified" and "by the right key". +# +# GitHub's published key file also carries 4AEE18F83AFDEB23, which EXPIRED on +# 2024-01-16 and is not this. Importing the file gets both; only this one is +# accepted. +SIGNING_FPR="968479A1AFF927E37D1A566BB5690EEEBB952194" + +TEXTFILE_DIR="${TEXTFILE_DIR:-/var/lib/node_exporter/textfile_collector}" +PROM="${TEXTFILE_DIR}/homelab-deploy.prom" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } +warn() { printf '\033[0;33mwarning:\033[0m %s\n' "$*" >&2; } +green(){ printf '\033[0;32m%s\033[0m\n' "$*"; } + +usage() { sed -n '/^# Usage:/,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } + +DRY_RUN=0 +ALLOW_UNSIGNED=0 +while (($#)); do + case "$1" in + --dry-run) DRY_RUN=1 ;; + --allow-unsigned) ALLOW_UNSIGNED=1 ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac + shift +done + +# The report-only switch, so the timer can be installed and watched before it is +# allowed to act. Folded into DRY_RUN rather than given a second code path — +# two ways to not-apply is two things to get wrong. +# +# APPLY_ENABLED is tracked SEPARATELY from DRY_RUN, and the distinction is the +# whole reason it exists. DRY_RUN is also set by --dry-run, which is a human +# asking a question; this is a property of how the host is configured. Only the +# second is worth recording, because only the second persists after the run and +# explains why a host stays behind. +# +# Without it, DeployBehind can only say "it is refusing, OR report-only is set", +# and telling those apart means someone with shell access reading a file in +# /etc that nothing else in this repository tracks. That is precisely the shape +# of unrecorded state #99 is about, so the mode goes in the record with +# everything else. +APPLY_ENABLED=1 +if [[ "${HOMELAB_CONVERGE_APPLY:-1}" == "0" ]]; then + info "HOMELAB_CONVERGE_APPLY=0 — reporting only, nothing will be applied" + DRY_RUN=1 + APPLY_ENABLED=0 +fi + +# --------------------------------------------------------------------------- +# The record +# --------------------------------------------------------------------------- +# +# Same contract as scripts/run-scheduled.sh, and for the same reasons: written +# to a temp in the SAME directory then renamed, because rename(2) is atomic +# within a filesystem and a truncating write exposes a half-file to the +# collector; mode set explicitly, because Alloy runs with cap_drop [ALL] and so +# obeys the mode; a missing directory warns and a non-writable one dies. +# +# A SEPARATE FILE from converge.prom, which run-scheduled.sh owns. The two say +# different things and have different lifetimes: run-scheduled.sh records +# whether the JOB ran, this records what the HOST is running. The second +# survives being meaningful even when the first says the job failed — a +# convergence that refused to move still knows the revision it refused at. +RECORD=1 +if [[ ! -d "${TEXTFILE_DIR}" ]]; then + RECORD=0 + warn "no textfile directory at ${TEXTFILE_DIR} — converging without recording what is deployed" + warn "on the monitoring host this means the timers were never installed: make install-timers" +elif [[ ! -w "${TEXTFILE_DIR}" ]]; then + die "${TEXTFILE_DIR} is not writable by $(id -un). +The host would converge and nothing would record what it converged to, which is +the failure this script exists to prevent. Fix the directory, then re-run: + sudo install -d -m 0755 -o $(id -un) -g $(id -gn) ${TEXTFILE_DIR}" +fi + +# Filled in as the run progresses, written once by record(). Every one has a +# value that is honest before anything has been measured: an unverified +# revision, an unknown lag of -1, and a tree assumed clean until looked at. +REVISION="" +COMMIT_TS=0 +BEHIND=-1 +DIRTY=0 +VERIFIED=0 + +recorded=0 +record() { + local tmp + ((recorded)) && return 0 + recorded=1 + ((RECORD)) || return 0 + [[ -n "${REVISION}" ]] || return 0 + + tmp="${TEXTFILE_DIR}/homelab-deploy.prom.$$" + cat > "${tmp}" </dev/null 2>&1 || die "git is not installed" + +# Read before any check about the STATE of this checkout — the branch it is on, +# whether it is clean, what it can fetch — so that every one of those refusals +# still records what the host is running. A refusal that leaves yesterday's +# metric in place is a refusal that reads as a healthy deployment. +# +# Three guards do run earlier and can exit before this line, and all three are +# cases where recording nothing is the correct outcome rather than a gap: +# +# - an unwritable TEXTFILE_DIR, where recording is impossible by definition +# and dying is the point; +# - REPO_ROOT != DEPLOY_ROOT, where HEAD belongs to some other checkout, and +# writing its revision as "what is deployed" would be an actively false +# statement about the host rather than a missing one; +# - git absent, where there is no revision to read. +# +# So the claim is narrower than "before anything that can fail", and stating it +# loosely was wrong: a maintainer reading the loose version would think the +# earlier guards were an oversight to fix. +REVISION="$(git rev-parse --short=12 HEAD)" +COMMIT_TS="$(git log -1 --format=%ct HEAD)" + +# --------------------------------------------------------------------------- +# Signature +# --------------------------------------------------------------------------- +# +# Defined here and applied to HEAD immediately, so homelab_deploy_verified is a +# claim about the revision the host IS RUNNING on every exit path — including +# the paths that give up before fetching anything. Evaluating it only alongside +# the fetched tip would make the metric mean "the last thing we were offered", +# which is a different and much less useful sentence. +verify_commit() { + local ref="$1" sig fpr + sig="$(git log -1 --format='%G?' "${ref}")" + fpr="$(git log -1 --format='%GF' "${ref}")" + [[ "${fpr}" == "${SIGNING_FPR}" ]] || return 1 + # G is a good signature from a key marked trusted; U is a good signature from + # a key that is not. Both are accepted, because ownertrust is a statement + # about a local keyring and the fingerprint above is the actual assertion — + # requiring G would mean every host had to run `gpg --lsign-key` as well as + # import, for no additional guarantee. + [[ "${sig}" == "G" || "${sig}" == "U" ]] || return 1 + return 0 +} + +verify_commit HEAD && VERIFIED=1 + +current_branch="$(git symbolic-ref --quiet --short HEAD || true)" +[[ "${current_branch}" == "${BRANCH}" ]] \ + || die "the deployment checkout is on '${current_branch:-a detached HEAD}', not ${BRANCH}. +Convergence only ever fast-forwards ${BRANCH}. Someone left this checkout +somewhere else; put it back deliberately rather than letting a timer do it: + git -C ${DEPLOY_ROOT} switch ${BRANCH}" + +# `origin` is not used for anything — the fetch names its own URL — but a +# checkout whose origin has been repointed is worth saying out loud, because it +# means someone has been editing the deployment host's git config. +origin_url="$(git remote get-url origin 2>/dev/null || true)" +case "${origin_url}" in + "${CANONICAL_URL}"|git@github.com:Gerrrt/HomeLab.git|https://github.com/Gerrrt/HomeLab) ;; + "") warn "no 'origin' remote configured — fetching ${CANONICAL_URL} regardless" ;; + *) warn "origin is ${origin_url}, which is not the canonical repository. + Nothing here reads it — the fetch below names ${CANONICAL_URL} explicitly — + but somebody changed it, and that is worth knowing." ;; +esac + +# --------------------------------------------------------------------------- +# Drift: has anything on the host diverged from what is committed? +# --------------------------------------------------------------------------- +# +# --porcelain skips ignored files, which is exactly right: .rendered/, .env, +# certificates/ and backups/ are all gitignored, all written by the deploy +# itself, and none of them is drift. What is left is a tracked file someone +# edited in place, or an untracked file someone dropped in the tree — both of +# which are the thing #99 says goes unnoticed until a deploy destroys it. +dirty_files="$(git status --porcelain --untracked-files=normal)" +if [[ -n "${dirty_files}" ]]; then + DIRTY=1 + printf '%s\n' "${dirty_files}" >&2 + die "the deployment checkout has uncommitted changes (above). + +Converging would overwrite them, which is exactly the silent loss #99 is about, +so this stops instead and will keep stopping — DeployDrifted and +ScheduledJobFailed will both be firing — until a human decides which it is: + + keep it git -C ${DEPLOY_ROOT} diff # then commit it, on a branch, via a pull request + drop it git -C ${DEPLOY_ROOT} checkout -- . # and remove any untracked files it listed + +There is no --force. Choosing is the whole point." +fi + +# --------------------------------------------------------------------------- +# Fetch +# --------------------------------------------------------------------------- +info "fetching ${BRANCH} from ${CANONICAL_URL}" +# --no-tags because nothing here reads a tag and a tag is another thing that can +# move. No --depth: a shallow fetch has no merge base, so the fast-forward +# assertion and the behind-count below would both be unanswerable. +git fetch --quiet --no-tags "${CANONICAL_URL}" "${BRANCH}" \ + || die "could not fetch ${BRANCH} from ${CANONICAL_URL}. +The host stays on ${REVISION}, which is the correct outcome of not knowing what +${BRANCH} says. If this persists, DeployBehind will not fire — nothing was +learned about how far behind the host is — but ScheduledJobFailed will." + +TARGET="$(git rev-parse FETCH_HEAD)" +BEHIND="$(git rev-list --count "HEAD..${TARGET}")" + +# --------------------------------------------------------------------------- +# Verify the tip before it becomes the deployed revision +# --------------------------------------------------------------------------- +target_verified=0 +verify_commit "${TARGET}" && target_verified=1 + +if ((target_verified == 0)); then + # Distinguish the two reasons, because they need different responses: a + # missing key is a setup step nobody did, and a missing signature is a commit + # that did not come through a pull request. + detail="signature: $(git log -1 --format='%G?' "${TARGET}"), key: $(git log -1 --format='%GK' "${TARGET}" || true)" + if ! gpg --batch --list-keys "${SIGNING_FPR}" >/dev/null 2>&1; then + hint="The signing key is not in $(id -un)'s keyring on this host, so nothing +CAN verify. Import it once — docs/runbooks/converge-the-host.md: + curl -fsSL https://github.com/web-flow.gpg | gpg --import +Then confirm the fingerprint it printed is ${SIGNING_FPR}." + else + hint="The key is present and this commit did not verify against it, which +means ${BRANCH} moved by something other than a GitHub merge — a direct push, or +a tip served by something that is not GitHub. Look at it before deploying it: + git -C ${DEPLOY_ROOT} log --show-signature -1 ${TARGET}" + fi + + if ((ALLOW_UNSIGNED)); then + warn "${TARGET} did not verify (${detail}) — continuing because --allow-unsigned was passed" + else + die "${TARGET} did not verify (${detail}). + +${hint} + +The host stays on ${REVISION}. To deploy it anyway, deliberately and by hand: + ${DEPLOY_ROOT}/scripts/converge.sh --allow-unsigned" + fi +fi + +# --------------------------------------------------------------------------- +# Converge +# --------------------------------------------------------------------------- +if [[ "${TARGET}" == "$(git rev-parse HEAD)" ]]; then + BEHIND=0 + green "converged — ${REVISION} is ${BRANCH}" + # Nothing rendered, no container touched, docker never called. This is the + # path an hourly cadence spends almost all of its time on. + exit 0 +fi + +# Fast-forward only. A non-fast-forward means `main` was rewritten or this +# checkout has commits of its own, and quietly resolving either one is how a +# deployment host ends up running something no branch points at. +git merge-base --is-ancestor HEAD "${TARGET}" \ + || die "${TARGET} is not a fast-forward from ${REVISION}. +Either ${BRANCH} was rewritten, or this checkout has local commits. Both need a +human — a timer that resolves this is a timer that can roll the host backwards +onto a revision someone deliberately replaced. + git -C ${DEPLOY_ROOT} log --oneline ${REVISION}..${TARGET} + git -C ${DEPLOY_ROOT} log --oneline ${TARGET}..${REVISION}" + +info "${BEHIND} commit(s) behind — ${REVISION} to $(git rev-parse --short=12 "${TARGET}")" +git --no-pager log --oneline --no-decorate "HEAD..${TARGET}" | sed 's/^/ /' >&2 + +if ((DRY_RUN)); then + # REVISION, COMMIT_TS and VERIFIED still describe HEAD, which is still what is + # deployed — the whole point of not applying. Only BEHIND changed, and it is + # the number that says so. + warn "dry run — not applying" + exit 0 +fi + +git merge --ff-only --quiet "${TARGET}" +REVISION="$(git rev-parse --short=12 HEAD)" +COMMIT_TS="$(git log -1 --format=%ct HEAD)" +BEHIND=0 +VERIFIED="${target_verified}" + +# `make up` and not a narrower command, on purpose. It renders the config, +# recreates whatever compose says changed, and runs reload-config.sh for the +# services that read their config once at startup — and it is what every runbook +# already tells a human to type, so there is exactly one deployment path and it +# is exercised both ways. +info "applying ${REVISION}" +make up + +green "converged to ${REVISION}" diff --git a/scripts/install-timers.sh b/scripts/install-timers.sh index 4b81591..63671f6 100755 --- a/scripts/install-timers.sh +++ b/scripts/install-timers.sh @@ -85,8 +85,15 @@ DEPLOY_ROOT="/home/robo/code/Gerrrt/HomeLab" # SecretsKeyBackupUnproven can still nag. The one thing that proves the secrets # are recoverable now has a deadline even though it has no schedule. # +# converge is the only hourly row, and the only one whose threshold is three +# times its period rather than two. It shares the `backups` lock with the two +# backup jobs, so a run that collides with the weekly archive can legitimately +# spend its whole 900s lock wait and then be an hour late; twice the period +# would alert on that, and being late for a reason is not the finding. +# # job unit prefix max_age make target JOBS=( + "converge homelab-converge 10800 converge" "backup-volumes homelab-backup-volumes 1209600 backup" "verify-backups homelab-verify-backups 259200 backup" "backup-firewall homelab-backup-firewall 259200 backup-firewall" @@ -360,7 +367,7 @@ for row in "${JOBS[@]}"; do done # Run each job once so the timers do not spend their first night looking like -# four jobs that have never run — and so the plumbing is proven now rather than +# jobs that have never run — and so the plumbing is proven now rather than # at 03:30. backup-volumes is excluded: it quiesces the monitoring stack, and # that is not something to do as a side effect of an install. if ((RUN_ONCE)); then diff --git a/stacks/observability/README.md b/stacks/observability/README.md index e5c25ef..dc6b257 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox.yaml probe targets — hot-reloaded, no restart needed - rules/*.rules.yaml 48 alert rules across host/network/ups/containers/blackbox/backup/ids + rules/*.rules.yaml 53 alert rules across host/network/ups/containers/blackbox/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability from outside the service alertmanager/ diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index 17f8a65..a0ec92c 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -20,9 +20,16 @@ # # homelab_job_max_age_seconds is written by scripts/install-timers.sh from the # JOBS table that also decides the cadence, so a threshold and a schedule cannot -# drift apart. It also means NO RULE HERE NAMES A JOB: adding a sixth timer is a -# row in that table, not an edit to this file. That is deliberate — a rule file -# listing job names is a list that silently stops growing. +# drift apart. It also means NO RULE HERE NAMES A JOB: adding a timer is a row in +# that table, not an edit to this file. That is deliberate — a rule file listing +# job names is a list that silently stops growing. +# +# That claim has since been tested rather than merely asserted. #99 added +# `converge`, an hourly job on a different cadence to everything above, and +# every rule in this file covered it with no edit at all. What #99 DID need was +# a separate file — prometheus/rules/deploy.rules.yaml — because the rules here +# describe whether a job RAN, and deployment also has results worth alerting on: +# which refusal, and how far behind the host is. # # WHY THERE IS NO absent() # diff --git a/stacks/observability/prometheus/rules/deploy.rules.yaml b/stacks/observability/prometheus/rules/deploy.rules.yaml new file mode 100644 index 0000000..54fb813 --- /dev/null +++ b/stacks/observability/prometheus/rules/deploy.rules.yaml @@ -0,0 +1,204 @@ +--- +# Whether this host is running what the repository says (#99). +# +# Deployment used to be `make up` typed into an SSH session. Nothing recorded +# what got deployed, so `make up` on an uncommitted tree and `make up` on `main` +# were indistinguishable; and nothing compared the host to the repository, so a +# config edited in place stayed edited until a deploy silently overwrote it. +# scripts/converge.sh now runs hourly and writes six gauges into the same +# textfile directory the maintenance jobs use. +# +# WHAT backup.rules.yaml ALREADY COVERS, AND IS NOT REPEATED HERE +# +# `converge` is a row in the JOBS table like any other, so ScheduledJobStale, +# ScheduledJobFailed and ScheduledJobNeverRan already say "the convergence +# stopped running" and "the convergence ran and failed". None of that is +# restated below. +# +# What those rules cannot say is anything about the RESULT. A convergence that +# correctly refuses to move — because the tree is dirty, or the tip did not +# verify — exits non-zero and is fully described by ScheduledJobFailed, but the +# interesting fact is WHICH refusal and how far behind the host now is. These +# five rules read the record converge.sh writes rather than its exit code. +# +# WHY THERE IS NO ALERT ON THE REVISION ITSELF +# +# homelab_deploy_revision_info exists to be read, not to be alerted on. "Is this +# the right commit" is a question with no threshold — the answer is a git log — +# and a rule that tried would either hardcode a revision or fire on every +# deploy. The alertable form of that question is DeployBehind, which asks the +# same thing against a moving target instead of a fixed one. +groups: + - name: deploy + interval: 60s + rules: + - alert: DeployDrifted + # Someone edited a file on the monitoring host. + # + # This is the alert #99 asked for by name. converge.sh refuses to + # fast-forward over an uncommitted change rather than overwriting it, so + # by the time this fires the edit is still there and still recoverable — + # which is the whole difference from the old model, where the edit was + # destroyed and the destruction was also invisible. + # + # 2h, not 30m. converge.sh rewrites this gauge once an hour, so a `for` + # shorter than the cadence asserts nothing beyond "one sample said so". + # Two hours means a second convergence looked and the edit was still + # uncommitted, which is the difference between someone mid-keystroke and + # a change nobody is going to commit. + expr: homelab_deploy_tree_dirty == 1 + for: 2h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "The deployment checkout has uncommitted changes" + description: >- + Something was edited on the monitoring host instead of in the + repository, and convergence has stopped until it is resolved — the + host is frozen on whatever it was already running. The change is + still there: `git -C /home/robo/code/Gerrrt/HomeLab status` and + `git diff` show it. Commit it through a pull request or discard it — + docs/runbooks/converge-the-host.md. + + - alert: DeployBehind + # The host knows about commits it has not applied. + # + # In steady state this is unfireable: converge.sh sets the gauge to 0 + # immediately after a successful fast-forward, so a run that applies + # cleanly never leaves a positive value behind. It fires when + # convergence is refusing (an unverified tip, a rewritten history), or + # when HOMELAB_CONVERGE_APPLY=0 has deliberately put the timer in + # report-only mode — in which case this alert IS the report and is + # expected to fire. + # + # Which of those two it is used to be unanswerable without shell access + # to read /etc/default/homelab-timers. DeployApplyDisabled below now + # fires alongside this one in the report-only case, so the pair reads as + # one sentence and neither rule needs to guess. + # + # > 0 and not != 0 on purpose. A failed fetch records -1 rather than a + # stale count, precisely so it cannot masquerade as a known lag; the + # fetch failing is ScheduledJobFailed's to report, and this rule stays + # quiet about a number it does not have. + # + # 3h matches the 10800s staleness budget scripts/install-timers.sh + # declares for the job, so a convergence that queued behind the weekly + # backup and then missed its next slot does not alert twice for one + # cause. + expr: homelab_deploy_behind_commits > 0 + for: 3h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "The host is {{ $value }} commit(s) behind main" + description: >- + Convergence has run and has not applied what it fetched. Either it + is refusing — check `journalctl -u homelab-converge.service -n 50` + for a dirty tree, a signature that did not verify, or a + non-fast-forward. If DeployApplyDisabled is firing too, it is + neither: the host is in report-only mode and this alert is that + mode's report. docs/runbooks/converge-the-host.md. + + - alert: DeployUnverified + # The deployed revision does not carry a good signature from the pinned + # key. + # + # Two very different causes, and the description names both because the + # metric cannot tell them apart. The dull one is that GitHub's web-flow + # key was never imported into robo's keyring, so nothing on this host + # CAN verify and convergence has been refusing since it was installed — + # the expected state between installing the timer and doing the one-time + # import. The interesting one is that somebody committed directly on the + # deployment host, which produces an unsigned HEAD and would otherwise + # look exactly like a normal deploy. + # + # Deliberately NOT gated on behind_commits. An unverifiable HEAD is a + # finding whether or not there is anything newer to move to. + expr: homelab_deploy_verified == 0 + for: 1h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "The deployed revision has no valid signature" + description: >- + Either GitHub's signing key is not in robo's keyring on this host — + the one-time setup step, and convergence is refusing to move until + it is done — or HEAD is a commit that did not come through a pull + request. `git -C /home/robo/code/Gerrrt/HomeLab log --show-signature -1` + distinguishes them. docs/runbooks/converge-the-host.md. + + - alert: DeployApplyDisabled + # The host is deliberately not deploying. + # + # This is the only rule here that reports a CHOICE rather than a fault, + # and it exists because the choice is made in /etc/default/homelab-timers + # — a file outside this repository, which nothing tracks and which is + # exactly the kind of thing that is still set eighteen months later + # because everyone who knew forgot. + # + # info, not warning: nothing is broken and nothing needs doing tonight. + # It routes to the info receiver alongside the other two info rules. + # + # It resolves by itself the moment the line is removed and the timer + # runs again, which is what makes it a better reminder than an issue — + # there is nothing to close, and no way to leave it stale. + # + # `for: 6h` because a deliberate mode is not urgent and a six-hour floor + # keeps it out of the way during the install itself, when converge is + # being primed and restarted by hand. + expr: homelab_deploy_apply_enabled == 0 + for: 6h + labels: + component: deploy + severity: info + category: correctness + annotations: + summary: "Convergence is in report-only mode and is deploying nothing" + description: >- + HOMELAB_CONVERGE_APPLY=0 is set in /etc/default/homelab-timers, so + this host fetches, verifies and records but applies nothing — + deployment is back to being something a human does. Intended as a + temporary state while the agent is watched. Remove the line and + `systemctl restart homelab-converge.timer` to let it act; this + resolves on the next run. docs/runbooks/converge-the-host.md. + + - alert: DeployMetricsAbsent + # The measuring apparatus, and not a duplicate of + # ScheduledJobMetricsAbsent. + # + # That rule watches homelab_job_max_age_seconds, which + # scripts/install-timers.sh writes into homelab-jobs.prom. These three + # rules read homelab-deploy.prom, which scripts/converge.sh writes. Two + # files, and node_exporter's textfile collector fails them + # independently: a malformed or deleted homelab-deploy.prom costs only + # this file's series, leaves node_textfile_scrape_error at 1 with every + # backup rule still green, and takes DeployDrifted, DeployBehind and + # DeployUnverified silently with it. Absence looking healthy is the one + # failure mode this repository keeps rebuilding the same guard against. + # + # revision_info is the series chosen because converge.sh writes it on + # every exit path that knows a revision at all, including the refusals. + # absent_over_time rather than absent, for the reason + # backup.rules.yaml's equivalent gives: six hours survives a Prometheus + # restart and a single missed scrape, where absent() fires on both. + expr: absent_over_time(homelab_deploy_revision_info[6h]) + for: 1h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "No deployment metrics have reached Prometheus for six hours" + description: >- + Nothing else in this file can fire while this is true — what is + deployed on this host is currently unknown. Check + node_textfile_scrape_error, then + /var/lib/node_exporter/textfile_collector/homelab-deploy.prom on the + monitoring host. If the timer was never installed, `make + install-timers` — docs/runbooks/converge-the-host.md. diff --git a/stacks/observability/prometheus/tests/deploy.test.yaml b/stacks/observability/prometheus/tests/deploy.test.yaml new file mode 100644 index 0000000..cb1ce94 --- /dev/null +++ b/stacks/observability/prometheus/tests/deploy.test.yaml @@ -0,0 +1,222 @@ +--- +# Unit tests for deploy.rules.yaml. +# +# Every firing case is paired with a silent one, per containers.test.yaml: a +# test that only ever expects silence would have passed against the unfireable +# rule in #63, and a test that only ever expects firing would pass against a +# rule that fires unconditionally. +# +# Two cases here exist to pin specific ways this file could break without +# `promtool check rules` noticing: +# +# 1. THE -1 SENTINEL. scripts/converge.sh records behind_commits as -1 when +# the fetch did not complete, so that "we do not know" cannot be read as +# "we are behind". A rule written `!= 0` instead of `> 0` passes every +# other test here and turns every network blip into a DeployBehind. The +# second case feeds -1 and asserts silence. +# +# 2. THE instance LABEL. These metrics come from the same textfile collector +# as homelab_job_*, so they carry the scrape target's `instance`. The +# firing cases assert it survives into the alert — without it a second +# monitored host would be indistinguishable from this one, which is the +# trap backup.test.yaml documents for the joined rules. +rule_files: + - ../rules/deploy.rules.yaml + +evaluation_interval: 1m + +tests: + # --- DeployDrifted: dirty, and clean ------------------------------------ + # + # Two hosts in one series set, one with an edited tree and one without, so a + # rule that ignored the value and fired on the series existing would fail. + - interval: 1m + input_series: + - series: 'homelab_deploy_tree_dirty{instance="prometheus"}' + values: "1+0x300" + - series: 'homelab_deploy_tree_dirty{instance="oracle"}' + values: "0+0x300" + alert_rule_test: + # `for: 2h`, so still only pending at 1h50m. This is what asserts the + # `for` is actually there — converge.sh rewrites the gauge hourly and a + # rule without it would alert on a single sample. + - eval_time: 1h50m + alertname: DeployDrifted + exp_alerts: [] + + - eval_time: 2h5m + alertname: DeployDrifted + exp_alerts: + - exp_labels: + alertname: DeployDrifted + instance: prometheus + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "The deployment checkout has uncommitted changes" + description: >- + Something was edited on the monitoring host instead of in the + repository, and convergence has stopped until it is resolved — + the host is frozen on whatever it was already running. The + change is still there: `git -C /home/robo/code/Gerrrt/HomeLab + status` and `git diff` show it. Commit it through a pull request + or discard it — docs/runbooks/converge-the-host.md. + + # --- DeployBehind: behind, converged, and "we do not know" --------------- + # + # The -1 case is the point of this block. A fetch that fails records -1, and + # `> 0` must treat that as silence while `!= 0` would not. + - interval: 1m + input_series: + - series: 'homelab_deploy_behind_commits{instance="prometheus"}' + values: "3+0x300" + - series: 'homelab_deploy_behind_commits{instance="oracle"}' + values: "0+0x300" + - series: 'homelab_deploy_behind_commits{instance="saruman"}' + values: "-1+0x300" + alert_rule_test: + - eval_time: 2h55m + alertname: DeployBehind + exp_alerts: [] + + # Only `prometheus` fires: `oracle` is converged and `saruman` could not + # fetch, which is ScheduledJobFailed's finding and not this rule's. + - eval_time: 3h5m + alertname: DeployBehind + exp_alerts: + - exp_labels: + alertname: DeployBehind + instance: prometheus + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "The host is 3 commit(s) behind main" + description: >- + Convergence has run and has not applied what it fetched. Either + it is refusing — check `journalctl -u homelab-converge.service + -n 50` for a dirty tree, a signature that did not verify, or a + non-fast-forward. If DeployApplyDisabled is firing too, it is + neither: the host is in report-only mode and this alert is that + mode's report. docs/runbooks/converge-the-host.md. + + # --- DeployUnverified: unsigned, and signed ----------------------------- + - interval: 1m + input_series: + - series: 'homelab_deploy_verified{instance="prometheus"}' + values: "0+0x180" + - series: 'homelab_deploy_verified{instance="oracle"}' + values: "1+0x180" + alert_rule_test: + - eval_time: 50m + alertname: DeployUnverified + exp_alerts: [] + + - eval_time: 1h5m + alertname: DeployUnverified + exp_alerts: + - exp_labels: + alertname: DeployUnverified + instance: prometheus + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "The deployed revision has no valid signature" + description: >- + Either GitHub's signing key is not in robo's keyring on this + host — the one-time setup step, and convergence is refusing to + move until it is done — or HEAD is a commit that did not come + through a pull request. `git -C /home/robo/code/Gerrrt/HomeLab + log --show-signature -1` distinguishes them. + docs/runbooks/converge-the-host.md. + + # --- DeployApplyDisabled: report-only, and applying -------------------- + # + # The paired silent case is the one that matters: a host that IS applying must + # not report itself as deliberately stopped, or the alert stops meaning + # anything the moment it is used for its intended week. + - interval: 1m + input_series: + - series: 'homelab_deploy_apply_enabled{instance="prometheus"}' + values: "0+0x480" + - series: 'homelab_deploy_apply_enabled{instance="oracle"}' + values: "1+0x480" + alert_rule_test: + - eval_time: 5h55m + alertname: DeployApplyDisabled + exp_alerts: [] + + - eval_time: 6h5m + alertname: DeployApplyDisabled + exp_alerts: + - exp_labels: + alertname: DeployApplyDisabled + instance: prometheus + component: deploy + severity: info + category: correctness + exp_annotations: + summary: "Convergence is in report-only mode and is deploying nothing" + description: >- + HOMELAB_CONVERGE_APPLY=0 is set in + /etc/default/homelab-timers, so this host fetches, verifies and + records but applies nothing — deployment is back to being + something a human does. Intended as a temporary state while the + agent is watched. Remove the line and `systemctl restart + homelab-converge.timer` to let it act; this resolves on the next + run. docs/runbooks/converge-the-host.md. + + # --- DeployMetricsAbsent: the series is there --------------------------- + # + # Paired with the block below, which is the same rule with nothing fed to it. + # Silence here is what proves the firing case is about absence and not about + # the rule being unconditional. + - interval: 1m + input_series: + - series: 'homelab_deploy_revision_info{instance="prometheus", revision="09d618ede580"}' + values: "1+0x600" + alert_rule_test: + - eval_time: 7h + alertname: DeployMetricsAbsent + exp_alerts: [] + + # --- DeployMetricsAbsent: homelab-deploy.prom stopped arriving ---------- + # + # No input_series at all. A backup metric IS supplied, to pin the thing this + # rule exists for: homelab-jobs.prom and homelab-deploy.prom are separate + # files that the textfile collector fails independently, so the schedule can + # look perfectly healthy while nothing knows what is deployed. + - interval: 1m + input_series: + - series: 'homelab_job_max_age_seconds{homelab_job="converge", instance="prometheus"}' + values: "10800+0x600" + alert_rule_test: + # `for: 1h` is the only delay this case can assert, and that is worth + # saying plainly: the series never exists here, so the 6h lookback is + # empty from t=0 and absent_over_time is true immediately. The six hours + # only buys anything against real data, where it is the gap that has to + # exceed the window. Measured rather than assumed — this case was + # originally written expecting a 6h delay and promtool disagreed. + - eval_time: 55m + alertname: DeployMetricsAbsent + exp_alerts: [] + + - eval_time: 1h5m + alertname: DeployMetricsAbsent + exp_alerts: + - exp_labels: + alertname: DeployMetricsAbsent + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "No deployment metrics have reached Prometheus for six hours" + description: >- + Nothing else in this file can fire while this is true — what is + deployed on this host is currently unknown. Check + node_textfile_scrape_error, then + /var/lib/node_exporter/textfile_collector/homelab-deploy.prom on + the monitoring host. If the timer was never installed, `make + install-timers` — docs/runbooks/converge-the-host.md. diff --git a/systemd/homelab-backup-volumes.service b/systemd/homelab-backup-volumes.service index 2d209a6..8341bbb 100644 --- a/systemd/homelab-backup-volumes.service +++ b/systemd/homelab-backup-volumes.service @@ -1,6 +1,6 @@ # Weekly quiesced backup of the observability stack's volumes. # -# This unit is the fully-commented one of the four; the others state only what +# This unit is the fully-commented one of the five; the others state only what # differs. Installed by scripts/install-timers.sh, which is also what asserts # that every ExecStart= here goes through run-scheduled.sh (#77). [Unit] @@ -27,7 +27,7 @@ SupplementaryGroups=docker # Load-bearing, not tidiness. backup-firewall.sh resolves backups/firewall and # .sops.yaml RELATIVE to the process's directory, so on that unit this line is # the difference between a backup and a crash. Same directive everywhere so the -# four units cannot disagree about where the repository is. +# five units cannot disagree about where the repository is. WorkingDirectory=/home/robo/code/Gerrrt/HomeLab Environment=STACK=observability # Explicit even though User= already sets $HOME from the passwd entry: the diff --git a/systemd/homelab-converge.service b/systemd/homelab-converge.service new file mode 100644 index 0000000..78749e3 --- /dev/null +++ b/systemd/homelab-converge.service @@ -0,0 +1,69 @@ +# Hourly convergence of this host onto what `main` says (#99). +# +# This is the one unit here that CHANGES the host rather than reading it, so +# two directives below are load-bearing in a way they are not elsewhere. +# +# SupplementaryGroups=docker, because converge.sh ends in `make up` and that +# talks to the daemon. Same reasoning as the backup units: a login shell picks +# the group up from /etc/group and a unit does not, so without it the failure +# appears only under systemd and never by hand. +# +# The `backups` lock, shared with homelab-backup-volumes and +# homelab-verify-backups. Not tidiness — homelab-backup-volumes STOPS the whole +# stack for the length of its archive, and a convergence that ran `make up` in +# that window would restart Prometheus, Loki and Grafana underneath a backup +# that is mid-read. The result would be an archive nothing had quiesced, which +# is exactly the kind of backup that looks fine until it is restored. +# +# See homelab-backup-volumes.service for why each of the remaining directives is +# set. +[Unit] +Description=HomeLab: converge this host onto the main branch +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/converge-the-host.md +Wants=docker.service network-online.target +After=docker.service network-online.target + +[Service] +Type=oneshot +User=robo +Group=robo +SupplementaryGroups=docker +WorkingDirectory=/home/robo/code/Gerrrt/HomeLab +Environment=STACK=observability +Environment=SOPS_AGE_KEY_FILE=/home/robo/.config/sops/age/keys.txt +# HOMELAB_CONVERGE_APPLY=0 goes here to make the timer report-only — it fetches, +# verifies and records how far behind the host is, and applies nothing. The +# leading - means an absent file is not an error, so the default is to apply. +EnvironmentFile=-/etc/default/homelab-timers +ExecStart=/home/robo/code/Gerrrt/HomeLab/scripts/run-scheduled.sh --job converge --lock backups --lock-wait 900 -- /usr/bin/make -C /home/robo/code/Gerrrt/HomeLab converge +# 900s of that is the lock wait. The convergence itself is a fetch on almost +# every run and a `make up` on the few that are not. +TimeoutStartSec=1800 +# Deliberately not Restart=on-failure, and the reason is sharper here than on +# the backup units. Every way this fails is a refusal — a dirty tree, an +# unverified tip, a history that was rewritten — and a refusal is a decision, +# not a transient error. Retrying a decision just makes it again. The timer +# comes back in an hour, and the alert fires either way. +Restart=no +Nice=10 +# No IOSchedulingClass=idle. `make up` recreating a container is latency-bound +# on the daemon, not throughput-bound on the disk, and deprioritising it only +# lengthens the window in which the stack is half-restarted. +ProtectHome=false +ProtectSystem=false +# Off, for the reason homelab-backup-volumes.service gives: the daemon resolves +# -v source paths in the HOST mount namespace, so a private /tmp silently +# bind-mounts the wrong directory into any container this starts. +PrivateTmp=false +NoNewPrivileges=true +RestrictSUIDSGID=true +LockPersonality=true +RestrictRealtime=true +StandardOutput=journal +StandardError=journal +SyslogIdentifier=homelab-converge + +# No [Install] section, on purpose — the timer is what enables this. A oneshot +# .service carrying WantedBy=multi-user.target would converge and restart the +# stack at every boot, which is a deploy nobody asked for at the least +# convenient moment. diff --git a/systemd/homelab-converge.timer b/systemd/homelab-converge.timer new file mode 100644 index 0000000..797a68f --- /dev/null +++ b/systemd/homelab-converge.timer @@ -0,0 +1,34 @@ +[Unit] +Description=HomeLab: hourly convergence onto main +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/converge-the-host.md + +[Timer] +Unit=homelab-converge.service +# Hourly, and hourly is a statement about how long a merged change may take to +# reach the host — an hour is short enough that "merged" and "deployed" are +# nearly the same word, and long enough that nothing here is chasing a webhook. +# +# The cost of the cadence is one fetch, not one deploy: converge.sh exits before +# it renders anything when the checkout is already at the fetched tip, so almost +# every one of these 24 daily runs touches no container at all. +# +# :25 rather than :00 or :30 because every other unit here is on one of those +# two marks, and the three that share the `backups` lock should not queue behind +# each other by default when they could simply not collide. +OnCalendar=*-*-* *:25:00 +# Five minutes, not the fifteen the other timers use. Jitter here spreads load +# on GitHub rather than on this host, one fetch is not load worth spreading, and +# a wide window makes "how stale can the host be" harder to state than it needs +# to be. +RandomizedDelaySec=300 +# True. A missed convergence — the host was off, or rebooting — should be picked +# up at boot rather than waiting for the next hour, because the state it is +# catching up to is the whole point. This is the opposite of the reasoning on +# homelab-backup-volumes.timer, where a replayed run quiesces the stack at an +# arbitrary hour; a replayed run here just deploys what should already be +# deployed. +Persistent=true +AccuracySec=1m + +[Install] +WantedBy=timers.target