From a43be0148eecc8b3230b47c5f66d7509ec35a668 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 4 Sep 2026 05:47:52 +0000 Subject: [PATCH 1/2] feat(deploy): converge the host on main hourly instead of deploying over SSH (#99) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deployment was `make up` typed into an SSH session. Nothing recorded what got deployed, so `make up` on an uncommitted tree and `make up` on `main` were indistinguishable; and nothing compared the host to the repository, so a config edited in place stayed edited until a deploy silently overwrote it. scripts/converge.sh runs hourly on #77's existing machinery — the same timer pattern, the same run-scheduled.sh wrapper, the same textfile directory. It fetches `main`, refuses to move unless the tip verifies, fast-forwards, and runs `make up`. It does not reimplement deploying: `make up` stays the one deployment path, so every runbook that names it stays true. The issue's stated blocker was an age key on a host that pulls from a public repository. Checked, and it dissolves: the key was already on that host and has decrypted every `make up` there since the stack was deployed — *public* means readable, and a key that decrypts local files is not endangered by strangers being able to read the repository. The real question is unattended execution, which is about write access, so the gate is a pinned signing fingerprint: - measured, not assumed — all 110 consecutive first-parent commits on `main` back to 2026-08-19 are GitHub-signed merges, and all 110 verify against 968479A1AFF927E37D1A566BB5690EEEBB952194; - fetches an anonymous https:// URL rather than `origin`, so the agent holds no credential and cannot push; - refuses a direct push to `main` and a non-fast-forward, and does not roll back. ADR-0019 states plainly what the gate does not buy: it cannot stop a compromised GitHub account, and the compensating control is the record, not the gate. A dirty tree is a hard stop with no --force. Refusing is the point — overwriting is what the old model did, and losing the edit while never reporting it is half of what #99 is about. Five gauges are recorded on every exit path including the refusals, so a run that declined to move still reports what the host is on. `converge` is an ordinary JOBS row, so ScheduledJobStale/Failed/NeverRan cover it with no rule edit — which tests backup.rules.yaml's claim that adding a timer is a row in a table. deploy.rules.yaml adds only what those cannot say: which refusal, and how far behind the host is. All four new rules have promtool unit tests. HOMELAB_CONVERGE_APPLY=0 in /etc/default/homelab-timers makes every run report-only, for watching it decide before letting it act. Verified against a real clone: converged no-op, behind + dry-run, dirty tree, missing key, unsigned tip, non-fast-forward, wrong branch, and wrong checkout. Co-Authored-By: Claude Opus 5 --- Makefile | 12 + README.md | 29 +- ...n-a-timer-instead-of-deploying-over-ssh.md | 237 ++++++++++ docs/observability.md | 10 +- docs/roadmap.md | 11 +- docs/runbooks/converge-the-host.md | 226 ++++++++++ docs/runbooks/deploy-stack.md | 30 +- docs/runbooks/schedule-maintenance.md | 21 +- docs/security.md | 2 +- scripts/converge.sh | 419 ++++++++++++++++++ scripts/install-timers.sh | 9 +- stacks/observability/README.md | 2 +- .../prometheus/rules/backup.rules.yaml | 13 +- .../prometheus/rules/deploy.rules.yaml | 164 +++++++ .../prometheus/tests/deploy.test.yaml | 186 ++++++++ systemd/homelab-backup-volumes.service | 4 +- systemd/homelab-converge.service | 69 +++ systemd/homelab-converge.timer | 34 ++ 18 files changed, 1448 insertions(+), 30 deletions(-) create mode 100644 docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md create mode 100644 docs/runbooks/converge-the-host.md create mode 100755 scripts/converge.sh create mode 100644 stacks/observability/prometheus/rules/deploy.rules.yaml create mode 100644 stacks/observability/prometheus/tests/deploy.test.yaml create mode 100644 systemd/homelab-converge.service create mode 100644 systemd/homelab-converge.timer diff --git a/Makefile b/Makefile index ff15bc5..7bf991e 100644 --- a/Makefile +++ b/Makefile @@ -46,6 +46,18 @@ down: ## Stop the stack (volumes are preserved) .PHONY: restart restart: down up ## Restart the stack +.PHONY: converge +converge: ## Fetch main, verify it, fast-forward and deploy (ARGS=--dry-run) + @# What the hourly timer runs, and what a human runs to deploy on purpose + @# without waiting for it. It ends in `make up` rather than replacing it, so + @# there is exactly one deployment path and both callers exercise it. + @# + @# It refuses to run anywhere but /home/robo/code/Gerrrt/HomeLab, for the + @# reason `make up` cares about and `make deploy-agent` does not: render + @# writes into the .rendered/ of the tree it is run from, and no container + @# mounts a worktree's copy. ARGS=--dry-run says what it would do. + ./scripts/converge.sh $(ARGS) + .PHONY: pull pull: ## Pull the pinned images $(COMPOSE) pull diff --git a/README.md b/README.md index 2dcd70c..4d52090 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ incident. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 140 panels, and - 61 alert rules — 48 metric-based in Prometheus, 13 log-based in Loki — sharing + 65 alert rules — 52 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -160,7 +160,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 48 alert rules +│ ├── prometheus/ # config, file_sd targets, 52 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host @@ -172,12 +172,12 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). ├── docs/ │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md -│ ├── adr/ # 14 architecture decision records -│ └── runbooks/ # deploy, add device, rotate creds, certs, key backup, -│ # purge, restore the firewall, restore the stack, -│ # ship firewall logs, verify the alert path, -│ # enable suricata, fit the UPS battery, -│ # add a host override +│ ├── adr/ # 19 architecture decision records +│ └── runbooks/ # deploy, converge, add device, rotate creds, certs, +│ # key backup, purge, restore the firewall, +│ # restore the stack, ship firewall logs, +│ # verify the alert path, enable suricata, +│ # fit the UPS battery, add a host override └── Makefile # make help ``` @@ -210,9 +210,18 @@ trust `certificates/ca.pem` — step 4 of that runbook. Full procedure, verification steps and troubleshooting in [`docs/runbooks/deploy-stack.md`](docs/runbooks/deploy-stack.md). +That is the first deploy. After it, the monitoring host deploys itself: a timer +runs `scripts/converge.sh` hourly, which fetches `main`, refuses it unless the +tip carries GitHub's signature, fast-forwards and runs the same `make up` — +recording what it deployed and refusing to overwrite anything edited on the host +([#99](https://github.com/Gerrrt/HomeLab/issues/99), +[ADR-0019](docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md), +[`docs/runbooks/converge-the-host.md`](docs/runbooks/converge-the-host.md)). + ```console $ make help up Render config and start the stack + converge Fetch main, verify it, fast-forward and deploy down Stop the stack (volumes are preserved) reload Hot-reload Prometheus, Alertmanager and snmp-exporter (no restart) secrets-init Generate an age keypair and create the encrypted secrets file @@ -225,8 +234,8 @@ $ make help ... ``` -The timers are what stop `backup`, `backup-firewall` and `snmp-verify` being -things someone has to remember, and the alert rules that come with them fire on a +The timers are what stop `backup`, `backup-firewall`, `snmp-verify` and now +deployment itself being things someone has to remember, and the alert rules that come with them fire on a job having *stopped being run* rather than only on one that failed ([#77](https://github.com/Gerrrt/HomeLab/issues/77)). One job deliberately has no timer: `secrets-verify-backup` needs a human to mount removable media, so it gets diff --git a/docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md b/docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md new file mode 100644 index 0000000..995c1d5 --- /dev/null +++ b/docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md @@ -0,0 +1,237 @@ +# ADR-0019: Converge on a timer instead of deploying over SSH + +**Status:** Accepted · 2026-09 + +## Context + +Deployment is `make up`, typed into an SSH session on `prometheus` +(`10.0.99.20`). [#99](https://github.com/Gerrrt/HomeLab/issues/99) asks for +something pull-based, so the host converges on the repository rather than being +pushed to, and names three reasons. + +**Nothing records what is deployed.** `make up` on an uncommitted working tree +and `make up` on `main` produce the same output and the same exit code. The +difference surfaces weeks later, as a configuration nobody can account for. +This is not hypothetical here: the header of +[`scripts/deploy-agent.sh`](../../scripts/deploy-agent.sh) is four paragraphs +about `oracle` running a different Alloy version, a different container shape +and a config three weeks stale, for two days, because nothing compared the host +to the repository. + +**Nothing detects drift.** A configuration edited on the host stays edited until +the next deploy overwrites it silently. Both halves are bad, and the second is +worse: the edit is destroyed *and* the destruction is invisible. + +**It is [#77](https://github.com/Gerrrt/HomeLab/issues/77) again.** "Nothing +schedules anything" and "nothing deploys anything" want the same three parts — +something on a timer, a wrapper that records what it did, and alert rules that +read the record. #77 already built all three. + +### The secrets constraint, re-examined + +The issue names the awkward part: `render-config.sh` needs the age key, so a +pull-based agent needs it too, "and that is a key sitting on a host that pulls +from a public repository." + +Checked rather than accepted, and it dissolves. The key is already there. It has +been since the stack was first deployed — [`.sops.yaml`](../../.sops.yaml) says +so in as many words, `~/.config/sops/age/keys.txt` is where it lives, and every +`make up` over SSH has decrypted with it on that host. A pull-based agent +running as the same user, on the same machine, reading the same file, adds no +exposure at all. + +The word doing the damage in that sentence is *public*, and public means +**readable**. A key that decrypts local files is not endangered by strangers +being able to read the repository; it would be endangered by strangers being +able to *write* it. So the constraint the issue was reaching for is real but +differently shaped: + +> A host that executes whatever a branch says, unattended, has replaced "do I +> trust this code" with "do I trust whoever can move that branch." + +That is the question this ADR has to answer, and it is a question about write +access, not about visibility. + +### What actually protects `main` today + +Measured on 2026-09-04, against this repository: + +| Property of `main` | Result | +| --- | --- | +| First-parent commits | 177 | +| Consecutive GitHub-signed merge commits from the tip | 110 | +| Oldest of that run | `2ea4cb4`, PR #33, 2026-08-19 | +| What is at 2026-08-19 | The git-history secret purge — the last force-push | +| Signing key on all 110 | `968479A1AFF927E37D1A566BB5690EEEBB952194` | +| `%G?` on all 110, with that key imported | `U` — good signature, untrusted keyring | + +Every advance of `main` for the last three weeks and 110 merges has been a merge +commit GitHub made and signed, because every change went through a pull request. +Nothing enforced that; it is simply how this repository has been worked. But it +is a property that can be *checked*, and checking it converts "trust the remote" +into "trust one fingerprint pinned on the host." + +### The options + +| Option | Why not | +| --- | --- | +| Flux, Argo CD | Both reconcile Kubernetes objects. There is no Kubernetes here and [ADR-0004](0004-one-compose-stack-per-host.md) is a decision not to have any. Adopting one to deploy a Compose file means adopting a cluster to run the operator that deploys the Compose file | +| `ansible-pull` | The closest fit, and still the wrong size. It brings a second configuration language, a second templating system and a second secrets story alongside SOPS, to schedule a `make` target that already exists. The playbook would be a wrapper around `make up` | +| A GitHub Actions self-hosted runner | Inverts the security story rather than improving it. A runner holds a registration token, keeps an outbound connection to GitHub, and runs whatever a workflow file says — and workflow files are in the repository being deployed. It is a push model with extra steps and a credential | +| A webhook receiver | A port to publish, a service to pin and back up, and a secret to rotate, so that deploys are prompt. [ADR-0012](0012-publish-only-ports-with-an-off-host-consumer.md) is the standing decision against publishing a port with no off-host consumer, and "prompt" is worth an hour at most here | +| `git pull` in cron | This, minus the record, minus the drift check, minus the refusals, and minus the alerting. The distance between that one line and what is decided below *is* the issue | + +## Decision + +**A script on an hourly systemd timer fetches `main`, verifies its signature, +fast-forwards, and runs `make up` — reusing #77's timer, wrapper and alert +machinery rather than introducing a second way to run things on a schedule.** + +[`scripts/converge.sh`](../../scripts/converge.sh), `homelab-converge.timer`, +one row in the `JOBS` table in +[`scripts/install-timers.sh`](../../scripts/install-timers.sh), and +[`prometheus/rules/deploy.rules.yaml`](../../stacks/observability/prometheus/rules/deploy.rules.yaml). + +### It decides when to deploy; it does not reimplement deploying + +The last thing `converge.sh` does is `make up`. It does not learn to render +config, start containers or reload Prometheus — `make up` already does all +three, and it is what every runbook tells a human to type. + +This keeps the blast radius of this ADR on the *decision* to deploy rather than +on deployment itself. Every existing runbook stays true, there is exactly one +deployment path, and both callers — the timer and the human — exercise it. + +### The gate is a fingerprint, not a remote + +Convergence refuses to move unless the fetched tip verifies against +`968479A1AFF927E37D1A566BB5690EEEBB952194`, pinned in the script. A full +fingerprint and not the 16-hex key id, because a key id is claimed by the +signature itself and a fingerprint is not. + +It also fetches an explicit `https://` URL rather than `origin`. `origin` is +SSH, and an unattended process using it would need a passphraseless key that can +also *push* to the repository this host executes. The repository is public, so +the agent needs no credential whatsoever — and a read-only URL that cannot push +is a better thing for a deployment host to hold than a key that can. + +**What the gate buys:** a commit pushed straight to `main` past the pull request +does not deploy, and neither does a tip served by anything that is not GitHub. + +**What it does not buy, stated plainly:** it does not stop a compromised GitHub +account. Someone who can open and merge a pull request gets a signature like +everybody else, and this host deploys it within the hour. That exposure is not +new — the operator ran `make up` from this checkout after pulling, which +executed exactly the same code — but the window changes, from "whenever someone +next deploys, having probably glanced at the diff" to "at most an hour, with +nobody looking." **The compensating control is the record, not the gate.** + +### Drift is refused, never overwritten + +An uncommitted change in the deployment checkout stops the run. It is not +overwritten, not stashed and not forced past; the run exits non-zero and keeps +doing so every hour, with `DeployDrifted` and `ScheduledJobFailed` both firing, +until a human commits the change or throws it away. + +Overwriting is what the old model did. Refusing is the entire point, so there is +no `--force`: `git checkout -- .` is one command and it belongs to the human. + +### What gets recorded + +`converge.sh` writes five gauges into the textfile directory +[`run-scheduled.sh`](../../scripts/run-scheduled.sh) already writes to, in its +own file, on every exit path — **including the refusals**, because a convergence +that declined to move still knows what the host is running. + +| Metric | Answers | +| --- | --- | +| `homelab_deploy_revision_info{revision}` | What is deployed | +| `homelab_deploy_commit_timestamp_seconds` | How old the running configuration is | +| `homelab_deploy_behind_commits` | How far behind `main` the host is; `-1` for "the fetch failed", so not-knowing cannot read as being-behind | +| `homelab_deploy_tree_dirty` | Whether anyone edited the host | +| `homelab_deploy_verified` | Whether the deployed revision carries a good signature | + +Because `converge` is an ordinary row in the `JOBS` table, `ScheduledJobStale`, +`ScheduledJobFailed` and `ScheduledJobNeverRan` cover "the convergence stopped +running" and "it ran and failed" with no new rule. `deploy.rules.yaml` adds only +what those cannot say — which *refusal*, and how far behind the host is as a +result. + +### Hourly, and the no-op path is free + +The cadence is a claim about how long a merged change may take to reach the +host, and an hour makes "merged" and "deployed" nearly the same word without +anything chasing a webhook. + +It is affordable because it costs one fetch, not one deploy: when the checkout +is already at the fetched tip and the tree is clean, `converge.sh` exits before +rendering anything and never calls Docker. Almost all 24 daily runs touch no +container. + +### Three things it deliberately does not do + +**It does not roll back.** Fast-forward only. A non-fast-forward means `main` +was rewritten or the checkout has local commits, and a timer that resolves +either one is a timer that can roll the host onto a revision somebody +deliberately replaced. + +**It does not converge anything but this host's stack.** `oracle` and `saruman` +run Alloy, which [`scripts/deploy-agent.sh`](../../scripts/deploy-agent.sh) +still pushes over SSH. Those hosts have no age key, no repository checkout and +no reason to grow either — and giving three machines a copy of this loop is a +larger decision than #99 asked for. + +**It does not check that running containers match their pins.** `make up` +converges the *configuration*; whether a running image still matches the digest +`compose.yaml` names is `make check-digests`'s question, and answering it here +would be two checks with one name. + +### Report-only exists, and it is one variable + +`HOMELAB_CONVERGE_APPLY=0` in `/etc/default/homelab-timers` — the file every +unit here already reads — makes every run fetch, verify and record while +applying nothing. `DeployBehind` then fires as the report, which is what that +mode is for. + +It is there because "let a timer restart my monitoring stack unattended" is a +reasonable thing to want to watch before allowing, and because the alternative +to a switch is not installing the timer, which is the state that produced #99. + +## Consequences + +- **`make up` over SSH still works and is still correct.** It is the escape + hatch, and it remains the documented way to deploy something urgently. What + changes is that the host no longer waits to be told. + +- **A merged pull request reaches the monitoring host within an hour, with no + human in the loop.** That is the point, and it is also the cost. Merging + becomes deploying, so a change that would break the stack breaks it at 03:25 + rather than when someone was watching. The mitigations are the ones this + repository already leans on — `make validate` in CI on every pull request, and + a stack whose failure is loud. + +- **One new setup step, and it fails loudly rather than silently.** GitHub's + web-flow key has to be imported into `robo`'s keyring once. Until it is, + nothing on the host can verify anything, convergence refuses every run, and + `DeployUnverified` says so. That is deliberate: the alternative — treating an + unverifiable commit as fine — would make the gate decorative. + +- **Editing a file on the monitoring host now stops deployment.** Previously it + was silently destroyed at the next deploy. This is strictly better and will + still be annoying the first time it happens at an inconvenient moment; the + runbook's answer is two commands. + +- **`homelab_deploy_revision_info` adds one series per deployed revision.** At + 177 first-parent commits in roughly six months, that is a rounding error + against a stack already holding cAdvisor's per-container series, and it buys + the ability to read what was deployed and when off a Grafana panel. + +- **The estate is now split two ways on deployment, on purpose.** This host + pulls; `oracle` and `saruman` are pushed to by `deploy-agent.sh`. That is not + a transitional state and no issue tracks unifying it — the hosts that hold no + key and no checkout are better served by a script that ships them files. + +- **`ScheduledJobFailed` becomes a routine sight during the first week.** Every + refusal is an exit code, so a missing key or a stray edited file will fire it. + That is the design, and it is the reason `deploy.rules.yaml` names which + refusal rather than leaving a bare non-zero exit to be interpreted. diff --git a/docs/observability.md b/docs/observability.md index e45d2d4..406c52e 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -256,7 +256,7 @@ separates a quiet stream from a stopped one. ## Alerting -61 rules in total: 48 metric-based in `prometheus/rules/`, and 13 log-based in +65 rules in total: 52 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -285,7 +285,7 @@ boot check. ### Metric-based (Prometheus) -48 rules across nine files in `prometheus/rules/`: +52 rules across ten files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -297,6 +297,7 @@ boot check. | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, and never-ran | +| `deploy.rules.yaml` | Whether this host is running what the repository says — an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` the host is. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0019](adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md)) | | `ids.rules.yaml` | Whether Suricata is running on each interface it is declared for, read from the firewall's process table over SNMP — the process metric `security.rules.yaml` says a log rule cannot be ([#90](https://github.com/Gerrrt/HomeLab/issues/90)) | `promtool check rules` validates that these parse. It does not — and cannot — @@ -306,11 +307,12 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is eighteen rules of 48 so far — the five +against the broken rule too. Coverage is twenty-two rules of 52 so far — the five in `blackbox.rules.yaml`, `ContainerHighMemory` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from [#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in -`backup.test.yaml`, `RemoteWriteJobStale`, and `SuricataStopped`. +`backup.test.yaml`, all four in `deploy.test.yaml`, `RemoteWriteJobStale`, +and `SuricataStopped`. The other 30 are still validated for syntax only, which is exactly the standing #63 had. Both numbers are checked by `scripts/check_docs.py` — the sentence they replaced claimed six and named two, and had been wrong for diff --git a/docs/roadmap.md b/docs/roadmap.md index bb35acd..15071de 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -238,7 +238,16 @@ what left this one unfireable for months. behind ADR-0008's sensitive tier. - **[#99](https://github.com/Gerrrt/HomeLab/issues/99) Move deployment from `make up` over SSH to something pull-based**, so the host converges on the repo - rather than being pushed to. + rather than being pushed to. Answered by + [ADR-0019](adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md): an + hourly timer running `scripts/converge.sh`, on #77's existing wrapper and + alert machinery. The issue's stated blocker — an age key on a host that pulls + from a public repository — turned out not to be one, because the key was + already on that host and *public* means readable. The real question was + unattended execution, and the answer is a pinned signing fingerprint plus a + record of every revision deployed. What is deliberately left out: `oracle` and + `saruman` are still pushed to with `deploy-agent.sh`, and nothing tracks + unifying that. - **[#100](https://github.com/Gerrrt/HomeLab/issues/100) Automate the Grafana dashboard export step** — the current loop is manual and therefore skipped under pressure. diff --git a/docs/runbooks/converge-the-host.md b/docs/runbooks/converge-the-host.md new file mode 100644 index 0000000..1ac55b0 --- /dev/null +++ b/docs/runbooks/converge-the-host.md @@ -0,0 +1,226 @@ +# Runbook: Let the host deploy itself + +How the monitoring host converges on `main`, how to set it up, and what to do +when it refuses. + +The decision and its reasoning are in +[ADR-0019](../adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md). +This is the operating half. The schedule this joins is +[`schedule-maintenance.md`](schedule-maintenance.md), and the manual deployment +it does not replace is [`deploy-stack.md`](deploy-stack.md). + +## Read this part first + +`scripts/converge.sh` runs hourly on `prometheus` and does five things: + +1. Fetches `main` from `https://github.com/Gerrrt/HomeLab.git` — anonymously, + with no credential, over a URL that cannot push. +2. Refuses to continue if the checkout has uncommitted changes. +3. Refuses to continue unless the fetched tip carries a good GPG signature from + GitHub's web-flow key. +4. Fast-forwards `main` — never a merge, never a rebase, never a rollback. +5. Runs `make up`, which is the same command a human runs. + +**A merged pull request is deployed within the hour.** That is the point of it. +If you need it sooner, `make converge` on the host does it now. + +**Every refusal is loud and leaves the host where it was.** Nothing here ever +overwrites a local edit, and nothing rolls the host backwards. + +> **This is not how `oracle` and `saruman` are deployed.** They run Alloy and +> are pushed to with [`make deploy-agent`](../../scripts/deploy-agent.sh). They +> have no repository checkout and no age key, and ADR-0019 §"Three things it +> deliberately does not do" is why that is not changing. + +## Set it up + +The timer is installed by `make install-timers` along with every other job — +see [`schedule-maintenance.md`](schedule-maintenance.md) §Install. There is one +step specific to this job, and it belongs **before** that: `make install-timers` +primes every job by running it once, so an unimported key means convergence +fails its very first run. + +### Import GitHub's signing key + +Convergence verifies every commit against one fingerprint. Until the key is in +`robo`'s keyring, nothing on this host can verify anything, so every run refuses +and `DeployUnverified` fires. Do this once, on the monitoring host, as `robo`: + +```bash +curl -fsSL https://github.com/web-flow.gpg | gpg --import +``` + +Then confirm you imported what you meant to. The fingerprint must be +`968479A1AFF927E37D1A566BB5690EEEBB952194`: + +```bash +gpg --fingerprint 968479A1AFF927E37D1A566BB5690EEEBB952194 +``` + +That file also contains `4AEE18F83AFDEB23`, which **expired on 2024-01-16** and +is not the key in use. Importing both is harmless — `converge.sh` accepts only +the fingerprint above — but do not confuse them if you are checking by eye. + +### Prove it before trusting it + +`--dry-run` does everything except the fast-forward and the `make up`: + +```bash +make converge ARGS=--dry-run +``` + +A converged host prints `converged — is main` and nothing else. A +host that is behind lists the commits it would apply and stops. + +### Optional: watch it for a week before letting it act + +To install the timer but keep it from changing anything, put this in +`/etc/default/homelab-timers` on the host: + +```bash +HOMELAB_CONVERGE_APPLY=0 +``` + +Every run then fetches, verifies and records, and applies nothing. +`DeployBehind` fires whenever the host is behind — in this mode that alert *is* +the report, and it firing is the expected state, not a fault. Remove the line +and `systemctl restart homelab-converge.timer` when you want it to act. + +## What it records + +Five gauges in `/var/lib/node_exporter/textfile_collector/homelab-deploy.prom`, +written on every exit path including the refusals, so a run that declined to +move still reports what the host is on. + +| Metric | Question it answers | +| --- | --- | +| `homelab_deploy_revision_info{revision}` | What is deployed | +| `homelab_deploy_commit_timestamp_seconds` | How old the running configuration is | +| `homelab_deploy_behind_commits` | How far behind `main`; `-1` means the fetch failed | +| `homelab_deploy_tree_dirty` | Whether someone edited a file on the host | +| `homelab_deploy_verified` | Whether the deployed revision has a valid signature | + +The quickest read of "what is this host running" is the journal, which Alloy +already ships to Loki: + +```bash +journalctl -u homelab-converge.service -n 20 --no-pager | grep homelab-deploy +``` + +`converge` is also an ordinary row in the `JOBS` table, so `ScheduledJobStale`, +`ScheduledJobFailed` and `ScheduledJobNeverRan` cover it exactly as they cover +the backups. + +## When it refuses + +Every one of these leaves the host running what it was already running. None of +them is an emergency, and none of them is fixed by re-running the timer. + +### The tree is dirty + +```text +error: the deployment checkout has uncommitted changes (above). +``` + +Someone edited a file on the monitoring host instead of in the repository. +Convergence has stopped and will stay stopped — this is the drift that used to +be destroyed silently at the next deploy. + +Look at it first, then choose. There is no third option and no `--force`: + +```bash +git -C /home/robo/code/Gerrrt/HomeLab status +``` + +```bash +git -C /home/robo/code/Gerrrt/HomeLab diff +``` + +To keep it, get it into the repository the normal way — a branch, a pull +request, CI. To discard it, `git checkout -- .` in that directory, and delete +any untracked files the status listed. + +### The tip did not verify + +```text +error: did not verify (signature: E, key: B5690EEEBB952194). +``` + +`signature: E` means gpg could not check it at all, which almost always means +the key was never imported — do the import above. + +```text +error: did not verify (signature: N, key: ). +``` + +`signature: N` means the commit carries no signature. `main` moved by something +other than a GitHub merge: a direct push past the pull request, or a tip served +by something that is not GitHub. **Look at it before you deploy it.** + +```bash +git -C /home/robo/code/Gerrrt/HomeLab log --show-signature -1 FETCH_HEAD +``` + +If it is genuinely yours and you want it anyway, deploy it deliberately and by +hand rather than teaching the timer to ignore signatures: + +```bash +/home/robo/code/Gerrrt/HomeLab/scripts/converge.sh --allow-unsigned +``` + +### It is not a fast-forward + +```text +error: is not a fast-forward from . +``` + +Either `main` was rewritten, or somebody committed on the deployment host. The +two commands the error prints tell you which — the second one lists commits the +host has that `main` does not. + +A rewritten `main` is a human decision to re-point the host at, and local +commits on the deployment checkout want rescuing to a branch before anything +else happens. Convergence deliberately resolves neither, because both +resolutions can roll the host onto a revision somebody replaced on purpose. + +### It refuses the directory + +```text +error: refusing to converge /home/robo/code/Gerrrt/HomeLab/.claude/worktrees/... +``` + +You ran it from a worktree or a second clone. `make render` writes into the +`.rendered/` of the tree it runs from, and no container mounts a worktree's +copy — so converging there would report success and change nothing. Run it from +`/home/robo/code/Gerrrt/HomeLab`. + +## If something goes wrong + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `DeployUnverified` right after install | GitHub's key is not in `robo`'s keyring, so nothing can verify | The import above. This is the expected state between installing the timer and doing it | +| `DeployUnverified` with the key present | `HEAD` is a commit that did not come through a pull request — usually someone committing on the host | `git log --show-signature -1` on the host. Get the commit onto a branch and merge it properly | +| `DeployDrifted` | A file was edited on the monitoring host | §"The tree is dirty". The edit is still there — this alert exists because it used to not be | +| `DeployBehind` and nothing else | Convergence is refusing, or is in report-only mode | `journalctl -u homelab-converge.service -n 50` names the refusal. If `HOMELAB_CONVERGE_APPLY=0` is set, this is the mode working | +| `DeployBehind` with `ScheduledJobFailed` | The refusal is real and recurring | The journal names it; every case is in §"When it refuses" | +| `DeployMetricsAbsent` | `homelab-deploy.prom` stopped arriving, while the backup metrics still do | Two separate files fail independently. Check `node_textfile_scrape_error`, then the file itself. If the timer was never installed, `make install-timers` | +| `homelab_job_last_exit_code{homelab_job="converge"}` is 75 | It never started — the weekly backup held the `backups` lock for the full 900s | Expected at most once a week, on Sunday. Persistent means a backup is hanging: `systemctl list-units 'homelab-*'` | +| The stack restarted at 03:25 and nobody deployed | Somebody merged a pull request | Working as designed — ADR-0019 §Consequences. `journalctl -u homelab-converge.service` names the revision | +| Convergence succeeds but a config change did not take | The container reads its config once at startup and nothing reloaded it | `make up` runs `reload-config.sh`, so this should not happen. If it does, it is a bug in that script's service list, not in convergence | +| `git fetch` fails every hour | No egress to github.com, or DNS | The host stays where it is, which is correct. `homelab_deploy_behind_commits` goes to `-1` rather than lying about the lag | + +## Turning it off + +The timer is one of several installed together; removing just this one: + +```bash +sudo systemctl disable --now homelab-converge.timer +``` + +The host then stays on whatever revision it is on until someone runs `make up` +or `make converge`, which is exactly the pre-#99 model. `homelab-deploy.prom` is +deliberately left in place — deleting it would make the host look like it had +never deployed rather than like it had stopped converging, and those are +different things. + +`make install-timers` puts it back. diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index bff123f..695a262 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -104,7 +104,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 48 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 52 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Seven dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken @@ -148,14 +148,36 @@ systemctl list-timers 'homelab-*' Full detail, including the one job that has no timer and never will, is in [`schedule-maintenance.md`](schedule-maintenance.md). +One of those timers deploys this host. `converge` runs hourly, fetches `main`, +verifies its signature and runs `make up` — so after this runbook, the section +below stops being something anybody has to do. It needs one setup step of its +own, a single GPG import: +[`converge-the-host.md`](converge-the-host.md). + ## Updating +**Normally you do not.** A merged pull request reaches this host within the hour +on its own ([ADR-0019](../adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md)). +What follows is how to deploy something *now* rather than waiting, and what the +timer is doing on your behalf. + +```bash +make converge # fetch main, verify it, fast-forward, make up — the timer's job, now +``` + +Or by hand, which is the escape hatch when the change is not on `main` yet: + ```bash git pull make validate make up # recreates changed services, then reloads config on the rest ``` +Note that a hand-deploy of something uncommitted leaves the tree dirty, which +**stops convergence** until it is committed or discarded — deliberately, because +the alternative is the timer destroying your edit at :25. `DeployDrifted` says so +within two hours. + `make up` recreates a container only when its *service definition* changes — a changed bind-mounted config file is invisible to `docker compose up -d`. So `make up` finishes by reloading Prometheus, Alertmanager and snmp-exporter from @@ -177,6 +199,12 @@ git revert make up ``` +Revert on a branch and merge it if you can: a revert committed straight to the +deployment checkout is an unsigned local commit, which is both a non-fast-forward +against `main` and an unverified `HEAD`, so convergence stops until `main` +catches up. Fine as an emergency measure — that is what the escape hatch is for — +but it is a state to leave, not to stay in. + Data volumes survive `make down` and `make up`. Only `make nuke` destroys them, it prompts, and it is recoverable from a backup set — see [`restore-the-stack.md`](restore-the-stack.md). diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 8ad9c8e..260f4d9 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -35,9 +35,13 @@ run, not a partial success. The same run bounds what accumulates on each side to change it. The copy needs a one-time key exchange between the two laptops, in [`restore-the-firewall.md`](restore-the-firewall.md) §0, and fails on purpose until that is done. The volume sets still do not leave -([#92](https://github.com/Gerrrt/HomeLab/issues/92)), and moving deployment to -something pull-based is [#99](https://github.com/Gerrrt/HomeLab/issues/99). -Until then the honest summary is that this host watches its own chores, and the +([#92](https://github.com/Gerrrt/HomeLab/issues/92)). Deployment itself is now +one of these jobs rather than something a human remembers to do — +[#99](https://github.com/Gerrrt/HomeLab/issues/99), +[ADR-0019](../adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md), and +[`converge-the-host.md`](converge-the-host.md) for the one setup step it needs +beyond this runbook. +The honest summary is still that this host watches its own chores, and the *external* cron-monitor described in [`verify-the-alert-path.md`](verify-the-alert-path.md) is the only thing watching the host. @@ -48,15 +52,19 @@ the host. | Job | Command | When | Alerts if not seen in | | --- | --- | --- | --- | +| `converge` | `make converge` | hourly, :25 | 3 hours | | `backup-volumes` | `make backup` | Sundays 03:30 | 14 days | | `verify-backups` | `make backup ARGS='--verify-only --all'` | daily 05:30 | 3 days | | `backup-firewall` | `make backup-firewall` | daily 04:30 | 3 days | | `snmp-verify` | `make snmp-verify` | Wednesdays 06:30 | 14 days | | `verify-key-backup` | **you**, `make secrets-verify-backup KEY=…` | no timer | 90 days | -Thresholds are roughly twice the period in every case, never once: a threshold -equal to the period fires on every run that slips past its jitter window, whereas -twice tolerates one missed run and not two. +Thresholds are roughly twice the period, never once: a threshold equal to the +period fires on every run that slips past its jitter window, whereas twice +tolerates one missed run and not two. `converge` is the one exception at three +times, because it shares the `backups` lock and a run that queues behind the +weekly archive can legitimately spend its full 900-second wait and then be an +hour late. Being late for a reason is not the finding. Both halves of that table live in one place. The cadence is in the `.timer` files, the threshold is in the `JOBS` table in @@ -227,6 +235,7 @@ expected rather than a second fault. | `docker info` fails only under systemd | The unit is missing `SupplementaryGroups=docker` | A login shell picks the group up from `/etc/group` and a unit does not, which is why this never reproduces by hand | | Timers exist but never fire | `WantedBy=timers.target` missing, or the timers were never enabled | `systemctl list-timers 'homelab-*'` shows nothing; re-run `make install-timers` | | `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The four other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` | +| `converge` fails every hour with a signature error | GitHub's signing key was never imported into `robo`'s keyring, so nothing on this host can verify | The one-time import in [`converge-the-host.md`](converge-the-host.md) §Set it up. Every other job here is unaffected | | `refusing to install from …` | You are in a worktree or a second clone | The units hardcode the deployment path. Install from `/home/robo/code/Gerrrt/HomeLab` | | `make validate` fails on the schedule | A cadence and its threshold disagree | `make check-timers` names the job and both numbers. Fix the `JOBS` table or the `.timer`, not the alert | | `make validate` fails with *no `homelab-*` units are installed* | The stack is running on this host but the schedule was never installed | Exactly the condition above, caught before an alert has to. Only a host running the stack is asked; a laptop with the repository checked out skips it | diff --git a/docs/security.md b/docs/security.md index af68bc2..5dc8da4 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | -| Losing visibility of a failure | 48 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 52 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | diff --git a/scripts/converge.sh b/scripts/converge.sh new file mode 100755 index 0000000..83f02be --- /dev/null +++ b/scripts/converge.sh @@ -0,0 +1,419 @@ +#!/usr/bin/env bash +# +# Bring the deployment checkout to what `main` says, and record what is +# deployed (#99). +# +# scripts/converge.sh [--dry-run] [--allow-unsigned] +# +# WHY THIS EXISTS +# +# Deployment was `make up` typed into an SSH session, which has three problems +# and only the first one is obvious. +# +# 1. Nothing recorded what was deployed. `make up` on an uncommitted working +# tree and `make up` on `main` produce the same output and the same exit +# code, and the difference surfaces weeks later as a config nobody can +# account for. `oracle` is the worked example — scripts/deploy-agent.sh's +# header is four paragraphs of one host quietly running something other +# than what the repository said, for two days, because nothing compared +# the two. +# +# 2. Nothing detected drift. A config edited on the host stayed edited until +# the next deploy overwrote it silently, so the edit was lost AND never +# seen. +# +# 3. It is the "nothing schedules anything" problem (#77) wearing a different +# hat, and #77 already built the answer: a timer, a wrapper that records +# the outcome, and alert rules that read the record. This reuses all +# three rather than introducing a second way to run things on a schedule. +# +# WHAT CONVERGENCE MEANS HERE, EXACTLY +# +# Fetch `main` from the canonical URL, refuse to move unless the tip carries a +# good signature from the pinned key, fast-forward, and run `make up`. That is +# the whole loop. It deliberately does NOT reimplement deployment: `make up` is +# still what renders the config and starts the stack, so every runbook that +# says `make up` stays true and this script's blast radius is the DECISION to +# deploy, not the deployment. +# +# The no-op path costs one fetch. When the checkout is already at the fetched +# tip and the tree is clean, nothing is rendered, no container is touched and +# docker is never called — which is what makes an hourly cadence reasonable. +# +# WHY IT FETCHES A URL AND NOT `origin` +# +# `origin` is git@github.com:Gerrrt/HomeLab.git — SSH, with a key that can also +# push. A systemd unit has no ssh-agent, so that path would need a +# passphraseless key readable by an unattended process, and that key would +# carry write access to the repository this host executes. +# +# The repository is public, so the agent needs no credential at all. It fetches +# an explicit https:// URL, which cannot push and cannot be redirected by a +# rewritten `remote.origin.url` in a checkout someone has already edited. The +# URL is pinned below and asserted against `origin` only as a sanity check, not +# trusted from it. +# +# WHY THE SIGNATURE GATE, AND WHAT IT DOES NOT BUY +# +# A host that executes whatever a branch says, unattended, has moved the +# question from "do I trust this code" to "do I trust whoever can move that +# branch". The gate narrows it back: every commit is checked against ONE +# fingerprint pinned in this file, and `main` only moves if the tip verifies. +# +# That is not a guess about how this repository works, it is a measured +# property of it. Every one of the last 110 first-parent commits on `main` — +# unbroken back to PR #33 on 2026-08-19, which is the history purge and the +# last time anything reached `main` other than through a pull request — is a +# merge commit GitHub made and signed. All 110 verify against the fingerprint +# below, with `%G?` of `U` and `%GF` equal to the pin. +# +# So the gate costs nothing today and refuses two things it should refuse: a +# commit pushed straight to `main` past the pull request, and a tip served by +# anything that is not GitHub. +# +# It does NOT stop a compromised GitHub account. An attacker who can open and +# merge a pull request gets a signature like anyone else, and this host will +# deploy it within the hour. That risk is real, it is not new — the operator +# ran `make up` from this checkout after pulling, which executed the same code +# — and what this change alters is the window: from "whenever someone next +# deploys" to "at most an hour", with no human glancing at the diff. The +# compensating control is the record, not the gate. Every convergence writes +# the revision it deployed, and DeployBehind / DeployUnverified / DeployDrifted +# in prometheus/rules/deploy.rules.yaml make an unexpected one visible. +# +# WHY A DIRTY TREE IS A HARD STOP +# +# Refusing is the point. Overwriting is what the old model did, and losing the +# edit while never reporting it is problem 2 above. So an uncommitted change in +# the deployment checkout stops the run, exits non-zero, and shows up as +# ScheduledJobFailed and DeployDrifted — loudly, every hour, until a human +# either commits it or throws it away. There is no --force. `git checkout -- .` +# is one command and it is the human's to type. +# +# Usage: +# scripts/converge.sh fetch, verify, fast-forward, make up +# scripts/converge.sh --dry-run say what it would do, change nothing +# scripts/converge.sh --allow-unsigned fast-forward past a failed signature +# +# Environment: +# TEXTFILE_DIR where homelab-deploy.prom goes +# (default /var/lib/node_exporter/textfile_collector) +# HOMELAB_CONVERGE_APPLY 0 makes every run report-only, as though --dry-run +# had been passed. Set in /etc/default/homelab-timers +# to watch the agent decide for a while before +# letting it act. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +# The checkout the stack actually runs from. Same constant, same reasoning and +# the same refusal as scripts/install-timers.sh: `make render` writes into +# .rendered/ under the tree it is run from, and no container mounts a worktree's +# copy — so converging a second clone would report success while changing +# nothing the stack can see. +DEPLOY_ROOT="/home/robo/code/Gerrrt/HomeLab" + +# Read-only, credential-free, and not taken from the checkout's own config. +CANONICAL_URL="https://github.com/Gerrrt/HomeLab.git" +BRANCH="main" + +# GitHub's web-flow signing key, full fingerprint. Not the 16-hex key id: a key +# id is claimed by the signature itself and a fingerprint is not. The `%GF` +# placeholder is empty unless gpg actually verified, so comparing it to this is +# one comparison that asserts both "verified" and "by the right key". +# +# GitHub's published key file also carries 4AEE18F83AFDEB23, which EXPIRED on +# 2024-01-16 and is not this. Importing the file gets both; only this one is +# accepted. +SIGNING_FPR="968479A1AFF927E37D1A566BB5690EEEBB952194" + +TEXTFILE_DIR="${TEXTFILE_DIR:-/var/lib/node_exporter/textfile_collector}" +PROM="${TEXTFILE_DIR}/homelab-deploy.prom" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } +warn() { printf '\033[0;33mwarning:\033[0m %s\n' "$*" >&2; } +green(){ printf '\033[0;32m%s\033[0m\n' "$*"; } + +usage() { sed -n '/^# Usage:/,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } + +DRY_RUN=0 +ALLOW_UNSIGNED=0 +while (($#)); do + case "$1" in + --dry-run) DRY_RUN=1 ;; + --allow-unsigned) ALLOW_UNSIGNED=1 ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac + shift +done + +# The report-only switch, so the timer can be installed and watched before it is +# allowed to act. Folded into DRY_RUN rather than given a second code path — +# two ways to not-apply is two things to get wrong. +if [[ "${HOMELAB_CONVERGE_APPLY:-1}" == "0" ]]; then + info "HOMELAB_CONVERGE_APPLY=0 — reporting only, nothing will be applied" + DRY_RUN=1 +fi + +# --------------------------------------------------------------------------- +# The record +# --------------------------------------------------------------------------- +# +# Same contract as scripts/run-scheduled.sh, and for the same reasons: written +# to a temp in the SAME directory then renamed, because rename(2) is atomic +# within a filesystem and a truncating write exposes a half-file to the +# collector; mode set explicitly, because Alloy runs with cap_drop [ALL] and so +# obeys the mode; a missing directory warns and a non-writable one dies. +# +# A SEPARATE FILE from converge.prom, which run-scheduled.sh owns. The two say +# different things and have different lifetimes: run-scheduled.sh records +# whether the JOB ran, this records what the HOST is running. The second +# survives being meaningful even when the first says the job failed — a +# convergence that refused to move still knows the revision it refused at. +RECORD=1 +if [[ ! -d "${TEXTFILE_DIR}" ]]; then + RECORD=0 + warn "no textfile directory at ${TEXTFILE_DIR} — converging without recording what is deployed" + warn "on the monitoring host this means the timers were never installed: make install-timers" +elif [[ ! -w "${TEXTFILE_DIR}" ]]; then + die "${TEXTFILE_DIR} is not writable by $(id -un). +The host would converge and nothing would record what it converged to, which is +the failure this script exists to prevent. Fix the directory, then re-run: + sudo install -d -m 0755 -o $(id -un) -g $(id -gn) ${TEXTFILE_DIR}" +fi + +# Filled in as the run progresses, written once by record(). Every one has a +# value that is honest before anything has been measured: an unverified +# revision, an unknown lag of -1, and a tree assumed clean until looked at. +REVISION="" +COMMIT_TS=0 +BEHIND=-1 +DIRTY=0 +VERIFIED=0 + +recorded=0 +record() { + local tmp + ((recorded)) && return 0 + recorded=1 + ((RECORD)) || return 0 + [[ -n "${REVISION}" ]] || return 0 + + tmp="${TEXTFILE_DIR}/homelab-deploy.prom.$$" + cat > "${tmp}" </dev/null 2>&1 || die "git is not installed" + +# Read before any check that can fail, so that EVERY exit path records what the +# host is on — including the ones that give up. A refusal that leaves yesterday's +# metric in place is a refusal that reads as a healthy deployment. +REVISION="$(git rev-parse --short=12 HEAD)" +COMMIT_TS="$(git log -1 --format=%ct HEAD)" + +# --------------------------------------------------------------------------- +# Signature +# --------------------------------------------------------------------------- +# +# Defined here and applied to HEAD immediately, so homelab_deploy_verified is a +# claim about the revision the host IS RUNNING on every exit path — including +# the paths that give up before fetching anything. Evaluating it only alongside +# the fetched tip would make the metric mean "the last thing we were offered", +# which is a different and much less useful sentence. +verify_commit() { + local ref="$1" sig fpr + sig="$(git log -1 --format='%G?' "${ref}")" + fpr="$(git log -1 --format='%GF' "${ref}")" + [[ "${fpr}" == "${SIGNING_FPR}" ]] || return 1 + # G is a good signature from a key marked trusted; U is a good signature from + # a key that is not. Both are accepted, because ownertrust is a statement + # about a local keyring and the fingerprint above is the actual assertion — + # requiring G would mean every host had to run `gpg --lsign-key` as well as + # import, for no additional guarantee. + [[ "${sig}" == "G" || "${sig}" == "U" ]] || return 1 + return 0 +} + +verify_commit HEAD && VERIFIED=1 + +current_branch="$(git symbolic-ref --quiet --short HEAD || true)" +[[ "${current_branch}" == "${BRANCH}" ]] \ + || die "the deployment checkout is on '${current_branch:-a detached HEAD}', not ${BRANCH}. +Convergence only ever fast-forwards ${BRANCH}. Someone left this checkout +somewhere else; put it back deliberately rather than letting a timer do it: + git -C ${DEPLOY_ROOT} switch ${BRANCH}" + +# `origin` is not used for anything — the fetch names its own URL — but a +# checkout whose origin has been repointed is worth saying out loud, because it +# means someone has been editing the deployment host's git config. +origin_url="$(git remote get-url origin 2>/dev/null || true)" +case "${origin_url}" in + "${CANONICAL_URL}"|git@github.com:Gerrrt/HomeLab.git|https://github.com/Gerrrt/HomeLab) ;; + "") warn "no 'origin' remote configured — fetching ${CANONICAL_URL} regardless" ;; + *) warn "origin is ${origin_url}, which is not the canonical repository. + Nothing here reads it — the fetch below names ${CANONICAL_URL} explicitly — + but somebody changed it, and that is worth knowing." ;; +esac + +# --------------------------------------------------------------------------- +# Drift: has anything on the host diverged from what is committed? +# --------------------------------------------------------------------------- +# +# --porcelain skips ignored files, which is exactly right: .rendered/, .env, +# certificates/ and backups/ are all gitignored, all written by the deploy +# itself, and none of them is drift. What is left is a tracked file someone +# edited in place, or an untracked file someone dropped in the tree — both of +# which are the thing #99 says goes unnoticed until a deploy destroys it. +dirty_files="$(git status --porcelain --untracked-files=normal)" +if [[ -n "${dirty_files}" ]]; then + DIRTY=1 + printf '%s\n' "${dirty_files}" >&2 + die "the deployment checkout has uncommitted changes (above). + +Converging would overwrite them, which is exactly the silent loss #99 is about, +so this stops instead and will keep stopping — DeployDrifted and +ScheduledJobFailed will both be firing — until a human decides which it is: + + keep it git -C ${DEPLOY_ROOT} diff # then commit it, on a branch, via a pull request + drop it git -C ${DEPLOY_ROOT} checkout -- . # and remove any untracked files it listed + +There is no --force. Choosing is the whole point." +fi + +# --------------------------------------------------------------------------- +# Fetch +# --------------------------------------------------------------------------- +info "fetching ${BRANCH} from ${CANONICAL_URL}" +# --no-tags because nothing here reads a tag and a tag is another thing that can +# move. No --depth: a shallow fetch has no merge base, so the fast-forward +# assertion and the behind-count below would both be unanswerable. +git fetch --quiet --no-tags "${CANONICAL_URL}" "${BRANCH}" \ + || die "could not fetch ${BRANCH} from ${CANONICAL_URL}. +The host stays on ${REVISION}, which is the correct outcome of not knowing what +${BRANCH} says. If this persists, DeployBehind will not fire — nothing was +learned about how far behind the host is — but ScheduledJobFailed will." + +TARGET="$(git rev-parse FETCH_HEAD)" +BEHIND="$(git rev-list --count "HEAD..${TARGET}")" + +# --------------------------------------------------------------------------- +# Verify the tip before it becomes the deployed revision +# --------------------------------------------------------------------------- +target_verified=0 +verify_commit "${TARGET}" && target_verified=1 + +if ((target_verified == 0)); then + # Distinguish the two reasons, because they need different responses: a + # missing key is a setup step nobody did, and a missing signature is a commit + # that did not come through a pull request. + detail="signature: $(git log -1 --format='%G?' "${TARGET}"), key: $(git log -1 --format='%GK' "${TARGET}" || true)" + if ! gpg --batch --list-keys "${SIGNING_FPR}" >/dev/null 2>&1; then + hint="The signing key is not in $(id -un)'s keyring on this host, so nothing +CAN verify. Import it once — docs/runbooks/converge-the-host.md: + curl -fsSL https://github.com/web-flow.gpg | gpg --import +Then confirm the fingerprint it printed is ${SIGNING_FPR}." + else + hint="The key is present and this commit did not verify against it, which +means ${BRANCH} moved by something other than a GitHub merge — a direct push, or +a tip served by something that is not GitHub. Look at it before deploying it: + git -C ${DEPLOY_ROOT} log --show-signature -1 ${TARGET}" + fi + + if ((ALLOW_UNSIGNED)); then + warn "${TARGET} did not verify (${detail}) — continuing because --allow-unsigned was passed" + else + die "${TARGET} did not verify (${detail}). + +${hint} + +The host stays on ${REVISION}. To deploy it anyway, deliberately and by hand: + ${DEPLOY_ROOT}/scripts/converge.sh --allow-unsigned" + fi +fi + +# --------------------------------------------------------------------------- +# Converge +# --------------------------------------------------------------------------- +if [[ "${TARGET}" == "$(git rev-parse HEAD)" ]]; then + BEHIND=0 + green "converged — ${REVISION} is ${BRANCH}" + # Nothing rendered, no container touched, docker never called. This is the + # path an hourly cadence spends almost all of its time on. + exit 0 +fi + +# Fast-forward only. A non-fast-forward means `main` was rewritten or this +# checkout has commits of its own, and quietly resolving either one is how a +# deployment host ends up running something no branch points at. +git merge-base --is-ancestor HEAD "${TARGET}" \ + || die "${TARGET} is not a fast-forward from ${REVISION}. +Either ${BRANCH} was rewritten, or this checkout has local commits. Both need a +human — a timer that resolves this is a timer that can roll the host backwards +onto a revision someone deliberately replaced. + git -C ${DEPLOY_ROOT} log --oneline ${REVISION}..${TARGET} + git -C ${DEPLOY_ROOT} log --oneline ${TARGET}..${REVISION}" + +info "${BEHIND} commit(s) behind — ${REVISION} to $(git rev-parse --short=12 "${TARGET}")" +git --no-pager log --oneline --no-decorate "HEAD..${TARGET}" | sed 's/^/ /' >&2 + +if ((DRY_RUN)); then + # REVISION, COMMIT_TS and VERIFIED still describe HEAD, which is still what is + # deployed — the whole point of not applying. Only BEHIND changed, and it is + # the number that says so. + warn "dry run — not applying" + exit 0 +fi + +git merge --ff-only --quiet "${TARGET}" +REVISION="$(git rev-parse --short=12 HEAD)" +COMMIT_TS="$(git log -1 --format=%ct HEAD)" +BEHIND=0 +VERIFIED="${target_verified}" + +# `make up` and not a narrower command, on purpose. It renders the config, +# recreates whatever compose says changed, and runs reload-config.sh for the +# services that read their config once at startup — and it is what every runbook +# already tells a human to type, so there is exactly one deployment path and it +# is exercised both ways. +info "applying ${REVISION}" +make up + +green "converged to ${REVISION}" diff --git a/scripts/install-timers.sh b/scripts/install-timers.sh index 24acb4b..30af76f 100755 --- a/scripts/install-timers.sh +++ b/scripts/install-timers.sh @@ -85,8 +85,15 @@ DEPLOY_ROOT="/home/robo/code/Gerrrt/HomeLab" # SecretsKeyBackupUnproven can still nag. The one thing that proves the secrets # are recoverable now has a deadline even though it has no schedule. # +# converge is the only hourly row, and the only one whose threshold is three +# times its period rather than two. It shares the `backups` lock with the two +# backup jobs, so a run that collides with the weekly archive can legitimately +# spend its whole 900s lock wait and then be an hour late; twice the period +# would alert on that, and being late for a reason is not the finding. +# # job unit prefix max_age make target JOBS=( + "converge homelab-converge 10800 converge" "backup-volumes homelab-backup-volumes 1209600 backup" "verify-backups homelab-verify-backups 259200 backup" "backup-firewall homelab-backup-firewall 259200 backup-firewall" @@ -359,7 +366,7 @@ for row in "${JOBS[@]}"; do done # Run each job once so the timers do not spend their first night looking like -# four jobs that have never run — and so the plumbing is proven now rather than +# jobs that have never run — and so the plumbing is proven now rather than # at 03:30. backup-volumes is excluded: it quiesces the monitoring stack, and # that is not something to do as a side effect of an install. if ((RUN_ONCE)); then diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 021ddf3..864b4cd 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox.yaml probe targets — hot-reloaded, no restart needed - rules/*.rules.yaml 48 alert rules across host/network/ups/containers/blackbox/backup/ids + rules/*.rules.yaml 52 alert rules across host/network/ups/containers/blackbox/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability from outside the service alertmanager/ diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index 17f8a65..a0ec92c 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -20,9 +20,16 @@ # # homelab_job_max_age_seconds is written by scripts/install-timers.sh from the # JOBS table that also decides the cadence, so a threshold and a schedule cannot -# drift apart. It also means NO RULE HERE NAMES A JOB: adding a sixth timer is a -# row in that table, not an edit to this file. That is deliberate — a rule file -# listing job names is a list that silently stops growing. +# drift apart. It also means NO RULE HERE NAMES A JOB: adding a timer is a row in +# that table, not an edit to this file. That is deliberate — a rule file listing +# job names is a list that silently stops growing. +# +# That claim has since been tested rather than merely asserted. #99 added +# `converge`, an hourly job on a different cadence to everything above, and +# every rule in this file covered it with no edit at all. What #99 DID need was +# a separate file — prometheus/rules/deploy.rules.yaml — because the rules here +# describe whether a job RAN, and deployment also has results worth alerting on: +# which refusal, and how far behind the host is. # # WHY THERE IS NO absent() # diff --git a/stacks/observability/prometheus/rules/deploy.rules.yaml b/stacks/observability/prometheus/rules/deploy.rules.yaml new file mode 100644 index 0000000..d9244f5 --- /dev/null +++ b/stacks/observability/prometheus/rules/deploy.rules.yaml @@ -0,0 +1,164 @@ +--- +# Whether this host is running what the repository says (#99). +# +# Deployment used to be `make up` typed into an SSH session. Nothing recorded +# what got deployed, so `make up` on an uncommitted tree and `make up` on `main` +# were indistinguishable; and nothing compared the host to the repository, so a +# config edited in place stayed edited until a deploy silently overwrote it. +# scripts/converge.sh now runs hourly and writes five gauges into the same +# textfile directory the maintenance jobs use. +# +# WHAT backup.rules.yaml ALREADY COVERS, AND IS NOT REPEATED HERE +# +# `converge` is a row in the JOBS table like any other, so ScheduledJobStale, +# ScheduledJobFailed and ScheduledJobNeverRan already say "the convergence +# stopped running" and "the convergence ran and failed". None of that is +# restated below. +# +# What those rules cannot say is anything about the RESULT. A convergence that +# correctly refuses to move — because the tree is dirty, or the tip did not +# verify — exits non-zero and is fully described by ScheduledJobFailed, but the +# interesting fact is WHICH refusal and how far behind the host now is. These +# four rules read the record converge.sh writes rather than its exit code. +# +# WHY THERE IS NO ALERT ON THE REVISION ITSELF +# +# homelab_deploy_revision_info exists to be read, not to be alerted on. "Is this +# the right commit" is a question with no threshold — the answer is a git log — +# and a rule that tried would either hardcode a revision or fire on every +# deploy. The alertable form of that question is DeployBehind, which asks the +# same thing against a moving target instead of a fixed one. +groups: + - name: deploy + interval: 60s + rules: + - alert: DeployDrifted + # Someone edited a file on the monitoring host. + # + # This is the alert #99 asked for by name. converge.sh refuses to + # fast-forward over an uncommitted change rather than overwriting it, so + # by the time this fires the edit is still there and still recoverable — + # which is the whole difference from the old model, where the edit was + # destroyed and the destruction was also invisible. + # + # 2h, not 30m. converge.sh rewrites this gauge once an hour, so a `for` + # shorter than the cadence asserts nothing beyond "one sample said so". + # Two hours means a second convergence looked and the edit was still + # uncommitted, which is the difference between someone mid-keystroke and + # a change nobody is going to commit. + expr: homelab_deploy_tree_dirty == 1 + for: 2h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "The deployment checkout has uncommitted changes" + description: >- + Something was edited on the monitoring host instead of in the + repository, and convergence has stopped until it is resolved — the + host is frozen on whatever it was already running. The change is + still there: `git -C /home/robo/code/Gerrrt/HomeLab status` and + `git diff` show it. Commit it through a pull request or discard it — + docs/runbooks/converge-the-host.md. + + - alert: DeployBehind + # The host knows about commits it has not applied. + # + # In steady state this is unfireable: converge.sh sets the gauge to 0 + # immediately after a successful fast-forward, so a run that applies + # cleanly never leaves a positive value behind. It fires when + # convergence is refusing (an unverified tip, a rewritten history), or + # when HOMELAB_CONVERGE_APPLY=0 has deliberately put the timer in + # report-only mode — in which case this alert IS the report and is + # expected to fire. + # + # > 0 and not != 0 on purpose. A failed fetch records -1 rather than a + # stale count, precisely so it cannot masquerade as a known lag; the + # fetch failing is ScheduledJobFailed's to report, and this rule stays + # quiet about a number it does not have. + # + # 3h matches the 10800s staleness budget scripts/install-timers.sh + # declares for the job, so a convergence that queued behind the weekly + # backup and then missed its next slot does not alert twice for one + # cause. + expr: homelab_deploy_behind_commits > 0 + for: 3h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "The host is {{ $value }} commit(s) behind main" + description: >- + Convergence has run and has not applied what it fetched. Either it + is refusing — check `journalctl -u homelab-converge.service -n 50` + for a dirty tree, a signature that did not verify, or a + non-fast-forward — or HOMELAB_CONVERGE_APPLY=0 is set in + /etc/default/homelab-timers and this is the report-only mode doing + its job. docs/runbooks/converge-the-host.md. + + - alert: DeployUnverified + # The deployed revision does not carry a good signature from the pinned + # key. + # + # Two very different causes, and the description names both because the + # metric cannot tell them apart. The dull one is that GitHub's web-flow + # key was never imported into robo's keyring, so nothing on this host + # CAN verify and convergence has been refusing since it was installed — + # the expected state between installing the timer and doing the one-time + # import. The interesting one is that somebody committed directly on the + # deployment host, which produces an unsigned HEAD and would otherwise + # look exactly like a normal deploy. + # + # Deliberately NOT gated on behind_commits. An unverifiable HEAD is a + # finding whether or not there is anything newer to move to. + expr: homelab_deploy_verified == 0 + for: 1h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "The deployed revision has no valid signature" + description: >- + Either GitHub's signing key is not in robo's keyring on this host — + the one-time setup step, and convergence is refusing to move until + it is done — or HEAD is a commit that did not come through a pull + request. `git -C /home/robo/code/Gerrrt/HomeLab log --show-signature -1` + distinguishes them. docs/runbooks/converge-the-host.md. + + - alert: DeployMetricsAbsent + # The measuring apparatus, and not a duplicate of + # ScheduledJobMetricsAbsent. + # + # That rule watches homelab_job_max_age_seconds, which + # scripts/install-timers.sh writes into homelab-jobs.prom. These three + # rules read homelab-deploy.prom, which scripts/converge.sh writes. Two + # files, and node_exporter's textfile collector fails them + # independently: a malformed or deleted homelab-deploy.prom costs only + # this file's series, leaves node_textfile_scrape_error at 1 with every + # backup rule still green, and takes DeployDrifted, DeployBehind and + # DeployUnverified silently with it. Absence looking healthy is the one + # failure mode this repository keeps rebuilding the same guard against. + # + # revision_info is the series chosen because converge.sh writes it on + # every exit path that knows a revision at all, including the refusals. + # absent_over_time rather than absent, for the reason + # backup.rules.yaml's equivalent gives: six hours survives a Prometheus + # restart and a single missed scrape, where absent() fires on both. + expr: absent_over_time(homelab_deploy_revision_info[6h]) + for: 1h + labels: + component: deploy + severity: warning + category: correctness + annotations: + summary: "No deployment metrics have reached Prometheus for six hours" + description: >- + Nothing else in this file can fire while this is true — what is + deployed on this host is currently unknown. Check + node_textfile_scrape_error, then + /var/lib/node_exporter/textfile_collector/homelab-deploy.prom on the + monitoring host. If the timer was never installed, `make + install-timers` — docs/runbooks/converge-the-host.md. diff --git a/stacks/observability/prometheus/tests/deploy.test.yaml b/stacks/observability/prometheus/tests/deploy.test.yaml new file mode 100644 index 0000000..b79a503 --- /dev/null +++ b/stacks/observability/prometheus/tests/deploy.test.yaml @@ -0,0 +1,186 @@ +--- +# Unit tests for deploy.rules.yaml. +# +# Every firing case is paired with a silent one, per containers.test.yaml: a +# test that only ever expects silence would have passed against the unfireable +# rule in #63, and a test that only ever expects firing would pass against a +# rule that fires unconditionally. +# +# Two cases here exist to pin specific ways this file could break without +# `promtool check rules` noticing: +# +# 1. THE -1 SENTINEL. scripts/converge.sh records behind_commits as -1 when +# the fetch did not complete, so that "we do not know" cannot be read as +# "we are behind". A rule written `!= 0` instead of `> 0` passes every +# other test here and turns every network blip into a DeployBehind. The +# second case feeds -1 and asserts silence. +# +# 2. THE instance LABEL. These metrics come from the same textfile collector +# as homelab_job_*, so they carry the scrape target's `instance`. The +# firing cases assert it survives into the alert — without it a second +# monitored host would be indistinguishable from this one, which is the +# trap backup.test.yaml documents for the joined rules. +rule_files: + - ../rules/deploy.rules.yaml + +evaluation_interval: 1m + +tests: + # --- DeployDrifted: dirty, and clean ------------------------------------ + # + # Two hosts in one series set, one with an edited tree and one without, so a + # rule that ignored the value and fired on the series existing would fail. + - interval: 1m + input_series: + - series: 'homelab_deploy_tree_dirty{instance="prometheus"}' + values: "1+0x300" + - series: 'homelab_deploy_tree_dirty{instance="oracle"}' + values: "0+0x300" + alert_rule_test: + # `for: 2h`, so still only pending at 1h50m. This is what asserts the + # `for` is actually there — converge.sh rewrites the gauge hourly and a + # rule without it would alert on a single sample. + - eval_time: 1h50m + alertname: DeployDrifted + exp_alerts: [] + + - eval_time: 2h5m + alertname: DeployDrifted + exp_alerts: + - exp_labels: + alertname: DeployDrifted + instance: prometheus + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "The deployment checkout has uncommitted changes" + description: >- + Something was edited on the monitoring host instead of in the + repository, and convergence has stopped until it is resolved — + the host is frozen on whatever it was already running. The + change is still there: `git -C /home/robo/code/Gerrrt/HomeLab + status` and `git diff` show it. Commit it through a pull request + or discard it — docs/runbooks/converge-the-host.md. + + # --- DeployBehind: behind, converged, and "we do not know" --------------- + # + # The -1 case is the point of this block. A fetch that fails records -1, and + # `> 0` must treat that as silence while `!= 0` would not. + - interval: 1m + input_series: + - series: 'homelab_deploy_behind_commits{instance="prometheus"}' + values: "3+0x300" + - series: 'homelab_deploy_behind_commits{instance="oracle"}' + values: "0+0x300" + - series: 'homelab_deploy_behind_commits{instance="saruman"}' + values: "-1+0x300" + alert_rule_test: + - eval_time: 2h55m + alertname: DeployBehind + exp_alerts: [] + + # Only `prometheus` fires: `oracle` is converged and `saruman` could not + # fetch, which is ScheduledJobFailed's finding and not this rule's. + - eval_time: 3h5m + alertname: DeployBehind + exp_alerts: + - exp_labels: + alertname: DeployBehind + instance: prometheus + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "The host is 3 commit(s) behind main" + description: >- + Convergence has run and has not applied what it fetched. Either + it is refusing — check `journalctl -u homelab-converge.service + -n 50` for a dirty tree, a signature that did not verify, or a + non-fast-forward — or HOMELAB_CONVERGE_APPLY=0 is set in + /etc/default/homelab-timers and this is the report-only mode + doing its job. docs/runbooks/converge-the-host.md. + + # --- DeployUnverified: unsigned, and signed ----------------------------- + - interval: 1m + input_series: + - series: 'homelab_deploy_verified{instance="prometheus"}' + values: "0+0x180" + - series: 'homelab_deploy_verified{instance="oracle"}' + values: "1+0x180" + alert_rule_test: + - eval_time: 50m + alertname: DeployUnverified + exp_alerts: [] + + - eval_time: 1h5m + alertname: DeployUnverified + exp_alerts: + - exp_labels: + alertname: DeployUnverified + instance: prometheus + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "The deployed revision has no valid signature" + description: >- + Either GitHub's signing key is not in robo's keyring on this + host — the one-time setup step, and convergence is refusing to + move until it is done — or HEAD is a commit that did not come + through a pull request. `git -C /home/robo/code/Gerrrt/HomeLab + log --show-signature -1` distinguishes them. + docs/runbooks/converge-the-host.md. + + # --- DeployMetricsAbsent: the series is there --------------------------- + # + # Paired with the block below, which is the same rule with nothing fed to it. + # Silence here is what proves the firing case is about absence and not about + # the rule being unconditional. + - interval: 1m + input_series: + - series: 'homelab_deploy_revision_info{instance="prometheus", revision="09d618ede580"}' + values: "1+0x600" + alert_rule_test: + - eval_time: 7h + alertname: DeployMetricsAbsent + exp_alerts: [] + + # --- DeployMetricsAbsent: homelab-deploy.prom stopped arriving ---------- + # + # No input_series at all. A backup metric IS supplied, to pin the thing this + # rule exists for: homelab-jobs.prom and homelab-deploy.prom are separate + # files that the textfile collector fails independently, so the schedule can + # look perfectly healthy while nothing knows what is deployed. + - interval: 1m + input_series: + - series: 'homelab_job_max_age_seconds{homelab_job="converge", instance="prometheus"}' + values: "10800+0x600" + alert_rule_test: + # `for: 1h` is the only delay this case can assert, and that is worth + # saying plainly: the series never exists here, so the 6h lookback is + # empty from t=0 and absent_over_time is true immediately. The six hours + # only buys anything against real data, where it is the gap that has to + # exceed the window. Measured rather than assumed — this case was + # originally written expecting a 6h delay and promtool disagreed. + - eval_time: 55m + alertname: DeployMetricsAbsent + exp_alerts: [] + + - eval_time: 1h5m + alertname: DeployMetricsAbsent + exp_alerts: + - exp_labels: + alertname: DeployMetricsAbsent + component: deploy + severity: warning + category: correctness + exp_annotations: + summary: "No deployment metrics have reached Prometheus for six hours" + description: >- + Nothing else in this file can fire while this is true — what is + deployed on this host is currently unknown. Check + node_textfile_scrape_error, then + /var/lib/node_exporter/textfile_collector/homelab-deploy.prom on + the monitoring host. If the timer was never installed, `make + install-timers` — docs/runbooks/converge-the-host.md. diff --git a/systemd/homelab-backup-volumes.service b/systemd/homelab-backup-volumes.service index 2d209a6..8341bbb 100644 --- a/systemd/homelab-backup-volumes.service +++ b/systemd/homelab-backup-volumes.service @@ -1,6 +1,6 @@ # Weekly quiesced backup of the observability stack's volumes. # -# This unit is the fully-commented one of the four; the others state only what +# This unit is the fully-commented one of the five; the others state only what # differs. Installed by scripts/install-timers.sh, which is also what asserts # that every ExecStart= here goes through run-scheduled.sh (#77). [Unit] @@ -27,7 +27,7 @@ SupplementaryGroups=docker # Load-bearing, not tidiness. backup-firewall.sh resolves backups/firewall and # .sops.yaml RELATIVE to the process's directory, so on that unit this line is # the difference between a backup and a crash. Same directive everywhere so the -# four units cannot disagree about where the repository is. +# five units cannot disagree about where the repository is. WorkingDirectory=/home/robo/code/Gerrrt/HomeLab Environment=STACK=observability # Explicit even though User= already sets $HOME from the passwd entry: the diff --git a/systemd/homelab-converge.service b/systemd/homelab-converge.service new file mode 100644 index 0000000..78749e3 --- /dev/null +++ b/systemd/homelab-converge.service @@ -0,0 +1,69 @@ +# Hourly convergence of this host onto what `main` says (#99). +# +# This is the one unit here that CHANGES the host rather than reading it, so +# two directives below are load-bearing in a way they are not elsewhere. +# +# SupplementaryGroups=docker, because converge.sh ends in `make up` and that +# talks to the daemon. Same reasoning as the backup units: a login shell picks +# the group up from /etc/group and a unit does not, so without it the failure +# appears only under systemd and never by hand. +# +# The `backups` lock, shared with homelab-backup-volumes and +# homelab-verify-backups. Not tidiness — homelab-backup-volumes STOPS the whole +# stack for the length of its archive, and a convergence that ran `make up` in +# that window would restart Prometheus, Loki and Grafana underneath a backup +# that is mid-read. The result would be an archive nothing had quiesced, which +# is exactly the kind of backup that looks fine until it is restored. +# +# See homelab-backup-volumes.service for why each of the remaining directives is +# set. +[Unit] +Description=HomeLab: converge this host onto the main branch +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/converge-the-host.md +Wants=docker.service network-online.target +After=docker.service network-online.target + +[Service] +Type=oneshot +User=robo +Group=robo +SupplementaryGroups=docker +WorkingDirectory=/home/robo/code/Gerrrt/HomeLab +Environment=STACK=observability +Environment=SOPS_AGE_KEY_FILE=/home/robo/.config/sops/age/keys.txt +# HOMELAB_CONVERGE_APPLY=0 goes here to make the timer report-only — it fetches, +# verifies and records how far behind the host is, and applies nothing. The +# leading - means an absent file is not an error, so the default is to apply. +EnvironmentFile=-/etc/default/homelab-timers +ExecStart=/home/robo/code/Gerrrt/HomeLab/scripts/run-scheduled.sh --job converge --lock backups --lock-wait 900 -- /usr/bin/make -C /home/robo/code/Gerrrt/HomeLab converge +# 900s of that is the lock wait. The convergence itself is a fetch on almost +# every run and a `make up` on the few that are not. +TimeoutStartSec=1800 +# Deliberately not Restart=on-failure, and the reason is sharper here than on +# the backup units. Every way this fails is a refusal — a dirty tree, an +# unverified tip, a history that was rewritten — and a refusal is a decision, +# not a transient error. Retrying a decision just makes it again. The timer +# comes back in an hour, and the alert fires either way. +Restart=no +Nice=10 +# No IOSchedulingClass=idle. `make up` recreating a container is latency-bound +# on the daemon, not throughput-bound on the disk, and deprioritising it only +# lengthens the window in which the stack is half-restarted. +ProtectHome=false +ProtectSystem=false +# Off, for the reason homelab-backup-volumes.service gives: the daemon resolves +# -v source paths in the HOST mount namespace, so a private /tmp silently +# bind-mounts the wrong directory into any container this starts. +PrivateTmp=false +NoNewPrivileges=true +RestrictSUIDSGID=true +LockPersonality=true +RestrictRealtime=true +StandardOutput=journal +StandardError=journal +SyslogIdentifier=homelab-converge + +# No [Install] section, on purpose — the timer is what enables this. A oneshot +# .service carrying WantedBy=multi-user.target would converge and restart the +# stack at every boot, which is a deploy nobody asked for at the least +# convenient moment. diff --git a/systemd/homelab-converge.timer b/systemd/homelab-converge.timer new file mode 100644 index 0000000..797a68f --- /dev/null +++ b/systemd/homelab-converge.timer @@ -0,0 +1,34 @@ +[Unit] +Description=HomeLab: hourly convergence onto main +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/converge-the-host.md + +[Timer] +Unit=homelab-converge.service +# Hourly, and hourly is a statement about how long a merged change may take to +# reach the host — an hour is short enough that "merged" and "deployed" are +# nearly the same word, and long enough that nothing here is chasing a webhook. +# +# The cost of the cadence is one fetch, not one deploy: converge.sh exits before +# it renders anything when the checkout is already at the fetched tip, so almost +# every one of these 24 daily runs touches no container at all. +# +# :25 rather than :00 or :30 because every other unit here is on one of those +# two marks, and the three that share the `backups` lock should not queue behind +# each other by default when they could simply not collide. +OnCalendar=*-*-* *:25:00 +# Five minutes, not the fifteen the other timers use. Jitter here spreads load +# on GitHub rather than on this host, one fetch is not load worth spreading, and +# a wide window makes "how stale can the host be" harder to state than it needs +# to be. +RandomizedDelaySec=300 +# True. A missed convergence — the host was off, or rebooting — should be picked +# up at boot rather than waiting for the next hour, because the state it is +# catching up to is the whole point. This is the opposite of the reasoning on +# homelab-backup-volumes.timer, where a replayed run quiesces the stack at an +# arbitrary hour; a replayed run here just deploys what should already be +# deployed. +Persistent=true +AccuracySec=1m + +[Install] +WantedBy=timers.target From b6a1cc5f7339621d5ade32a3aca153234cdc5e8d Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 4 Sep 2026 13:33:51 +0000 Subject: [PATCH 2/2] feat(deploy): record the apply mode, and ship report-only (#99) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Report-only is the chosen rollout: the timer goes on, and applying is switched on once it has been watched deciding correctly for a while. That made a gap in the previous commit visible. HOMELAB_CONVERGE_APPLY=0 lives in /etc/default/homelab-timers, a file outside this repository that nothing tracks, and DeployBehind could only say "it is refusing, OR report-only is set" — leaving an operator to go and read a file over ssh to tell a deliberate mode apart from a genuine refusal. A mode that exists only in /etc on one host is unrecorded state, which is the exact thing #99 is about. So the mode joins the record. homelab_deploy_apply_enabled is written with the other five gauges, tracked separately from DRY_RUN because --dry-run is a human asking a question while this is a property of how the host is configured — a manual dry run still reports apply=1, since the timer would have applied. DeployApplyDisabled (info) reports it, and is deliberately an alert rather than a roadmap entry for "remember to turn this on": it fires six hours in and resolves by itself on the first run after the line is removed. There is nothing to close and no way to leave it stale. DeployBehind now points at it instead of guessing, so the two read as one sentence when both fire. Verified against a real clone: HOMELAB_CONVERGE_APPLY=0 records apply=0 and applies nothing; --dry-run alone records apply=1. Co-Authored-By: Claude Opus 5 --- README.md | 4 +- ...n-a-timer-instead-of-deploying-over-ssh.md | 41 ++++++++++----- docs/observability.md | 8 +-- docs/runbooks/converge-the-host.md | 50 +++++++++++++++---- docs/runbooks/deploy-stack.md | 2 +- docs/security.md | 2 +- scripts/converge.sh | 26 ++++++++-- stacks/observability/README.md | 2 +- .../prometheus/rules/deploy.rules.yaml | 50 +++++++++++++++++-- .../prometheus/tests/deploy.test.yaml | 42 ++++++++++++++-- 10 files changed, 185 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 4d52090..ba7bfeb 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ incident. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 140 panels, and - 65 alert rules — 52 metric-based in Prometheus, 13 log-based in Loki — sharing + 66 alert rules — 53 metric-based in Prometheus, 13 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -160,7 +160,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 52 alert rules +│ ├── prometheus/ # config, file_sd targets, 53 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 13 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md b/docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md index 995c1d5..c7f0af7 100644 --- a/docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md +++ b/docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md @@ -150,6 +150,7 @@ that declined to move still knows what the host is running. | `homelab_deploy_behind_commits` | How far behind `main` the host is; `-1` for "the fetch failed", so not-knowing cannot read as being-behind | | `homelab_deploy_tree_dirty` | Whether anyone edited the host | | `homelab_deploy_verified` | Whether the deployed revision carries a good signature | +| `homelab_deploy_apply_enabled` | Whether the host applies what it fetches, or is in report-only mode | Because `converge` is an ordinary row in the `JOBS` table, `ScheduledJobStale`, `ScheduledJobFailed` and `ScheduledJobNeverRan` cover "the convergence stopped @@ -186,16 +187,30 @@ converges the *configuration*; whether a running image still matches the digest `compose.yaml` names is `make check-digests`'s question, and answering it here would be two checks with one name. -### Report-only exists, and it is one variable +### Report-only exists, it is one variable, and it is how this ships `HOMELAB_CONVERGE_APPLY=0` in `/etc/default/homelab-timers` — the file every unit here already reads — makes every run fetch, verify and record while -applying nothing. `DeployBehind` then fires as the report, which is what that -mode is for. - -It is there because "let a timer restart my monitoring stack unattended" is a -reasonable thing to want to watch before allowing, and because the alternative -to a switch is not installing the timer, which is the state that produced #99. +applying nothing. + +**That is the chosen rollout.** The timer goes on in report-only, and applying +is switched on once it has been watched deciding correctly for a while. "Let a +timer restart my monitoring stack unattended" is a reasonable thing to want to +see working first, and the alternative to a switch is not installing the timer +at all, which is the state that produced #99. + +The mode is a gauge, not just a file. `homelab_deploy_apply_enabled` is written +with the rest of the record, because a mode that lives only in `/etc` on one +host is unrecorded state — the exact thing this ADR exists to stop — and +because without it `DeployBehind` could only say "it is refusing, OR +report-only is set" and leave the operator to go and read a file. + +`DeployApplyDisabled` (info) reports the mode, and it is deliberately an alert +rather than an issue on the roadmap: it fires six hours in, and it **resolves by +itself** on the first run after the line is removed. There is nothing to close +and no way to leave it stale, which is the failure mode a tracked task for +"remember to turn this on" would have. Being temporary is enforced by the thing +itself rather than by anyone remembering. ## Consequences @@ -204,11 +219,13 @@ to a switch is not installing the timer, which is the state that produced #99. changes is that the host no longer waits to be told. - **A merged pull request reaches the monitoring host within an hour, with no - human in the loop.** That is the point, and it is also the cost. Merging - becomes deploying, so a change that would break the stack breaks it at 03:25 - rather than when someone was watching. The mitigations are the ones this - repository already leans on — `make validate` in CI on every pull request, and - a stack whose failure is loud. + human in the loop — once report-only is switched off.** That is the point, and + it is also the cost. Merging becomes deploying, so a change that would break + the stack breaks it at 03:25 rather than when someone was watching. The + mitigations are the ones this repository already leans on — `make validate` in + CI on every pull request, and a stack whose failure is loud. Until the switch + is flipped, `DeployBehind` and `DeployApplyDisabled` fire as a pair and + deployment stays manual. - **One new setup step, and it fails loudly rather than silently.** GitHub's web-flow key has to be imported into `robo`'s keyring once. Until it is, diff --git a/docs/observability.md b/docs/observability.md index 406c52e..3db43aa 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -256,7 +256,7 @@ separates a quiet stream from a stopped one. ## Alerting -65 rules in total: 52 metric-based in `prometheus/rules/`, and 13 log-based in +66 rules in total: 53 metric-based in `prometheus/rules/`, and 13 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -285,7 +285,7 @@ boot check. ### Metric-based (Prometheus) -52 rules across ten files in `prometheus/rules/`: +53 rules across ten files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -307,11 +307,11 @@ and healthy and could not fire for any input ([#63](https://github.com/Gerrrt/Ho `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is twenty-two rules of 52 so far — the five +against the broken rule too. Coverage is twenty-three rules of 53 so far — the five in `blackbox.rules.yaml`, `ContainerHighMemory` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from [#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in -`backup.test.yaml`, all four in `deploy.test.yaml`, `RemoteWriteJobStale`, +`backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, and `SuricataStopped`. The other 30 are still validated for syntax only, which is exactly the standing #63 had. Both numbers are checked by `scripts/check_docs.py` — the diff --git a/docs/runbooks/converge-the-host.md b/docs/runbooks/converge-the-host.md index 1ac55b0..8988bfb 100644 --- a/docs/runbooks/converge-the-host.md +++ b/docs/runbooks/converge-the-host.md @@ -72,23 +72,50 @@ make converge ARGS=--dry-run A converged host prints `converged — is main` and nothing else. A host that is behind lists the commits it would apply and stops. -### Optional: watch it for a week before letting it act +### Start in report-only mode -To install the timer but keep it from changing anything, put this in -`/etc/default/homelab-timers` on the host: +**This is how it is being rolled out.** The timer is installed and watched for a +while before it is allowed to change anything. Do this before +`make install-timers`, as root: ```bash -HOMELAB_CONVERGE_APPLY=0 +printf 'HOMELAB_CONVERGE_APPLY=0\n' | sudo tee -a /etc/default/homelab-timers ``` -Every run then fetches, verifies and records, and applies nothing. -`DeployBehind` fires whenever the host is behind — in this mode that alert *is* -the report, and it firing is the expected state, not a fault. Remove the line -and `systemctl restart homelab-converge.timer` when you want it to act. +Every run then fetches, verifies and records, and applies nothing — deployment +stays a thing a human does with `make up`. + +Two alerts describe that state together, and reading them as a pair is the +point: + +- `DeployApplyDisabled` (info) says the host is deliberately not deploying. It + fires six hours in and **resolves by itself** on the first run after the line + is removed. +- `DeployBehind` (warning) says how far behind it has got. In this mode that + alert *is* the report and firing is expected, not a fault. + +When only `DeployBehind` is firing, report-only is **not** the explanation and +something is genuinely refusing — see §When it refuses. + +### Letting it act + +Remove the line, then restart the timer: + +```bash +sudo sed -i '/^HOMELAB_CONVERGE_APPLY=0$/d' /etc/default/homelab-timers +``` + +```bash +sudo systemctl restart homelab-converge.timer && sudo systemctl start homelab-converge.service +``` + +`DeployApplyDisabled` resolves on that run. Nothing else needs doing: the first +convergence catches up however many commits have accumulated, in one +fast-forward. ## What it records -Five gauges in `/var/lib/node_exporter/textfile_collector/homelab-deploy.prom`, +Six gauges in `/var/lib/node_exporter/textfile_collector/homelab-deploy.prom`, written on every exit path including the refusals, so a run that declined to move still reports what the host is on. @@ -99,6 +126,7 @@ move still reports what the host is on. | `homelab_deploy_behind_commits` | How far behind `main`; `-1` means the fetch failed | | `homelab_deploy_tree_dirty` | Whether someone edited a file on the host | | `homelab_deploy_verified` | Whether the deployed revision has a valid signature | +| `homelab_deploy_apply_enabled` | Whether this host applies what it fetches, or is in report-only mode | The quickest read of "what is this host running" is the journal, which Alloy already ships to Loki: @@ -201,7 +229,9 @@ copy — so converging there would report success and change nothing. Run it fro | `DeployUnverified` right after install | GitHub's key is not in `robo`'s keyring, so nothing can verify | The import above. This is the expected state between installing the timer and doing it | | `DeployUnverified` with the key present | `HEAD` is a commit that did not come through a pull request — usually someone committing on the host | `git log --show-signature -1` on the host. Get the commit onto a branch and merge it properly | | `DeployDrifted` | A file was edited on the monitoring host | §"The tree is dirty". The edit is still there — this alert exists because it used to not be | -| `DeployBehind` and nothing else | Convergence is refusing, or is in report-only mode | `journalctl -u homelab-converge.service -n 50` names the refusal. If `HOMELAB_CONVERGE_APPLY=0` is set, this is the mode working | +| `DeployBehind` **with** `DeployApplyDisabled` | Report-only mode — the host is fetching and recording but not applying | Working as intended. §Letting it act when you want it to deploy | +| `DeployBehind` **without** `DeployApplyDisabled` | Convergence is genuinely refusing | `journalctl -u homelab-converge.service -n 50` names the refusal; every case is in §When it refuses | +| `DeployApplyDisabled` you did not expect | Somebody set `HOMELAB_CONVERGE_APPLY=0` and it was forgotten | That is what this alert is for. `grep CONVERGE /etc/default/homelab-timers` | | `DeployBehind` with `ScheduledJobFailed` | The refusal is real and recurring | The journal names it; every case is in §"When it refuses" | | `DeployMetricsAbsent` | `homelab-deploy.prom` stopped arriving, while the backup metrics still do | Two separate files fail independently. Check `node_textfile_scrape_error`, then the file itself. If the timer was never installed, `make install-timers` | | `homelab_job_last_exit_code{homelab_job="converge"}` is 75 | It never started — the weekly backup held the `backups` lock for the full 900s | Expected at most once a week, on Sunday. Persistent means a backup is hanging: `systemctl list-units 'homelab-*'` | diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 695a262..203381f 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -104,7 +104,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 52 rules loaded, none in error. +2. **Prometheus → Status → Rules.** 53 rules loaded, none in error. 3. **Grafana → Dashboards → HomeLab.** Seven dashboards, populated. 4. **Grafana → Explore → Loki**, run `{host=~".+"}`. Logs should be arriving. 5. Confirm level normalisation is working — this has been silently broken diff --git a/docs/security.md b/docs/security.md index 5dc8da4..41b25e7 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | -| Losing visibility of a failure | 52 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 53 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | diff --git a/scripts/converge.sh b/scripts/converge.sh index 83f02be..0643903 100755 --- a/scripts/converge.sh +++ b/scripts/converge.sh @@ -101,7 +101,10 @@ # HOMELAB_CONVERGE_APPLY 0 makes every run report-only, as though --dry-run # had been passed. Set in /etc/default/homelab-timers # to watch the agent decide for a while before -# letting it act. +# letting it act. Recorded as +# homelab_deploy_apply_enabled, so the mode is +# visible from Prometheus rather than only from a +# file on the host. set -euo pipefail @@ -153,9 +156,23 @@ done # The report-only switch, so the timer can be installed and watched before it is # allowed to act. Folded into DRY_RUN rather than given a second code path — # two ways to not-apply is two things to get wrong. +# +# APPLY_ENABLED is tracked SEPARATELY from DRY_RUN, and the distinction is the +# whole reason it exists. DRY_RUN is also set by --dry-run, which is a human +# asking a question; this is a property of how the host is configured. Only the +# second is worth recording, because only the second persists after the run and +# explains why a host stays behind. +# +# Without it, DeployBehind can only say "it is refusing, OR report-only is set", +# and telling those apart means someone with shell access reading a file in +# /etc that nothing else in this repository tracks. That is precisely the shape +# of unrecorded state #99 is about, so the mode goes in the record with +# everything else. +APPLY_ENABLED=1 if [[ "${HOMELAB_CONVERGE_APPLY:-1}" == "0" ]]; then info "HOMELAB_CONVERGE_APPLY=0 — reporting only, nothing will be applied" DRY_RUN=1 + APPLY_ENABLED=0 fi # --------------------------------------------------------------------------- @@ -219,14 +236,17 @@ homelab_deploy_tree_dirty ${DIRTY} # HELP homelab_deploy_verified 1 when the deployed revision carries a good signature from the pinned key. # TYPE homelab_deploy_verified gauge homelab_deploy_verified ${VERIFIED} +# HELP homelab_deploy_apply_enabled 1 when this host applies what it fetches. 0 is report-only, set by HOMELAB_CONVERGE_APPLY=0. +# TYPE homelab_deploy_apply_enabled gauge +homelab_deploy_apply_enabled ${APPLY_ENABLED} EOF chmod 0644 "${tmp}" mv -f "${tmp}" "${PROM}" # One structured line for the journal, which Alloy already ships to Loki with # a `unit` label — findable with LogQL without parsing anything above it. - printf 'homelab-deploy revision=%s behind=%s dirty=%s verified=%s\n' \ - "${REVISION}" "${BEHIND}" "${DIRTY}" "${VERIFIED}" + printf 'homelab-deploy revision=%s behind=%s dirty=%s verified=%s apply=%s\n' \ + "${REVISION}" "${BEHIND}" "${DIRTY}" "${VERIFIED}" "${APPLY_ENABLED}" } trap record EXIT diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 864b4cd..c04391c 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox.yaml probe targets — hot-reloaded, no restart needed - rules/*.rules.yaml 52 alert rules across host/network/ups/containers/blackbox/backup/ids/deploy + rules/*.rules.yaml 53 alert rules across host/network/ups/containers/blackbox/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability from outside the service alertmanager/ diff --git a/stacks/observability/prometheus/rules/deploy.rules.yaml b/stacks/observability/prometheus/rules/deploy.rules.yaml index d9244f5..54fb813 100644 --- a/stacks/observability/prometheus/rules/deploy.rules.yaml +++ b/stacks/observability/prometheus/rules/deploy.rules.yaml @@ -5,7 +5,7 @@ # what got deployed, so `make up` on an uncommitted tree and `make up` on `main` # were indistinguishable; and nothing compared the host to the repository, so a # config edited in place stayed edited until a deploy silently overwrote it. -# scripts/converge.sh now runs hourly and writes five gauges into the same +# scripts/converge.sh now runs hourly and writes six gauges into the same # textfile directory the maintenance jobs use. # # WHAT backup.rules.yaml ALREADY COVERS, AND IS NOT REPEATED HERE @@ -19,7 +19,7 @@ # correctly refuses to move — because the tree is dirty, or the tip did not # verify — exits non-zero and is fully described by ScheduledJobFailed, but the # interesting fact is WHICH refusal and how far behind the host now is. These -# four rules read the record converge.sh writes rather than its exit code. +# five rules read the record converge.sh writes rather than its exit code. # # WHY THERE IS NO ALERT ON THE REVISION ITSELF # @@ -73,6 +73,11 @@ groups: # report-only mode — in which case this alert IS the report and is # expected to fire. # + # Which of those two it is used to be unanswerable without shell access + # to read /etc/default/homelab-timers. DeployApplyDisabled below now + # fires alongside this one in the report-only case, so the pair reads as + # one sentence and neither rule needs to guess. + # # > 0 and not != 0 on purpose. A failed fetch records -1 rather than a # stale count, precisely so it cannot masquerade as a known lag; the # fetch failing is ScheduledJobFailed's to report, and this rule stays @@ -94,9 +99,9 @@ groups: Convergence has run and has not applied what it fetched. Either it is refusing — check `journalctl -u homelab-converge.service -n 50` for a dirty tree, a signature that did not verify, or a - non-fast-forward — or HOMELAB_CONVERGE_APPLY=0 is set in - /etc/default/homelab-timers and this is the report-only mode doing - its job. docs/runbooks/converge-the-host.md. + non-fast-forward. If DeployApplyDisabled is firing too, it is + neither: the host is in report-only mode and this alert is that + mode's report. docs/runbooks/converge-the-host.md. - alert: DeployUnverified # The deployed revision does not carry a good signature from the pinned @@ -128,6 +133,41 @@ groups: request. `git -C /home/robo/code/Gerrrt/HomeLab log --show-signature -1` distinguishes them. docs/runbooks/converge-the-host.md. + - alert: DeployApplyDisabled + # The host is deliberately not deploying. + # + # This is the only rule here that reports a CHOICE rather than a fault, + # and it exists because the choice is made in /etc/default/homelab-timers + # — a file outside this repository, which nothing tracks and which is + # exactly the kind of thing that is still set eighteen months later + # because everyone who knew forgot. + # + # info, not warning: nothing is broken and nothing needs doing tonight. + # It routes to the info receiver alongside the other two info rules. + # + # It resolves by itself the moment the line is removed and the timer + # runs again, which is what makes it a better reminder than an issue — + # there is nothing to close, and no way to leave it stale. + # + # `for: 6h` because a deliberate mode is not urgent and a six-hour floor + # keeps it out of the way during the install itself, when converge is + # being primed and restarted by hand. + expr: homelab_deploy_apply_enabled == 0 + for: 6h + labels: + component: deploy + severity: info + category: correctness + annotations: + summary: "Convergence is in report-only mode and is deploying nothing" + description: >- + HOMELAB_CONVERGE_APPLY=0 is set in /etc/default/homelab-timers, so + this host fetches, verifies and records but applies nothing — + deployment is back to being something a human does. Intended as a + temporary state while the agent is watched. Remove the line and + `systemctl restart homelab-converge.timer` to let it act; this + resolves on the next run. docs/runbooks/converge-the-host.md. + - alert: DeployMetricsAbsent # The measuring apparatus, and not a duplicate of # ScheduledJobMetricsAbsent. diff --git a/stacks/observability/prometheus/tests/deploy.test.yaml b/stacks/observability/prometheus/tests/deploy.test.yaml index b79a503..cb1ce94 100644 --- a/stacks/observability/prometheus/tests/deploy.test.yaml +++ b/stacks/observability/prometheus/tests/deploy.test.yaml @@ -97,9 +97,9 @@ tests: Convergence has run and has not applied what it fetched. Either it is refusing — check `journalctl -u homelab-converge.service -n 50` for a dirty tree, a signature that did not verify, or a - non-fast-forward — or HOMELAB_CONVERGE_APPLY=0 is set in - /etc/default/homelab-timers and this is the report-only mode - doing its job. docs/runbooks/converge-the-host.md. + non-fast-forward. If DeployApplyDisabled is firing too, it is + neither: the host is in report-only mode and this alert is that + mode's report. docs/runbooks/converge-the-host.md. # --- DeployUnverified: unsigned, and signed ----------------------------- - interval: 1m @@ -132,6 +132,42 @@ tests: log --show-signature -1` distinguishes them. docs/runbooks/converge-the-host.md. + # --- DeployApplyDisabled: report-only, and applying -------------------- + # + # The paired silent case is the one that matters: a host that IS applying must + # not report itself as deliberately stopped, or the alert stops meaning + # anything the moment it is used for its intended week. + - interval: 1m + input_series: + - series: 'homelab_deploy_apply_enabled{instance="prometheus"}' + values: "0+0x480" + - series: 'homelab_deploy_apply_enabled{instance="oracle"}' + values: "1+0x480" + alert_rule_test: + - eval_time: 5h55m + alertname: DeployApplyDisabled + exp_alerts: [] + + - eval_time: 6h5m + alertname: DeployApplyDisabled + exp_alerts: + - exp_labels: + alertname: DeployApplyDisabled + instance: prometheus + component: deploy + severity: info + category: correctness + exp_annotations: + summary: "Convergence is in report-only mode and is deploying nothing" + description: >- + HOMELAB_CONVERGE_APPLY=0 is set in + /etc/default/homelab-timers, so this host fetches, verifies and + records but applies nothing — deployment is back to being + something a human does. Intended as a temporary state while the + agent is watched. Remove the line and `systemctl restart + homelab-converge.timer` to let it act; this resolves on the next + run. docs/runbooks/converge-the-host.md. + # --- DeployMetricsAbsent: the series is there --------------------------- # # Paired with the block below, which is the same rule with nothing fed to it.