From ada33fb56ef8f97b6f1c302313ec37883b5bc46b Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 4 Sep 2026 14:07:32 +0000 Subject: [PATCH] docs(runbook): say that report-only mode makes every merge a manual apply (#99) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The report-only section explained what the mode does and how to leave it, and never said what it costs while it is on: nothing is applied, so a merge lands on disk and reaches no running process. That is easy to miss precisely because a merge in this mode LOOKS like it landed. `git pull` moves the checkout, the files are there, and the containers bind-mount the directories those files came from. Nothing has re-read them. It caught this change out on its first day, which is why the note is a worked example rather than a caution. The merge installing the agent also added deploy.rules.yaml, so Prometheus ran with 48 alerting rules where the repository had 53 — and the five missing ones were the rules that watch convergence, DeployApplyDisabled among them. The mode had disabled the alerting that reports the mode, and nothing said so. `make reload` fixed it in one hot reload. Also records the trap underneath it: Prometheus re-reads rule files at startup and on POST /-/reload and at nothing else, while Loki polls its rule directory and updates on its own. So Loki being current is actively misleading evidence about Prometheus, and #98's Loki rules did arrive by themselves on the same day Prometheus's did not. deploy-stack.md §Updating gets the same correction. It opens "**Normally you do not.** A merged pull request reaches this host within the hour on its own", which is false while report-only is set — and that is the document someone reads when they want to know whether they still have to deploy. The verification command is `jq`, not `grep -c`, and that is not stylistic: the rules API returns the whole document on one line, so `grep -c` counts the line and answers 1 however many rules are loaded. Both forms were run against the live stack; the first draft of this runbook shipped the broken one. Co-Authored-By: Claude Opus 5 --- docs/runbooks/converge-the-host.md | 37 ++++++++++++++++++++++++++++++ docs/runbooks/deploy-stack.md | 6 +++++ 2 files changed, 43 insertions(+) diff --git a/docs/runbooks/converge-the-host.md b/docs/runbooks/converge-the-host.md index ed9a405..dbf6e08 100644 --- a/docs/runbooks/converge-the-host.md +++ b/docs/runbooks/converge-the-host.md @@ -85,6 +85,41 @@ printf 'HOMELAB_CONVERGE_APPLY=0\n' | sudo tee -a /etc/default/homelab-timers Every run then fetches, verifies and records, and applies nothing — deployment stays a thing a human does with `make up`. +> **Every merge needs a manual apply while this is set, including config that +> would otherwise need no deploy at all.** This is the part that is easy to miss, +> because a merge in this mode looks like it landed: `git pull` moves the +> checkout, the files are on disk, and the containers bind-mount the directories +> they came from. Nothing has re-read them. +> +> It caught this change out on its first day. The merge that installed the agent +> also added `deploy.rules.yaml`, so Prometheus sat with 48 alerting rules where +> the repository had 53 — and the five missing ones were the rules that watch +> convergence, `DeployApplyDisabled` among them. The mode had silently disabled +> the alerting that reports the mode. `make reload` fixed it in one hot reload, +> no container restart: +> +> ```bash +> cd /home/robo/code/Gerrrt/HomeLab && make reload +> ``` +> +> Prometheus re-reads rule files at startup and on `POST /-/reload`, and at +> nothing else. Loki is the exception that makes this easy to get wrong — it +> polls its rule directory, so its rules DO arrive on their own, and seeing them +> update is not evidence that Prometheus has. +> +> So while report-only is on, treat `make up` — or `make reload` for a +> config-only change — as part of merging, not as an optional follow-up. The +> check is one command: +> +> ```bash +> curl -s localhost:9090/api/v1/rules | jq '[.data.groups[].rules[] | select(.type=="alerting")] | length' +> ``` +> +> Compare it against `grep -c '^ - alert:' stacks/observability/prometheus/rules/*.rules.yaml`. +> Note the `jq` rather than a `grep -c` on the response: the API returns the +> whole document on one line, so `grep -c` counts that line and answers `1` +> however many rules are loaded. + Two alerts describe that state together, and reading them as a pair is the point: @@ -232,6 +267,8 @@ copy — so converging there would report success and change nothing. Run it fro | `DeployBehind` **with** `DeployApplyDisabled` | Report-only mode — the host is fetching and recording but not applying | Working as intended. §Letting it act when you want it to deploy | | `DeployBehind` **without** `DeployApplyDisabled` | Convergence is genuinely refusing | `journalctl -u homelab-converge.service -n 50` names the refusal; every case is in §When it refuses | | `DeployApplyDisabled` you did not expect | Somebody set `HOMELAB_CONVERGE_APPLY=0` and it was forgotten | That is what this alert is for. `grep CONVERGE /etc/default/homelab-timers` | +| A merged change is on disk but the stack does not have it | Report-only mode applies nothing, and Prometheus re-reads rule files only on reload | `make reload`, or `make up` if compose or a rendered file changed. Expected in this mode — §Start in report-only mode | +| Prometheus has fewer alerting rules than the repository | The same thing: a rule file landed and nothing reloaded | `make reload`. Loki polls its rule directory and updates on its own, so Loki being current is not evidence that Prometheus is | | `DeployBehind` with `ScheduledJobFailed` | The refusal is real and recurring | The journal names it; every case is in §"When it refuses" | | `DeployMetricsAbsent` | `homelab-deploy.prom` stopped arriving, while the backup metrics still do | Two separate files fail independently. Check `node_textfile_scrape_error`, then the file itself. If the timer was never installed, `make install-timers` | | `homelab_job_last_exit_code{homelab_job="converge"}` is 75 | It never started — the weekly backup held the `backups` lock for the full 900s | Expected at most once a week, on Sunday. Persistent means a backup is hanging: `systemctl list-units 'homelab-*'` | diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 0995658..885e501 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -161,6 +161,12 @@ on its own ([ADR-0021](../adr/0021-converge-on-a-timer-instead-of-deploying-over What follows is how to deploy something *now* rather than waiting, and what the timer is doing on your behalf. +**Unless the host is in report-only mode**, in which case none of that is true +and every merge needs one of the commands below — +[`converge-the-host.md`](converge-the-host.md) §Start in report-only mode says +how to tell, and why a merge in that mode looks like it landed when it has not. +That is the mode the agent was rolled out in. + ```bash make converge # fetch main, verify it, fast-forward, make up — the timer's job, now ```