Converge the host on main hourly instead of deploying over SSH (#99)

[Gerrrt]

Closes #99.

Deployment was make up typed into an SSH session. Nothing recorded what got deployed, so make up on an uncommitted tree and make up on main were indistinguishable; and nothing compared the host to the repository, so a config edited in place stayed edited until a deploy silently overwrote it.

scripts/converge.sh runs hourly on #77's existing machinery — the same timer pattern, the same run-scheduled.sh wrapper, the same textfile directory. It fetches main, refuses to move unless the tip verifies, fast-forwards, and runs make up.

It does not reimplement deploying. make up stays the one deployment path, so every runbook that names it stays true and the blast radius is the decision to deploy.

Ships in report-only mode

HOMELAB_CONVERGE_APPLY=0 in /etc/default/homelab-timers. The timer fetches, verifies and records; applying is switched on once it has been watched deciding correctly for a while.

The mode is a gauge, not just a file. A mode that lives only in /etc on one host is unrecorded state — the exact thing this PR is about — and without it DeployBehind could only say "it is refusing, OR report-only is set", leaving an operator to go read a file over ssh to tell a deliberate mode from a genuine refusal. So homelab_deploy_apply_enabled joins the record, and DeployApplyDisabled (info) reports it.

That alert is deliberately not a roadmap entry for "remember to turn this on": it fires six hours in and resolves by itself on the first run after the line is removed. Nothing to close, no way to leave it stale.

The secrets constraint dissolved

The issue's stated blocker was an age key on a host that pulls from a public repository. Checked, and it turns out not to be one: the key was already on that host, at ~/.config/sops/age/keys.txt, and has decrypted every make up there since the stack was deployed. Public means readable, and a key that decrypts local files is not endangered by strangers being able to read the repository.

The real question is unattended execution, which is about write access. So the gate is a pinned signing fingerprint — measured, not assumed:

Property of main, checked 2026-09-04	Result
Consecutive GitHub-signed merge commits from the tip	110
Oldest of that run	PR #33, 2026-08-19 — the history purge
Verify against 968479A1AFF927E37D1A566BB5690EEEBB952194	110 of 110

It also fetches an anonymous https:// URL rather than origin, so the agent holds no credential and cannot push. ADR-0021 states plainly what this does not buy: it cannot stop a compromised GitHub account, and the compensating control is the record, not the gate.

A dirty tree is a hard stop

No --force. Refusing is the point — overwriting is what the old model did, and losing the edit while never reporting it is half of what #99 is about.

What gets recorded

Six gauges, written on every exit path including the refusals, so a run that declined to move still reports what the host is on: the deployed revision, its commit time, how far behind main, whether the tree is dirty, whether the revision verified, and whether the host is applying at all.

converge is an ordinary JOBS row, so ScheduledJobStale/Failed/NeverRan cover it with no rule edit — which tests backup.rules.yaml's claim that adding a timer is a row in a table rather than an edit to a rule file. deploy.rules.yaml adds only what those cannot say: which refusal, and how far behind. All five new rules have promtool unit tests.

Verification

make validate passes (2 skips, both worktree-only). Every path exercised against a real clone: converged no-op, behind + --dry-run, dirty tree, missing key, unsigned tip, non-fast-forward, wrong branch, wrong checkout, and both apply modes.

Two manual steps on merge

Both on the monitoring host, before make install-timers (install primes each job once):

curl -fsSL https://github.com/web-flow.gpg | gpg --import

printf 'HOMELAB_CONVERGE_APPLY=0\n' | sudo tee -a /etc/default/homelab-timers

Until the key is imported, convergence refuses every run and DeployUnverified says so — deliberately, since treating an unverifiable commit as fine would make the gate decorative. docs/runbooks/converge-the-host.md §Set it up.

🤖 Generated with Claude Code

---

Note: this PR also unblocks main

main is currently failing its own ADR numbering guard — #98 (device joins) and #101 (lab stack) both merged as ADR-0019, 23 minutes apart. check_adr_numbers fails on any tree containing both, so this blocks every open PR, not just this one.

check_docs.py states the rule in its failure text — renumber the one that landed second — so the resolution is arithmetic, by landing order on main:

Merge	Landed	ADR
da0a071 (#271, device joins)	06:33	stays 0019
c7071c0 (#270, lab stack)	06:40	becomes 0020
this branch (unmerged)	—	becomes 0021

The converge ADR yields to both, since merged work should not renumber around an open branch. If you would rather renumber differently, say so and I will redo it — nothing here depends on the number.

[Copilot]

🔵 Needs a closer look

It introduces unattended deployment behavior on a production-like host, so a final human review is warranted despite only minor inline issues found.

Pull request overview

Introduces a pull-based, hourly convergence mechanism so the monitoring host fast-forwards to main and deploys via the existing make up path, while recording deployment state as Prometheus textfile metrics and alerting on drift/verification/lag.

Changes:

• Adds homelab-converge systemd timer/service plus a new scripts/converge.sh and make converge entrypoint.
• Adds deploy.rules.yaml + promtool tests to alert on deploy drift, verification failures, lag behind main, and missing deploy metrics.
• Updates operational documentation/runbooks/ADR and various rule-count summaries to reflect the new deploy alerting surface.

File summaries

File	Description
systemd/homelab-converge.timer	Defines hourly convergence cadence (with jitter and persistence).
systemd/homelab-converge.service	Runs convergence via the scheduled wrapper with the shared backups lock.
systemd/homelab-backup-volumes.service	Updates commentary to reflect an additional systemd unit in the set.
stacks/observability/README.md	Updates rule count and categorization to include deploy rules.
stacks/observability/prometheus/tests/deploy.test.yaml	Adds promtool unit tests covering the new deploy alert rules.
stacks/observability/prometheus/rules/deploy.rules.yaml	Introduces alert rules based on deployment convergence metrics.
stacks/observability/prometheus/rules/backup.rules.yaml	Updates documentation to reflect that deploy results live in a separate rules file.
scripts/install-timers.sh	Adds converge to the JOBS table and timer installation/priming flow.
scripts/converge.sh	Implements fetch/verify/fast-forward + make up, and writes deployment gauges.
README.md	Updates repo summary, topology counts, and documents the new self-deploy behavior.
Makefile	Adds make converge target as the human/timer entrypoint.
docs/security.md	Updates alert-rule count reference.
docs/runbooks/schedule-maintenance.md	Adds converge to the scheduled-jobs table and troubleshooting.
docs/runbooks/deploy-stack.md	Documents convergence as the normal update path post-initial deploy.
docs/runbooks/converge-the-host.md	Adds a dedicated runbook for setup, refusal modes, and disablement.
docs/roadmap.md	Marks #99 as answered and summarizes the chosen approach.
docs/observability.md	Updates alerting counts and documents deploy rules/test coverage.
docs/adr/0019-converge-on-a-timer-instead-of-deploying-over-ssh.md	Adds ADR capturing rationale, risks, and operational consequences.

Review details

• Files reviewed: 18/18 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.