diff --git a/.sops.yaml b/.sops.yaml index 70f5198..030b54a 100644 --- a/.sops.yaml +++ b/.sops.yaml @@ -22,6 +22,19 @@ # # Run `make secrets-init` on a fresh machine to generate a keypair and write its # public half into this file. +# +# ADDING A RECIPIENT +# +# `make secrets-add-recipient PUBKEY=age1...`, not a hand-edit. Editing this +# file without running `sops updatekeys` produces a recipient the repository +# advertises as a recovery path which cannot decrypt anything, and that is the +# DEFAULT outcome of doing it by hand — see +# docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md. +# +# Recipients are comma-separated, written one per line in the folded scalar. +# sops reads the value as a single string, splits it on commas and trims each +# part, so the line break is cosmetic and the trailing comma is not. Measured on +# sops 3.9.4; a wrong guess here parses as YAML and decrypts nothing. creation_rules: # The lab stack, FIRST — SOPS takes the first rule whose path_regex matches, diff --git a/Makefile b/Makefile index bd9444d..ca38b5a 100644 --- a/Makefile +++ b/Makefile @@ -102,6 +102,29 @@ secrets-edit: ## Edit the encrypted secrets in $$EDITOR @# unencrypted disk. scripts/secrets-edit.sh silences the editor first. ./scripts/secrets-edit.sh $(STACK) +.PHONY: secrets-add-recipient +secrets-add-recipient: ## Add a second age recipient and re-key (PUBKEY=age1...) + @# Under Secrets and not Maintenance because it is part of setting the + @# secrets up, but it is the one target here that rewrites a committed file + @# — .sops.yaml and secrets/$(STACK).sops.yaml both change and must be + @# committed together. ADR-0024 says why more than one recipient exists. + @# + @# PUBKEY rather than ARGS, for the reason secrets-verify-backup takes KEY: + @# exactly one argument, required, and an empty ARGS would reach the script + @# as no argument at all and print usage, which reads like a broken target. + @# + @# The PUBLIC half only. The private half of the key being added must never + @# be generated on, copied to, or pasted into this host — that is the whole + @# property the second recipient exists to have. + @[[ -n "$(PUBKEY)" ]] || { \ + printf '\033[0;31merror:\033[0m PUBKEY is required\n' >&2; \ + printf 'Generate the keypair where it will LIVE, then bring back its public half:\n' >&2; \ + printf ' make secrets-add-recipient PUBKEY=age1...\n' >&2; \ + printf 'See docs/runbooks/back-up-the-age-key.md\n' >&2; \ + exit 2; \ + } + ./scripts/add-recipient.sh "$(PUBKEY)" $(STACK) + .PHONY: secrets-show secrets-show: ## Print the decrypted secrets to stdout (careful) sops --decrypt $(SECRETS) diff --git a/docs/adr/0015-give-oracle-the-off-host-jobs.md b/docs/adr/0015-give-oracle-the-off-host-jobs.md index 1725839..f1464eb 100644 --- a/docs/adr/0015-give-oracle-the-off-host-jobs.md +++ b/docs/adr/0015-give-oracle-the-off-host-jobs.md @@ -2,6 +2,16 @@ **Status:** Accepted · 2026-09 +> [!NOTE] +> "Not a second age recipient", below, is narrowed by +> [ADR-0024](0024-hold-a-second-age-recipient-and-prove-each-one-separately.md). +> The rejection of **`oracle`** as a recipient is unchanged and still the design +> — this machine holds ciphertext and no key. What ADR-0024 sets aside is the +> broader framing quoted with it, that a second recipient is worth having only +> once the lab stops being a one-person project: that makes a loss problem read +> as a headcount one. A second recipient, held offline and off this estate, is +> now the design. The text here is left as written, per ADR-0001. + ## Context [#94](https://github.com/Gerrrt/HomeLab/issues/94) opens with "`oracle` diff --git a/docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md b/docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md new file mode 100644 index 0000000..e7aca56 --- /dev/null +++ b/docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md @@ -0,0 +1,180 @@ +# ADR-0024: Hold a second age recipient, and prove each one separately + +**Status:** Accepted · 2026-09 + +## Context + +[#106](https://github.com/Gerrrt/HomeLab/issues/106) reopens a question +[`back-up-the-age-key.md`](../runbooks/back-up-the-age-key.md) had already +answered, and is right to. That runbook closes with: + +> One key, one person. … It is worth doing on the day the lab stops being a +> one-person project, and not before — every extra recipient is another key that +> can leak. + +The issue's objection is that **the risk being described is not about team +size**. Nobody joining is what makes a second *holder* worth having; it has +nothing to do with whether the key survives a disk failure. Losing the key does +not degrade anything — every encrypted value in git history becomes permanently +undecryptable, and the recovery path is re-deriving each credential from the +device it belongs to. That is four SNMP rotations on live hardware, one of which +(`neo`) cannot persist a community deletion and needs a reboot, which in turn +cannot happen during working hours because that switch carries every VLAN. The +cost of the failure is measured in scheduled outages on someone else's calendar, +not in effort. + +[ADR-0015](0015-give-oracle-the-off-host-jobs.md) then answered the same +question the other way, four months later, and it is worth being precise about +what it actually rejected: + +> **Not a second age recipient.** … A second key on a powered, network-attached +> host in the same room, administered by the same person, adds no person and no +> offline copy. It only adds a key. Worse, it would be *this* host. … Giving it +> a private key that decrypts the estate's secrets makes it the one place where +> the backups and the means to open them sit on the same 5400 rpm disk. + +**Both of those paragraphs stand.** That argument is about `oracle`, and it does +not generalise: it turns on the machine being powered, network-attached, in the +same room, and already holding the ciphertext it would then be able to read. +[ADR-0023](0023-keep-the-household-recovery-path-outside-the-estate.md) has +since sharpened the same point in a different context — `oracle` is "the same +VLAN, the same rack, the same power feed and the same room", and every failure +worth insuring against reaches both boxes in one event. A key held offline, off +this estate, shares none of those properties. `oracle` remains a machine that +holds ciphertext and no key. + +### The argument #106 does not make, and it is the one that decides this + +For a one-person lab, a second *recipient* and a second *copy of the existing +key* are almost the same object. Either way there are two secret artefacts, each +of which alone decrypts everything, and losing both is total loss. The +cryptography does not care which one is chosen, and the confidentiality cost — +"another key that can leak" — is identical, because a second copy is also +another thing that can leak. + +They differ in exactly one respect, and it is not a cryptographic one: + +**`.sops.yaml` records a recipient in git. Nothing records a copy.** + +`verify-key-backup.sh` reads the public half out of the backup being tested and +matches it against the recipients of the encrypted file, so it can say *which* +recovery path a given run just proved. Two copies of one key are indistinguishable +to it and to everything else here — the same public half, the same match, the +same green result. Prove one and the tooling reports, accurately as far as it +can tell, that the backup works. + +That is the distinction this repository has already decided it cares about, in +this exact domain. [`secrets/README.md`](../../secrets/README.md) rejects a +gitignored `.env` and gives the reason: + +> A gitignored `.env` keeps secrets out of the repository, but it also keeps +> them out of any backup, review or history. + +And [`run-scheduled.sh`](../../scripts/run-scheduled.sh) exists because: + +> the requirement is not "run the job", it is "make NOT having run the job +> observable". + +A second copy of the key in a drawer is the `.env` of key backup. It works, and +nothing in this repository can see it, check it, or notice when it goes bad. + +### What that exposes about the existing deadline + +`SecretsKeyBackupUnproven` fires on +`homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup"}`, +which is one series. `run-scheduled.sh` writes it on every successful run +whatever `KEY=` pointed at. With one recipient that is exactly right and the +alert means what it says. + +With two it stops being true. Proving either copy resets the ninety-day clock +for both, so the second can rot behind a green alert — and the alert would be +*more* wrong the more recovery paths existed, which is the opposite of what +adding them is for. This is not a consequence of the decision below; it is a +defect the decision would introduce if left alone, and it is the reason the +mechanism is not, as #106 puts it, free. + +## Decision + +**The estate's secrets are encrypted to more than one age recipient, and every +recipient carries its own ninety-day proof.** Three parts. + +The mechanism and the per-recipient deadline are built with this ADR. **The +second keypair itself is not**, and cannot be: its private half must be +generated where it will live, which is somewhere this repository cannot reach — +see part 1. Until the operator does that, the estate has one recipient and every +check below behaves exactly as it did before, which is the honest state to leave +it in rather than pretending a key exists. + +**1. A second recipient, held offline and off this estate.** Its private half is +generated on the medium or machine that will keep it and never touches the +monitoring host — `scripts/add-recipient.sh` refuses a public key whose private +half it can find at `~/.config/sops/age/keys.txt`, because a key generated here +and added here is a second copy on the disk being insured wearing the costume of +a second recovery path. It is not `oracle`, for ADR-0015's reasons, restated +above and unchanged. Where it goes instead is an operational choice recorded in +`back-up-the-age-key.md`; the only constraint this ADR imposes is that it must +not fail at the same time as the first copy, which rules out the same drawer as +firmly as it rules out the same disk. + +**2. `make secrets-add-recipient PUBKEY=age1...` is how one is added.** Public +half only. It resolves which `creation_rule` governs the stack by looking at the +recipients the encrypted file *already uses*, rather than re-implementing sops' +first-match-wins `path_regex` resolution — which makes writing a key into the +wrong rule structurally impossible, and that matters because +[ADR-0020](0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) gives +`lab` a rule of its own precisely so a lab-guest key cannot decrypt the estate's +SNMP communities. It re-keys in the same run and rolls `.sops.yaml` back if that +fails, because the intermediate state — a recipient this repository advertises +as a recovery path which cannot decrypt anything — is worse than either end. + +**3. The proof is per recipient, read out of the ciphertext.** +`scripts/key-recipients.sh` emits +`homelab_key_recipient_last_proof_timestamp_seconds` with one series per +recipient, and `SecretsKeyBackupUnproven` fires on that instead. A recipient +that has never been verified is recorded as `0`, not omitted, so it is loud +rather than invisible — the same choice `run-scheduled.sh` makes for a job that +has never run. + +The recipient list comes from the `sops:` block inside +`secrets/.sops.yaml` and **not** from `.sops.yaml`. Those answer different +questions: `.sops.yaml` is the policy for the next encryption, and the file's own +metadata is the set of keys that can open the bytes on disk. They diverge for +exactly as long as it takes somebody to add a recipient and forget +`sops updatekeys`, which is a window in which the repository advertises a +recovery path that does not exist. Reading the ciphertext means every check here +is a statement about what can actually be recovered. + +## Consequences + +- **The confidentiality cost is real and is accepted.** There is now a second + private key that decrypts every secret in this repository, and ADR-0015's + "another key that can leak" applies to it in full. What changes the balance is + that the alternative being compared against is not "one key" — it is "one key + and an unaudited second copy of it", which carries the same exposure and + cannot be checked. +- **Revocation is still rotation.** Removing a recipient and re-keying protects + future values only; every historical ciphertext in git remains readable by the + removed key. A leaked second recipient means rotating every credential, exactly + as a leaked first one does. Nothing here improves that and the runbook says so. +- **The ninety-day deadline gets stricter on its own.** The threshold is still + declared once, in the `JOBS` table in `install-timers.sh`, but it now applies + to each recipient independently. Adding a recipient adds an alert that fires + immediately and keeps firing until that specific copy has been mounted and + tested. That is the intended behaviour and it is also the main ongoing cost: + two copies means two trips to wherever they are kept, four times a year. +- **`ScheduledJobNeverRan` still covers the cold start.** Before any + verification has ever happened, `key-recipients.sh` has not written its file + and there are no per-recipient series to fire on, so the generic rule speaks — + unchanged from today. +- **This does not close the "one person" half of #106.** The title says "one + holder and one copy" and this ADR answers the copy. A second holder is still a + question about who else should be able to open the estate's secrets, which is + a decision about people and is not made here. What changes is that the + mechanism for it now exists and is exercised: adding a second *holder* later is + the same command with someone else's public key. +- **`make render` is the canary if sops changes.** Recipients are written one per + line in a folded scalar, comma-separated, which relies on sops trimming each + entry. That was measured on sops 3.9.4 rather than assumed, and + `add-recipient.sh` re-reads the file after re-keying so a future regression + fails at add time. If it ever regressed silently instead, decryption on the + deployment host is where it would surface. diff --git a/docs/observability.md b/docs/observability.md index b832400..8c0f53a 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -504,7 +504,15 @@ checking, with the key that is on that machine, against the disk that is in it. disk or a fire. The only job that proves off-host recoverability is `secrets-verify-backup`, and it is precisely the one that cannot be automated — it needs a human to mount removable media, so `SecretsKeyBackupUnproven` nags at -ninety days instead. One output does leave: `backup-firewall` copies each export +ninety days instead. That alert is the one rule in `backup.rules.yaml` not keyed +on `homelab_job`: +[ADR-0024](adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +allows the secrets to be encrypted to more than one age recipient, so it fires +per recipient off `homelab_key_recipient_last_proof_timestamp_seconds` rather +than off the job. One timestamp for every copy would mean proving either one +vouched for the other, which is backwards when the whole point of the second +copy is that it fails independently. With a single recipient it behaves exactly +as it always has. One output does leave: `backup-firewall` copies each export to `oracle` and fails if it cannot, so its failure alert doubles as "the config has stopped leaving this host". The volume sets do not leave; that is [#92](https://github.com/Gerrrt/HomeLab/issues/92). diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md index df29b49..5f8c2e9 100644 --- a/docs/runbooks/back-up-the-age-key.md +++ b/docs/runbooks/back-up-the-age-key.md @@ -203,28 +203,96 @@ the live key by device and inode on purpose: what gets tested has to be a copy o removable media, and no timer can mount that. So it is enforced from the other end. `make secrets-verify-backup` records the -timestamp of a successful run, and `SecretsKeyBackupUnproven` fires when that -proof passes ninety days old — routed to the normal alert channel like any other -warning. Until the first verification there is no timestamp at all and -`ScheduledJobNeverRan` says so instead, which is the honest reading of a key -backup nobody has ever tested. +timestamp of a successful run **against the recipient it just proved**, and +`SecretsKeyBackupUnproven` fires when any recipient's proof passes ninety days +old — routed to the normal alert channel like any other warning. Until the first +verification there is no timestamp at all and `ScheduledJobNeverRan` says so +instead, which is the honest reading of a key backup nobody has ever tested. + +The per-recipient part is [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +and it only starts to matter once there is more than one. With one recipient +this is the same alert it always was. With two, one timestamp for both would +mean that proving either copy vouched for the other — which is exactly backwards, +because the point of the second copy is that it fails independently. Nagging is not as good as running it. It is a great deal better than remembering. The threshold lives in the `JOBS` table in [`install-timers.sh`](../../scripts/install-timers.sh); the mechanism is in [`schedule-maintenance.md`](schedule-maintenance.md). +## Adding a second recipient + +The section this replaces said a second recipient was "worth doing on the day +the lab stops being a one-person project, and not before". +[#106](https://github.com/Gerrrt/HomeLab/issues/106) pointed out that this +frames a loss problem as a headcount problem. Nobody joining is what makes a +second *holder* worth having; it has nothing to do with whether the key survives +a disk dying. [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +decides the question on the loss grounds and explains why a second *recipient* +beats a second *copy* — briefly: `.sops.yaml` records a recipient and nothing +records a copy, so the tooling can name and separately nag about a recipient and +cannot tell two copies of one key apart. + +Generate the keypair **on the machine or the medium that will keep it**. Not +here. A key generated on the monitoring host and then added as a recipient is a +second copy on the disk you are insuring, and `make secrets-add-recipient` +refuses it for that reason. + +```bash +# ...somewhere that is not this host: +umask 077 +age-keygen -o /path/to/its/keys.txt +age-keygen -y /path/to/its/keys.txt # the PUBLIC half — this is what travels +``` + +Then, on the monitoring host, with only that public half: + +```bash +make secrets-add-recipient PUBKEY=age1... +``` + +That writes the key into the `creation_rule` this stack's file already uses and +re-keys the file in the same run. sops shows the group change and asks before +committing to it; answering no rolls `.sops.yaml` back, so the two never +disagree. Both files change and belong in one commit: + +```bash +git add .sops.yaml secrets/observability.sops.yaml +``` + +The new recipient starts unproven, and says so: `SecretsKeyBackupUnproven` fires +against it immediately and names it. Clear it the same way as the first — +`make secrets-verify-backup KEY=/path/to/the/new/copy` — and note that this is +the point of the whole exercise. **Proving one recipient does not clear another.** +Each copy has to be mounted and tested on its own, which is four trips a year +rather than two. + +The table in [Copy the key](#1-copy-the-key) and the disqualifiers in +[Confirm the copy is not republishing itself](#3-confirm-the-copy-is-not-republishing-itself) +apply to the second copy as much as the first, plus one rule that only exists +once there are two: **it must not fail +at the same time as the first.** Two USB sticks in the same drawer is one copy +with extra steps, and so is a second recipient whose private half is in the same +password-manager vault as the first — the failure mode of a vault is losing the +master password, and that takes everything in it at once. + ## What this still does not solve -One key, one person, one copy plus the original. If the answer to "who else can -recover this" needs to be more than one, the mechanism already exists: generate a -second keypair that lives only offline, add its public half to `.sops.yaml` as an -additional recipient, and re-key with -`sops updatekeys secrets/observability.sops.yaml` from a host that can already -decrypt. That is the same procedure as bringing a second host in, described in -[`secrets/README.md`](../../secrets/README.md). It is worth doing on the day the -lab stops being a one-person project, and not before — every extra recipient is -another key that can leak. +**One person.** ADR-0024 answers the copy and deliberately does not answer the +holder: whether somebody else should be able to open the estate's secrets is a +question about people, not about mechanism. What has changed is that the +mechanism is now built and exercised — a second holder is the same +`make secrets-add-recipient` with their public key, not a procedure to work out +on the day it matters. + +**Revocation is still rotation.** Removing a recipient and re-keying protects +values encrypted from then on. Every historical ciphertext in git stays readable +by the removed key, because the commits are still there. A second recipient that +leaks means rotating every credential — four SNMP communities on hardware, the +Grafana password, the four notification URLs and the renderer token — exactly as +a leaked first one does. That is why "every extra recipient is another key that +can leak" is still the right thing to weigh; ADR-0024 weighs it and accepts it +against an alternative that carries the same exposure unaudited. ## If something goes wrong diff --git a/docs/runbooks/successor-handover.md b/docs/runbooks/successor-handover.md index a618308..eb3ae0a 100644 --- a/docs/runbooks/successor-handover.md +++ b/docs/runbooks/successor-handover.md @@ -193,7 +193,7 @@ inherits without knowing. | --- | --- | --- | | **Alert delivery to a destination you do not own** | Immediately, and silently | You do not. This is step 2 above, and it is the reason it is step 2 | | **The external heartbeat watcher** — a free-tier cron-monitor on somebody else's account | Whenever that account lapses | Nothing here can tell you. A watcher on this host would fail with the thing it watches, which is why it is off-host and therefore outside anything this repository can check | -| **The age key backup goes unproven** | 90 days after the last verification | `SecretsKeyBackupUnproven`, routed to the normal alert channel. Until the first verification there is no timestamp at all and `ScheduledJobNeverRan` says so instead | +| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). Until the first verification there is no timestamp at all and `ScheduledJobNeverRan` says so instead | | **Grafana's leaf certificate** | 825 days from issue; the APC card's own certificate expires on its own clock | `TlsCertificateExpiringSoon` at 30 days, `TlsCertificateExpiryImminent` at 7 — read off the served handshake by `blackbox-exporter`, not off a file. Let it lapse and `up{job="grafana"}` goes to 0 as well | | **The UPS battery pack** | A pack was fitted 2026-08-28 and passed its self-test; packs are consumables and this one is on a biweekly test schedule | `UpsSelfTestFailed` and `UpsBatteryUnproven` key on the self-test result, which is the single honest signal this card emits — every charge, runtime and alarm value it reports was fabricated while the bay was empty. Two things remain open: the card's test *schedule* is unwatched ([#249](https://github.com/Gerrrt/HomeLab/issues/249)), and `upsBasicBatteryLastReplaceDate` still reads a pre-fit date, so it is not a usable record of the pack's age | | **Mains power to the monitoring path** | Any cut | The rack is on the UPS; the switch carrying `prometheus` and `oracle` is not, so both laptops keep running and go deaf. Stated in [`security.md`](../security.md#threat-model) | @@ -243,15 +243,36 @@ outgoing operator directly: ### If you do not have the age key Say so out loud before anything else. This is -[#106](https://github.com/Gerrrt/HomeLab/issues/106) — one key, one holder, one -copy plus the original — and it is open. Without it every encrypted value in -this repository and its history is permanently undecryptable, and the recovery -path is re-deriving each credential from the device it belongs to: four SNMP -rotations on hardware, one of which cannot persist a community deletion and -needs the switch rebooted to change. The mechanism for a second recipient costs -nothing (`.sops.yaml` takes a list, and `sops updatekeys` re-keys the file from -any host that can already decrypt); what it has always needed is somewhere to -put the second key. A handover is that somewhere. +[#106](https://github.com/Gerrrt/HomeLab/issues/106). Without the key every +encrypted value in this repository and its history is permanently undecryptable, +and the recovery path is re-deriving each credential from the device it belongs +to: four SNMP rotations on hardware, one of which cannot persist a community +deletion and needs the switch rebooted to change. + +Check first whether it is actually gone. `.sops.yaml` may list more than one +recipient — [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +made a second one the design, and a second *copy* may exist that is not the one +you were handed: + +```bash +grep -A3 creation_rules .sops.yaml # every key that can open the secrets +``` + +If a recipient there is one somebody else holds, the secrets are recoverable and +this is a phone call rather than a rotation. + +**A handover is the moment to add one.** ADR-0024 answers the copy and +deliberately leaves the holder open, which is exactly the question a successor +is. Generate the keypair where *they* will keep it, and add its public half from +a host that can already decrypt: + +```bash +make secrets-add-recipient PUBKEY=age1... +``` + +That is not free — every extra recipient is another key that can leak, and +removing one later protects future values only, because the history stays +readable by it. ADR-0024 weighs both. ### If you do have it diff --git a/scripts/add-recipient.sh b/scripts/add-recipient.sh new file mode 100755 index 0000000..2c0052c --- /dev/null +++ b/scripts/add-recipient.sh @@ -0,0 +1,253 @@ +#!/usr/bin/env bash +# +# Add a second age recipient to a stack's secrets: register its public half in +# .sops.yaml and re-key the encrypted file so the new key can actually open it. +# +# ADR-0024 decides that this repository holds more than one recipient and why. +# This is the mechanism, and it exists rather than "edit .sops.yaml and run sops +# updatekeys" for the two reasons bootstrap.sh already refuses to be a one-liner: +# +# - .sops.yaml has more than one creation_rule, and writing a key into the +# wrong one is silent. ADR-0020 gives `lab` a rule of its own precisely so a +# lab-guest key cannot decrypt the estate's SNMP communities; a hand-edit +# into the general rule undoes that and still looks like it worked. +# - Editing .sops.yaml WITHOUT running `sops updatekeys` produces the worst +# possible state: a recipient this repository advertises as a recovery path +# which cannot decrypt anything. That is not a hypothetical failure, it is +# the default outcome of doing this by hand and forgetting the second step. +# +# ONLY THE PUBLIC HALF EVER REACHES THIS SCRIPT. The private half of the key +# being added must never exist on this host — see the guard in section 2, and +# docs/runbooks/back-up-the-age-key.md for where it should live instead. +# +# Usage: scripts/add-recipient.sh [stack] +# make secrets-add-recipient PUBKEY=age1... + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +PUBKEY="${1:-}" +STACK="${2:-observability}" +SECRETS_FILE="${REPO_ROOT}/secrets/${STACK}.sops.yaml" +SOPS_CONFIG="${REPO_ROOT}/.sops.yaml" +LIVE_KEY="${SOPS_AGE_KEY_FILE:-${HOME}/.config/sops/age/keys.txt}" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } +warn() { printf '\033[0;33m!!\033[0m %s\n' "$*"; } + +if [[ -z "${PUBKEY}" ]]; then + cat >&2 < [stack] + or: make secrets-add-recipient PUBKEY=age1... + +Adds a recipient to secrets/${STACK}.sops.yaml. Public half only — generate the +keypair on the machine that will HOLD it, never here. +Procedure: docs/runbooks/back-up-the-age-key.md +EOF + exit 2 +fi + +command -v sops >/dev/null 2>&1 || die "sops not found. + https://github.com/getsops/sops/releases" + +[[ -f "${SECRETS_FILE}" ]] || die "no encrypted secrets at secrets/${STACK}.sops.yaml +Run 'make secrets-init' first." +[[ -f "${SOPS_CONFIG}" ]] || die "no ${SOPS_CONFIG}" + +# --------------------------------------------------------------------------- +# 1. Is it an age public key at all? +# --------------------------------------------------------------------------- +# Checked before anything is written, because the failure mode of a typo is a +# .sops.yaml that sops refuses to parse — which breaks `make render` and takes +# the stack down at the next converge, for a value nobody can decrypt with +# anyway. +# +# An age X25519 public key is bech32 with the `age1` HRP and is always 62 +# characters. Length is checked rather than left to the regex so that a +# truncated paste is named as truncated. +[[ "${PUBKEY}" =~ ^age1[a-z0-9]+$ ]] \ + || die "not an age public key: ${PUBKEY} +It should start with 'age1' and contain only lowercase letters and digits. +A private key starts with AGE-SECRET-KEY-1 and must never be pasted here." + +((${#PUBKEY} == 62)) \ + || die "'${PUBKEY}' is ${#PUBKEY} characters; an age public key is 62. +Usually a truncated or wrapped paste. Take it from 'age-keygen -y ' on +the machine that holds the private half." + +# --------------------------------------------------------------------------- +# 2. Is it already a recipient? +# --------------------------------------------------------------------------- +mapfile -t CURRENT < <("${REPO_ROOT}/scripts/key-recipients.sh" --list --stack "${STACK}") + +# key-recipients.sh dies with its own message when the file lists none, but it +# dies inside a process substitution, whose exit status this shell never sees. +# Without this the next line would index an empty array and report a bash error +# instead of the real one. +((${#CURRENT[@]})) || die "could not read the recipients of secrets/${STACK}.sops.yaml — see above" + +if printf '%s\n' "${CURRENT[@]}" | grep -qxF "${PUBKEY}"; then + info "secrets/${STACK}.sops.yaml is already encrypted to ${PUBKEY}" + info "nothing to do" + exit 0 +fi + +# --------------------------------------------------------------------------- +# 3. It must not be a key this host already holds +# --------------------------------------------------------------------------- +# The whole value of a second recipient is that its private half is somewhere +# this machine is not. A key generated here and then added here is a second copy +# on the disk being insured, wearing the costume of a second recovery path — and +# it would pass every other check in this script and in verify-key-backup.sh. +# +# ADR-0015 rejects `oracle` as a recipient on the same argument one host over: +# the backups and the means to open them must not share a disk. +if [[ -r "${LIVE_KEY}" ]] && command -v age-keygen >/dev/null 2>&1; then + if age-keygen -y "${LIVE_KEY}" 2>/dev/null | grep -qxF "${PUBKEY}"; then + die "that is this host's own key. + +Its private half is at ${LIVE_KEY}, on the disk these secrets already live on. +Adding it as a second recipient records a recovery path that dies with the +machine it is supposed to survive. + +Generate the keypair on the machine or the medium that will hold it, and bring +only the public half back here: age-keygen -y /path/to/its/keys.txt" + fi +fi + +# --------------------------------------------------------------------------- +# 4. Which creation_rule does this stack's file actually use? +# --------------------------------------------------------------------------- +# Derived from the file rather than by re-implementing sops' first-match-wins +# path_regex resolution. The rule that governs this file is, by definition, the +# one listing the recipients the file is already encrypted to — so the anchor is +# a fact about the ciphertext, not a guess about the policy. +# +# That also makes the ADR-0020 mistake structurally impossible: the lab rule +# does not list the estate's recipient, so `--stack observability` cannot land +# in it, and vice versa. +ANCHOR="${CURRENT[0]}" + +mapfile -t ANCHOR_LINES < <(grep -nE "^[[:space:]]+${ANCHOR},?[[:space:]]*$" "${SOPS_CONFIG}" | cut -d: -f1) + +if ((${#ANCHOR_LINES[@]} == 0)); then + die "secrets/${STACK}.sops.yaml is encrypted to + ${ANCHOR} +but no creation_rule in .sops.yaml lists that key. + +The policy and the ciphertext disagree, and this script cannot tell which one is +right. Fix .sops.yaml by hand — the recipients the file actually uses are: +$(printf ' %s\n' "${CURRENT[@]}")" +fi + +if ((${#ANCHOR_LINES[@]} > 1)); then + die "${ANCHOR} +appears in ${#ANCHOR_LINES[@]} creation_rules in .sops.yaml (lines: ${ANCHOR_LINES[*]}). + +One key covering two rules is the collapse those rules exist to prevent — +bootstrap.sh refuses the same thing. Untangle .sops.yaml by hand first." +fi + +LINE="${ANCHOR_LINES[0]}" +ANCHOR_TEXT="$(sed -n "${LINE}p" "${SOPS_CONFIG}")" + +# The anchor line matched ^[[:space:]]+age1...,?[[:space:]]*$ to get here, so +# these two are total: everything before the key is the indent, and the key +# itself is what the new line has to line up with. +INDENT="${ANCHOR_TEXT%%age1*}" +ANCHOR_TEXT="${ANCHOR_TEXT%"${ANCHOR_TEXT##*[![:space:]]}"}" + +# --------------------------------------------------------------------------- +# 5. Write it in +# --------------------------------------------------------------------------- +# One recipient per line, comma-separated. sops reads `age:` as a single string +# and splits it on commas, trimming each part — so a folded scalar that puts one +# key per line is the same value as one long line, and reviews far better. +# Measured on sops 3.9.4 rather than assumed, because a wrong guess here is a +# .sops.yaml that parses as YAML and decrypts nothing. +# +# The anchor keeps whatever trailing comma it had: if it already ended in one it +# was not the last entry, and the new line needs one too. +if [[ "${ANCHOR_TEXT}" == *, ]]; then + NEW_LINE="${INDENT}${PUBKEY}," + REWRITTEN="${ANCHOR_TEXT}" +else + NEW_LINE="${INDENT}${PUBKEY}" + REWRITTEN="${ANCHOR_TEXT}," +fi + +BACKUP="$(mktemp)" +cp "${SOPS_CONFIG}" "${BACKUP}" +# Restores .sops.yaml on any exit before the re-key is confirmed. A half-applied +# change here is the state described at the top of this file: an advertised +# recovery path that cannot decrypt. +restore() { cp "${BACKUP}" "${SOPS_CONFIG}"; } +trap 'restore; rm -f "${BACKUP}"' EXIT INT TERM + +python3 - "${SOPS_CONFIG}" "${LINE}" "${REWRITTEN}" "${NEW_LINE}" <<'PY' +import sys +path, line, rewritten, new_line = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4] +lines = open(path).read().split("\n") +lines[line - 1] = rewritten +lines.insert(line, new_line) +open(path, "w").write("\n".join(lines)) +PY + +info "added ${PUBKEY} to .sops.yaml" + +# --------------------------------------------------------------------------- +# 6. Re-key, which is the half that does the work +# --------------------------------------------------------------------------- +# Needs the private half of a key that can ALREADY decrypt — normally the live +# key on this host. sops prints the group change and asks; that prompt is worth +# keeping, because it is the last point at which a wrong recipient is cheap. +info "re-keying secrets/${STACK}.sops.yaml" +if ! sops updatekeys "${SECRETS_FILE}"; then + die "sops updatekeys failed — .sops.yaml has been rolled back. + +Nothing changed. The usual cause is that this host cannot decrypt the file, and +re-keying requires a key that can. Run this where the existing key is." +fi + +# --------------------------------------------------------------------------- +# 7. Do not trust the exit status alone +# --------------------------------------------------------------------------- +# The same rule bootstrap.sh and verify-key-backup.sh apply. `updatekeys` exits +# 0 when the user answers no at its prompt, which leaves .sops.yaml advertising +# a recipient the ciphertext has never heard of. +if ! "${REPO_ROOT}/scripts/key-recipients.sh" --list --stack "${STACK}" | grep -qxF "${PUBKEY}"; then + die "sops exited 0 but secrets/${STACK}.sops.yaml is still not encrypted to +${PUBKEY} — answering 'no' at the prompt does exactly this. + +.sops.yaml has been rolled back so the two still agree. Re-run when ready." +fi + +trap - EXIT INT TERM +rm -f "${BACKUP}" + +# The new recipient starts life unproven, and says so out loud rather than +# waiting for the next verification of some OTHER key to notice it exists. +# key-recipients.sh writes 0 for it, SecretsKeyBackupUnproven reads that as +# "never", and the nagging starts now instead of in ninety days. +"${REPO_ROOT}/scripts/key-recipients.sh" --record --stack "${STACK}" || true + +cat <.sops.yaml is fact: the recipients this file is +# encrypted to right now. They differ for exactly as long as it takes somebody +# to add a key and forget `sops updatekeys` — a window in which .sops.yaml +# promises a recovery path that does not exist, and every check reading it +# reports a backup that cannot open anything. +# +# So this reads the file. No decryption is involved: the recipient list is +# plaintext metadata, which is the same reason the key names are. +# +# WHY THE TIMESTAMPS ARE PER RECIPIENT +# +# ADR-0024. `homelab_job_last_success_timestamp_seconds{homelab_job= +# "verify-key-backup"}` is one series, and `make secrets-verify-backup` records +# it whichever key was mounted. With one recipient that is exactly right. With +# two it says "a key was proved" when the question is "was THIS key proved" — +# so proving one resets the ninety-day deadline for both and the other is free +# to rot behind a green alert. One series per recipient is the smallest thing +# that cannot lie about that. +# +# A recipient that has never been verified is recorded as 0 rather than omitted. +# time() - 0 is about 1.8 billion seconds, which exceeds every threshold, so +# "never proved" and "not proved lately" are one alert — the same choice +# run-scheduled.sh makes and for the same reason. +# +# Usage: +# scripts/key-recipients.sh --list [--stack ] +# scripts/key-recipients.sh --record [--stack ] [--proved ] +# +# Environment: +# TEXTFILE_DIR where the .prom files go +# (default /var/lib/node_exporter/textfile_collector) + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +TEXTFILE_DIR="${TEXTFILE_DIR:-/var/lib/node_exporter/textfile_collector}" + +STACK="observability" +MODE="" +PROVED="" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +warn() { printf '\033[0;33m!!\033[0m %s\n' "$*" >&2; } + +usage() { sed -n '/^# Usage:/,/^$/p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } + +while (($#)); do + case "$1" in + --list) MODE="list"; shift ;; + --record) MODE="record"; shift ;; + --stack) STACK="${2:-}"; shift 2 ;; + --proved) PROVED="${2:-}"; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac +done + +[[ -n "${MODE}" ]] || { usage >&2; die "--list or --record is required"; } + +# Same shape run-scheduled.sh enforces on a job name, for the same reason: this +# becomes a path segment and a Prometheus label value. +[[ "${STACK}" =~ ^[a-z][a-z0-9-]{0,30}$ ]] \ + || die "stack name '${STACK}' must match ^[a-z][a-z0-9-]{0,30}\$" + +SECRETS_FILE="${REPO_ROOT}/secrets/${STACK}.sops.yaml" +[[ -f "${SECRETS_FILE}" ]] || die "no encrypted secrets at secrets/${STACK}.sops.yaml" + +# --------------------------------------------------------------------------- +# The recipients, read out of the file's own metadata +# --------------------------------------------------------------------------- +# grep and not a YAML parser. The `recipient:` keys sit inside the `sops:` block +# and nowhere else in these files — every other value is ciphertext — and a +# bech32 age public key cannot appear in an ENC[...] literal or an armoured age +# blob, both of which are base64. Keeping this dependency-free matters because +# verify-key-backup.sh calls it on a host that has just been rebuilt from bare +# metal, where python3 and PyYAML are not yet a given. +mapfile -t RECIPIENTS < <(grep -oE '^[[:space:]]*(-[[:space:]]+)?recipient:[[:space:]]*age1[a-z0-9]+' "${SECRETS_FILE}" \ + | grep -oE 'age1[a-z0-9]+' | sort -u) + +((${#RECIPIENTS[@]})) \ + || die "secrets/${STACK}.sops.yaml lists no age recipients. +Either it is not SOPS-encrypted, or it is encrypted to a KMS this repository +does not use. Check: grep -A3 '^sops:' secrets/${STACK}.sops.yaml" + +if [[ "${MODE}" == "list" ]]; then + printf '%s\n' "${RECIPIENTS[@]}" + exit 0 +fi + +# --------------------------------------------------------------------------- +# Record +# --------------------------------------------------------------------------- +if [[ -n "${PROVED}" ]]; then + [[ "${PROVED}" =~ ^age1[a-z0-9]+$ ]] || die "--proved is not an age public key: ${PROVED}" + # Refused rather than silently ignored. Reaching here means a key decrypted + # the file while not being one of the recipients the file records, which is + # not a thing that can happen — so it is a bug in the caller, and writing a + # proof row for a recipient that does not exist would age out and fire an + # alert naming a key nobody can find. + printf '%s\n' "${RECIPIENTS[@]}" | grep -qxF "${PROVED}" \ + || die "${PROVED} is not a recipient of secrets/${STACK}.sops.yaml" +fi + +# Same two failure modes run-scheduled.sh distinguishes, and the same answers: +# a missing directory is a human on a workstation and must not break the +# command; an unwritable one is a broken monitoring host and must not be +# reported as a recorded outcome. +if [[ ! -d "${TEXTFILE_DIR}" ]]; then + warn "no textfile directory at ${TEXTFILE_DIR} — not recording which recipients are proved" + exit 0 +elif [[ ! -w "${TEXTFILE_DIR}" ]]; then + die "${TEXTFILE_DIR} is not writable by $(id -un). +Fix the directory, then re-run: + sudo install -d -m 0755 -o $(id -un) -g $(id -gn) ${TEXTFILE_DIR}" +fi + +PROM="${TEXTFILE_DIR}/key-recipients.prom" +NOW="$(date +%s)" + +# Carry forward what is already there, exactly as run-scheduled.sh carries +# forward prior_success: this file is rewritten in full on every run, so a +# recipient's proof would otherwise be lost the moment a different one was +# verified — which is the failure this whole file exists to prevent. +prior_for() { + local want="$1" + [[ -r "${PROM}" ]] || { printf '0'; return; } + awk -v want="${want}" ' + $0 ~ /^homelab_key_recipient_last_proof_timestamp_seconds\{/ { + if (index($0, "recipient=\"" want "\"")) value = $NF + } + END { print (value ~ /^[0-9]+$/) ? value : "0" } + ' "${PROM}" +} + +# Every recipient of every stack shares one file, because node_exporter merges +# the directory and a metric name may carry only one HELP string across it. A +# second stack writing its own file would collide on that, not on the series. +# So rows for other stacks are carried through untouched. +others="" +if [[ -r "${PROM}" ]]; then + others="$(grep -F 'homelab_key_recipient_last_proof_timestamp_seconds{' "${PROM}" \ + | grep -vF "stack=\"${STACK}\"" || true)" +fi + +tmp="${PROM}.$$" +{ + printf '# HELP homelab_key_recipient_last_proof_timestamp_seconds Unix time an offline copy of this age recipient private key was last proved to decrypt the stack secrets. 0 means never.\n' + printf '# TYPE homelab_key_recipient_last_proof_timestamp_seconds gauge\n' + [[ -n "${others}" ]] && printf '%s\n' "${others}" + for recipient in "${RECIPIENTS[@]}"; do + if [[ "${recipient}" == "${PROVED}" ]]; then + ts="${NOW}" + else + ts="$(prior_for "${recipient}")" + fi + printf 'homelab_key_recipient_last_proof_timestamp_seconds{stack="%s",recipient="%s"} %s\n' \ + "${STACK}" "${recipient}" "${ts}" + done +} > "${tmp}" + +# 0644 explicitly and rename to publish, for the two reasons run-scheduled.sh +# gives: a 0600 .prom is invisible to the collector, and a truncate-in-place +# exposes a half-written file to a scrape. +chmod 0644 "${tmp}" +mv -f "${tmp}" "${PROM}" diff --git a/scripts/verify-key-backup.sh b/scripts/verify-key-backup.sh index 2fb86bb..5b15f22 100755 --- a/scripts/verify-key-backup.sh +++ b/scripts/verify-key-backup.sh @@ -152,10 +152,21 @@ An age key file holds a line beginning AGE-SECRET-KEY-1. If you transcribed this from paper, check for a truncated or wrapped line." fi +# Matched against the RECIPIENTS OF THE FILE, not against .sops.yaml. +# +# .sops.yaml is the policy for the next encryption; the `sops:` block inside the +# encrypted file is the list of keys that can open the bytes on disk. They +# diverge for exactly as long as it takes somebody to add a recipient and forget +# `sops updatekeys` — and in that window this check used to pass on a key that +# then failed at section 4 with "the secret half is damaged or truncated", +# sending you to re-copy a backup that was never the problem. +mapfile -t RECIPIENTS < <("${REPO_ROOT}/scripts/key-recipients.sh" --list --stack "${STACK}") +((${#RECIPIENTS[@]})) || die "could not read the recipients of secrets/${STACK}.sops.yaml — see above" + matched="" while IFS= read -r pub; do [[ -n "${pub}" ]] || continue - if grep -qF "${pub}" "${SOPS_CONFIG}"; then + if printf '%s\n' "${RECIPIENTS[@]}" | grep -qxF "${pub}"; then matched="${pub}" break fi @@ -164,14 +175,18 @@ done <<< "${PUBLIC_KEYS}" if [[ -z "${matched}" ]]; then die "this is a valid age key, but not one the secrets are encrypted to. - backup holds: $(printf '%s' "${PUBLIC_KEYS}" | tr '\n' ' ') - .sops.yaml wants: $(grep -oE 'age1[a-z0-9]+' "${SOPS_CONFIG}" | tr '\n' ' ') + backup holds: $(printf '%s' "${PUBLIC_KEYS}" | tr '\n' ' ') + the file is encrypted to: $(printf '%s ' "${RECIPIENTS[@]}") + .sops.yaml lists: $(grep -oE 'age1[a-z0-9]+' "${SOPS_CONFIG}" | tr '\n' ' ') + +A freshly generated keypair looks exactly like this. If the first two lines +agree and the third does not, .sops.yaml has drifted from the ciphertext. -A freshly generated keypair looks exactly like this. If you meant to add a new -recipient rather than restore an old one, that is 'sops updatekeys' — see -secrets/README.md." +If you meant to ADD this key as a recipient rather than restore an old one: + make secrets-add-recipient PUBKEY=$(printf '%s' "${PUBLIC_KEYS}" | head -n1)" fi -info "recipient matches .sops.yaml: ${matched}" +info "recipient ${matched}" +info "this file has ${#RECIPIENTS[@]} recipient(s); this run proves one of them" # --------------------------------------------------------------------------- # 4. The real test: decrypt with nothing but the backup available @@ -253,8 +268,35 @@ if [[ -n "${mode}" && "${mode: -2}" != "00" ]]; then warn "mode ${mode} — group or other can read this copy: chmod 600 $(printf '%q' "${BACKUP_ABS}")" fi +# --------------------------------------------------------------------------- +# 7. Record WHICH recipient was proved +# --------------------------------------------------------------------------- +# ADR-0024. run-scheduled.sh wraps this call and records one timestamp under +# homelab_job="verify-key-backup" whichever key was mounted, so with more than +# one recipient it answers "a key was proved" to the question "was THIS key +# proved" — and proving one silences the ninety-day deadline for all of them. +# key-recipients.sh keeps a series per recipient, which is what +# SecretsKeyBackupUnproven actually reads. +# +# Before the ok line and allowed to be fatal, deliberately. An unrecorded proof +# is the failure #77 is about: the job ran, nothing can see that it ran, and the +# gap is invisible until the day it matters. +"${REPO_ROOT}/scripts/key-recipients.sh" --record --stack "${STACK}" --proved "${matched}" + printf '\033[0;32mok\033[0m — %s decrypts secrets/%s.sops.yaml (%d/%d keys)\n' \ "$(basename "${BACKUP}")" "${STACK}" "${found}" "${#REQUIRED[@]}" printf ' Proven: this key, on its own, recovers every secret in the repo.\n' + +# The line the single-series metric could never say. Proving one recipient used +# to read as proving the backup; with more than one it proves exactly one of +# them, and the other copies are still only as good as the last time somebody +# went and got them out. +if ((${#RECIPIENTS[@]} > 1)); then + printf ' Not proven: the other %d recipient(s) of this file. Each needs its own\n' \ + "$((${#RECIPIENTS[@]} - 1))" + printf ' run against its own copy — SecretsKeyBackupUnproven names any that go\n' + printf ' ninety days without one.\n' +fi + printf ' Not proven: that where you keep it will still exist after a fire,\n' printf ' a theft, or a forgotten password. That part is your judgement.\n' diff --git a/secrets/README.md b/secrets/README.md index 55f2010..125bb53 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -17,10 +17,24 @@ decrypted only in memory at deploy time. > > On a host that holds none of those private keys, `make render` fails at > decryption rather than starting the stack with defaults. To bring a second -> host in, run `make secrets-init` there to generate its keypair, add its public -> half to `.sops.yaml` as an additional recipient, and re-key the file with -> `sops updatekeys secrets/observability.sops.yaml` from a host that can already -> decrypt it. `secrets-init` refuses to overwrite an existing encrypted file. +> host in, run `make secrets-init` there to generate its keypair, bring back the +> public half it prints, and add it here: +> +> ```bash +> make secrets-add-recipient PUBKEY=age1... +> ``` +> +> That writes the key into the `creation_rule` this stack's file already uses — +> resolved from the file's own recipients, so it cannot land in another stack's +> rule — and re-keys in the same run. `secrets-init` refuses to overwrite an +> existing encrypted file, and `secrets-add-recipient` refuses a key whose +> private half is already on this host. +> +> A recipient that is a *backup* rather than a host follows the same procedure +> and one extra rule: generate it where it will live, never here. +> [ADR-0024](../docs/adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +> and [`back-up-the-age-key.md`](../docs/runbooks/back-up-the-age-key.md) cover +> why the estate keeps more than one and how each is proved separately. ## Why encrypted-in-git rather than a gitignored `.env` diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index a0ec92c..6e00c30 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -174,23 +174,44 @@ groups: # a real off-host copy. That means a human has to mount removable media, # and no timer can do it. What a timer CAN do is notice that nobody has. # - # The threshold comes from the same declaration series as everything - # else, so the ninety days lives in scripts/install-timers.sh and nowhere - # else. Until the first verification there is no last_success series at - # all and ScheduledJobNeverRan is what speaks instead. + # ONE SERIES PER RECIPIENT, and this is the only rule in the file that + # is not keyed on homelab_job. ADR-0024 added a second age recipient, and + # the moment there were two, homelab_job_last_success_timestamp_seconds + # {homelab_job="verify-key-backup"} became a wrong answer: it is a single + # series that run-scheduled.sh writes whichever key was mounted, so + # proving one copy reset the ninety-day deadline for every copy and the + # other could rot behind a green alert. scripts/key-recipients.sh emits + # homelab_key_recipient_last_proof_timestamp_seconds per recipient, read + # out of the encrypted file's own metadata, and this fires per recipient. + # + # The threshold still comes from the same declaration series as + # everything else, so the ninety days lives in scripts/install-timers.sh + # and nowhere else. It is joined on(instance) rather than on(homelab_job) + # because the left side has no homelab_job label to join on — both series + # are written into the same textfile directory on the same host. + # group_left() for the reason ScheduledJobStale gives above: without it + # the result carries only the labels named in on(), and `recipient` — the + # entire point of this rule — would be silently dropped. + # + # A recipient that has never been verified is recorded as 0, not omitted, + # so it fires here with an absurd duration rather than being invisible. + # ScheduledJobNeverRan still speaks for the case before ANY verification + # has happened, because key-recipients.sh has not written its file yet + # and there are no per-recipient series to fire on. expr: | - (time() - homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup"}) - > on(homelab_job) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + (time() - homelab_key_recipient_last_proof_timestamp_seconds) + > on(instance) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} for: 1h labels: component: backup severity: warning category: correctness annotations: - summary: "The age key backup has not been verified in {{ $value | humanizeDuration }}" + summary: "No proven backup of age recipient {{ $labels.recipient }} in {{ $value | humanizeDuration }}" description: >- - Nothing else here proves the secrets are recoverable from anywhere - but this host — every other job runs on the same machine, with the - same key, against the same disk. Get the offline copy out and run - `make secrets-verify-backup KEY=`: + Nothing else here proves the {{ $labels.stack }} secrets are + recoverable from anywhere but this host — every other job runs on the + same machine, with the same key, against the same disk. Proving a + DIFFERENT recipient does not clear this one. Get that copy out and + run `make secrets-verify-backup KEY=`: docs/runbooks/back-up-the-age-key.md. diff --git a/stacks/observability/prometheus/tests/backup.test.yaml b/stacks/observability/prometheus/tests/backup.test.yaml index 0c2011d..db55774 100644 --- a/stacks/observability/prometheus/tests/backup.test.yaml +++ b/stacks/observability/prometheus/tests/backup.test.yaml @@ -89,24 +89,55 @@ tests: alertname: ScheduledJobStale exp_alerts: [] + # And SecretsKeyBackupUnproven does not fire off the JOB series either. + # It reads the per-recipient series and nothing else, so a host where + # verify-key-backup has run but key-recipients.sh has not written its file + # is covered by ScheduledJobNeverRan, not by a rule that would have to + # invent a recipient label it does not have. + - eval_time: 2h30m + alertname: SecretsKeyBackupUnproven + exp_alerts: [] + + # --- SecretsKeyBackupUnproven: one alert per recipient -------------------- + # + # The assertion ADR-0024 turns on. Two recipients, one of them verified + # 30 minutes ago and one never — the shape the estate is in the moment a + # second key is added. Exactly one alert, naming the unproven key. + # + # Under the single-series rule this replaced, the fresh proof of `age1aaa` + # was the only series there was, so this state was silent. That is the whole + # defect: proving one copy vouched for every copy. + - interval: 1m + input_series: + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="observability", + recipient="age1aaa", instance="prometheus"}' + values: "9000+0x180" + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="observability", + recipient="age1bbb", instance="prometheus"}' + values: "0+0x180" + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup", instance="prometheus"}' + values: "3600+0x180" + alert_rule_test: - eval_time: 2h30m alertname: SecretsKeyBackupUnproven exp_alerts: - exp_labels: alertname: SecretsKeyBackupUnproven - homelab_job: verify-key-backup + stack: observability + recipient: age1bbb instance: prometheus component: backup severity: warning category: correctness exp_annotations: - summary: "The age key backup has not been verified in 2h 30m 0s" + summary: "No proven backup of age recipient age1bbb in 2h 30m 0s" description: >- - Nothing else here proves the secrets are recoverable from - anywhere but this host — every other job runs on the same - machine, with the same key, against the same disk. Get the - offline copy out and run `make secrets-verify-backup - KEY=`: docs/runbooks/back-up-the-age-key.md. + Nothing else here proves the observability secrets are + recoverable from anywhere but this host — every other job runs + on the same machine, with the same key, against the same disk. + Proving a DIFFERENT recipient does not clear this one. Get that + copy out and run `make secrets-verify-backup KEY=`: + docs/runbooks/back-up-the-age-key.md. # --- ScheduledJobFailed: fires on non-zero, silent on zero --------------- - interval: 1m