From a876aeaf32c5a041566eb85e64a9fdfc36dc6537 Mon Sep 17 00:00:00 2001
From: Bartek Tofel <tofel.b@gmail.com>
Date: Fri, 4 Sep 2026 17:44:06 +0200
Subject: [PATCH 1/2] chore: add centralized docs

---
 grafana-alertcheck/docs/_category_.yaml       |  8 ++
 grafana-alertcheck/docs/advanced.md           | 41 +++++++++
 .../docs/how-alerts-are-evaluated.md          | 85 +++++++++++++++++
 grafana-alertcheck/docs/index.md              | 83 +++++++++++++++++
 .../docs/reference/_category_.yaml            |  8 ++
 grafana-alertcheck/docs/reference/cli.md      | 91 +++++++++++++++++++
 6 files changed, 316 insertions(+)
 create mode 100644 grafana-alertcheck/docs/_category_.yaml
 create mode 100644 grafana-alertcheck/docs/advanced.md
 create mode 100644 grafana-alertcheck/docs/how-alerts-are-evaluated.md
 create mode 100644 grafana-alertcheck/docs/index.md
 create mode 100644 grafana-alertcheck/docs/reference/_category_.yaml
 create mode 100644 grafana-alertcheck/docs/reference/cli.md

diff --git a/grafana-alertcheck/docs/_category_.yaml b/grafana-alertcheck/docs/_category_.yaml
new file mode 100644
index 000000000..3cbc431c4
--- /dev/null
+++ b/grafana-alertcheck/docs/_category_.yaml
@@ -0,0 +1,8 @@
+position: 1
+label: 'Grafana Alertcheck'
+collapsible: true
+collapsed: false
+link:
+  type: generated-index
+  slug: /platform-services/devex/cicd/grafana-alertcheck/index
+  description: 'CD quality gate for Grafana alerts: watch, classify, and gate releases.'
diff --git a/grafana-alertcheck/docs/advanced.md b/grafana-alertcheck/docs/advanced.md
new file mode 100644
index 000000000..073292cd6
--- /dev/null
+++ b/grafana-alertcheck/docs/advanced.md
@@ -0,0 +1,41 @@
+---
+id: grafana-alertcheck-advanced
+title: Check budget and scheduling
+sidebar_label: Budget and scheduling
+sidebar_position: 2
+description: Why grafana-alertcheck schedules per rule, how the request budget works, and why it never queries state history.
+---
+
+# Check budget and scheduling
+
+## Per-rule schedules, never a global cycle
+
+Each rule polls at its **own** cadence, `--poll-interval` (default: half the rule's own evaluation interval). There is deliberately no single global minimum-interval cycle.
+
+One rule at `intervalSeconds=10` beside twenty at `300` keeps a 5 s cadence for itself and 150 s for the other twenty — not a 5 s cycle for all of them, which would be a 60× request bloat at ~1.8 s per request and would fail to start on a reasonable fleet.
+
+The scheduler staggers each rule's initial next-due time across its cadence, and serves due rules **earliest-due-first**, so a tight rule never queues behind slack ones.
+
+## The check budget
+
+The gate records one observation of every rule up front and checks the schedule against those **measured** latencies (payload sizes vary ~230× across rules, so a fixed estimate is meaningless). It errors at start — before waiting — if any of three conditions hold:
+
+- **Utilization** — total request rate exceeds `--concurrency`.
+- **Per-rule** — one rule's request can't fit its own cadence.
+- **Burst bound** — the slowest request exceeds the fleet's tightest cadence, which can open a mid-run gap.
+
+The error names the three levers only: raise `--concurrency`, raise `--poll-interval`, or watch fewer alerts. It never prescribes a single interval.
+
+## Why the gate never queries state history
+
+Querying Grafana's alert state history after the fact fails closed *in the wrong direction* — it returns "pass" when the truth is unknown:
+
+- History stores **transitions**, not states. An alert firing through the whole window has its only record *before* the window.
+- The annotations API **does not serve Loki-backed history** at all.
+- An empty result is indistinguishable from a healthy one: no alert fired, the backend differs, retention removed data, the token lacked permission — all look identical.
+- There is **no coverage signal** — nothing proves the history is complete to time T.
+- Artifact transitions (`Paused`, `RuleDeleted`, `Updated`, `MissingSeries`) look like recoveries.
+
+Instead, `watch` records its own evidence live and the log becomes the source of truth. The trade-off: the gate can miss an episode shorter than a rule's poll interval, though `activeAt` still surfaces sub-interval onsets for instances still active at a poll.
+
+A corollary of recording fresh: there is no replay. Re-running a failed job is a new deploy with a new `from` and a new recording — never a re-classification of old evidence.
diff --git a/grafana-alertcheck/docs/how-alerts-are-evaluated.md b/grafana-alertcheck/docs/how-alerts-are-evaluated.md
new file mode 100644
index 000000000..7c9cb3b6a
--- /dev/null
+++ b/grafana-alertcheck/docs/how-alerts-are-evaluated.md
@@ -0,0 +1,85 @@
+---
+id: grafana-alertcheck-evaluation
+title: How alerts are evaluated
+sidebar_label: How alerts are evaluated
+sidebar_position: 1
+description: The verdict model, instance timelines, and coverage proof behind grafana-alertcheck.
+---
+
+# How alerts are evaluated
+
+Both `watch`+`check` (recorder mode) and `check` alone (single-step mode) converge on the same input: a flat list of polls. Everything below runs over that list; the mode only changes where the polls came from.
+
+## Instance states
+
+Grafana reports instance states in two vocabularies (`Alerting`/`Normal` at instance level, `firing`/`inactive` at rule level). The gate normalizes every instance to one canonical set:
+
+| Canonical | Meaning |
+| --------- | ------- |
+| `normal`  | Healthy |
+| `firing`  | The condition is true and `for` has elapsed |
+| `pending` | The condition is true, `for` has not elapsed |
+| `nodata`  | The query returned no series (synthetic instance) |
+| `error`   | The query failed (synthetic instance) |
+
+A rule's **rule-level** `state` and `health` are kept verbatim and only reported — they are never classified. The **instance** state is what the classifier reasons about.
+
+A "bad" instance is one whose canonical state is in `--states` (default `firing`). `pending` and `nodata` are excluded by default.
+
+## Verdict model
+
+For each instance the gate builds a timeline of bad spans over `[from, to]`, then takes the worst outcome across a rule's instances as the rule's outcome.
+
+| Outcome | Shape | Exit |
+| ------- | ----- | ---- |
+| `clean` | Good throughout, observed throughout | → 0 |
+| `newly_bad` | Entered a bad state **inside** the window | → 1 |
+| `persistently_bad` | Bad at `from`, still bad at `to` | → 1 |
+| `recovered` | Bad at `from`, cleared before `to`, stayed clear | → 0 |
+| `flapping` | Cleared, then became bad again | → 1 |
+| `skipped` | Paused **before** the window opened | reported, not observable |
+| `unobservable` | Coverage gap / sustained `health=error` / stale / absent | → 2 |
+
+`recovered` has **no deadline** — an alert that clears at minute 58 of a 60-minute window still passes. The total bad time is reported as `BadFor`; the removed deadline is replaced by that measured value rather than a derived limit.
+
+### Preexisting policy
+
+For an instance already bad when `from` opened, `--preexisting` decides:
+
+- `fail-unless-recovered` (default) — clears and stays clear → pass; never clears → fail.
+- `fail` — any preexisting instance fails, recovered or not.
+- `ignore` — preexisting instances are disregarded; only new episodes fail.
+
+## Cleared vs vanished
+
+When an instance leaves the bad set, the gate looks it up **in the same response**:
+
+- Present as `normal` → `cleared` (a real recovery).
+- Absent, or present as `normal (MissingSeries)` → `vanished` (a discontinuity, **not** a recovery).
+
+A vanished instance that was bad stays `persistently_bad`. A metric that stops being emitted is not evidence of health — this is deliberate and can surprise users whose fix is to remove a metric rather than drive it to a good value.
+
+## Coverage proof
+
+Before classifying, `check` must **prove** continuous coverage of `[from, to]` for each alert. Nine checks run; any failure makes the rule `unobservable`:
+
+1. **Sentinel** — a clean recorder stop, timestamped at or after `to + transitionGrace`. A recorder that died mid-window looks exactly like a coverage gap and is one.
+2. **`from` bounds** — `from` earlier than the recording start is unprovable.
+3. **Heartbeat gap** — any gap larger than `maxGap` (= 2 × poll cadence) inside the window. Data at both ends with a hole between is not enough.
+4. **`health=error`** — a contiguous run longer than `healthGrace` consumes coverage; a short blip is a note.
+5. **`health=nodata`** — a note, never fatal (unless `--nodata-is-unobservable`).
+6. **Liveness** — `grafana_now − lastEvaluation` must not exceed `evalStaleAfter`. This is an **absolute** check, never a "did it increase since the last poll" delta.
+7. **In-window pause** — a poll reporting `isPaused` mid-window is `unobservable` (the primary pause detector).
+8. **Rule absent** — an authoritative `2xx` with no matching rule.
+9. **`KeepLast`** — a note naming a stale-state blind spot.
+
+## Health: `error` vs `nodata`
+
+- `health=error` means the query **failed** — a malfunction. Sustained past `healthGrace`, it makes the rule `unobservable`.
+- `health=nodata` means the query **ran and returned no series** — indistinguishable from a quiet system. It is not fatal by default; most of a fleet runs `no_data_state: OK`.
+
+## The drain wait and `transitionGrace`
+
+A condition that arises just before `to` becomes `firing` only at the first evaluation after its `for` elapses. `transitionGrace` (derived from the watched rules' `for` values) extends the classification bound past `to` so such a surfacing condition is caught. After collection, a **drain wait** polls until each rule has evaluated through `to + transitionGrace` (bounded by `drainTimeout`); a rule that never does is `unobservable`.
+
+Run time = `(to − from) + transitionGrace + drainTimeout`. This is printed at start, and the grace is warned about when it exceeds a quarter of the window — the window may be too short for the alert's `for`.
diff --git a/grafana-alertcheck/docs/index.md b/grafana-alertcheck/docs/index.md
new file mode 100644
index 000000000..3e5bee11f
--- /dev/null
+++ b/grafana-alertcheck/docs/index.md
@@ -0,0 +1,83 @@
+---
+id: grafana-alertcheck-index
+title: Grafana Alertcheck
+sidebar_label: Overview
+sidebar_position: 0
+description: A CD quality gate that bookends a release with alert-state observation and answers whether any watched Grafana alert was bad during the release window.
+---
+
+# Grafana Alertcheck
+
+`grafana-alertcheck` is a CD quality gate for Grafana alerts. It bookends a release with two commands — `watch` (record) and `check` (classify) — and answers one question:
+
+> A release finished at time T. Was any of these Grafana alerts in a bad state during the next N minutes?
+
+The contract is `watch → your work → check`. Between the two you run whatever you want (deploy, tests, migration); the gate only observes, then classifies.
+
+It **fails closed**: if it cannot get an answer, it stops the release. It never passes an unproven window.
+
+## How it works, in one paragraph
+
+`watch` starts a background recorder that polls each named alert and appends snapshots to a JSONL log. Your work then emits two RFC3339 timestamps — `from` (when the change landed) and `to` (when the work ended). `check` proves continuous coverage of `[from, to]`, builds a state timeline per alert, classifies it, and exits `0`, `1`, or `2`.
+
+## Install
+
+```bash
+go install github.com/smartcontractkit/chainlink-testing-framework/grafana-alertcheck/cmd/grafana-alertcheck@latest
+```
+
+Connection details come from the environment — the token is env-only, never a flag:
+
+```bash
+export GRAFANA_URL=https://grafana.example.com
+export GRAFANA_TOKEN=…
+```
+
+Requires Grafana >= 13.0.0 and < 14.0.0. Outside that range the gate exits `2`.
+
+## Quickstart — recorder mode
+
+```bash
+grafana-alertcheck watch --out /tmp/run.jsonl --alerts alerts.txt
+./deploy.sh   # emits deployed_at=<RFC3339> when the rollout is stable
+./verify.sh   # emits finished_at=<RFC3339> when the work is done
+grafana-alertcheck check --in /tmp/run.jsonl --from "$deployed_at" --to "$finished_at"
+```
+
+`alerts.txt` holds one alert name per line. See [Naming alerts](./reference/cli#naming-alerts).
+
+`watch` returns only after the recorder has observed every named, non-paused alert once and reported ready — so auth, name-resolution, and parse failures surface **before** your deploy runs.
+
+## Quickstart — single-step mode
+
+Skip the recorder and observe the window inline, from inside `check` itself:
+
+```bash
+grafana-alertcheck check --alerts alerts.txt --to "$finished_at"
+```
+
+In single-step mode the window starts at `check`'s first observation; if you give no `--from`, the interval before that first observation is declared as a blind spot with a warning (not an error).
+
+## Exit codes
+
+| Code | Meaning |
+| ---- | ------- |
+| `0`  | Pass — no violations |
+| `1`  | Violations (including a paused-rule-only `--min-observed` shortfall) |
+| `2`  | The gate could not check — config, auth, resolution, a coverage gap, health/staleness, the drain limit, transport, … |
+
+An error is never a pass: `2` wins over any violation found alongside it.
+
+## Common surprises
+
+- A **paused rule fails by default** — even one someone else paused. Use `--allow-paused`.
+- A fix that **stops emitting a metric is not a recovery** — the instance vanishes, which is a discontinuity, not health.
+- The gate checks alert **state and health**, not notification delivery — a silenced alert that still fires fails.
+- `recovered` has **no deadline** — a bad-at-`from` alert that clears by `to` passes; set `--preexisting fail` to forbid it.
+- A **retry is a new deploy**, not a replay — re-running the job re-records against a new `from`.
+
+## More
+
+- [How alerts are evaluated](./how-alerts-are-evaluated) — the verdict model and coverage proof
+- [Check budget and scheduling](./advanced) — why the schedule and budget look the way they do, and why history isn't queried
+- [CLI reference](./reference/cli) — every subcommand and flag
diff --git a/grafana-alertcheck/docs/reference/_category_.yaml b/grafana-alertcheck/docs/reference/_category_.yaml
new file mode 100644
index 000000000..2e5d50946
--- /dev/null
+++ b/grafana-alertcheck/docs/reference/_category_.yaml
@@ -0,0 +1,8 @@
+position: 3
+label: Reference
+collapsible: true
+collapsed: false
+link:
+  type: generated-index
+  slug: /platform-services/devex/cicd/grafana-alertcheck/reference
+  description: 'CLI reference for grafana-alertcheck.'
diff --git a/grafana-alertcheck/docs/reference/cli.md b/grafana-alertcheck/docs/reference/cli.md
new file mode 100644
index 000000000..1530c508d
--- /dev/null
+++ b/grafana-alertcheck/docs/reference/cli.md
@@ -0,0 +1,91 @@
+---
+id: grafana-alertcheck-cli
+title: CLI reference
+sidebar_label: CLI reference
+sidebar_position: 0
+description: Full reference for the grafana-alertcheck CLI: watch, check, list, environment, naming, and output.
+---
+
+# CLI reference
+
+```
+grafana-alertcheck <list|watch|check>
+```
+
+Connection details are always from the environment: `GRAFANA_URL` and `GRAFANA_TOKEN`. The token is never a flag and never logged.
+
+## `list`
+
+Lists every rule from the ruler endpoint — kind, folder, group, title, uid. Useful to check auth and to find `uid:` names.
+
+```bash
+grafana-alertcheck list
+```
+
+## `watch` — record
+
+```bash
+grafana-alertcheck watch --out <file> [--pidfile F] [--daemon-log F] \
+  --alerts <file|-> [--folder F] [--poll-interval D] [--concurrency N] [--until RFC3339]
+```
+
+| Flag | Default | Meaning |
+| ---- | ------- | ------- |
+| `--out` | — | JSONL log path (required) |
+| `--pidfile` | `<out>.pid` | Where the recorder's pid is written |
+| `--daemon-log` | `<out>.daemon.log` | stdout/stderr sink for the detached recorder |
+| `--alerts` | — | File of alert names, one per line, or `-` for stdin (required) |
+| `--folder` | — | Default folder to scope unqualified names |
+| `--poll-interval` | half the rule's interval | Override every rule's cadence (never clamped) |
+| `--concurrency` | `1` | Max concurrent requests to Grafana |
+| `--until` | run until signalled | Optional hard stop |
+
+`watch` writes the header, observes every non-paused rule once, checks the budget, then detaches a background recorder and returns. Recording is **unfiltered** — there is no `--states` here, so the same log can be re-classified later under different `--states` without re-recording.
+
+## `check` — classify
+
+```bash
+grafana-alertcheck check [--in <file>] [--pidfile F] --from RFC3339 --to RFC3339 \
+  [--alerts ...] [--folder F] [--states ...] [--preexisting ...] [--min-observed N] \
+  [--allow-paused] [--nodata-is-unobservable] [--concurrency N] [--output json]
+```
+
+| Flag | Default | Meaning |
+| ---- | ------- | ------- |
+| `--in` | — | Log recorded by `watch`; empty selects single-step mode |
+| `--pidfile` | `<in>.pid` | Recorder to stop before reading `--in` |
+| `--from` | see below | Moment the deploy finished |
+| `--to` | — | End of the window (required) |
+| `--alerts` | — | Required **without** `--in`; refused **with** `--in` |
+| `--states` | `firing` | Comma-separated bad states: `firing,pending,nodata,error` |
+| `--preexisting` | `fail-unless-recovered` | `fail-unless-recovered` \| `fail` \| `ignore` |
+| `--min-observed` | every resolved rule | Minimum rules that must be observed |
+| `--allow-paused` | `false` | Don't count pre-window-paused rules against `--min-observed` |
+| `--nodata-is-unobservable` | `false` | Treat sustained `health=nodata` as unobservable |
+| `--concurrency` | `1` | Max concurrent requests |
+| `--output` | `table` | `json` also writes the machine-readable result to stdout |
+
+`--from` and `--to` are RFC3339 with an explicit offset and must come from your work — `from` from the deploy step, `to` from the step that finishes. In recorder mode an absent `--from` is a hard error; in single-step mode it falls back (with a warning) to the start of the step.
+
+## Naming alerts
+
+Alert names take one of four forms:
+
+| Form | Meaning |
+| ---- | ------- |
+| `HighErrorRate` | Title only, scoped by `--folder` |
+| `Platform/HighErrorRate` | Folder + title |
+| `Platform/api/HighErrorRate` | Folder + group + title (always unique) |
+| `uid:abc123` | Exact uid (present on both endpoints) |
+
+Datasource-managed and recording rules are refused with a specific error. A name matching multiple rules errors listing every candidate with the copyable `Folder/Group/Title` and its `uid:` form. A no-match errors with case-insensitive substring suggestions and points at `list`. Duplicate names that resolve to the same uid collapse to one (a note, not an error).
+
+## Output and exit codes
+
+The human table goes to **stderr**: `RESULTS` (one row per rule), `VIOLATIONS` (one per violation), and `THRESHOLDS` (each rule's `maxGap`/`healthGrace`/`evalStaleAfter` plus global `transitionGrace`/`drainTimeout` and the largest measured clock skew). `--output json` writes the result to stdout.
+
+| Code | Meaning |
+| ---- | ------- |
+| `0`  | Pass |
+| `1`  | Violations |
+| `2`  | Could not check — every library error, never a pass |

From 9fa719905a9c1dcf0fca569548ca29421b333372 Mon Sep 17 00:00:00 2001
From: Bartek Tofel <tofel.b@gmail.com>
Date: Mon, 7 Sep 2026 11:33:48 +0200
Subject: [PATCH 2/2] chore: further update docs

---
 grafana-alertcheck/README.md                  | 46 ++++++++-
 grafana-alertcheck/docs/architecture.md       | 68 +++++++++++++
 grafana-alertcheck/docs/index.md              |  4 +
 .../docs/reference/log-format.md              | 98 +++++++++++++++++++
 4 files changed, 213 insertions(+), 3 deletions(-)
 create mode 100644 grafana-alertcheck/docs/architecture.md
 create mode 100644 grafana-alertcheck/docs/reference/log-format.md

diff --git a/grafana-alertcheck/README.md b/grafana-alertcheck/README.md
index 43cc03fc4..96aa0e999 100644
--- a/grafana-alertcheck/README.md
+++ b/grafana-alertcheck/README.md
@@ -1,6 +1,46 @@
 # grafana-alertcheck
 
-A CD quality gate for Grafana alerts: bookend a release with `watch` (record) and `check` (classify) to
-answer whether any watched alert was in a bad state during the release window.
+A CD quality gate for Grafana alerts. It bookends a release with two commands — `watch` (record) and
+`check` (classify) — and answers whether any watched alert was in a bad state during the release window.
 
-Under construction.
+```
+watch  → your work → check
+```
+
+`watch` starts a background recorder that polls each named alert into a JSONL log. After the work emits a
+`from`/`to` pair, `check` proves continuous coverage of that window, classifies each alert's state
+timeline, and exits `0`, `1`, or `2`.
+
+It **fails closed**: if it cannot get an answer, it stops the release — never a pass on an unproven window.
+
+## Quickstart
+
+```bash
+export GRAFANA_URL=https://grafana.example.com
+export GRAFANA_TOKEN=…
+
+grafana-alertcheck watch --out /tmp/run.jsonl --alerts alerts.txt
+./deploy.sh   # emits deployed_at=<RFC3339>
+./verify.sh   # emits finished_at=<RFC3339>
+grafana-alertcheck check --in /tmp/run.jsonl --from "$deployed_at" --to "$finished_at"
+```
+
+Requires Grafana >= 13.0.0 and < 14.0.0. Connection details come from the environment only — the token is
+never a flag.
+
+## Documentation
+
+| Doc | Covers |
+| --- | ------ |
+| [`docs/index.md`](./docs/index.md) | Overview, quickstarts, exit codes, common surprises |
+| [`docs/how-alerts-are-evaluated.md`](./docs/how-alerts-are-evaluated.md) | Verdict model, coverage proof, health/liveness |
+| [`docs/advanced.md`](./docs/advanced.md) | Check budget, scheduling, why history isn't queried |
+| [`docs/architecture.md`](./docs/architecture.md) | Design invariants, the pure-function seam, recorder lifecycle |
+| [`docs/reference/cli.md`](./docs/reference/cli.md) | Full CLI reference — subcommands, flags, naming |
+| [`docs/reference/log-format.md`](./docs/reference/log-format.md) | The JSONL log schema, for debugging artifacts |
+
+## Build
+
+```bash
+go build ./... && go test ./...
+```
diff --git a/grafana-alertcheck/docs/architecture.md b/grafana-alertcheck/docs/architecture.md
new file mode 100644
index 000000000..ca493220e
--- /dev/null
+++ b/grafana-alertcheck/docs/architecture.md
@@ -0,0 +1,68 @@
+---
+id: grafana-alertcheck-architecture
+title: Architecture
+sidebar_label: Architecture
+sidebar_position: 3
+description: The design invariants, pure-function seam, and recorder lifecycle of grafana-alertcheck, for maintainers.
+---
+
+# Architecture
+
+This page documents the invariants and seams a maintainer must not break. It exists because most of them are the difference between a gate that fails closed and one that silently passes broken windows.
+
+## Fail-closed invariants
+
+The gate must stop the release if it cannot get an answer. Every rule below is a specific instance of that:
+
+- **An error is never a pass.** A pass is exactly `len(Violations) == 0 && err == nil`. Every error path leaves `err` non-nil, and the CLI maps that to exit `2` unconditionally.
+- **Inability beats violation.** Any `unobservable` rule is exit `2`, even alongside a real violation found first.
+- **Absent never means normal.** An instance that leaves the bad set is looked up in the *same* response: present as `normal` → cleared; absent (or `MissingSeries`) → vanished (a discontinuity, not a recovery).
+- **Staleness is absolute.** `grafana_now − lastEvaluation` is compared against a threshold, never "did it increase since the last poll" — a delta check reports stale on ~half the polls of a healthy rule.
+- **`grafana_now` is the response `Date` header.** Never the runner clock, in any comparison against a Grafana timestamp.
+- **No early exit.** `check` collects to `to + transitionGrace` before classifying once.
+- **No replay.** No run-id key, no artifact download, no state between attempts. A retry is a new deploy.
+
+## The pure-function seam
+
+All correctness lives in two phases written as **pure functions** over a flat list of polls — no HTTP, no files, no clock, no goroutines:
+
+```
+HTTP ──> Source ──> []StateRule ──> reduce ──> []Poll ──> proveCoverage ──> decide ──> Result
+                            │
+                       JSONL log ──> ReadLog ──┘
+```
+
+- `proveCoverage` (the nine coverage checks) and `decide` (the instance timelines and outcomes) are pure; tests drive them with `[]Poll` literals and a fake `Clock`, with no sleeping or fixture server.
+- `Check`/`Watch` are I/O shells: HTTP, signals, the pidfile, file reads, the countdown print. The only test doubles needed are the `Source` and `Clock` interfaces.
+- `Policy` is the narrowed view of `Config` that reaches the pure layer — classification knobs and the window, no URL and no token. The token must never cross that line, which is the cheapest guarantee it never lands in an error string or a result.
+
+## Strict parsing as the version guard
+
+Both API responses are parsed strictly: a missing or unparseable **required** field (`health`, `state`, `lastEvaluation`, `interval`) is an error, never a zero value. Optional keys (`alerts`, `totals`, `labels`, `keepFiringFor`) are absent-tolerant, and unknown keys are ignored — so Grafana can add fields without breaking the parser, but removing one fails loudly.
+
+This, plus the declared supported range (Grafana >= 13.0.0, < 14.0.0), is how a deprecation or schema change is caught instead of silently misread.
+
+## The recorder lifecycle
+
+`watch` detaches a background recorder so observation survives the step boundary:
+
+1. Parent resolves names, writes the header, observes every non-paused rule once, checks the budget.
+2. Parent re-execs itself as the child (`--daemon-child`) under a new session/process group, stdout/stderr to the daemon log.
+3. Child re-reads the header, reopens the log `O_APPEND`, takes the exclusive `flock`, and writes one readiness byte on `--ready-fd`.
+4. Parent writes the pidfile **after** the readiness report, then returns.
+
+Two authorities, only one of which is evidence:
+
+- The **pidfile** says a recording ever started (written only after ready, removed on failure). It can go stale — a pid gets reused.
+- The **flock** says a writer exists *now*. The kernel drops it on exit, so the lock is always authoritative.
+
+On a clean stop (SIGTERM/SIGINT/`--until`) the child finishes the in-flight write, appends the `stopped` sentinel, fsyncs, and exits. A hard error writes no sentinel — so a recorder that died reads exactly like a coverage gap, because it is one.
+
+`check` signals via the pidfile, waits for the **lock** to release (never the pid), and only then reads the log once. Reading while a writer can still append can only produce a shorter window than was recorded.
+
+## The log is the source of truth
+
+`watch` records raw evidence, so nothing trusts a state that could become unreachable. Two consequences a maintainer must preserve:
+
+- The **header is authoritative for recording facts** (the cadence actually used, the URL, the alert set); the ruler API is authoritative for **rule facts** (`for`, `intervalSeconds`, kind). `check` always re-resolves definitions fresh and never reconstructs them from the header — the header duplicates `for`/`interval` only so the uploaded artifact is self-describing.
+- The **cadence authority** is the header's `poll_every_seconds`, not the definitions. Re-deriving it would compare gaps recorded at an override cadence against default-cadence thresholds — fail-open in the faster-override direction.
diff --git a/grafana-alertcheck/docs/index.md b/grafana-alertcheck/docs/index.md
index 3e5bee11f..e8289a664 100644
--- a/grafana-alertcheck/docs/index.md
+++ b/grafana-alertcheck/docs/index.md
@@ -75,9 +75,13 @@ An error is never a pass: `2` wins over any violation found alongside it.
 - The gate checks alert **state and health**, not notification delivery — a silenced alert that still fires fails.
 - `recovered` has **no deadline** — a bad-at-`from` alert that clears by `to` passes; set `--preexisting fail` to forbid it.
 - A **retry is a new deploy**, not a replay — re-running the job re-records against a new `from`.
+- `watch` and `check` must run in **one job, one runner, one filesystem** — nothing persists across jobs or attempts.
+- The gate **never exits early** — a violation at minute 2 still holds the runner to `to + transitionGrace + drainTimeout`; size the job timeout to the planned run time the gate prints at start.
 
 ## More
 
 - [How alerts are evaluated](./how-alerts-are-evaluated) — the verdict model and coverage proof
 - [Check budget and scheduling](./advanced) — why the schedule and budget look the way they do, and why history isn't queried
+- [Architecture](./architecture) — design invariants and the recorder lifecycle, for maintainers
 - [CLI reference](./reference/cli) — every subcommand and flag
+- [Log format](./reference/log-format) — the JSONL log schema, for debugging artifacts
diff --git a/grafana-alertcheck/docs/reference/log-format.md b/grafana-alertcheck/docs/reference/log-format.md
new file mode 100644
index 000000000..66e071770
--- /dev/null
+++ b/grafana-alertcheck/docs/reference/log-format.md
@@ -0,0 +1,98 @@
+---
+id: grafana-alertcheck-log-format
+title: Log format
+sidebar_label: Log format
+sidebar_position: 1
+description: The JSONL log schema written by watch and read by check, for debugging the forensic artifact.
+---
+
+# Log format
+
+`watch` records evidence to a JSONL log — one JSON object per line. A poll record *is* the heartbeat; there is no separate heartbeat type.
+
+## Record types
+
+Exactly three:
+
+| `type` | Meaning |
+| ------ | ------- |
+| `header` | Line 1 — identity and the alert set |
+| `poll` | One reduced observation of one rule |
+| `stopped` | The sentinel, written on a clean stop only |
+
+The header must be line 1, appear once, and carry `schema_version` `1` (any other value is a read error). Any unparseable line — including the last, or one after the sentinel — makes the log unreadable: a truncated log is evidence the recorder was killed, and must not pass.
+
+## Header
+
+```json
+{
+  "type": "header",
+  "schema_version": 1,
+  "url": "https://grafana.example.com",
+  "grafana_version": "13.1.0",
+  "started_at": "2026-09-07T10:00:00Z",
+  "rules": [
+    {
+      "uid": "rule0000001",
+      "title": "HighErrorRate",
+      "folder": "Platform",
+      "group": "api",
+      "for_seconds": 300,
+      "interval_seconds": 60,
+      "is_paused": false,
+      "no_data_state": "OK",
+      "exec_err_state": "OK",
+      "poll_every_seconds": 30
+    }
+  ]
+}
+```
+
+- `url` and `rules` are the log's identity — `check` validates them against the current environment and a fresh ruler read.
+- `is_paused` records the pause state at record start (the moment `skipped` means).
+- `poll_every_seconds` is the cadence the recording **actually used** (after any `--poll-interval` override). `check` derives `maxGap` from it, never from `interval_seconds`.
+- `for_seconds`, `interval_seconds`, `no_data_state`, `exec_err_state` are forensic only — `check` re-resolves definitions and never reads them back.
+
+## Poll
+
+```json
+{
+  "type": "poll",
+  "rule_uid": "rule0000001",
+  "grafana_now": "2026-09-07T10:00:30Z",
+  "skew_ms": 20,
+  "skew_bound_ms": 40,
+  "latency_ms": 123,
+  "found": true,
+  "state": "inactive",
+  "health": "ok",
+  "last_evaluation": "2026-09-07T10:00:28Z",
+  "is_paused": false,
+  "histogram": { "alerting": 0, "normal": 2004 },
+  "reasons": { "NoData": 1091 },
+  "abnormal": [ { "labels": { "env": "prod" }, "state": "firing", "active_at": "2026-09-07T09:50:00Z", "value": "1.5" } ],
+  "cleared": [ "env=prod\u0001..." ],
+  "vanished": []
+}
+```
+
+Field notes:
+
+- `grafana_now` is the response's `Date` header — never the runner clock.
+- `skew_ms`/`skew_bound_ms` are the per-poll clock-skew estimate and its uncertainty (RTT/2), in milliseconds for compactness only.
+- `found: false` is an authoritative `2xx` in which this rule was absent — a transport failure is retried and never becomes a poll.
+- `state`, `health`, `last_error` are raw rule-level strings, reporting-only.
+- `histogram` is a verbatim copy of the response `totals`; written, never analysed.
+- `reasons` counts non-empty instance reasons (`NoData`, `Error`, `KeepLast`, …); composite states stay visible only here.
+- `abnormal` holds only instances whose **canonical** state is not `normal`.
+- `cleared`/`vanished` are instance keys that left the bad set, resolved against the same response: `cleared` = a real recovery; `vanished` = a discontinuity, never a recovery.
+
+Instance keys are a sorted `k=v\n` join of labels, so they correlate across polls without hashing.
+
+## Stopped
+
+```json
+{ "type": "stopped", "at": "2026-09-07T10:10:30Z" }
+```
+
+`at` is the recorder's own stop time. `check` compares it against `to + transitionGrace`; absent or earlier is `unobservable` — never a pass.