diff --git a/.github/workflows/docs-drift-check.yml b/.github/workflows/docs-drift-check.yml index 50a318b9b2..87f929eec9 100644 --- a/.github/workflows/docs-drift-check.yml +++ b/.github/workflows/docs-drift-check.yml @@ -38,6 +38,22 @@ name: Docs Drift Check # run the tool on their own worktree. A reader who did that got a different list and # reported a correct row as a false positive, and the investigation of the non-existent # defect cost a full round. Same posture, one more thing the run knows and used not to say. +# +# THE RECALL DENOMINATOR (#13306, maintainer ruling 2026-08-31) — the OTHER half of the +# structural-invisibility paragraphs above. #11356/#11357 are about a page that IS already +# in this tool's scope and still goes unlisted (it restates a rule without naming what this +# tool anchors on). Separately, and wider: scripts/docs-audit/affected-docs.mjs's own header +# declares the CORPUS this whole check claims to cover — hand-written docs, excluding +# `content/docs/references/**` (generated, never hand-edited; excluding it is this check's +# constructive design, not an omission). Any recall figure for this check must use THAT +# corpus as its denominator; one computed against all edits under content/docs measures how +# often the docs generator ran in the sampling window, not this check — measured 2026-08-30, +# 31 of 46 ground-truth entries in the one figure ever computed that way were generated +# pages this check can never list (method and correction in #13306). This check stays +# advisory-only (see the top of this file) and the ceiling does not license widening the +# corpus to improve the number. Full statement in affected-docs.mjs's header — this +# paragraph only points at it so the two files compose instead of each stating their own +# half. on: pull_request: diff --git a/scripts/docs-audit/affected-docs.mjs b/scripts/docs-audit/affected-docs.mjs index 61e0f44c61..5abd9b27b6 100644 --- a/scripts/docs-audit/affected-docs.mjs +++ b/scripts/docs-audit/affected-docs.mjs @@ -13,6 +13,33 @@ // Scope: hand-written docs only = content/docs/**/*.mdx MINUS content/docs/references/** // (references are generated from packages/spec and handled by a separate regenerate pass). // +// THE ABOVE SCOPE IS THE RECALL DENOMINATOR (#13306, maintainer ruling 2026-08-31). Any +// recall figure for this tool — "of the docs pages that should have been listed, how many +// were" — MUST use the scope above (hand-written docs, generated pages excluded) as its +// denominator. Excluding `content/docs/references/**` is not a gap this tool happens to +// have: those pages are AUTO-GENERATED and nobody hand-edits them, so telling an author +// "you may have affected this page" would be WRONG ADVICE, not missing advice — the +// exclusion is constructive, by design, same as the Scope line states. +// +// ⇒ a recall ratio computed against a WIDER denominator — every edit under content/docs, +// generated pages included — is not measuring this tool at all. It is measuring how often +// the docs generator happened to run inside whatever window was sampled, because every one +// of those runs counts as a "miss" this tool could structurally never have avoided. This is +// exactly how the one such figure ever computed got it wrong: measured 2026-08-30 over a +// 91-commit window ending at `c4ecf0c49` (method, replay and the corrected re-derivation in +// #13306), 31 of its 46 ground-truth entries were `content/docs/references/**` pages this +// tool cannot list on any run, at any recall — the ratio moved by a factor of 2.7 just from +// widening the sampling window, which is the signature of a denominator not measuring the +// thing. +// +// ⛔ Maintainer ruling: this ceiling is honest, not a defect, and does not license widening +// the corpus to improve the number — that trades a real regression (prompting authors about +// pages they must never touch) for a paper gain. The tool's advisory-only posture (see +// .github/workflows/docs-drift-check.yml's own header) is unchanged. That file documents +// the OTHER half of "why a listed number can still miss something" — a page already IN this +// scope that goes unlisted because it restates a rule without naming what this tool anchors +// on. Read both; they compose into one picture, not two competing ones. +// // DERIVATION (#9192): a doc is "affected" when it NAMES SOMETHING THE CHANGE TOUCHED — // an ANCHOR — not when it merely mentions the changed package. //