From 019236bed193d6ab64add5a9644a9f405cc4f6a7 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 02:50:22 +0000 Subject: [PATCH] docs(docs-audit): state the recall denominator in both headers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #13306 (maintainer ruling 2026-08-31): a recall figure for affected-docs.mjs must use the corpus it claims to cover — hand-written docs, excluding content/docs/references/** — as its denominator. Excluding generated pages is constructive design, not an omission: they are never hand-edited, so prompting an author about one is wrong advice. The one figure ever computed against a wider denominator (all content/docs edits) was measuring how often the docs generator ran in the sampling window, not this tool's recall (31 of its 46 ground-truth entries were generated pages the tool can never list). Adds that statement to affected-docs.mjs's own header (the authoritative half) and a short cross-reference in docs-drift-check.yml's header (which already documents the separate, page-level structural-invisibility class from #11356/#11357) so the two halves compose instead of each stating their own thing. No behavior change; no new recall figure computed. --- .github/workflows/docs-drift-check.yml | 16 +++++++++++++++ scripts/docs-audit/affected-docs.mjs | 27 ++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) diff --git a/.github/workflows/docs-drift-check.yml b/.github/workflows/docs-drift-check.yml index 50a318b9b2..87f929eec9 100644 --- a/.github/workflows/docs-drift-check.yml +++ b/.github/workflows/docs-drift-check.yml @@ -38,6 +38,22 @@ name: Docs Drift Check # run the tool on their own worktree. A reader who did that got a different list and # reported a correct row as a false positive, and the investigation of the non-existent # defect cost a full round. Same posture, one more thing the run knows and used not to say. +# +# THE RECALL DENOMINATOR (#13306, maintainer ruling 2026-08-31) — the OTHER half of the +# structural-invisibility paragraphs above. #11356/#11357 are about a page that IS already +# in this tool's scope and still goes unlisted (it restates a rule without naming what this +# tool anchors on). Separately, and wider: scripts/docs-audit/affected-docs.mjs's own header +# declares the CORPUS this whole check claims to cover — hand-written docs, excluding +# `content/docs/references/**` (generated, never hand-edited; excluding it is this check's +# constructive design, not an omission). Any recall figure for this check must use THAT +# corpus as its denominator; one computed against all edits under content/docs measures how +# often the docs generator ran in the sampling window, not this check — measured 2026-08-30, +# 31 of 46 ground-truth entries in the one figure ever computed that way were generated +# pages this check can never list (method and correction in #13306). This check stays +# advisory-only (see the top of this file) and the ceiling does not license widening the +# corpus to improve the number. Full statement in affected-docs.mjs's header — this +# paragraph only points at it so the two files compose instead of each stating their own +# half. on: pull_request: diff --git a/scripts/docs-audit/affected-docs.mjs b/scripts/docs-audit/affected-docs.mjs index 61e0f44c61..5abd9b27b6 100644 --- a/scripts/docs-audit/affected-docs.mjs +++ b/scripts/docs-audit/affected-docs.mjs @@ -13,6 +13,33 @@ // Scope: hand-written docs only = content/docs/**/*.mdx MINUS content/docs/references/** // (references are generated from packages/spec and handled by a separate regenerate pass). // +// THE ABOVE SCOPE IS THE RECALL DENOMINATOR (#13306, maintainer ruling 2026-08-31). Any +// recall figure for this tool — "of the docs pages that should have been listed, how many +// were" — MUST use the scope above (hand-written docs, generated pages excluded) as its +// denominator. Excluding `content/docs/references/**` is not a gap this tool happens to +// have: those pages are AUTO-GENERATED and nobody hand-edits them, so telling an author +// "you may have affected this page" would be WRONG ADVICE, not missing advice — the +// exclusion is constructive, by design, same as the Scope line states. +// +// ⇒ a recall ratio computed against a WIDER denominator — every edit under content/docs, +// generated pages included — is not measuring this tool at all. It is measuring how often +// the docs generator happened to run inside whatever window was sampled, because every one +// of those runs counts as a "miss" this tool could structurally never have avoided. This is +// exactly how the one such figure ever computed got it wrong: measured 2026-08-30 over a +// 91-commit window ending at `c4ecf0c49` (method, replay and the corrected re-derivation in +// #13306), 31 of its 46 ground-truth entries were `content/docs/references/**` pages this +// tool cannot list on any run, at any recall — the ratio moved by a factor of 2.7 just from +// widening the sampling window, which is the signature of a denominator not measuring the +// thing. +// +// ⛔ Maintainer ruling: this ceiling is honest, not a defect, and does not license widening +// the corpus to improve the number — that trades a real regression (prompting authors about +// pages they must never touch) for a paper gain. The tool's advisory-only posture (see +// .github/workflows/docs-drift-check.yml's own header) is unchanged. That file documents +// the OTHER half of "why a listed number can still miss something" — a page already IN this +// scope that goes unlisted because it restates a rule without naming what this tool anchors +// on. Read both; they compose into one picture, not two competing ones. +// // DERIVATION (#9192): a doc is "affected" when it NAMES SOMETHING THE CHANGE TOUCHED — // an ANCHOR — not when it merely mentions the changed package. //