Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/docs-drift-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,22 @@ name: Docs Drift Check
# run the tool on their own worktree. A reader who did that got a different list and
# reported a correct row as a false positive, and the investigation of the non-existent
# defect cost a full round. Same posture, one more thing the run knows and used not to say.
#
# THE RECALL DENOMINATOR (#13306, maintainer ruling 2026-08-31) — the OTHER half of the
# structural-invisibility paragraphs above. #11356/#11357 are about a page that IS already
# in this tool's scope and still goes unlisted (it restates a rule without naming what this
# tool anchors on). Separately, and wider: scripts/docs-audit/affected-docs.mjs's own header
# declares the CORPUS this whole check claims to cover — hand-written docs, excluding
# `content/docs/references/**` (generated, never hand-edited; excluding it is this check's
# constructive design, not an omission). Any recall figure for this check must use THAT
# corpus as its denominator; one computed against all edits under content/docs measures how
# often the docs generator ran in the sampling window, not this check — measured 2026-08-30,
# 31 of 46 ground-truth entries in the one figure ever computed that way were generated
# pages this check can never list (method and correction in #13306). This check stays
# advisory-only (see the top of this file) and the ceiling does not license widening the
# corpus to improve the number. Full statement in affected-docs.mjs's header — this
# paragraph only points at it so the two files compose instead of each stating their own
# half.

on:
pull_request:
Expand Down
27 changes: 27 additions & 0 deletions scripts/docs-audit/affected-docs.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,33 @@
// Scope: hand-written docs only = content/docs/**/*.mdx MINUS content/docs/references/**
// (references are generated from packages/spec and handled by a separate regenerate pass).
//
// THE ABOVE SCOPE IS THE RECALL DENOMINATOR (#13306, maintainer ruling 2026-08-31). Any
// recall figure for this tool — "of the docs pages that should have been listed, how many
// were" — MUST use the scope above (hand-written docs, generated pages excluded) as its
// denominator. Excluding `content/docs/references/**` is not a gap this tool happens to
// have: those pages are AUTO-GENERATED and nobody hand-edits them, so telling an author
// "you may have affected this page" would be WRONG ADVICE, not missing advice — the
// exclusion is constructive, by design, same as the Scope line states.
//
// ⇒ a recall ratio computed against a WIDER denominator — every edit under content/docs,
// generated pages included — is not measuring this tool at all. It is measuring how often
// the docs generator happened to run inside whatever window was sampled, because every one
// of those runs counts as a "miss" this tool could structurally never have avoided. This is
// exactly how the one such figure ever computed got it wrong: measured 2026-08-30 over a
// 91-commit window ending at `c4ecf0c49` (method, replay and the corrected re-derivation in
// #13306), 31 of its 46 ground-truth entries were `content/docs/references/**` pages this
// tool cannot list on any run, at any recall — the ratio moved by a factor of 2.7 just from
// widening the sampling window, which is the signature of a denominator not measuring the
// thing.
//
// ⛔ Maintainer ruling: this ceiling is honest, not a defect, and does not license widening
// the corpus to improve the number — that trades a real regression (prompting authors about
// pages they must never touch) for a paper gain. The tool's advisory-only posture (see
// .github/workflows/docs-drift-check.yml's own header) is unchanged. That file documents
// the OTHER half of "why a listed number can still miss something" — a page already IN this
// scope that goes unlisted because it restates a rule without naming what this tool anchors
// on. Read both; they compose into one picture, not two competing ones.
//
// DERIVATION (#9192): a doc is "affected" when it NAMES SOMETHING THE CHANGE TOUCHED —
// an ANCHOR — not when it merely mentions the changed package.
//
Expand Down
Loading