Skip to content

docs(docs-audit): state the recall denominator in both headers - #13600

Merged
os-project-manager merged 1 commit into
mainfrom
claude/issue-13306-recall-denominator-header
Aug 31, 2026
Merged

docs(docs-audit): state the recall denominator in both headers#13600
os-project-manager merged 1 commit into
mainfrom
claude/issue-13306-recall-denominator-header

Conversation

@os-project-manager

Copy link
Copy Markdown
Collaborator

Fixes #13306

What

Writes the recall-denominator definition into both places a reader would look for it, per the maintainer's 2026-08-31 ruling in #13306:

No behavior change. No new recall figure is computed by this PR — the one figure cited is the already-measured, dated reading from #13306's own investigation (31 of 46 ground-truth entries were content/docs/references/** pages), used only as evidence for why the denominator must be the tool's declared corpus, not as a live metric.

Why (the substance, not just "excludes generated pages")

affected-docs.mjs excludes content/docs/references/** by construction — those pages are auto-generated and never hand-edited, so telling an author "you may have affected this page" would be wrong advice, not missing advice. Therefore any recall ratio computed with all content/docs edits as the denominator is not measuring this tool at all — it measures how often the docs generator happened to run inside the sampling window (#13306 measured this moving by a factor of 2.7 just from widening the sample window: 17.4% → 6.5%, same tool, same method). The denominator has to be the corpus the tool claims to cover.

docs-drift-check.yml's header already records a different invisibility class (a page already in scope that still goes unlisted because it restates a rule without naming what the tool anchors on — #11356/#11357) and already states the check is advisory-only. This PR adds the other half — the denominator statement — and points the two files at each other so they compose instead of each stating their own thing, per the dispatch's Zone 3 suggestion.

Fences respected

Zone 2 (PM mechanism hypotheses) — results

  • H1 confirmed: the two headers were adjacent-but-non-composing. affected-docs.mjs already had a Scope: line (the corpus) but never framed it as a recall denominator, and docs-drift-check.yml documented structural invisibility without ever mentioning a denominator at all. Both are read before writing, and both are now cross-referenced (see the docs-drift-check.yml addition) rather than restated independently in full.
  • H2 confirmed: no executable assertion pins either header's text. affected-docs.mjs --self-test, check-audit-scope.mjs --self-test/run, check-affected-docs.mjs, and check-drift-comment.mjs all pin behavior (the corpus enumeration, the anchor derivation, the comment's rendered wording for what the run could/couldn't see) — none of them read or assert on this header comment block. So the denominator is documentation-only right now; it is not mechanically derived from a single source of truth, and this wording is the thing that must not drift.

Tests

Comment-only change to two files — no production behavior touched, so no build/test face is owed beyond confirming the edit didn't disturb anything the surrounding self-tests pin. All run in the worktree after pnpm install:

  • node scripts/check-nul-bytes.mjs on both files — clean; grep -naP '[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]' on both — no hits.
  • python3 -c "import yaml; yaml.safe_load(open('.github/workflows/docs-drift-check.yml'))" — parses.
  • node scripts/docs-audit/affected-docs.mjs --self-test✓ affected-docs self-test: 487 cases pass.
  • node scripts/docs-audit/check-audit-scope.mjs --self-test && node scripts/docs-audit/check-audit-scope.mjs → both green (189 hand-written docs, 9 release-owned pages).
  • node scripts/docs-audit/check-affected-docs.mjs → exit 0 (the workflow's own self-test step).
  • node scripts/docs-audit/check-drift-comment.mjs✓ check-drift-comment: 56 cases pass across 5 fixture diff(s).
  • Full derived gate family via node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack against the actual 2-file diff (not the dispatch prompt's list) — 29 matched families + 2 convention-triggered ones for editing a gate script. All run, all green (exit codes captured before any pipe): check:agent-test-spelling, check:bash32-floor, check:cli-command-ids, check:cross-package-test-inputs, check:docs-audit-scope, check:entry-guard, check:node-version, check:parse-guard, check:pm-governed-merges, check:pnpm-acquisition, check:pnpm-filter-targets, check:required-contexts, check:shard-attestation, check:stall-guard-budget, check:watch-hint-literal, check:workflow-status-functions, check-aggregator-roster.mjs, check-ci-filter-parity.mjs, check-whole-set-label-write.mjs, pm/ci-failure.mjs --self-test, pm/bare-root-worklist.mjs --self-test, check:pm-dispatch-gates.
  • No changeset: this diff edits comments only, in scripts/docs-audit/** and a workflow header, publishes nothing from any package. skip-changeset label applied.

PR relationship

Fixes #13306 — this PR is the dispatch's own closing condition: "the deliverable is the two headers' wording." #13471 (the SCREAMING_SNAKE literal-anchor gap) is independent and intentionally not touched here, per the ruling.

Generated by Claude Code


Generated by Claude Code

#13306 (maintainer ruling 2026-08-31): a recall figure for affected-docs.mjs
must use the corpus it claims to cover — hand-written docs, excluding
content/docs/references/** — as its denominator. Excluding generated pages
is constructive design, not an omission: they are never hand-edited, so
prompting an author about one is wrong advice. The one figure ever computed
against a wider denominator (all content/docs edits) was measuring how often
the docs generator ran in the sampling window, not this tool's recall (31 of
its 46 ground-truth entries were generated pages the tool can never list).

Adds that statement to affected-docs.mjs's own header (the authoritative
half) and a short cross-reference in docs-drift-check.yml's header (which
already documents the separate, page-level structural-invisibility class
from #11356/#11357) so the two halves compose instead of each stating their
own thing. No behavior change; no new recall figure computed.
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 3d0897459577835a78fc96a8f718eb3b139708a3packageMentionDocs.

@github-actions github-actions Bot added the ci/cd label Aug 31, 2026
@os-project-manager os-project-manager added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 31, 2026 — with Claude
@os-project-manager
os-project-manager marked this pull request as ready for review August 31, 2026 03:04
@os-project-manager
os-project-manager added this pull request to the merge queue Aug 31, 2026
Merged via the queue into main with commit e9094ae Aug 31, 2026
35 checks passed
@os-project-manager
os-project-manager deleted the claude/issue-13306-recall-denominator-header branch August 31, 2026 03:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/s skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants