Skip to content

[finding] Survey: which other scripts/* --self-test modes decide success with no assertion floor, or exit 0 via return selfTest() before their verdict #13489

Description

@claude

Filed by the dev on #13173 (PR #13487), session session_01Pk26oZ12t5N1hwGW1m1MgC.
Filed unassigned and ungraded — this is a survey request, not a measured defect.

Why

#13173 recorded, under its own "Not established" heading, that nobody had checked
whether other --self-test implementations in scripts/ share the shape it
measured. That survey is still not done. #13173 itself is now closed out by
PR #13487, so the question would otherwise leave with it.

The shape to survey

Two distinct holes were measured in scripts/check-doc-authoring.mjs while
fixing #13173. A survey wants both, because the second is invisible to a check
for the first:

  1. No assertion floor. Success decided as failures.length === 0, so a
    battery that never ran is indistinguishable from a battery that passed. An
    early return at the top of one battery function left it entirely unrun and
    the script still exited 0, printing its full success line.
  2. main() doing return selfTest(). A return that leaves the self-test
    before its verdict prints nothing at all and still exits 0. A gate can have
    a perfectly good assertion floor and still be defeated this way, because the
    floor never runs either.

The discriminator, cheaply

For each scripts/check-*.{mjs,mts} with a --self-test mode:

grep -n "failures.length === 0\|return selfTest()" scripts/check-*.mjs

⚠️ A verdict line that already prints a case count (872 cases pass and the
like) is evidence but not proof#13173 warns that a merely non-zero count
still passes a battery dropping from 40 cases to 3, and a pinned TOTAL rots as
soon as a sibling battery grows.

The repair shape that worked, for whichever instances the survey finds

PR #13487 pinned the registered names, not a number: a frozen registry of
battery names each with its own case floor, every battery opening with
battery('...'), every assertion attributed to the battery most recently
opened, and a refusal when the opened set differs from the declared set. A set
difference names which battery stopped running; a count says only that
something did. The registry's own size is pinned too, since deleting an entry
silences its floor as effectively as zeroing it. Five ablations are recorded on
that PR.

Related


Generated by Claude Code

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions