Skip to content

skills(ai): delete the planned-eval stub and drop its ceiling row (#14296 item 2) - #14652

Merged
os-zhuang merged 2 commits into
mainfrom
claude/issue-14305-ai-eval-stub-removal
Sep 2, 2026
Merged

skills(ai): delete the planned-eval stub and drop its ceiling row (#14296 item 2)#14652
os-zhuang merged 2 commits into
mainfrom
claude/issue-14305-ai-eval-stub-removal

Conversation

@os-litant

Copy link
Copy Markdown
Collaborator

Fixes #14305

Round 2 of this card. Round 1 (PR #14463, merged) landed every RESTRUCTURE finding in skills/objectstack-ai/SKILL.md and deferred one row — AI-H-01 (with AI-H-02, which lives in the same file) — to the batch-1 ruling. That is the only deferral the seat's ACCEPT names, so this PR closes the card.

The ruling

#14296 item 2 = A, comment 5507419465 (09:27Z), quoted from the dispatch:

the five planned-eval stubs are deleted, their ceilings step down, real fixtures stay, no new eval packages

Same two-file shape as the merged precedents PR #14585 (query) and PR #14578 (automation).

Pre-deletion verification

The ruling protects real fixtures, so the directory was enumerated before anything was touched:

$ find skills/objectstack-ai/evals
skills/objectstack-ai/evals
skills/objectstack-ai/evals/README.md

One file, and it is the planned-eval stub, not a fixture: 44 lines whose Status section reads ⚠️ **Not yet implemented** — This is a placeholder for future development. Its "Structure" fence lists five eval files. None of the five exists anywhere in the tree:

$ for f in test-trigger-conditions.md test-surface-affinity.md \
           test-json-schema-parameters.md test-strict-unknown-keys.md \
           test-knowledge-source-filters.md; do ... find . -name "$f" ... ; done
test-trigger-conditions.md               matches=0
test-surface-affinity.md                 matches=0
test-json-schema-parameters.md           matches=0
test-strict-unknown-keys.md              matches=0
test-knowledge-source-filters.md         matches=0

Nothing reads the file. The whole tree holds exactly one reference to the path, and it is the ceiling row this PR drops:

$ git grep -n "objectstack-ai/evals"
scripts/check-skills-token-ratchet.mjs:440:  ['skills/objectstack-ai/evals/README.md', 315],

No gate, no script under scripts/ or packages/spec/scripts/, and no test reads evals/ for this package.

The diff — two files

file change lines tokens
skills/objectstack-ai/evals/README.md deleted −44 −315
scripts/check-skills-token-ratchet.mjs one ceiling row dropped −1

No other row changed, no ceiling raised, no prose edited, no file added, no other package touched.

Governed-surface size readings (skills/**)

Token counts use the ratchet's own convention, ceil(utf8 bytes / 4).

reading before after delta
the edited file, skills/objectstack-ai/evals/README.md 44 lines / 315 tok deleted −44 lines / −315 tok
the whole package, skills/objectstack-ai/** 506 lines / 6,610 tok 462 lines / 6,295 tok −44 lines / −315 tok (−4.8%)

The package total is SKILL.md (417 lines / 5,476 tok) plus the generated references/_index.md (45 lines / 819 tok), both byte-identical to origin/main.

Reverse verification — red, then green

Both readings are byte-exact from this branch; exit codes were captured before any pipe.

Leg 1 — file deleted, ceiling row still present. Expected direction: red, because the ratchet treats an unreadable input as a failure rather than a skip.

$ git rm skills/objectstack-ai/evals/README.md
$ node scripts/check-skills-token-ratchet.mjs ; echo "EXIT=$?"
✗ check-skills-token-ratchet: cannot read skills/objectstack-ai/evals/README.md — red, not a skip (#4690).
...
✗ check-skills-token-ratchet: 1 of 42 published bundle file(s) failed their check.
EXIT=1

Leg 2 — ceiling row dropped. Green:

$ node scripts/check-skills-token-ratchet.mjs ; echo "EXIT=$?"
✓ check-skills-token-ratchet: 30 authored bundle file(s) within their ceilings; 11 generator-owned file(s) measured, not ratcheted.
EXIT=0

The script's own suite. The ratchet script is edited by this PR, so its self-test is owed on top of the derived families:

$ node scripts/check-skills-token-ratchet.mjs --self-test ; echo "EXIT=$?"
✓ check-skills-token-ratchet self-test: 64 cases pass.
EXIT=0

git grep -l check-skills-token-ratchet over test files returns nothing — the built-in --self-test is the whole of that script's suite.

Scope note — one thing deliberately NOT edited

Dropping the row leaves its section header standing over an empty section:

  // the remaining skills' eval notes
]);

The header used to carry two rows; the i18n flight removed the other one earlier today, and this PR removes the last. Deleting the header is a prose edit, which this flight's scope explicitly excludes, so it is left exactly as it stands and flagged here instead of being tidied silently. It is inert — the map is keyed by path, and a comment over zero rows changes no verdict. If the seat wants it gone, it is a one-line follow-up.

Gates

Derived after the edit — and again after the merge below, at the final head. Both derivations return the same 24 commands. The final one:

$ node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands
dispatch-gates: gate list derived from the tree of 'objectstack-ai/objectstack' at commit ffdbdcd7
  --repo 'objectstack-ai/objectstack' checked against this checkout's 'origin' remote — it holds.
dispatch-gates: change set derived from git — 2 path(s) vs merge base 20b883918 of 'origin/main' and HEAD
  · scripts/check-skills-token-ratchet.mjs
  · skills/objectstack-ai/evals/README.md
dispatch-gates --commands: 24 command(s) — 18 pnpm, 6 direct node (22 matched by path, 2 by change KIND).

All 24 were run, plus check:skill-docs which the dispatch names explicitly — 25 commands, one acquisition of the shared verify lock (OS_VERIFY_LOCK_SLOT=issue-14305-r2). Every exit code was captured before any pipe.

Union re-run at head ffdbdcd7 — the final commit on this branch:

# gate exit its own verdict line
1 node scripts/check-ci-filter-parity.mjs 0 OK: all 130 declared cross-package glob(s) (92 unique) are covered by core or crosspkg, every crosspkg entry still covers one, and the test job's if: still names both filters.
2 node scripts/check-cross-package-test-inputs.mjs 0 OK: 25 package(s) read outside themselves, all declared, and turbo.json hashes every declared glob.
3 node scripts/check-shard-attestation.mjs 0 ✓ check-shard-attestation: 2 aggregate gate(s) count 3 declared leg(s) across 3 attesting job(s).
4 node scripts/check-skills-token-ratchet.mjs 0 ✓ check-skills-token-ratchet: 30 authored bundle file(s) within their ceilings; 11 generator-owned file(s) measured, not ratcheted.
5 node scripts/check-test-completeness.mjs 3 check-test-completeness: PREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named.
6 node scripts/pm/bare-root-worklist.mjs --self-test 0 OK self-test: 57 live row(s), 49 unreachable as spelled, 49 recorded verdict(s) — none stale, none missing, none contradicted. 23 record(s) carry a spelling and every one of 14 distinct …
7 pnpm --filter @objectstack/lint run check:doc-formula-expressions 3 check-doc-formula-expressions: PREREQUISITE NOT MET — the workspace package @objectstack/formula is not built
8 pnpm check:agent-test-spelling 0 ✓ check-agent-test-spelling: 0 violations — 424 file(s) · 5645 bare -- token(s) · 1380 launcher-rooted run(s) · 9 separator(s) JUDGED · 5 vitest-backed script name(s) derived from 81 ma…
9 pnpm check:bash32-floor 0 ✓ check-bash32-floor: 26 tracked shell file(s) under scripts/, .claude/hooks/, .githooks/** name no bash 4+ construct outside a comment, a guarded ${VAR:-} read, or a non-command posi…
10 pnpm check:cli-command-ids 0 ✓ check-cli-command-ids: 315 command-id literal(s) across 112 file(s) outside packages/cli all resolve to a real command path (73 ids derived; 3 declared fixture exemptions, 0 baselined v…
11 pnpm check:corpus-claim-drift 0 check-corpus-claim-drift: OK, no new claim sites beside a pinned spelling.
12 pnpm check:cross-package-test-inputs 0 OK: 25 package(s) read outside themselves, all declared, and turbo.json hashes every declared glob.
13 pnpm check:doc-authoring 0 ✓ doc authoring guard: sibling-package prose ids hold the baseline — 831 pinned site(s) across 231 file(s), 82825 string(s) read in 1124 parsed source(s), no growth, no burn-down unrecorded.
14 pnpm check:entry-guard 0 ✓ check:entry-guard: 199 scripts/ file(s) — every entry guard goes through invoked-as.mjs; 150 export bindings, 148 of them inert on import (2 known-unsafe, ⛔ SHRINK-ONLY).
15 pnpm check:parse-guard 0 ✓ check:parse-guard: 198 scripts/ file(s) — every TypeScript parse goes through ts-parse.mjs.
16 pnpm check:pm-dispatch-gates 0 ✓ dispatch-gates self-test: 1240 cases pass.
17 pnpm check:pm-governed-merges 0 ✓ check-governed-merges --self-test: 243 assertions (the unified governed predicate + near misses, subject→PR spellings, window parsing, the #12633 landing window — the QS-7 regression pi…
18 pnpm check:pnpm-filter-targets 0 ✓ check:pnpm-filter-targets: 142/181 --filter occurrence(s) across 32 file(s) resolve against 79 workspace package(s); 39 not judged (2 foreign, 17 interpolated, 20 path); 47 more in co…
19 pnpm check:ratchet-remedy-authority 0 OK check-ratchet-remedy-authority: 184 scripts swept (scripts/*.{mjs,mts}); 12 mark the expanding remedy ⛔ MAINTAINER-ONLY, 6 turn it down outright, 166 hand out no ratchet-expanding rem…
20 pnpm check:role-word 0 check-role-word: OK, no new occurrences of the reserved word.
21 pnpm check:skill-compatibility 0 ✓ check-skill-compatibility-version: 11 SKILL.md file(s) reconciled against 79 workspace packages
22 pnpm check:skill-frame-sync 0 ✓ check-skill-frame-sync: 4 copies of the decision frame are structurally isomorphic across 3 files
23 pnpm check:skill-identifier-liveness 0 check-skill-identifier-liveness OK — Leg 1: 465 citation(s) over 40 published file(s) checked against 93441 implementation word tokens (3 ledgered exemption(s)); Leg 2: 8 registered exhau…
24 pnpm check:watch-hint-literal 0 ✓ check-watch-hint-literal: 45 declaration(s) across 4 rostered name(s) -- ROOT_DIR_WATCH_HINTS 29, ROOT_FILE_WATCH_HINTS 9, ROOT_WATCH_HINTS 2, DECLARED_WATCH_HINTS 5 -- every one an arr…
25 pnpm --filter @objectstack/spec check:skill-docs 0 ✓ content/docs/ai/skills-reference.mdx

NOT MEASURED — 2 of 25

Both exit 3, both by design, and this flight touches neither subject. CI supplies both.

gate exit the prerequisite it names
node scripts/check-test-completeness.mjs 3 check-test-completeness: PREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named.
pnpm --filter @objectstack/lint run check:doc-formula-expressions 3 check-doc-formula-expressions: PREREQUISITE NOT MET — the workspace package @objectstack/formula is not built

An exit 3 is recorded as NOT MEASURED, never as a pass and never as a flake.

The two the dispatch asked to see proved unaffected

✓ check-skill-compatibility-version: 11 SKILL.md file(s) reconciled against 79 workspace packages
check-skill-identifier-liveness OK — Leg 1: 465 citation(s) over 40 published file(s) checked
  against 93441 implementation word tokens (3 ledgered exemption(s)); Leg 2: 8 registered
  exhaustive section(s), 0 ledgered gap(s).

And the generated catalog needs no regeneration — neither generated file lists evals/ for this package, so the deletion leaves them in sync:

$ pnpm --filter @objectstack/spec check:skill-docs
✓ skills/README.md
✓ content/docs/ai/skills-reference.mdx
✅ Skill docs in sync

Base

Branch cut from origin/main at ca48cf37. origin/main then moved to 20b88391, which touches .claude/skills/pm-dispatch/SKILL.md and scripts/pm/dispatch-gates.mjs — neither of this PR's two paths — so it was merged in rather than rebased, and the family was re-derived at the merge head and re-run whole. Against origin/main the branch is still exactly the two files:

$ git diff --stat origin/main...HEAD
 scripts/check-skills-token-ratchet.mjs |  1 -
 skills/objectstack-ai/evals/README.md  | 44 ----------------------------------
 2 files changed, 45 deletions(-)

Governed surface (skills/**), so this PR stays draft; review requests and the merge are not the dev's step.


Generated by Claude Code

#14296 item 2 = A (comment 5507419465): the planned-eval stubs are
deleted and their ceilings step down; real fixtures stay, and no new
eval packages are created. This is round 2 of #14305 -- round 1 (PR
#14463, merged) implemented every RESTRUCTURE finding in
skills/objectstack-ai/SKILL.md and deferred AI-H-01 / AI-H-02, the
evals/README.md planned-structure stub, to this ruling.

Deletes skills/objectstack-ai/evals/README.md (315 tokens, 44 lines --
a placeholder whose "Structure" fence lists five eval files that do not
exist anywhere in the repo) and drops its one ceiling row from
scripts/check-skills-token-ratchet.mjs. That row was the only reference
to the path in the tree, so nothing else reads the file. No other row
changes, no ceiling raised.

skills/objectstack-ai/evals/ held nothing but the stub, so the ruling's
"real fixtures stay" clause has no subject here.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LraLgQVGq8egUwfYZpbYt1
@os-zhuang
os-zhuang marked this pull request as ready for review September 2, 2026 22:48
@os-zhuang
os-zhuang enabled auto-merge September 2, 2026 22:49
@os-zhuang
os-zhuang added this pull request to the merge queue Sep 2, 2026
Merged via the queue into main with commit f681b92 Sep 2, 2026
36 checks passed
@os-zhuang
os-zhuang deleted the claude/issue-14305-ai-eval-stub-removal branch September 2, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/s skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

3 participants