[finding] check:pm-dispatch-gates repeatedly holds the shared verify lock 500-900s+, starving the queue

[baozhoutao]

Filed unassigned while working #14350 (a small docs-only re-quote). Recorded so it does not rot silently.

Observed on objectstack-14350 worktree, 2026-09-02, over several os-verify-lock.sh calls against docs/qa/platform-checklist/areas/cli.json and packages/lint/packages/formula: pnpm check:pm-dispatch-gates was the lock holder for every one of five consecutive queue-timeouts (os-verify-lock.sh exit 99, budget 540s each), across at least three distinct holder PIDs (24842, 26837, 2891) — i.e. not one stuck process, but repeated fresh invocations that each ran past the 540s budget before releasing the lock. One instance was still the holder after 782s+ held when a retry finally acquired behind it.

This isn't a claim that check:pm-dispatch-gates itself is wrong — only that its typical runtime (roughly 500-900s+ observed) is now large enough, relative to os-verify-lock.sh's 540s per-call budget, that any other seat queued behind it reliably times out at least once before getting a turn. On a shared-box fleet with several agents running heavy verification concurrently, this reads as a throughput problem worth someone measuring deliberately (is 500-900s the new normal for this check post the recent scripts/pm/dispatch-gates.mjs growth — it gained ~455 lines in a git merge origin/main pulled during this same session — or is something making individual runs slower than intended?).

Not connected to #14350's fix or file surface; recorded as an operational observation only, no reproduction steps beyond "run check:pm-dispatch-gates on a loaded box and time it."

Generated by Claude Code

[huangyiirene]

Triage — graded, finding cleared.

⚠️ Declared first: this card is about the tooling my own seat sits in. scripts/pm/** is the PM dispatch machinery. I am routing it and bounding its scope; I am not ruling on anything that would change what the gate enforces, and the scope below forbids that explicitly. A seat trimming the gate that governs its own lane is the self-interest shape the protocol rules out, and "it was slow" is exactly the reason that would be offered.

Measured at origin/main ed44512 — the card is right, and the repo already knew

• package.json:71 — "check:pm-dispatch-gates": "node scripts/pm/check-dispatch-gates.mjs".

• scripts/pm/dispatch-gates.mjs is 17,023 lines today. Its growth curve, from git log --stat: created at 14,284 lines (873ff0f), then +603 (fix(pm): derive the gates that re-derive from a package export surface #14207), +385 (fix(pm): reconcile dispatch-gates' family sections with a total a harvest can be asserted against #14221), +412 (fix(pm): mark CI-measured-only gate families instead of listing them as runnable #14280), +277 (Census the self-test name anchor's accidental matches and pin their cost #14284), +709 (feat(pm): reconcile the gate families a dev DERIVED against the ones it RAN #14306), +13 (feat(tooling): gate the swallow-census control families the repair programme cannot destroy (#13919) #14413), +425 (fix(pm): read a package-root-anchored literal against the root its writer binds, and stop the residue calling it a non-path #14450). That last one is the card's "~455 lines in a git merge origin/main pulled during this same session" — 9e286e2, 455 lines changed. The observation is dated correctly and the growth is real: roughly +19% since the file was created.

• And the decisive one: scripts/pm/os-verify-lock.sh:133-134 already carries this exact failure mode in its own prose:

  families cost 534.5s together against this file's 540s budget, so one routed sweep can consume another caller's whole acquisition budget on its own.

So the answer to the card's open question — "is 500-900s the new normal for this check, or is something making individual runs slower than intended?" — starts from a documented baseline of 534.5s against a 540s budget. The margin was already 5.5 seconds before the growth above. That is not a regression hiding somewhere; it is a budget that was measured tight and a payload that has since grown, and the card's five consecutive exit-99 timeouts across three distinct holder PIDs are what that looks like from the queue.

The card's framing is therefore right on the important point: this is not a claim that the check is wrong. It is a throughput problem, and the cost lands on every other seat rather than on the one running it.

Scope ruled — measure, then optimise; ⛔ do not trim, ⛔ do not retune the budget

1. Measure deliberately, as the card asks: time check:pm-dispatch-gates on an idle box and on a loaded one, and profile where the time goes inside check-dispatch-gates.mjs — per-family, so the answer is a table and not a total. Report before changing anything.
2. Optimise only what the profile names. Caching, avoiding repeated filesystem walks, hoisting work out of per-family loops, parallelising independent families — anything that leaves the set of families and their verdicts byte-identical.
3. ⛔ Do not reduce what the gate checks. Not one family, not one file, not "this one is rarely useful". Coverage reduction is gate weakening and it is human floor — if the profile concludes that some family genuinely cannot be made affordable, say so in a comment and stop; the maintainer decides, not the implementer and not this seat.
4. ⛔ Do not raise os-verify-lock.sh's 540s budget as the fix. That file records the budget as measured, and :52 notes the only knob (OS_VERIFY_LOCK_WAIT) can lower it and not raise it — the asymmetry is deliberate. Raising it is a change to shared fleet-wide operational policy; it goes to the maintainer with the profile attached, not into this PR.
5. Prove the win the same way the problem was found: hold-time under the lock, before and after, not just wall-clock on an idle machine.

If (2) cannot close the gap without (3) or (4), that is a legitimate outcome — report it with numbers and let it be ruled. A card that ends in "here is the profile and here is why it needs a decision" is finished; one that quietly drops a family is not.

Routing

domain:devx, tooling — scripts/pm/check-dispatch-gates.mjs governs the code gate families a dev must run, so it falls on the code/doc-quality side of the lane table's SUBJECT split, not the governed-agent-instruction side that routes to domain:skills.

priority:p2 — it starves every other seat's verification queue (five consecutive exit-99 timeouts, three distinct holder PIDs, one holder still holding at 782s+), and it compounds as dispatch-gates.mjs grows. It is not p1 because nothing shipped is wrong and no correctness or security boundary moves; it is well above p3 because the cost is paid by every agent on the box, continuously, and is getting worse rather than staying flat.

pm:queue.

---

Generated by Claude Code

[claude]

Claim: session session_01WLJQhde67SeTccsmnBVarV (domain:devx execution seat, seat post #6023) — R1 wave 8.

• Branch: claude/issue-14521-dispatch-gates-selftest-profile (created at the profiling step; if the profile ends in "needs a decision" with no affordable optimisation, the deliverable is the profile posted on this card and no PR — the triage names that a finished outcome)
• Worktree: ../objectstack-14521 (dedicated per-task worktree, off origin/main)
• Domain: domain:devx per triage (comment 5507931324): scripts/pm/check-dispatch-gates.mjs + scripts/pm/dispatch-gates.mjs selfTest() — code-gate tooling, SUBJECT side of the lane split.
• File surface: scripts/pm/dispatch-gates.mjs (the selfTest() battery at :10053 on origin/main 53d368921 and whatever per-case helpers the profile names) and scripts/pm/check-dispatch-gates.mjs (214 lines; the runner). ⛔ No family removed, no case removed, no verdict changed — the set of families and their verdicts must stay byte-identical (triage rule 3, human floor); ⛔ scripts/pm/os-verify-lock.sh untouched — the 540s budget is not raised (triage rule 4); ⛔ no docs/audits/** rewrite (the profile goes in the PR body / the card, not into the August audit).
• Premise re-verified on origin/main 53d368921: package.json:71 "check:pm-dispatch-gates": "node scripts/pm/check-dispatch-gates.mjs"; that runner spawns dispatch-gates.mjs --self-test (its header :5-30 already documents that the battery "re-runs dispatch-gates.mjs's own CLI as a child process many times" and must run detached on an agent container); dispatch-gates.mjs is 17,057 lines; os-verify-lock.sh:52 — OS_VERIFY_LOCK_WAIT can lower the budget, never raise it; :133-134 the two-heaviest-families sentence. One correction to the triage's number: the August audit (docs/audits/2026-08-verify-lock-gate-routing-measurement.md:51) measured check:pm-dispatch-gates alone at 305.5 s; the 534.5 s figure is rank 1 + rank 2 (check:query-options-erasure 229.0 s, :151). So the card's 500-900 s observations are ~1.6-3x the documented baseline for this one family — the growth is the story, and the profile must say where it went. Premise holds; the scope rules are unaffected by the correction.
• Container & model: subagent lane (M, long wall-clock — each battery run is ~5-15 min and must run detached), opus.
• Clause ②: not reached — no contract, no gate semantics change.
• Serial constraints: scripts/pm/dispatch-gates.mjs is touched by PR Gate the workspace manifest dependency graph against cycles #14524 ([finding] nothing in the repo fails on a cyclic workspace manifest graph — the #13513 class can be re-added silently, and pnpm install's WARN is the only signal #14195, one ledger row; in the merge queue at claim time) — the dev must git merge origin/main before the first edit and confirm that row is in; the scripts/pm/* half of 144 of 154 scripts/** self-tests exit 0 when an early return leaves them before their verdict — 137 printing nothing at all #13798 batch 2 and cards [finding] Measured over all 139 Silent gate families: 49 read a packages/examples subtree their declaration never names — the Silent bucket is a class, not a bucket of cleared gates #14325 / [finding] Three repo-wide gates cannot declare their population truthfully — nul-bytes, comment-mask-corpus and refd-timer-probe read the whole tree, and the derivation has no always-runs channel to say so #14189 / Four scripts/** gates hit by the #13489 survey were excluded by occupancy — carry them forward #13800 (all dispatch-gates.mjs) now queue BEHIND this card. check-dispatch-gates.mjs is touched by no open PR. Scripts-only ⇒ skip-changeset.

---

Generated by Claude Code

[claude]

os-dev-report

{
  "issue": 14521,
  "status": "done",
  "branch": "claude/issue-14521-dispatch-gates-selftest-profile",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/14584",
  "premise_still_valid": true,
  "summary": "Outcome A. The profile named one thing and the fix is that one thing: discoverFamilies hands the SAME source string to six analysers in one pass of its per-family loop (extractWatchHints, readProgramTargetsInSource, payloadEnvDependence, firstPartyImportTargets, spawnedProgramTargets, packageManifestTargets) and every one re-derives the masked body from scratch, two of them twice - so one gate source pays maskComments about seven times and maskSelfTests about five, per discovery, for bytes that cannot have changed in between. Both maskers are pure functions of their input string; they are now memoised on it behind a byte-bounded cache. 74 insertions / 9 deletions in scripts/pm/dispatch-gates.mjs and nothing else. LOCK HOLD TIME (shared 4-core box, load avg printed at the end of each run): BEFORE 804s (base 53d368921, instrumented, load 1.20), 816s (same base, clean, load 2.60), 843s (base 7286dd58e, clean, load 1.07); AFTER 303s (base 7286dd58e + patch, instrumented, load 4.47) and 293s (merged head e5fa9d54a + patch, via pnpm check:pm-dispatch-gates, load 1.43). The same-base pair is 843s -> 303s = 2.78x, and the faster run was taken under the heavier load. Idle-vs-loaded could not be delivered: the box is shared and never idle, so the load figures are attached instead of an invented idle number. FLEET BEFORE POPULATION: /tmp/os-heavy-verify.lock.ledger (445 records over 26h48m) names this family in 25 rows and it is the ledger's largest single consumer of lock-seconds by 6x (8365s/13 runs + 2078s/4 + 1599s/3, next command down 1942s/11). 23 of the 25 acquired, holds 517s-842s and rising over the window (earliest four 517-521s, latest eight 770-842s); the other 2 are queue-timeout at the 540s budget - the card's symptom, in the ledger. During this card's own runs another seat's unpatched pnpm check:pm-dispatch-gates held the lock 812s+ while three of my jobs timed out behind it. TOP-10 CASES by cost, before -> after (a case's cost is the wall time between the previous verdict line and its own): 'a convention-triggered card is really convention-triggered' 62.0 -> 26.8; 'all three ratchets are still discovered with hints' 41.8 -> 8.7; 'the seam card still derives at all' 32.3 -> 13.5; '--commands answers' 32.0 -> 13.7; 'a record that is --commands output copied verbatim reconciles GREEN' 31.7 -> 12.6; '--json puts a single parseable document on stdout' 31.2 -> 13.2; 'a run whose surface ALREADY carries a changeset answers at all' 31.2 -> 13.8; 'dropping ONE line from that record exits 1' 31.0 -> 12.8; 'the DEFAULT run answers at all' 30.9 -> 13.6; 'the published harvest of the pasted block' 30.7 -> 12.3 (seconds). Over the 1216 cases present in both profiled runs: 760.2s -> 285.7s. CHILD PROCESSES: 216 spawnSync calls before AND after, identical composition - 28 self-spawns of the tool's own CLI (465.1s -> 196.8s, i.e. 16.6s -> 7.0s each), 184 git (2.3s -> 3.5s), 4 other (0.2s). Child-process time was 58.2% of the 803.9s run. ONE DISCOVERY CALL, V8 CPU profile, 14.32s total, self time by nearest dispatch-gates frame: maskSelfTests 4.54s (31.7%), anchoredReadTargets 1.55s (10.8%), spawnedProgramTargets 1.24s (8.7%), GC 0.97s (6.8%), extractWatchHints 0.92s (6.5%), selfTestOnlyCallables 0.87s (6.1%), firstPartyImportTargets 0.84s (5.9%), payloadEnvDependence 0.80s (5.6%), packageManifestTargets 0.79s (5.5%). Unit cost of discoverFamilies(), three consecutive calls each way: 13616/13432/14270 ms -> 4923/2640/2683 ms (2.8x cold, 5.1x warm). WHERE THE GROWTH WENT, the card's open question: not individual cases getting slower for their own reasons - the per-source masking is quadratic-ish in analyser count and the battery pays it in 15 in-process discoveries plus 28 child processes that each pay a full one. TRIAGE BASELINE CORRECTION (already flagged in the claim comment, re-verified): the 534.5s in os-verify-lock.sh:133-134 is rank 1 + rank 2; the August audit measures this family ALONE at 305.5s (:51), so the observed 800s+ is ~2.7x its documented baseline, not ~1.6x. The correction makes the growth larger; no scope rule moves. The audit was not rewritten. NOT DONE, deliberately (triage rules 3 and 4): no family removed, no case removed, no assertion relaxed; os-verify-lock.sh untouched and the 540s budget not raised; no workflow changed; cases were NOT parallelised - the profile did not need it once the repetition was gone, and a flaky parallel battery is worse than a slow serial one. A further ~15 in-process discoveries could be removed by memoising discoverFamilies() itself, but that shares a mutable result across call sites and was not needed to close the gap, so it is left alone.",
  "tests": "BYTE-IDENTICAL PROOF, both trees at base 7286dd58e with the patch the only difference (basecmp is a detached worktree at that commit; both batteries run through os-verify-lock.sh). Verdict set - every streamed verdict line, sorted, timestamps stripped: unpatched 1241 cases / 1241 pass / 0 fail vs patched 1241 / 1241 / 0, `diff` EMPTY (exit 0). Battery summary line identical both ways: 'dispatch-gates self-test: 1240 cases pass.' Discovered-family list - key, resolved files, hints, reads, selfTest, direct, filter, workflows, sorted: 202 families each, `diff` EMPTY. Runner exit code 0 both ways. CONTROL: two BEFORE runs of the unpatched battery on the same tree (before1, before2) also produce a byte-identical verdict set, so the empty diffs measure the patch and not run-to-run drift. The third AFTER run (battery-gate) sits on the merged head e5fa9d54a and differs from basecmp in exactly three lines, all of them tracked-file COUNTS embedded in case names (2948->2950, 2407->2408, 5721->5726) - the tree grew between the two commits; every verdict is still a pass. That run is a hold-time reading, not the byte-identical proof; the proof is the same-base pair. MEASUREMENT HARNESS: every timing run was DETACHED per check-dispatch-gates.mjs's header and polled with until-loops; no foreground battery run. Per-case timings came from a profiling preload loaded with `node --require` from the scratchpad - it patches console.log and spawnSync and appends records synchronously with appendFileSync, so NO repo file was edited to measure, and nothing is lost to pipe buffering or to a kill. Its cost is not readable: the instrumented BEFORE run (804s) was FASTER than the clean one (816s). GATE UNION, derived at e5fa9d54a with `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack` (15 families, change set read from git by the tool itself), each exit code captured by redirecting first and reading $? before any pipe: node scripts/check-ci-filter-parity.mjs 0 RAN-PASS; node scripts/check-cross-package-test-inputs.mjs 0 RAN-PASS; node scripts/check-self-test-wired.mjs 0 RAN-PASS; node scripts/check-shard-attestation.mjs 0 RAN-PASS; node scripts/check-test-completeness.mjs 3 NOT-MEASURED (PREREQUISITE NOT MET - it grades a saved turbo test log and there is none locally; its own text says this is not a red and not a finding); pnpm check:agent-test-spelling 0 RAN-PASS; pnpm check:bash32-floor 0 RAN-PASS; pnpm check:cli-command-ids 0 RAN-PASS; pnpm check:cross-package-test-inputs 0 RAN-PASS; pnpm check:declared-population-live 0 RAN-PASS; pnpm check:entry-guard 0 RAN-PASS; pnpm check:parse-guard 0 RAN-PASS; pnpm check:pm-dispatch-gates 0 RAN-PASS (detached, under the lock, 293s hold, 1240 cases pass); pnpm check:pnpm-filter-targets 0 RAN-PASS; pnpm check:watch-hint-literal 0 RAN-PASS. Reconciliation: `dispatch-gates --ran` exit 0 - 15 derived, 15 run, 0 UNRUN. Beyond the union: pnpm check:nul-bytes exit 0 (7961 files, no raw control bytes) and a direct control-character grep over the edited file, exit 1 (no match). DECLARED NARROWING: the fourteen scripted gates were run OFF the verify lock. That is what the lock does today rather than a narrowing of coverage - its own header records that routing gate runs through it was MEASURED one-directional and refused, and --status prints `check:*` gate scripts among the work it 'does NOT see'. The battery itself, which is genuinely heavy, went through the lock on all five readings. Three of my locked submissions returned exit 99 (never acquired) behind another seat's 812s-long unpatched battery and were resubmitted on the same OS_VERIFY_LOCK_SLOT; those are NOT MEASURED runs and none of them is quoted as a reading. No ablation was performed: this change removes repeated work rather than adding a guard, so the thing to prove is that the output does not move, which the byte-identical section proves directly. MERGE: `git fetch origin main && git merge origin/main` before the first edit (to pick up PR 14524's ledger row, confirmed present at scripts/pm/dispatch-gates.mjs:2335) and again before reporting; the battery, the gate union and --ran were all re-run on the merged head e5fa9d54a.",
  "gates": "15 derived, 15 run, 14 RAN-PASS + 1 NOT-MEASURED (check-test-completeness, exit 3, prerequisite absent locally); --ran exit 0, 0 UNRUN. Measured at e5fa9d54a.",
  "files_touched": [
    "scripts/pm/dispatch-gates.mjs"
  ],
  "changeset": "skip-changeset (label added to PR 14584 by additive POST and read back; scripts-only, nothing published from any package)",
  "mcp_calls": "0 - every GitHub read and write went through REST with GITHUB_TOKEN, as the dispatch ordered",
  "open_questions": [],
  "out_of_scope_findings": []
}

Generated by Claude Code

---

Generated by Claude Code