perf(CC-573): single streaming jq pass for pmctl run-stats - #547
Merged
Conversation
Profile: jq invocations = N+2, ~34ms/event; real events.jsonl (6642 lines) makes `pmctl run-stats --json` time out past 2 minutes. Same shape as the pre-CC-364 trace tail. P2 hygiene. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
pmctl_run_stats_extract_line spawned one jq per events.jsonl line: jq invocations = N+2, ~34ms/event (100->3.0s, 300->8.8s, 900->30.2s), and the real 6642-line events.jsonl made `run-stats --json` exceed a 2-minute timeout. Same shape CC-364 removed from trace tail. Scan phase is now one `jq -R` streaming pass over the concatenated archive+active stream, emitting a TSV row per run.* event that passes the --since bound (predicate identical to the former shell check); the bash loop folds rows into the _rs_* maps without per-line jq. New helpers pmctl_run_stats_filter_program + pmctl_run_stats_scan_stream replace extract_line / process_line / scan_path / scan_gzip_path. Measured: jq invocations 102/302/902 -> 2/2/2, wall 3.0-30.2s -> 0.19s flat. Output byte-identical (jq -S normalized JSON + human) to origin/main across a run with --since filtering and a rotated archive. New case_run_stats_single_jq_pass: counting jq shim asserts the invocation tally is equal (and non-zero) for a 20-run and a 200-run partition. test-pmctl-run-stats.sh 16 passed. shellcheck clean. The archive-glob + concat + one-jq-pass idiom is now duplicated with pmctl-trace.sh (~12 lines); extraction considered and deferred at two consumers (differing jq programs, output consumers, gzip-unavailable signalling) -- noted in the file header. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
pr-gate round 1 was GO with one critic advisory (critic-F001, medium): the jq-count test guards the perf invariant but nothing pins field-level report semantics, so a TSV-decode change could drift a cell without moving the 200-run total. This is CC-573's own Req 5. Add case_run_stats_streaming_matches_reference: a heterogeneous fixture (valid terminals, partial, nonzero exit, cancelled, missing-terminal, fallback_used, pre-`--since` row, two malformed lines, a non-run event, an archived-only run; three adapters) compared byte-for-byte (jq -S) against the golden JSON captured from origin/main. Golden frozen as a fixture per the CC-364 method rather than re-running two implementations. test-pmctl-run-stats.sh 17 passed. shellcheck clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
Index row -> done + pr:#547; body gets a Closure 2026-08-27 note with the 102/302/902 -> 2/2/2 jq-invocation result, byte-identity evidence, and the Req 6 (shared primitive) deferral rationale. validate.sh, test-pmctl-backlog, test-archive-closed-backlog, test-schema-task-mirrors-backlog all pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CC-573 —
pmctl run-statssingle streaming jq passProblem
pmctl_run_stats_extract_lineran onejq -rperevents.jsonlline.Profile (jq-counting shim + wall clock):
Linear at ~34ms/event. The real state store's
events.jsonl(6642 lines)made
pmctl run-stats --jsonexceed a 2-minute foreground timeout(~225s extrapolated). run-stats is v1.0 readiness evidence tooling (CC-358
DoD) and could not complete on real data. Same shape CC-364 removed from
pmctl trace tail.Change
Scan phase is now one
jq -Rstreaming pass over the concatenatedarchive + activestream. It emits a TSV row perrun.*event that passesthe
--sincebound (predicate identical to the former shell check:dropped only when a bound is set AND ts is non-empty AND ts < bound); the
bash loop folds rows into the
_rs_*maps with no per-line jq. New helperspmctl_run_stats_filter_program+pmctl_run_stats_scan_streamreplaceextract_line/process_line/scan_path/scan_gzip_path.Result: jq invocations 102/302/902 → 2/2/2; wall 3.0–30.2s → 0.19s
flat. Output byte-identical (
jq -Snormalized JSON + human) toorigin/mainacross a run with--sincefiltering and a rotated archive.Behavior parity (unchanged)
--sincelexicographic ISO-8601 filter, malformed-row silent skip(run-stats has never counted them), archive-inclusive scan +
gzip-unavailable
_metafallback,--by-adapter, output schema, CLI.fallback_usedevent signal (CC-358) untouched.Reuse note
The archive-glob + gzip-check + concat-then-one-jq-pass idiom (~12 lines) is
now duplicated with
pmctl-trace.sh. Extraction into a shared events-scanprimitive was considered and deferred: the two jq programs and output
consumers differ, and the gzip-unavailable signalling diverges (trace tail
read_archives=0; run-statsarchive_scanned=false+_meta). At twoconsumers the shared seam isn't clearly worth the callback indirection —
documented in the file header; revisit if a third consumer appears. No
ticket filed.
Tests
case_run_stats_single_jq_pass— counting jq shim asserts the invocationtally is equal (and non-zero) for a 20-run and a 200-run partition
(O(1) in event count).
case_run_stats_streaming_matches_reference— heterogeneous fixture(valid terminals, partial, nonzero exit, cancelled, missing-terminal,
fallback_used, pre-
--sincerow, two malformed lines, a non-run event,an archived-only run; three adapters) compared byte-for-byte (
jq -S)against the golden JSON frozen from
origin/main.test-pmctl-run-stats.sh17 passed; full suiterun-all-tests.sh105 passed, 0 failed, 0 skipped.
PR-gate
2 rounds. Round 1 GO with one critic advisory (medium: no field-level
equivalence oracle — CC-573's own Req 5); added the golden test; round 2
sequential, clean GO — critic / qa-tester / architecture / security all
approve, zero findings.
BACKLOG
CC-573 index row + body marked
✅ donewith aClosure 2026-08-27 (pr:#547)note in this PR.validate.sh,test-pmctl-backlog,test-archive-closed-backlog,test-schema-task-mirrors-backlogall pass.🤖 Generated with Claude Code