Skip to content

perf(CC-573): single streaming jq pass for pmctl run-stats - #547

Merged
screenleon merged 4 commits into
mainfrom
perf/cc-573-run-stats-single-jq-pass
Aug 27, 2026
Merged

perf(CC-573): single streaming jq pass for pmctl run-stats#547
screenleon merged 4 commits into
mainfrom
perf/cc-573-run-stats-single-jq-pass

Conversation

@screenleon

Copy link
Copy Markdown
Owner

CC-573 — pmctl run-stats single streaming jq pass

Problem

pmctl_run_stats_extract_line ran one jq -r per events.jsonl line.
Profile (jq-counting shim + wall clock):

N events jq invocations wall
100 102 3.0s
300 302 8.8s
900 902 30.2s

Linear at ~34ms/event. The real state store's events.jsonl (6642 lines)
made pmctl run-stats --json exceed a 2-minute foreground timeout
(~225s extrapolated). run-stats is v1.0 readiness evidence tooling (CC-358
DoD) and could not complete on real data. Same shape CC-364 removed from
pmctl trace tail.

Change

Scan phase is now one jq -R streaming pass over the concatenated
archive + active stream. It emits a TSV row per run.* event that passes
the --since bound (predicate identical to the former shell check:
dropped only when a bound is set AND ts is non-empty AND ts < bound); the
bash loop folds rows into the _rs_* maps with no per-line jq. New helpers
pmctl_run_stats_filter_program + pmctl_run_stats_scan_stream replace
extract_line / process_line / scan_path / scan_gzip_path.

Result: jq invocations 102/302/902 → 2/2/2; wall 3.0–30.2s → 0.19s
flat
. Output byte-identical (jq -S normalized JSON + human) to
origin/main across a run with --since filtering and a rotated archive.

Behavior parity (unchanged)

--since lexicographic ISO-8601 filter, malformed-row silent skip
(run-stats has never counted them), archive-inclusive scan +
gzip-unavailable _meta fallback, --by-adapter, output schema, CLI.
fallback_used event signal (CC-358) untouched.

Reuse note

The archive-glob + gzip-check + concat-then-one-jq-pass idiom (~12 lines) is
now duplicated with pmctl-trace.sh. Extraction into a shared events-scan
primitive was considered and deferred: the two jq programs and output
consumers differ, and the gzip-unavailable signalling diverges (trace tail
read_archives=0; run-stats archive_scanned=false + _meta). At two
consumers the shared seam isn't clearly worth the callback indirection —
documented in the file header; revisit if a third consumer appears. No
ticket filed.

Tests

  • case_run_stats_single_jq_pass — counting jq shim asserts the invocation
    tally is equal (and non-zero) for a 20-run and a 200-run partition
    (O(1) in event count).
  • case_run_stats_streaming_matches_reference — heterogeneous fixture
    (valid terminals, partial, nonzero exit, cancelled, missing-terminal,
    fallback_used, pre---since row, two malformed lines, a non-run event,
    an archived-only run; three adapters) compared byte-for-byte (jq -S)
    against the golden JSON frozen from origin/main.
  • test-pmctl-run-stats.sh 17 passed; full suite run-all-tests.sh
    105 passed, 0 failed, 0 skipped.

PR-gate

2 rounds. Round 1 GO with one critic advisory (medium: no field-level
equivalence oracle — CC-573's own Req 5); added the golden test; round 2
sequential, clean GO — critic / qa-tester / architecture / security all
approve, zero findings.

BACKLOG

CC-573 index row + body marked ✅ done with a Closure 2026-08-27 (pr:#547) note in this PR. validate.sh, test-pmctl-backlog,
test-archive-closed-backlog, test-schema-task-mirrors-backlog all pass.

🤖 Generated with Claude Code

screenleon and others added 4 commits August 27, 2026 15:36
Profile: jq invocations = N+2, ~34ms/event; real events.jsonl (6642 lines)
makes `pmctl run-stats --json` time out past 2 minutes. Same shape as the
pre-CC-364 trace tail. P2 hygiene.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
pmctl_run_stats_extract_line spawned one jq per events.jsonl line: jq
invocations = N+2, ~34ms/event (100->3.0s, 300->8.8s, 900->30.2s), and
the real 6642-line events.jsonl made `run-stats --json` exceed a 2-minute
timeout. Same shape CC-364 removed from trace tail.

Scan phase is now one `jq -R` streaming pass over the concatenated
archive+active stream, emitting a TSV row per run.* event that passes the
--since bound (predicate identical to the former shell check); the bash
loop folds rows into the _rs_* maps without per-line jq. New helpers
pmctl_run_stats_filter_program + pmctl_run_stats_scan_stream replace
extract_line / process_line / scan_path / scan_gzip_path.

Measured: jq invocations 102/302/902 -> 2/2/2, wall 3.0-30.2s -> 0.19s
flat. Output byte-identical (jq -S normalized JSON + human) to origin/main
across a run with --since filtering and a rotated archive.

New case_run_stats_single_jq_pass: counting jq shim asserts the invocation
tally is equal (and non-zero) for a 20-run and a 200-run partition.
test-pmctl-run-stats.sh 16 passed. shellcheck clean.

The archive-glob + concat + one-jq-pass idiom is now duplicated with
pmctl-trace.sh (~12 lines); extraction considered and deferred at two
consumers (differing jq programs, output consumers, gzip-unavailable
signalling) -- noted in the file header.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
pr-gate round 1 was GO with one critic advisory (critic-F001, medium):
the jq-count test guards the perf invariant but nothing pins field-level
report semantics, so a TSV-decode change could drift a cell without moving
the 200-run total. This is CC-573's own Req 5.

Add case_run_stats_streaming_matches_reference: a heterogeneous fixture
(valid terminals, partial, nonzero exit, cancelled, missing-terminal,
fallback_used, pre-`--since` row, two malformed lines, a non-run event, an
archived-only run; three adapters) compared byte-for-byte (jq -S) against
the golden JSON captured from origin/main. Golden frozen as a fixture per
the CC-364 method rather than re-running two implementations.

test-pmctl-run-stats.sh 17 passed. shellcheck clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
Index row -> done + pr:#547; body gets a Closure 2026-08-27 note with the
102/302/902 -> 2/2/2 jq-invocation result, byte-identity evidence, and the
Req 6 (shared primitive) deferral rationale. validate.sh, test-pmctl-backlog,
test-archive-closed-backlog, test-schema-task-mirrors-backlog all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFPwSUfmLGh6KJLYFBArTS
@screenleon
screenleon merged commit b8a746c into main Aug 27, 2026
66 checks passed
@screenleon
screenleon deleted the perf/cc-573-run-stats-single-jq-pass branch August 27, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant