Fix #85: sum per-suite test counts instead of reporting the last one - #86
Merged
Conversation
regression-test.sh reported "7/0 passed" for a green run of 178 assertions across 11 suites. Root cause: count_before() ended in `tail -1`, taking the last "<N> passed" in the log. Correct for Jest/Playwright, which emit one summary per run; wrong for this repo's unit command, which loops over a dozen independent bash scripts that each print their own "Results: N passed, M failed". The denominator was worse — no bash suite prints the word "total", so UNIT_TOTAL was always the literal 0. Changes: - count_before() sums matches via awk instead of tail -1. It still returns EMPTY when nothing matched, which the no-parseable-summary guard depends on to distinguish "runner never ran" from a real zero. - UNIT_TOTAL is derived from passed+failed only when the runner reports no total, and only AFTER that guard, so an unparseable run cannot synthesise 0+0=0 and read as green. A total the runner did report is never overwritten — Jest's includes skipped tests. - Mirrored both into .agent/scripts/regression-test.sh per the parallel maintenance requirement in CLAUDE.md. Scope: reporting only, not a false green. UNIT_EXIT comes from PIPESTATUS, and the failure path is unaffected because `|| exit 1` makes a failing script the last summary in the log. Verification: two new fixtures in tests/test_regression_test_config.sh pin both directions (green suites sum to 26/26; failing suites sum to 9/14 and still exit non-zero). Suite 21/21. Full regression now reports 178/178, exit 0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
laird
added a commit
that referenced
this pull request
Aug 8, 2026
…n nothing (#78) * Show suite status, not just counts, in the run summary (#73) Both summary lines printed only "$PASSED/$TOTAL passed" in green, discarding $UNIT_STATUS. On a failed run — especially the "no parseable test summary" case — the reason was written to the report file and never shown, so an operator watching the run saw a green "0/0 passed" and no explanation. Now prints the status and colours the label by outcome, so a red run is visibly red at the point a human is actually looking. Verification: tests/test_regression_test_config.sh 16/16 (the last assertion covers exactly this — an unparseable summary must be *reported*, not merely exit non-zero); all tests/test_*.sh 18/18; pytest 59/59; BUILD_OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix #85: sum per-suite test counts instead of reporting the last one (#86) regression-test.sh reported "7/0 passed" for a green run of 178 assertions across 11 suites. Root cause: count_before() ended in `tail -1`, taking the last "<N> passed" in the log. Correct for Jest/Playwright, which emit one summary per run; wrong for this repo's unit command, which loops over a dozen independent bash scripts that each print their own "Results: N passed, M failed". The denominator was worse — no bash suite prints the word "total", so UNIT_TOTAL was always the literal 0. Changes: - count_before() sums matches via awk instead of tail -1. It still returns EMPTY when nothing matched, which the no-parseable-summary guard depends on to distinguish "runner never ran" from a real zero. - UNIT_TOTAL is derived from passed+failed only when the runner reports no total, and only AFTER that guard, so an unparseable run cannot synthesise 0+0=0 and read as green. A total the runner did report is never overwritten — Jest's includes skipped tests. - Mirrored both into .agent/scripts/regression-test.sh per the parallel maintenance requirement in CLAUDE.md. Scope: reporting only, not a false green. UNIT_EXIT comes from PIPESTATUS, and the failure path is unaffected because `|| exit 1` makes a failing script the last summary in the log. Verification: two new fixtures in tests/test_regression_test_config.sh pin both directions (green suites sum to 26/26; failing suites sum to 9/14 and still exit non-zero). Suite 21/21. Full regression now reports 178/178, exit 0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #85
Stacked on #78 (
feature/issue-73-master). Based on that branch so the diff shows only this work; merge #78 first.Root Cause
count_before()ended intail -1, taking the last"<N> passed"in the log. That is correct for Jest/Playwright, which emit one summary per run. This repo's unit command is a loop over a dozen independent bash scripts, each printing its ownResults: N passed, M failed— so a green run of 178 assertions reported the last suite's 7. The denominator was worse: no bash suite prints the wordtotal, soUNIT_TOTALwas always the literal0.Console read
Unit Tests: ✅ PASSED (7/0 passed).Scope — reporting only, not a false green
Worth stating so this is not read as a repeat of #73:
UNIT_EXITcomes fromPIPESTATUS[0], not from these counts, so a red run still goes red. The regression-test.sh reports "All tests passed" after running zero tests, and writes reports into a directory named tests/test_*.sh #73 fix holds.if [ "$UNIT_FAILED" -gt 0 ]reads the same parser, but|| exit 1in the suite command makes a failing script the last summary in the log, so the count was already right exactly when it mattered.What was wrong is the number a human reads, in the console and in
docs/test/regression-reports/*.md— understated ~25x, over a nonsensical/0.Changes
count_before()sums matches withawkinstead oftail -1. It still returns empty when nothing matched, which the no-parseable-summary guard depends on to distinguish "the runner never ran" from a genuine zero.UNIT_TOTALis derived frompassed + failedonly when the runner reports no total, and only after that guard — otherwise an unparseable run would synthesise0 + 0 = 0and read as a valid green. A total the runner did report is never overwritten, since Jest's legitimately counts skipped tests..agent/scripts/regression-test.shper the parallel maintenance requirement in CLAUDE.md.Verification
Two new fixtures in
tests/test_regression_test_config.shpin both directions:26/26 passed, not7/09/14 passedand still exit non-zeroNoted, not fixed here
.agent/scripts/regression-test.shis missing the whole of #73 — noPIPESTATUSread, no all-skipped guard, no report-dir scoping — so the mirror can still report a false green. That is #78's / #71's territory, deliberately left alone to keep this diff to one defect.🤖 Generated with Claude Code