Skip to content

Fix #85: sum per-suite test counts instead of reporting the last one - #86

Merged
laird merged 1 commit into
feature/issue-73-masterfrom
feature/issue-85
Aug 8, 2026
Merged

Fix #85: sum per-suite test counts instead of reporting the last one#86
laird merged 1 commit into
feature/issue-73-masterfrom
feature/issue-85

Conversation

@laird

@laird laird commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Closes #85

Stacked on #78 (feature/issue-73-master). Based on that branch so the diff shows only this work; merge #78 first.

Root Cause

count_before() ended in tail -1, taking the last "<N> passed" in the log. That is correct for Jest/Playwright, which emit one summary per run. This repo's unit command is a loop over a dozen independent bash scripts, each printing its own Results: N passed, M failed — so a green run of 178 assertions reported the last suite's 7. The denominator was worse: no bash suite prints the word total, so UNIT_TOTAL was always the literal 0.

Console read Unit Tests: ✅ PASSED (7/0 passed).

Scope — reporting only, not a false green

Worth stating so this is not read as a repeat of #73:

What was wrong is the number a human reads, in the console and in docs/test/regression-reports/*.md — understated ~25x, over a nonsensical /0.

Changes

  • count_before() sums matches with awk instead of tail -1. It still returns empty when nothing matched, which the no-parseable-summary guard depends on to distinguish "the runner never ran" from a genuine zero.
  • UNIT_TOTAL is derived from passed + failed only when the runner reports no total, and only after that guard — otherwise an unparseable run would synthesise 0 + 0 = 0 and read as a valid green. A total the runner did report is never overwritten, since Jest's legitimately counts skipped tests.
  • Mirrored both into .agent/scripts/regression-test.sh per the parallel maintenance requirement in CLAUDE.md.
  • E2E needed no separate change: it already derives its own total, so it inherits the summing fix.

Verification

Two new fixtures in tests/test_regression_test_config.sh pin both directions:

  • three green suites (6 + 13 + 7) must report 26/26 passed, not 7/0
  • two failing suites (4+2, 5+3) must report 9/14 passed and still exit non-zero
$ bash tests/test_regression_test_config.sh
=== 21 passed, 0 failed ===        # 19/2 red before the fix

$ bash -n plugins/autocoder/scripts/*.sh && bash -n .agent/scripts/*.sh \
    && python3 -m py_compile plugins/autocoder/scripts/*.py
✅ build OK

$ bash plugins/autocoder/scripts/regression-test.sh   # exit 0
  Unit Tests:  ✅ PASSED (178/178 passed)             # was (7/0 passed)
  E2E Tests:   ⏭️  SKIPPED (not configured) (0/0 passed)

Noted, not fixed here

.agent/scripts/regression-test.sh is missing the whole of #73 — no PIPESTATUS read, no all-skipped guard, no report-dir scoping — so the mirror can still report a false green. That is #78's / #71's territory, deliberately left alone to keep this diff to one defect.

🤖 Generated with Claude Code

regression-test.sh reported "7/0 passed" for a green run of 178
assertions across 11 suites.

Root cause: count_before() ended in `tail -1`, taking the last
"<N> passed" in the log. Correct for Jest/Playwright, which emit one
summary per run; wrong for this repo's unit command, which loops over a
dozen independent bash scripts that each print their own
"Results: N passed, M failed". The denominator was worse — no bash suite
prints the word "total", so UNIT_TOTAL was always the literal 0.

Changes:
- count_before() sums matches via awk instead of tail -1. It still
  returns EMPTY when nothing matched, which the no-parseable-summary
  guard depends on to distinguish "runner never ran" from a real zero.
- UNIT_TOTAL is derived from passed+failed only when the runner reports
  no total, and only AFTER that guard, so an unparseable run cannot
  synthesise 0+0=0 and read as green. A total the runner did report is
  never overwritten — Jest's includes skipped tests.
- Mirrored both into .agent/scripts/regression-test.sh per the parallel
  maintenance requirement in CLAUDE.md.

Scope: reporting only, not a false green. UNIT_EXIT comes from
PIPESTATUS, and the failure path is unaffected because `|| exit 1` makes
a failing script the last summary in the log.

Verification: two new fixtures in tests/test_regression_test_config.sh
pin both directions (green suites sum to 26/26; failing suites sum to
9/14 and still exit non-zero). Suite 21/21. Full regression now reports
178/178, exit 0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@laird
laird merged commit bdaab01 into feature/issue-73-master Aug 8, 2026
4 checks passed
@laird
laird deleted the feature/issue-85 branch August 8, 2026 23:22
laird added a commit that referenced this pull request Aug 8, 2026
…n nothing (#78)

* Show suite status, not just counts, in the run summary (#73)

Both summary lines printed only "$PASSED/$TOTAL passed" in green, discarding
$UNIT_STATUS. On a failed run — especially the "no parseable test summary"
case — the reason was written to the report file and never shown, so an
operator watching the run saw a green "0/0 passed" and no explanation.

Now prints the status and colours the label by outcome, so a red run is
visibly red at the point a human is actually looking.

Verification: tests/test_regression_test_config.sh 16/16 (the last assertion
covers exactly this — an unparseable summary must be *reported*, not merely
exit non-zero); all tests/test_*.sh 18/18; pytest 59/59; BUILD_OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix #85: sum per-suite test counts instead of reporting the last one (#86)

regression-test.sh reported "7/0 passed" for a green run of 178
assertions across 11 suites.

Root cause: count_before() ended in `tail -1`, taking the last
"<N> passed" in the log. Correct for Jest/Playwright, which emit one
summary per run; wrong for this repo's unit command, which loops over a
dozen independent bash scripts that each print their own
"Results: N passed, M failed". The denominator was worse — no bash suite
prints the word "total", so UNIT_TOTAL was always the literal 0.

Changes:
- count_before() sums matches via awk instead of tail -1. It still
  returns EMPTY when nothing matched, which the no-parseable-summary
  guard depends on to distinguish "runner never ran" from a real zero.
- UNIT_TOTAL is derived from passed+failed only when the runner reports
  no total, and only AFTER that guard, so an unparseable run cannot
  synthesise 0+0=0 and read as green. A total the runner did report is
  never overwritten — Jest's includes skipped tests.
- Mirrored both into .agent/scripts/regression-test.sh per the parallel
  maintenance requirement in CLAUDE.md.

Scope: reporting only, not a false green. UNIT_EXIT comes from
PIPESTATUS, and the failure path is unaffected because `|| exit 1` makes
a failing script the last summary in the log.

Verification: two new fixtures in tests/test_regression_test_config.sh
pin both directions (green suites sum to 26/26; failing suites sum to
9/14 and still exit non-zero). Suite 21/21. Full regression now reports
178/178, exit 0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant