Skip to content

Fix #97: scope REPORT_DIR to the Test Reports section - #98

Closed
laird wants to merge 1 commit into
masterfrom
feature/issue-97
Closed

Fix #97: scope REPORT_DIR to the Test Reports section#98
laird wants to merge 1 commit into
masterfrom
feature/issue-97

Conversation

@laird

@laird laird commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Closes #97

Root Cause

regression-test.sh selected the report directory with a global first match:

REPORT_DIR=$(grep "Location: " CLAUDE.md | sed '...' | head -1)

CLAUDE.md has two Location: lines and the Unit Tests one comes first, so REPORT_DIR became tests/test_*.sh and mkdir -p created a directory with that literal name.

Impact

Worse than a misplaced report — the directory shadows the project's own documented test glob. Once it exists:

for t in tests/test_*.sh; do bash "$t" || exit 1; done

yields the directory, bash exits 126, and the unit suite goes red with no failing test behind it. Reports also never reached docs/test/regression-reports/.

Solution

  • Scope the lookup to the **Test Reports** section rather than taking the first global match.
  • Reject a resolved REPORT_DIR that still contains glob metacharacters (*?[) — that is a parse failure, not a directory name — falling back to the default.

Changes Made

  • plugins/autocoder/scripts/regression-test.sh — section-scoped lookup + metacharacter guard.
  • tests/test_regression_report_dir.sh (new) — asserts no glob-named path is ever created and the report lands in docs/test/regression-reports/. Two scenarios: the real CLAUDE.md section ordering, and a Test Reports location that is itself a glob, so the guard cannot rot into dead code.

Verification

  • Test written first, confirmed red: FAIL: regression-test.sh created a path literally named 'tests/test_*.sh'.
  • Both guards proven non-vacuous by reverting each independently and observing the matching failure.
  • 18/18 shell unit tests pass; bash -n and py_compile clean.
  • End-to-end against the real CLAUDE.md: report now written to docs/test/regression-reports/, no stray directory.

Scope

Distinct from #73/#85 (unit-test counting and exit codes); this is the REPORT_DIR selection path only. Independent of #96 — both branch from master and touch different files.

Note: on this base the harness still reports 0/0 passed as green — that is the #73/#85 defect, unchanged here and still behind the blocked #74 stack.

🤖 Generated with Claude Code

regression-test.sh picked the report directory with a global
`grep "Location: " CLAUDE.md | head -1`. CLAUDE.md has several such lines and
the Unit Tests one comes first, so REPORT_DIR became `tests/test_*.sh` and
mkdir -p created a directory with that literal name.

That directory shadows the project's own documented test glob: once it exists,
`for t in tests/test_*.sh; do bash "$t" || exit 1; done` yields the directory,
bash exits 126, and the unit suite goes red with no failing test behind it.
Reports also never reached docs/test/regression-reports/.

Changes:
- regression-test.sh: scope the lookup to the "**Test Reports**" section, and
  reject a resolved REPORT_DIR still containing glob metacharacters (*?[) —
  that is a parse failure, not a directory name — falling back to the default.
- tests/test_regression_report_dir.sh: new. Asserts no glob-named path is ever
  created and the report lands in docs/test/regression-reports/. Covers two
  scenarios: the real CLAUDE.md section ordering, and a Test Reports location
  that is itself a glob (exercising the metacharacter guard directly).

Verification: both assertions confirmed non-vacuous by reverting each guard and
observing the matching failure. 18/18 shell unit tests pass; bash -n and
py_compile clean. End-to-end run against the real CLAUDE.md now writes to
docs/test/regression-reports/ and creates no stray directory.

Distinct from #73/#85, which cover unit-test counting and exit codes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@laird

laird commented Aug 16, 2026

Copy link
Copy Markdown
Owner Author

Superseded — master already fixed this (see #97, closed with the master line numbers). Closing without merging to avoid re-applying a stale branch's version of this fix over master's current one.

@laird laird closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

regression-test.sh writes reports into a directory literally named 'tests/test_*.sh', breaking the documented unit-test glob

1 participant