Skip to content

fix(CC-571): sqlite atomic-script gap in _ctx_fts_rebuild + _ctx_index_file - #539

Merged
screenleon merged 3 commits into
mainfrom
fix/CC-571-fts-rebuild-transaction
Aug 26, 2026
Merged

fix(CC-571): sqlite atomic-script gap in _ctx_fts_rebuild + _ctx_index_file#539
screenleon merged 3 commits into
mainfrom
fix/CC-571-fts-rebuild-transaction

Conversation

@screenleon

Copy link
Copy Markdown
Owner

Summary

_ctx_fts_rebuild's DROP+CREATE+INSERT sequence had no BEGIN/COMMIT, and critically no -bail: direct reproduction confirmed the sqlite3 CLI's default behavior on a mid-script SQL error is to print the error and keep executing subsequent statements — it does not stop — so a bare BEGIN...COMMIT alone would still reach and execute COMMIT after silently skipping the failed statement, committing a half-built content_fts. Both callers (pmctl_context_index, pmctl_context_update) also called it unchecked, reporting success regardless.

Fixed by wrapping in BEGIN IMMEDIATE/COMMIT with sqlite3 -bail: an error now aborts before COMMIT, leaving the transaction open; the process exit closes the connection, triggering an automatic ROLLBACK that leaves the previous content_fts fully intact. Both callers now check the return value and report the degradation — including in the final stdout summary line itself, not just stderr (see gate findings below) — instead of an unqualified success message. Non-fatal, since FTS is a best-effort acceleration layer (LIKE fallback remains available).

Scope expanded during the mandatory reuse/simplify pass: the altitude review found the identical pattern in _ctx_index_file (pmctl_context_update's other call site), which is more severe since it writes the primary files/symbols/file_chunks data, not an acceleration layer. Direct testing also found a third, independent bug there: the function's trailing rm -f "$tmpf" masked sqlite3's real exit code (the function always returned rm's status, ~always 0, regardless of whether sqlite3 succeeded). Both functions now share one _ctx_sqlite_exec_atomic helper (also closing a reuse-review finding that the fix reinvented a pattern already established in memory.sh's memory_usage_commit rather than citing it), and _ctx_index_file's caller now treats its failure as fatal — unlike FTS, a failed primary-index write must not be reported as "re-indexed".

Gate findings fixed (2 rounds)

  • critic-F001 (round 1): stderr carried the degradation diagnostic, but stdout still printed a bare "N indexed, M skipped" / "re-indexed " — a contradictory summary for any caller that only looks at stdout or exit code. Fixed by appending the degradation qualifier directly onto the stdout summary line.
  • critic-F001 (round 2): when content_fts didn't exist before the rebuild attempt (first-time build), the message wrongly said "existing (now stale) FTS index retained" — there was no existing index to retain. Fixed by branching the message on whether content_fts existed beforehand.

Verification

  • pmctl gate run --executor codex --policy genericFinal: GO (critic/qa-tester/architecture-reviewer/security-reviewer, tier standard, 5 rounds total — 2 real findings fixed, remaining rounds were gate-executor synthesis protocol instability unrelated to the code, diagnosed via supervisor-stdout.log).
  • tests/shell/test-pmctl-context.sh — 168 passed, 0 failed (7 new/strengthened cases covering: FTS rollback preserves the old index, _ctx_index_file's return code reflects sqlite3's real result, both callers report failure honestly in their stdout summary, and the first-time-build-failure wording).
  • tests/bin/run-tests.sh --base main — 9/9 diff-selected suites passed (incl. lint-scripts).
  • tests/bin/run-all-tests.sh104 passed, 0 failed, 0 skipped (full sign-off).

🤖 Generated with Claude Code

https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

screenleon and others added 3 commits August 26, 2026 10:16
…x_file

_ctx_fts_rebuild's DROP+CREATE+INSERT sequence had no BEGIN/COMMIT, and
critically no -bail: direct reproduction confirmed the sqlite3 CLI's
default behavior on a mid-script SQL error is to print the error and keep
executing subsequent statements -- it does NOT stop -- so a bare
BEGIN...COMMIT alone would still reach and execute COMMIT after silently
skipping the failed statement, committing a half-built content_fts. Both
callers (pmctl_context_index, pmctl_context_update) also called it
unchecked, reporting success regardless.

Fixed by wrapping in BEGIN IMMEDIATE/COMMIT with `sqlite3 -bail`: an error
now aborts before COMMIT, leaving the transaction open; the process exit
closes the connection, triggering an automatic ROLLBACK that leaves the
previous content_fts fully intact. Both callers now check the return value
and report an honest "FTS index rebuild failed; stale index retained,
LIKE fallback still available" instead of a bare success line -- non-fatal,
since FTS is a best-effort acceleration layer.

Scope expanded during the mandatory reuse/simplify pass: the altitude
review found the identical pattern in _ctx_index_file (pmctl_context_update's
other call site), which is more severe since it writes the PRIMARY
files/symbols/file_chunks data, not an acceleration layer. Direct testing
also found a third, independent bug there: the function's trailing
`rm -f "$tmpf"` masked sqlite3's real exit code (the function always
returned rm's status, ~always 0, regardless of whether sqlite3 succeeded).
Both functions now share one `_ctx_sqlite_exec_atomic` helper (closing a
reuse-review finding that the fix reinvented a pattern already established
in memory.sh's memory_usage_commit rather than citing/sharing it), and
_ctx_index_file's caller now treats its failure as fatal -- unlike FTS, a
failed primary-index write must not be reported as "re-indexed".

5 new regression tests cover: FTS rollback preserves the old index,
_ctx_index_file's return code reflects sqlite3's real result, and both
callers report failure honestly rather than silently claiming success.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx
…ne itself

Gate finding critic-F001 (round 1, gate-20260826-011700-b139d2): printing
the FTS-rebuild-failure diagnostic on stderr while stdout still read a bare
"context index: N indexed, M skipped" / "context update: re-indexed <path>"
is a contradictory summary -- a caller that only looks at stdout, or only
checks exit code 0, saw nothing but success. This violated CC-571's own
Requirement 2 ("failure must not print success-worded text").

Both callers now append a degradation qualifier directly onto the final
summary line when the FTS rebuild fails, in addition to the existing
stderr diagnostic. Updated both regression tests to assert the summary
line itself discloses degradation, not just that stderr mentions it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx
Gate finding critic-F001 (round 2, gate-20260826-021038-ac0bc2): when
content_fts did not exist before a rebuild attempt (a first-time index
build), a failed rebuild's rollback leaves no FTS table at all -- the
existing "existing (now stale) FTS index retained" message was false in
that case, since there was no existing index to retain.

Both callers now check whether content_fts existed before attempting the
rebuild (pmctl_context_index already captured this via _fts_present;
pmctl_context_update now captures it too) and branch the message: "stale
index retained" only when one genuinely existed, otherwise "no FTS index
available, LIKE fallback only".

Strengthened case_context_index_reports_fts_rebuild_failure_honestly (which
already exercised a fresh fixture repo -- exactly the first-time-build
scenario) to assert the correct branch's wording and that content_fts is
actually absent afterward, per the finding's own verification_expectation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx
@screenleon
screenleon merged commit 36569bb into main Aug 26, 2026
64 checks passed
@screenleon
screenleon deleted the fix/CC-571-fts-rebuild-transaction branch August 26, 2026 04:18
screenleon added a commit that referenced this pull request Aug 26, 2026
…lt path (#541)

* docs(BACKLOG): mark CC-571 done, record PR #539 reference

Status flip missed at merge time -- PR #539 shipped and merged, but
BACKLOG.md's index row and body heading were never flipped from active to
done. This is the fourth time this exact pattern has recurred in one
session (after CC-567, CC-533, CC-015) -- this time by the same agent that
had just fixed the other three, which is its own lesson: the fix has to
happen inside the PR before merge, not as a mental note for after.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

* fix(CC-572): pr-gate synthesis retry removes (not truncates) the result path

Diagnosed from repeated production protocol failures during the CC-571
gate saga: apply_patch failures ("invalid patch: multiple operations
target <file>", "invalid hunk", "Failed to find expected lines") kept
recurring on the synthesis correction retry, in both sequential and
parallel mode.

Root cause: the reviewer-protocol retry (which never showed this failure)
always writes to a brand-new path (reviewer-<name>-<ts>-retry1.md) --
always an unambiguous "Add File" for whatever patch tool the executor
uses. Both synthesis retries instead re-dispatch to the SAME fixed
$OUTPUT_FILE path. The sequential retry already truncated it
(`: > "$OUTPUT_FILE"`) before this fix, and STILL hit the identical
failure -- confirmed by direct trace inspection that a path which still
EXISTS on disk, even at 0 bytes, can lead the executor's patch tool to
choose an "Update File" operation (which locates existing content to
edit) instead of "Add File". An update against empty/rewritten content
then fails outright, since there's no content to locate -- a hard
patch-tool failure unrelated to the actual synthesis content. The
parallel retry didn't even truncate; it left the first attempt's full
content in place.

Fixed by removing the path entirely (not truncating it) before both
retries, via a new shared `gate_clear_retry_target` helper next to
`gate_dispatch_command`. This forces an unambiguous "Add File" the same
way the already-reliable reviewer-retry path's brand-new filename does.

Added a test-only capture hook (CODEX_GATE_CAPTURE_OUTPUT_EXISTS_DIR) that
records, at the very start of each synthesis dispatch, whether the result
path exists on disk at that instant -- and two regression tests (one per
mode) asserting the first dispatch sees "exists" (reviewer content already
written) and the retry sees "absent". All existing synthesis-protocol
(14) and sequential-protocol (4) tests still pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

* fix(CC-572): add missing test docstrings on the two mode wrappers

lint-test-docstrings requires each test_* function to carry its own
'# Behavior:' marker directly above its declaration -- the shared helper's
docstring didn't satisfy that for the two thin per-mode wrappers, which
caused tests/bin/run-all-tests.sh to fail at phase 0 and skip 99 suites.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

* docs(BACKLOG): mark CC-572 done in the same PR that ships it

Following the lesson from CC-524/CC-567/CC-533/CC-015/CC-571: flip the
status inside the shipping PR, not as a follow-up after merge.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

* docs(BACKLOG): record PR #541 reference for CC-572

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

* docs(CC-572): trim comments/ticket text to why, not what

gate_clear_retry_target's own comment restated what the function does
(the name already says that); trimmed to keep only the why (the
Update-vs-Add-File patch-tool failure mode). BACKLOG.md's Requirement/
Update sections similarly dropped line-number-level implementation detail
that duplicates the code and would go stale as soon as the code moves --
kept only the outcome and the reasoning.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

* docs(CC-572): remove ticket-number references from code comments

Ticket refs belong in BACKLOG.md/DECISIONS.md, not in code/test comments,
which should describe current behavior on their own terms. Removed all
CC-572 mentions from runtime/bin/pr-gate.sh and tests/shell/test-pr-gate.sh
(comments and the _cc572_-prefixed variable names), rewording each to state
what the code does/why without the ticket citation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
screenleon added a commit that referenced this pull request Aug 26, 2026
…ns (#544)

* fix(ship): give the permanent-test admission record a PR body template slot

Second reading of CC-554's admission-recording rule (effective since
pr:#530): across a 12-PR evidence window, both qualifying instances found
(pr:#539's 7 new regression cases, pr:#541's 2 new test functions) added a
permanent test as a gate-finding remedy but recorded zero admission-criteria
citations in the PR body. The rule text lives in Step 2.5, several screens
before the PR body actually gets composed in Step 4 -- a free-floating
prose reminder read once during remediation is not reliably recalled by the
time the PR body template gets filled in. Add an explicit
`Permanent test admissions:` field to that template so its omission is
conspicuous (fill it in, or write `none`) instead of silent, and add a
regression assertion that the template carries the field.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

* fixup: clarify none is not a substitute for an alternative-path record

Two gate reviewers (critic, architecture-reviewer) independently flagged
that the template's `| none` fallback read as usable whenever no new
permanent test was added -- including a round whose finding was resolved
via a recorded alternative, which Step 2.5 still requires a line for.
Reworded so `none` applies only when no finding this round was assessed
against Step 2.5 at all, and added regression assertions for both the
alternative-path requirement and the none-is-not-a-substitute wording.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0177ds9nxEMtsh3bwpBxNHYx

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant