Skip to content

feat(AF-937): persist the effective executed SQL on the query snapshot - #1091

Merged
babltiga merged 2 commits into
mainfrom
feature/AF-937-snapshot-effective-sql
Sep 24, 2026
Merged

babltiga merged 2 commits into
mainfrom
feature/AF-937-snapshot-effective-sql

Conversation

@babltiga

Copy link
Copy Markdown
Contributor

Closes #937

What

Persists the statement as it actually executed — row-security predicates and soft-delete rewrites spliced in, every bound value left as a ? placeholder — on the immutable query snapshot, so an auditor sees what ran instead of reconstructing it from policy rows that may since have been edited or deleted.

  • Storage: new nullable query_snapshots.effective_sql (V187), updatable = false. RowSecurityRewriter already deparses predicate values as JdbcParameter (?), so no redaction code was needed — tests pin that no bound value ever appears.
  • Null rule: NULL when the rewrite returned the submitted SQL unchanged (no policy, no soft-delete). A soft-delete DELETE → UPDATE is stored — that's what ran.
  • Transactional batches (the open question): one text column; every statement's effective form joined by ; + newline, NULL when no statement was rewritten. Keeps the read UI and diff simple.
  • Plumbing: effectiveSql on core.api.SelectExecutionResult / UpdateExecutionResult → QueryExecutedEvent → QuerySnapshotListener → recordOnExecution(id, effectiveSql). A SELECT result-cache hit is re-stamped with this execution's effective SQL.
  • Engine plugins (MongoDB, Redis, …) always store NULL: they splice filters into native commands, so there is no redacted form. Pre-audit: persist the effective executed SQL on the query snapshot #937 record constructors are kept, so published plugin jars stay binary-compatible — no engine re-pin.
  • Read-back: effective_sql on GET /queries/{id} (existing gate: submitter or QUERY_VIEW_ALL); query detail SQL card gets a Submitted / Effective / Diff toggle reusing SqlDiffView; regulatory audit trail JSON + signed CSV (effective_sql column) + signed PDF (Effective SQL column) + auditor dashboard column.

Acceptance: a policied query stores the spliced predicate with ? (backend Postgres IT + e2e); an unpolicied query stores NULL; deleting the policy afterwards leaves the stored statement unchanged (e2e deletes the policy and re-reads).

Docs & website

  • docs/03-data-model.md, docs/04-api-spec.md, docs/05-backend.md, docs/06-frontend.md, docs/07-security.md
  • website/docs/configuration/datasources/index.html, website/docs/configuration/audit-compliance/index.html, website/sitemap.xml (dates bumped)
  • help-corpus/ regenerated

Verification

  • Backend mvn -o verify -Pcoverage: 10,270 tests, 0 failures (incl. ApplicationModulesTest, ApiPackageDependencyTest, Spotless, Checkstyle).
  • Frontend lint / typecheck / test:coverage (96.0% lines / 87.7% branches) / build; website guards; help-corpus drift clean.
  • E2E (local stack): row-security-policies.spec.ts (extended), auditor-compliance.spec.ts, query-replay.spec.ts — 9/9 passed.

Review notes

Independent reviewers (af-verifier, af-reviewer, af-java-reviewer, af-frontend-reviewer, af-content-reviewer) — no Blockers. Addressed in the second commit: soft-delete wording (af-content-reviewer, af-frontend-reviewer), a positive + org-scoped GET /queries/{id} integration case (af-java-reviewer), the auditor "—" fallback assertion and a class-free Segmented selector (af-frontend-reviewer).

Surviving concerns, for a human decision:

  • Submitter sees the predicate's shape (af-reviewer). Per the issue ("behind the existing permissions") effective_sql is visible to the submitter, so an analyst can now see which column filters them, the operator, how many ? an IN list carries, and 1 = 0 when their attribute didn't resolve — never the values. Documented in docs/07-security.md. If policy structure should stay admin-only, gating the field on QUERY_VIEW_ALL is a one-line change.
  • Coverage gaps, deliberate (af-java-reviewer, af-reviewer): request-group members (incl. schema-change promotions) execute without a snapshot, so they get no effective_sql (pre-existing, documented); the MCP query-detail tool does not carry the field.
  • API spec example shows "effective_sql": null although the backend omits nulls — kept to match the file's existing house style (af-content-reviewer nit).

Screenshots

Query detail — submitted SQL with the new toggle
Query detail — effective SQL with the row-security predicate and ? placeholder
Query detail — side-by-side diff of submitted vs effective
Auditor dashboard — regulatory audit trail with the Effective SQL column

@github-actions

Copy link
Copy Markdown
Contributor

Frontend Test Results

    1 files  ±0    309 suites  +1   6m 30s ⏱️ - 5m 30s
2 631 tests +3  2 631 ✅ +3  0 💤 ±0  0 ❌ ±0 
2 632 runs  +3  2 632 ✅ +3  0 💤 ±0  0 ❌ ±0 

Results for commit e767753. ± Comparison against base commit e310bfd.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage Report for Frontend Coverage (frontend)

Status Category Percentage Covered / Total
🟢 Lines 96.03% (🎯 90%) 3658 / 3809
🟢 Statements 94.65% (🎯 90%) 4059 / 4288
🟢 Functions 94.02% (🎯 90%) 1118 / 1189
🟢 Branches 87.69% (🎯 80%) 2430 / 2771
File CoverageNo changed files found.
Generated in workflow #1404 for commit e767753 by the Vitest Coverage Report Action

@github-actions

Copy link
Copy Markdown
Contributor

Backend Test Results

10 271 tests  +16   10 271 ✅ +16   11m 23s ⏱️ -17s
 1 141 suites + 1        0 💤 ± 0 
 1 141 files   + 1        0 ❌ ± 0 

Results for commit e767753. ± Comparison against base commit e310bfd.

@github-actions

Copy link
Copy Markdown
Contributor

Backend Code Coverage

Overall Project 94.74% 🍏
Files changed 100% 🍏

File Coverage
QuerySnapshotListener.java 100% 🍏
QuerySnapshotMapper.java 100% 🍏
RegulatoryAuditTrailRow.java 100% 🍏
QueryExecutedEvent.java 100% 🍏
UpdateExecutionResult.java 100% 🍏
SelectExecutionResult.java 100% 🍏
QuerySnapshotView.java 100% 🍏
DefaultComplianceReportService.java 96.83% 🍏
DefaultQueryExecutor.java 95.18% 🍏
DefaultQueryLifecycleService.java 94.65% 🍏
DefaultQuerySnapshotService.java 93.94% 🍏
QueryReadController.java 91.78% 🍏
QueryDetailResponse.java 84.85% 🍏
CompliancePdfWriter.java 80.7% 🍏
ComplianceReportResponse.java 78.93% 🍏
ComplianceCsvWriter.java 63.44% 🍏

@babltiga
babltiga merged commit a8e3809 into main Sep 24, 2026
35 checks passed
@babltiga
babltiga deleted the feature/AF-937-snapshot-effective-sql branch September 24, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

audit: persist the effective executed SQL on the query snapshot

1 participant