feat(AF-942): per-user data-volume budgets - #1105
Merged
Merged
Conversation
Datasource-scoped budgets bound the rows and result bytes each targeted user may read over a rolling window. Reads are capped at the allowance left; an exhausted budget rejects the SELECT or forces human review, and only a review the exhausted budget itself forced may run past it. Usage is an append-only ledger charged by query execution, request-group members and sample data, pruned by a locked job. Adds the data_budget_used_percent routing operand, the DATA_BUDGET decision step, DATA_BUDGET_MANAGE, audit actions and two notification events. Refs #942
Datasource settings gain a Data budgets tab; the query editor shows the caller's remaining allowance; the users page opens a per-user Data usage drawer. Adds the routing operand, trace step, truncation label, bell cases and an e2e spec. Refs #942
The Data budgets tab pushes Discovery/Classification out of the visible tab strip, where a physical click is swallowed; use the clickTab helper. Refs #942
Contributor
Frontend Test Results 1 files ± 0 319 suites +5 13m 28s ⏱️ -12s Results for commit b4c722e. ± Comparison against base commit cbee5ad. This pull request removes 2 and adds 40 tests. Note that renamed tests count towards both. |
Contributor
Coverage Report for Frontend Coverage (frontend)
File Coverage
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #942
What
Per-user data-volume budgets: a datasource-scoped limit on the rows and/or result bytes each targeted user may read over a rolling window (1 hour–31 days, default 24 h), with an action once it is used up. This is the slow-exfiltration control that per-query caps can't be: until now, a user allowed 1,000 rows per query could read a million in an afternoon.
These were the open questions in the issue; each was settled before implementation:
max_queries_per_dayprecedent — no reset job, no timezone)data_budget_usage) rather than sums overquery_requests, so request groups and table previews count tootruncated_reason=DATA_BUDGET); once exhausted, per-budgetREJECTorREQUIRE_REVIEWSemantics
applies_toroles / groups / users, the same model as row-limit policies (proxy: per-table row limits #934); an empty scope means everyone. Each user gets their own allowance: a group target is not a shared pool. When several budgets apply, the most constrained one wins, andREJECTbeatsREQUIRE_REVIEW.REJECTbudget moves the queryPENDING_AI → REJECTED, decided right after the proxy: bytes-scanned cost caps #941 bytes cap. An exhaustedREQUIRE_REVIEWbudget suppresses every auto-approve path, like a SQL review BLOCK, and never softensAUTO_REJECT. There is a newDATA_BUDGETdecision-trace step and an audit action,QUERY_DATA_BUDGET_ENFORCED.REJECTmeansAPPROVED → FAILED. UnderREQUIRE_REVIEWthe query runs only if the exhausted budget itself forced its review (query_requests.data_budget_review_forced, V195), so an approval obtained while allowance remained never lifts the budget. Request-group members fail closed. Table previews answer 409DATA_BUDGET_EXHAUSTED. Break-glass is counted but never capped or refused.data_budget_used_percent. Escalates heavy readers before they hit the limit; fails closed.DATA_BUDGET_THRESHOLD_REACHED) and on exhaustion (DATA_BUDGET_EXHAUSTED, which also goes toDATA_BUDGET_MANAGEholders).Interface surface
V193addsdata_budgets,data_budget_usageand two PG enums.V194seeds theDATA_BUDGET_MANAGEpermission.V195addsquery_requests.data_budget_review_forced./api/v1/datasources/{id}/data-budgets(CRUD),…/data-budgets/me, and/api/v1/admin/users/{id}/data-budget-usage.ACCESSFLOW_CORE_DATA_BUDGET_USAGE_RETENTION(defaultP32D) andACCESSFLOW_CORE_DATA_BUDGET_PRUNE_INTERVAL(defaultPT1H) driveDataBudgetUsagePruneJob, which holds a@SchedulerLock.QueryExecutionRequest.maxResultBytesOverrideandSelectExecutionResult.resultBytesare new; the previous canonical constructors are kept, so published engine plugins stay binary-compatible.Docs & website
docs/:03-data-model,04-api-spec,05-backend(including the job registry),06-frontend,07-security,08-notifications,09-deploymentand13-mcp.CLAUDE.md(the state machine) andREADME.md.website/docs/configuration/{datasources,review-workflows,users-roles}/index.htmlandwebsite/README.md.help-corpus/: regenerated.Verification
mvn verify -Pcoverageran 10,747 tests. The 2 failures were stale step-index assertions inAdminAccessSimulationControllerIntegrationTest(the trace gained a step); they are fixed and re-run green. After the review fixes, the affected suites were re-run green, including the newDataBudgetEnforcementIntegrationTestandDataBudgetControllerIntegrationTest(Testcontainers) andApplicationModulesTest/ApiPackageDependencyTest/MessagesParityTest.test:coverageran 2,738/2,738 at 94.7% statements / 87.8% branches.e2e/tests/data-budgets.spec.tscovers truncation, escalation to review, rejection, the editor indicator, the settings tab and the admin drawer. It passed 3×, repeated, together withdiscovery.spec.ts.discovery.spec.ts, was caused by this change: the new tab pushes Discovery out of the visible tab strip. It is fixed by switching to the existingclickTabhelper.Review notes
Five independent reviewers ran (
af-verifier,af-reviewer,af-java-reviewer,af-frontend-reviewer,af-content-reviewer). What they found and what was done:Fixed:
data_budget_review_forcedstamp.pg_advisory_xact_lock.tabpanel.window_minutes; a table-preview row cut was not attributed to the budget; the website used the wrong breach-action label.Surviving concerns, not addressed:
measureBytesre-estimates every SELECT row even when no budget applies./editorwithout the datasource pre-selected.admin-users-crud(row edit),reviews-rejectandws-realtimefailed on this machine. The users table is wider than the 1280px viewport, and a sticky onboarding panel overlays rows. These specs exercise flows this change does not touch, but I did not prove the failures also happen onmain.admin-review-plansfailed once in the parallel run and passed when re-run alone.Screenshots