diff --git a/CLAUDE.md b/CLAUDE.md index 9c4f1983c..bd25b7f27 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -349,6 +349,13 @@ nullable or has a DEFAULT. `ALTER TYPE … ADD VALUE` needs a `.sql.conf` sideca PENDING_AI → PENDING_REVIEW (bytes-scanned cap, no estimate, bytes_cap_missing_estimate= REQUIRE_REVIEW — #941; suppresses the same auto-approve paths as a SQL review BLOCK, never softens AUTO_REJECT) + PENDING_AI → REJECTED (data budget — #942; a SELECT whose submitter has used up an applying + per-user data budget with breach_action=REJECT. Decided right after the + bytes-scanned cap, no routing_decision row, QueryAutoRejectedEvent with a + null policy id; audited as QUERY_DATA_BUDGET_ENFORCED) + PENDING_AI → PENDING_REVIEW (data budget used up, breach_action=REQUIRE_REVIEW — #942; suppresses + the same auto-approve paths as a SQL review BLOCK, never softens + AUTO_REJECT) PENDING_REVIEW → APPROVED or REJECTED (external ticket resolution — AF-453; a channel with bidirectional_sync=true maps a ServiceNow/Jira ticket resolution onto a decision via workflow.api.ExternalDecisionService. System-attributed: @@ -373,7 +380,10 @@ nullable or has a DEFAULT. `ALTER TYPE … ADD VALUE` needs a `.sql.conf` sideca APPROVED → EXECUTED (break-glass run — audit action QUERY_BREAK_GLASS_EXECUTED — AF-385) APPROVED → FAILED (execution error; also the bytes-scanned cap re-checked just before execution refusing the run — #941, scheduled / recurring / - break-glass included) + break-glass included; an exhausted data budget re-checked there — + #942, REJECT always, REQUIRE_REVIEW unless the exhausted budget + itself forced the review (data_budget_review_forced); break-glass + is counted but never capped or refused by a budget) ``` Illegal transitions must throw a domain exception, not silently succeed. **Break-glass / diff --git a/README.md b/README.md index be8f70ec9..94cc351d0 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ A glance at the day-to-day flows engineers and approvers actually use. - **Proxy-first execution** — no user ever holds production credentials; the proxy holds them encrypted and opens connections only after approval. Single SQL statements run with autocommit; multi-statement INSERT/UPDATE/DELETE batches wrapped in `BEGIN; … COMMIT;` execute atomically inside one JDBC transaction (mixed SELECT/DML batches are rejected at parse time), with homogeneous INSERT runs collapsed into JDBC `executeBatch()` for bulk-load throughput. Optional **multi-replica read load balancing**: attach any number of replica endpoints to a datasource and SELECT traffic round-robins across the healthy ones — per-node health checks with circuit-breaker failover skip downed replicas, and only full replica-set exhaustion falls back to the primary (with an audit row). Optional **SELECT result caching**: opt a datasource into a Redis-backed result cache (per-datasource TTL) keyed over the security-rewritten query — masking and row-level security still apply — and invalidated on any proxied write to a referenced table. - **Configurable review workflows** — per-datasource review plans, multi-stage sequential approval chains, optional auto-approve for reads, approval timeouts with auto-reject. Reviewers can be scoped **per-datasource** (directly or via groups) so different teams see only the queues that belong to them. Reviewers going away can set an **out-of-office delegation** naming a colleague to cover their review duty for a window — across queries, governed API calls, and grouped requests — with every decision recording both identities. A delegate can never act on the delegator's own requests, delegation never grants a permission they lack, and it does not chain. Plans can also **escalate** a request that nobody has decided on to the reviewers at its current stage plus your admins before the approval timeout auto-rejects it, and **nudge** those same reviewers on a cadence — both optional, both notify-only, so waiting never changes who may approve. -- **Policy-as-code routing** — ordered, attribute-based routing policies decide a query's path after AI analysis and before reviewers see it: **auto-approve**, **auto-reject**, **require N approvals**, or **escalate**. Conditions match on query type, referenced tables (glob), AI risk level / score, requester role or group, time-of-day / day-of-week, WHERE / LIMIT presence, **query shape** (joins, set operations, subqueries, CTEs, GROUP BY / HAVING, aggregates, window functions), the transactional flag, the pre-flight estimate (**estimated rows, estimated bytes scanned**, scan type), and the submission **client context** — source IP / CIDR, user-agent, time-since-last-approval, and CI/CD origin (API key or `X-AccessFlow-CI` header) — combined with AND / OR / NOT. Client-context conditions **fail closed** (missing context never auto-approves), so an off-network or stale-approval query escalates to stricter review instead. First match by priority wins; on no match the query falls through to the datasource's review plan. Every automated decision is recorded in the audit log. +- **Policy-as-code routing** — ordered, attribute-based routing policies decide a query's path after AI analysis and before reviewers see it: **auto-approve**, **auto-reject**, **require N approvals**, or **escalate**. Conditions match on query type, referenced tables (glob), AI risk level / score, requester role or group, time-of-day / day-of-week, WHERE / LIMIT presence, **query shape** (joins, set operations, subqueries, CTEs, GROUP BY / HAVING, aggregates, window functions), the transactional flag, the pre-flight estimate (**estimated rows, estimated bytes scanned**, scan type), the submitter's **data-budget usage**, and the submission **client context** — source IP / CIDR, user-agent, time-since-last-approval, and CI/CD origin (API key or `X-AccessFlow-CI` header) — combined with AND / OR / NOT. Client-context conditions **fail closed** (missing context never auto-approves), so an off-network or stale-approval query escalates to stricter review instead. First match by priority wins; on no match the query falls through to the datasource's review plan. Every automated decision is recorded in the audit log. - **Policy simulator** — dry-run a draft routing, row-security, or masking policy against your own historical query traffic before you save it. AccessFlow replays the window **twice** — once against your current policies, once with the draft applied — and reports the difference between those two runs: how many past queries would change, which users would be affected, and which queries a row predicate would newly filter, deny, or reject outright. It is strictly read-only — no connection to your database, nothing executed, nothing persisted — and it names its own approximations (memberships are read as they are now; an engine that cannot classify a query shape offline is reported as *unclassifiable*, never as safe) instead of implying a precision the data does not have. - **Deterministic SQL review rules (#860)** — a named rule catalog that judges every SQL query alongside the AI verdict and routing policies, and shows its findings **in the editor as you type**. Fourteen built-in rules derived from the parsed statement alone (missing `WHERE` on `UPDATE` / `DELETE`, an always-true `WHERE`, `SELECT *`, unbounded reads, cross joins, leading-wildcard `LIKE`, `DROP` / `TRUNCATE` / DDL, banned functions, protected tables by glob, DML outside a transaction), each at an admin-set **`OFF` / `WARN` / `BLOCK`** severity configured per **environment** (`DEVELOPMENT` / `TEST` / `STAGING` / `PRODUCTION`, plus an organisation default). **`BLOCK` escalates, it never rejects**: a blocking finding suppresses every auto-approve path and sends the query to a human reviewer; a routing auto-reject still rejects. Evaluated synchronously at submission so findings exist even when AI analysis is off or fails, rendered in each reader's language on the query detail, review queue, break-glass retro-review and request-group detail; break-glass runs record findings but are never gated by them, and non-relational engines report *not applicable*. See [`docs/19-sql-review.md`](https://github.com/bablsoft/accessflow/blob/main/docs/19-sql-review.md). - **Just-in-time (JIT) access requests** — users self-request temporary, scoped access to a datasource (read/write/DDL, optional schema/table scope) or an API connection (read/write, optional operation allow-list) for an ISO-8601 duration. Requests flow through the same approval engine, a time-boxed permission is granted on approval, and a clustered scheduler auto-revokes it on expiry (admins can also revoke early). A grant can opt into **query pre-approval**: while it is active, queries it covers skip human review and are auto-approved with the grant recorded as the approval provenance — routing policies, high-risk AI verdicts, and behavioural anomalies still override. @@ -85,6 +85,7 @@ A glance at the day-to-day flows engineers and approvers actually use. - **Query playground / dry-run sandbox** — preview a query's impact before submitting it for review. A **Dry run** action returns the engine's execution plan (node type, estimated rows, cost, filters) and a best-effort estimated row impact **without executing or mutating data**, shown in the editor next to the AI panel. It runs through the same governance as a real execution (datasource access, table allow-list, row-level security) but creates no review. Dialect-aware across every engine with a plan concept — relational `EXPLAIN` (PostgreSQL / MySQL / MariaDB / Oracle / SQL Server), MongoDB `explain`, Couchbase / Neo4j `EXPLAIN`, Elasticsearch / OpenSearch query validation; engines without one degrade gracefully. Cloud warehouses additionally report the estimated bytes the query would scan (BigQuery's native dry-run job) — the direct cost signal for bytes-billed engines. SELECT dry-runs prefer the read replica when configured. - **Pre-flight cost & blast-radius estimation (AF-624)** — every submitted query automatically gets a persisted cost estimate before review: the engine's own `EXPLAIN` plan (estimated rows, scan type, cost) plus, for UPDATE/DELETE, an **exact affected-row count** computed with a governed, non-mutating count (relational `SELECT COUNT(*)` rewrite; MongoDB `countDocuments`, SQL++ `COUNT(*)`, Cypher `count(*)`, Elasticsearch `_count`). Reviewers see it on the query detail page ("this DELETE touches ~2.4M rows via Seq Scan"), routing policies can match on `estimated_rows` / `scan_type` (e.g. auto-escalate full-scan DELETEs over 100k rows), and the estimate is folded into the AI analyzer's prompt so risk scoring reflects the actual blast radius. Engines without a plan concept degrade gracefully to an "estimate unavailable" state that never blocks submission. - **Bytes-scanned cost caps** — on BigQuery, Snowflake and Databricks, where a ten-row result can scan a terabyte, set a maximum bytes-scanned per query on the datasource and, tighter, per grant. A query whose pre-flight scan estimate exceeds the cap is refused before it runs, with a message naming both numbers; the cap is checked again right before execution, so scheduled, recurring, grouped and break-glass runs honour it too. When no estimate exists, the datasource decides explicitly: send the query to a person, or refuse it. For an advisory signal instead, route on `estimated_bytes_scanned` to escalate expensive queries. +- **Per-user data-volume budgets** — bound how much data each person can read from a datasource over a rolling window (an hour up to 31 days): a row limit, a result-size limit, or both, for everyone or for chosen roles, groups or users — each person gets their own allowance, never a shared pool. While allowance remains, a query's result is capped to what is left; once it is used up, new reads are refused or sent to human review, per budget. Every delivered read counts — interactive, scheduled, recurring, grouped, table previews and break-glass (which is counted but never capped or blocked). Users see their remaining allowance in the query editor, get a warning past a threshold, and admins are told when someone runs out — the slow-exfiltration control that per-query caps cannot be. Route on `data_budget_used_percent` to escalate heavy readers before they hit the limit. - **Approval-likelihood prediction (AF-645)** — a triage signal for busy review queues. AccessFlow trains a small per-organization statistical model on your own historical review decisions and shows reviewers the probability that a query gets approved — a bare percentage badge on the review queue, and a labelled "Historical approval likelihood: 78%" card on the query detail page. No LLM call and no external service — it's plain logistic regression over data you already collect (query type, AI risk verdict, cost estimate, time of day, per-submitter and per-datasource approval history), retrained daily. **Advisory only**: it never approves or rejects anything, and it is never an input to routing, grant coverage, or any other decision path. Auto-approved, break-glass, and grant-covered queries are excluded from training because they carry no reviewer judgment, and until an organization has enough decided history for the model to clear its quality gate the UI says "not enough review history yet" rather than showing a number. - **Text-to-query** — opt-in per datasource. Users describe what they want in plain language ("order numbers for the last 5 days") and the AI drafts a schema-grounded query into the editor — in the datasource engine's **native query language** (SQL for relational engines, plus MongoDB shell/JSON, Cypher, CQL, Elasticsearch Query DSL, redis-cli, SQL++, and PartiQL for the NoSQL engines). Reuses the datasource's AI configuration and is grounded in its introspected schema (restricted columns are never referenced). The generated query is only a draft — it's still submitted through the normal pipeline, so AI risk analysis and human review always apply. - **RAG knowledge base** — opt-in per AI configuration. Attach knowledge documents (data-governance policies, naming conventions, schema notes); AccessFlow embeds them with a dedicated embedding model and, at analysis / text-to-SQL time, retrieves the most relevant chunks and injects them into the prompt — so analysis reflects your house rules. Pluggable vector store: in-app **pgvector** (self-contained, auto-provisioned where the DB role permits — and optional: AccessFlow still starts if the extension is absent, with the in-app store disabled) or external **Qdrant**. Retrieval is best-effort and never blocks analysis. diff --git a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java index 415bec54b..1f2976bb9 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java +++ b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java @@ -140,6 +140,15 @@ public enum AuditAction { * {@code estimated_bytes} and {@code outcome}. */ QUERY_BYTES_SCANNED_CAP_ENFORCED, + /** + * An exhausted data-volume budget (#942) refused a query or turned an automatic approval into + * human review ({@code stage=decision}), or refused an execution ({@code stage=execution}). + */ + QUERY_DATA_BUDGET_ENFORCED, + /** A data-volume budget (#942) was created, updated or deleted. */ + DATA_BUDGET_CREATED, + DATA_BUDGET_UPDATED, + DATA_BUDGET_DELETED, ROW_SECURITY_POLICY_CREATED, ROW_SECURITY_POLICY_UPDATED, ROW_SECURITY_POLICY_DELETED, diff --git a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java index fc969c3c5..48ba16088 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java +++ b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java @@ -37,6 +37,7 @@ public enum AuditResourceType { SERVICE_ACCOUNT("service_account"), ROW_SECURITY_POLICY("row_security_policy"), ROW_LIMIT_POLICY("row_limit_policy"), + DATA_BUDGET("data_budget"), CONNECTOR("connector"), QUERY_COMMENT("query_comment"), DATA_CLASSIFICATION_TAG("data_classification_tag"), diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetAdminService.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetAdminService.java new file mode 100644 index 000000000..5b8bc0b51 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetAdminService.java @@ -0,0 +1,21 @@ +package com.bablsoft.accessflow.core.api; + +import java.util.List; +import java.util.UUID; + +/** + * Admin CRUD for per-user data-volume budgets on a datasource (#942). All methods are + * organization-scoped: a datasource outside {@code organizationId} raises + * {@link DatasourceNotFoundException}; {@code applies_to} targets must belong to the organization. + */ +public interface DataBudgetAdminService { + + List listForDatasource(UUID datasourceId, UUID organizationId); + + DataBudgetView create(UUID datasourceId, UUID organizationId, DataBudgetCommand command); + + DataBudgetView update(UUID budgetId, UUID datasourceId, UUID organizationId, + DataBudgetCommand command); + + void delete(UUID budgetId, UUID datasourceId, UUID organizationId); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetBreachAction.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetBreachAction.java new file mode 100644 index 000000000..0da63f7f6 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetBreachAction.java @@ -0,0 +1,20 @@ +package com.bablsoft.accessflow.core.api; + +/** + * What happens to a SELECT once a data budget (#942) is exhausted. {@link #REJECT} beats + * {@link #REQUIRE_REVIEW} when several exhausted budgets disagree. + */ +public enum DataBudgetBreachAction { + REJECT, + REQUIRE_REVIEW; + + public static DataBudgetBreachAction strictest(DataBudgetBreachAction a, DataBudgetBreachAction b) { + if (a == null) { + return b; + } + if (b == null) { + return a; + } + return a == REJECT || b == REJECT ? REJECT : REQUIRE_REVIEW; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetCommand.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetCommand.java new file mode 100644 index 000000000..f1ec83386 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetCommand.java @@ -0,0 +1,22 @@ +package com.bablsoft.accessflow.core.api; + +import java.util.List; +import java.util.UUID; + +/** + * Create/update payload for a data budget (#942). Updates are total: every field is re-applied. + * At least one of {@code maxRows} / {@code maxBytes} must be set; {@code windowMinutes} and + * {@code breachAction} fall back to 1440 and {@link DataBudgetBreachAction#REQUIRE_REVIEW}. + */ +public record DataBudgetCommand( + String name, + Long maxRows, + Long maxBytes, + Integer windowMinutes, + DataBudgetBreachAction breachAction, + Integer warnThresholdPercent, + List appliesToRoles, + List appliesToGroupIds, + List appliesToUserIds, + Boolean enabled) { +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetConsumption.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetConsumption.java new file mode 100644 index 000000000..914943302 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetConsumption.java @@ -0,0 +1,46 @@ +package com.bablsoft.accessflow.core.api; + +import java.util.UUID; + +/** + * One budget's consumption for one user over its trailing window (#942). A null limit means the + * budget does not bound that metric; the matching {@code remaining*} is then null too. + */ +public record DataBudgetConsumption( + UUID budgetId, + String name, + Long maxRows, + Long maxBytes, + int windowMinutes, + DataBudgetBreachAction breachAction, + Integer warnThresholdPercent, + long usedRows, + long usedBytes) { + + public Long remainingRows() { + return maxRows == null ? null : Math.max(0, maxRows - usedRows); + } + + public Long remainingBytes() { + return maxBytes == null ? null : Math.max(0, maxBytes - usedBytes); + } + + public boolean exhausted() { + return (maxRows != null && usedRows >= maxRows) || (maxBytes != null && usedBytes >= maxBytes); + } + + /** The larger of the rows and bytes percentages used, unclamped (may exceed 100). */ + public double usedPercent() { + return Math.max(percent(usedRows, maxRows), percent(usedBytes, maxBytes)); + } + + /** This consumption after {@code rows} / {@code bytes} more were read. */ + public DataBudgetConsumption plus(long rows, long bytes) { + return new DataBudgetConsumption(budgetId, name, maxRows, maxBytes, windowMinutes, + breachAction, warnThresholdPercent, usedRows + rows, usedBytes + bytes); + } + + private static double percent(long used, Long limit) { + return limit == null ? 0d : used * 100d / limit; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetException.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetException.java new file mode 100644 index 000000000..001480f15 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetException.java @@ -0,0 +1,9 @@ +package com.bablsoft.accessflow.core.api; + +public sealed class DataBudgetException extends RuntimeException + permits DataBudgetNotFoundException, IllegalDataBudgetException { + + protected DataBudgetException(String message) { + super(message); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetExhaustedException.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetExhaustedException.java new file mode 100644 index 000000000..f934f3326 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetExhaustedException.java @@ -0,0 +1,21 @@ +package com.bablsoft.accessflow.core.api; + +/** + * A read was refused because the reader's data budget (#942) is exhausted — at execution under + * {@link DataBudgetBreachAction#REJECT} (or {@code REQUIRE_REVIEW} without a human approval), and + * on the sample-data path, which has no review to escalate to. The message is the caller's + * localized explanation; execution paths record it as the failure reason. + */ +public class DataBudgetExhaustedException extends RuntimeException { + + private final transient DataBudgetConsumption budget; + + public DataBudgetExhaustedException(String message, DataBudgetConsumption budget) { + super(message); + this.budget = budget; + } + + public DataBudgetConsumption budget() { + return budget; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetNotFoundException.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetNotFoundException.java new file mode 100644 index 000000000..f71fecd75 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetNotFoundException.java @@ -0,0 +1,10 @@ +package com.bablsoft.accessflow.core.api; + +import java.util.UUID; + +public final class DataBudgetNotFoundException extends DataBudgetException { + + public DataBudgetNotFoundException(UUID id) { + super("Data budget not found: " + id); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetStatus.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetStatus.java new file mode 100644 index 000000000..09697fd22 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetStatus.java @@ -0,0 +1,72 @@ +package com.bablsoft.accessflow.core.api; + +import java.util.List; +import java.util.Objects; +import java.util.UUID; + +/** + * A user's effective data-budget standing on one datasource (#942): every enabled budget that + * applies to them, each over its own window. The effective view is the most constrained budget — + * the smallest remaining allowance per metric, exhausted when any budget is, and the strictest + * breach action among the exhausted ones. An empty status means no budget applies. + */ +public record DataBudgetStatus(UUID datasourceId, String datasourceName, + List budgets) { + + public DataBudgetStatus { + budgets = budgets == null ? List.of() : List.copyOf(budgets); + } + + public static DataBudgetStatus none(UUID datasourceId) { + return new DataBudgetStatus(datasourceId, null, List.of()); + } + + public boolean isEmpty() { + return budgets.isEmpty(); + } + + public boolean exhausted() { + return budgets.stream().anyMatch(DataBudgetConsumption::exhausted); + } + + /** The strictest action among exhausted budgets; null while none is exhausted. */ + public DataBudgetBreachAction breachAction() { + DataBudgetBreachAction action = null; + for (var budget : budgets) { + if (budget.exhausted()) { + action = DataBudgetBreachAction.strictest(action, budget.breachAction()); + } + } + return action; + } + + /** The exhausted budget that decides {@link #breachAction()}; null while none is exhausted. */ + public DataBudgetConsumption decidingBudget() { + var action = breachAction(); + if (action == null) { + return null; + } + return budgets.stream() + .filter(b -> b.exhausted() && b.breachAction() == action) + .findFirst() + .orElse(null); + } + + /** Smallest remaining row allowance across budgets bounding rows; null when none does. */ + public Long remainingRows() { + return budgets.stream().map(DataBudgetConsumption::remainingRows).filter(Objects::nonNull) + .min(Long::compare).orElse(null); + } + + /** Smallest remaining byte allowance across budgets bounding bytes; null when none does. */ + public Long remainingBytes() { + return budgets.stream().map(DataBudgetConsumption::remainingBytes).filter(Objects::nonNull) + .min(Long::compare).orElse(null); + } + + /** The highest percentage used across budgets; null when no budget applies. */ + public Double usedPercent() { + return budgets.stream().map(DataBudgetConsumption::usedPercent) + .max(Double::compare).orElse(null); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetStatusService.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetStatusService.java new file mode 100644 index 000000000..9a57e0c6e --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetStatusService.java @@ -0,0 +1,17 @@ +package com.bablsoft.accessflow.core.api; + +import java.util.List; +import java.util.UUID; + +/** + * Reads a user's data-budget standing (#942) from the append-only usage ledger — a trailing-window + * sum per budget, no counter table and no reset job. + */ +public interface DataBudgetStatusService { + + /** The user's standing on one datasource; {@link DataBudgetStatus#isEmpty()} when unbudgeted. */ + DataBudgetStatus statusFor(UUID datasourceId, UUID userId); + + /** The user's standing on every datasource of the organization where a budget applies. */ + List statusesForUser(UUID organizationId, UUID userId); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageRecord.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageRecord.java new file mode 100644 index 000000000..008526ad2 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageRecord.java @@ -0,0 +1,26 @@ +package com.bablsoft.accessflow.core.api; + +import java.util.Objects; +import java.util.UUID; + +/** + * One delivered SELECT result to charge against a user's data budgets (#942): the rows and + * estimated result bytes the user actually received, after row security and every cap. + */ +public record DataBudgetUsageRecord( + UUID userId, + UUID datasourceId, + long rowsRead, + long bytesRead, + DataBudgetUsageSource source, + UUID queryRequestId, + UUID requestGroupId) { + + public DataBudgetUsageRecord { + Objects.requireNonNull(userId, "userId"); + Objects.requireNonNull(datasourceId, "datasourceId"); + Objects.requireNonNull(source, "source"); + rowsRead = Math.max(0, rowsRead); + bytesRead = Math.max(0, bytesRead); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageService.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageService.java new file mode 100644 index 000000000..3cd346afa --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageService.java @@ -0,0 +1,13 @@ +package com.bablsoft.accessflow.core.api; + +/** + * Charges delivered reads to the usage ledger (#942). A read on a datasource where no budget + * applies to the user writes nothing. Crossing a budget's warn threshold or its limit publishes a + * {@code core.events.DataBudgetThresholdCrossedEvent}, computed statelessly from the before/after + * consumption, so no dedup table is needed. Runs in its own transaction: a failure never affects + * the read that was already delivered, and callers log rather than propagate it. + */ +public interface DataBudgetUsageService { + + void record(DataBudgetUsageRecord usage); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageSource.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageSource.java new file mode 100644 index 000000000..ade5bad30 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetUsageSource.java @@ -0,0 +1,8 @@ +package com.bablsoft.accessflow.core.api; + +/** Which read path delivered the rows a data-budget ledger entry (#942) counts. */ +public enum DataBudgetUsageSource { + QUERY, + REQUEST_GROUP, + SAMPLE_DATA +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetView.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetView.java new file mode 100644 index 000000000..aa8c178e8 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/DataBudgetView.java @@ -0,0 +1,28 @@ +package com.bablsoft.accessflow.core.api; + +import java.time.Instant; +import java.util.List; +import java.util.UUID; + +public record DataBudgetView( + UUID id, + UUID datasourceId, + String name, + Long maxRows, + Long maxBytes, + int windowMinutes, + DataBudgetBreachAction breachAction, + Integer warnThresholdPercent, + List appliesToRoles, + List appliesToGroupIds, + List appliesToUserIds, + boolean enabled, + Instant createdAt, + Instant updatedAt) { + + public DataBudgetView { + appliesToRoles = appliesToRoles == null ? List.of() : List.copyOf(appliesToRoles); + appliesToGroupIds = appliesToGroupIds == null ? List.of() : List.copyOf(appliesToGroupIds); + appliesToUserIds = appliesToUserIds == null ? List.of() : List.copyOf(appliesToUserIds); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/IllegalDataBudgetException.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/IllegalDataBudgetException.java new file mode 100644 index 000000000..b83618119 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/IllegalDataBudgetException.java @@ -0,0 +1,13 @@ +package com.bablsoft.accessflow.core.api; + +/** + * Raised when a data-budget create/update request is structurally invalid — no limit, a limit or + * window out of range, an unknown {@code applies_to} role, or a user/group outside the + * organization. The {@code message} is a resolved, localized string. + */ +public final class IllegalDataBudgetException extends DataBudgetException { + + public IllegalDataBudgetException(String message) { + super(message); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/Permission.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/Permission.java index e0aa10dc4..d075b402a 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/api/Permission.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/Permission.java @@ -65,6 +65,8 @@ public enum Permission { EXPORT_POLICY_MANAGE(PermissionGroup.DATA_POLICIES), /** Manage per-table row-limit policies (#934). */ ROW_LIMIT_POLICY_MANAGE(PermissionGroup.DATA_POLICIES), + /** Manage per-user data-volume budgets (#942). */ + DATA_BUDGET_MANAGE(PermissionGroup.DATA_POLICIES), /** Manage review plans. */ REVIEW_PLAN_MANAGE(PermissionGroup.WORKFLOW_ADMIN), diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryExecutionRequest.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryExecutionRequest.java index 2ef079184..2c325b3f7 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryExecutionRequest.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryExecutionRequest.java @@ -12,6 +12,10 @@ * {@code schema.table} or bare {@code table}) so the proxy can index cached SELECT results for * write-invalidation. An empty set means "tables unknown": SELECTs are then never cached and * writes purge the whole datasource cache (fail-safe). + * + *

{@code maxResultBytesOverride} (#942) is the reader's remaining data-budget byte allowance: + * the proxy trims a SELECT result past it (the first row is always kept) and marks it truncated + * with {@link SelectExecutionResult#TRUNCATED_DATA_BUDGET}. {@code null} leaves only the global cap. */ public record QueryExecutionRequest( UUID datasourceId, @@ -25,7 +29,8 @@ public record QueryExecutionRequest( boolean transactional, List statements, List softDeleteDirectives, - Set referencedTables) { + Set referencedTables, + Long maxResultBytesOverride) { public QueryExecutionRequest { Objects.requireNonNull(datasourceId, "datasourceId"); @@ -36,6 +41,9 @@ public record QueryExecutionRequest( if (maxRowsOverride != null && maxRowsOverride <= 0) { throw new IllegalArgumentException("maxRowsOverride must be positive"); } + if (maxResultBytesOverride != null && maxResultBytesOverride <= 0) { + throw new IllegalArgumentException("maxResultBytesOverride must be positive"); + } if (statementTimeoutOverride != null && (statementTimeoutOverride.isNegative() || statementTimeoutOverride.isZero())) { throw new IllegalArgumentException("statementTimeoutOverride must be positive"); @@ -66,6 +74,39 @@ public record QueryExecutionRequest( } } + /** Backward-compatible constructor without a result-byte override (#942). */ + public QueryExecutionRequest(UUID datasourceId, String sql, QueryType queryType, + Integer maxRowsOverride, Duration statementTimeoutOverride, + List restrictedColumns, List columnMasks, + List rowSecurityPredicates, + boolean transactional, List statements, + List softDeleteDirectives, + Set referencedTables) { + this(datasourceId, sql, queryType, maxRowsOverride, statementTimeoutOverride, + restrictedColumns, columnMasks, rowSecurityPredicates, transactional, statements, + softDeleteDirectives, referencedTables, null); + } + + /** + * Returns a copy whose row and result-byte caps are lowered to the given allowance (#942); + * a null argument leaves that cap unchanged, and neither can ever raise an existing cap. + */ + public QueryExecutionRequest withAllowance(Long maxRows, Long maxBytes) { + Integer rows = maxRowsOverride; + if (maxRows != null) { + int allowance = (int) Math.min(Integer.MAX_VALUE, Math.max(1, maxRows)); + rows = rows == null ? allowance : Math.min(rows, allowance); + } + Long bytes = maxResultBytesOverride; + if (maxBytes != null) { + long allowance = Math.max(1, maxBytes); + bytes = bytes == null ? allowance : Math.min(bytes, allowance); + } + return new QueryExecutionRequest(datasourceId, sql, queryType, rows, statementTimeoutOverride, + restrictedColumns, columnMasks, rowSecurityPredicates, transactional, statements, + softDeleteDirectives, referencedTables, bytes); + } + /** Backward-compatible constructor without referenced tables (defaults to unknown). */ public QueryExecutionRequest(UUID datasourceId, String sql, QueryType queryType, Integer maxRowsOverride, Duration statementTimeoutOverride, @@ -75,7 +116,7 @@ public QueryExecutionRequest(UUID datasourceId, String sql, QueryType queryType, List softDeleteDirectives) { this(datasourceId, sql, queryType, maxRowsOverride, statementTimeoutOverride, restrictedColumns, columnMasks, rowSecurityPredicates, transactional, statements, - softDeleteDirectives, Set.of()); + softDeleteDirectives, Set.of(), null); } public QueryExecutionRequest(UUID datasourceId, String sql, QueryType queryType, diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryRequestStateService.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryRequestStateService.java index 826e2d896..abbd024ca 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryRequestStateService.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/QueryRequestStateService.java @@ -33,6 +33,18 @@ public interface QueryRequestStateService { void recordBytesScannedCap(UUID queryRequestId, long limit, BytesScannedCapSource source, BytesScannedCapOutcome outcome); + /** + * Stamps that an exhausted {@code REQUIRE_REVIEW} data budget (#942) forced the query into + * human review as it left {@code PENDING_AI}. A plain stamp, not a transition. + */ + void recordDataBudgetReviewForced(UUID queryRequestId); + + /** + * Whether an exhausted data budget forced this query into review (#942) — the only case in which + * an approved query may run past an exhausted {@code REQUIRE_REVIEW} budget. + */ + boolean isDataBudgetReviewForced(UUID queryRequestId); + /** * Inserts an {@code APPROVED} {@link com.bablsoft.accessflow.core.api.DecisionType} row * for the given reviewer/stage and, if the per-stage threshold is now met AND it was the diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/api/SelectExecutionResult.java b/backend/src/main/java/com/bablsoft/accessflow/core/api/SelectExecutionResult.java index 78f377c1a..bd563f8cd 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/api/SelectExecutionResult.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/api/SelectExecutionResult.java @@ -14,12 +14,19 @@ public record SelectExecutionResult( Set appliedMaskingPolicyIds, Set appliedRowSecurityPolicyIds, String truncatedReason, - String effectiveSql) implements QueryExecutionResult { + String effectiveSql, + long resultBytes) implements QueryExecutionResult { /** {@link #truncatedReason()} value when the configured row cap cut the result short. */ public static final String TRUNCATED_ROW_LIMIT = "ROW_LIMIT"; /** {@link #truncatedReason()} value when the configured byte cap cut the result short. */ public static final String TRUNCATED_BYTE_LIMIT = "BYTE_LIMIT"; + /** + * {@link #truncatedReason()} value when the reader's remaining data-budget allowance (#942) + * cut the result short — set by the proxy's result-byte override trim and by the workflow when + * the budget's row allowance was the binding row cap. + */ + public static final String TRUNCATED_DATA_BUDGET = "DATA_BUDGET"; public SelectExecutionResult { columns = List.copyOf(columns); @@ -32,14 +39,14 @@ public record SelectExecutionResult( public SelectExecutionResult(List columns, List> rows, long rowCount, boolean truncated, Duration duration) { - this(columns, rows, rowCount, truncated, duration, Set.of(), Set.of(), null, null); + this(columns, rows, rowCount, truncated, duration, Set.of(), Set.of(), null, null, 0L); } public SelectExecutionResult(List columns, List> rows, long rowCount, boolean truncated, Duration duration, Set appliedMaskingPolicyIds) { this(columns, rows, rowCount, truncated, duration, appliedMaskingPolicyIds, Set.of(), null, - null); + null, 0L); } public SelectExecutionResult(List columns, List> rows, long rowCount, @@ -47,7 +54,7 @@ public SelectExecutionResult(List columns, List> rows Set appliedMaskingPolicyIds, Set appliedRowSecurityPolicyIds) { this(columns, rows, rowCount, truncated, duration, appliedMaskingPolicyIds, - appliedRowSecurityPolicyIds, null, null); + appliedRowSecurityPolicyIds, null, null, 0L); } /** Pre-#937 canonical shape — kept so published engine plugins stay binary-compatible. */ @@ -56,13 +63,23 @@ public SelectExecutionResult(List columns, List> rows Set appliedMaskingPolicyIds, Set appliedRowSecurityPolicyIds, String truncatedReason) { this(columns, rows, rowCount, truncated, duration, appliedMaskingPolicyIds, - appliedRowSecurityPolicyIds, truncatedReason, null); + appliedRowSecurityPolicyIds, truncatedReason, null, 0L); + } + + /** Pre-#942 canonical shape — kept so published engine plugins stay binary-compatible. */ + public SelectExecutionResult(List columns, List> rows, long rowCount, + boolean truncated, Duration duration, + Set appliedMaskingPolicyIds, + Set appliedRowSecurityPolicyIds, String truncatedReason, + String effectiveSql) { + this(columns, rows, rowCount, truncated, duration, appliedMaskingPolicyIds, + appliedRowSecurityPolicyIds, truncatedReason, effectiveSql, 0L); } /** Returns a copy of this result with the given row-security policy ids attached. */ public SelectExecutionResult withRowSecurityPolicyIds(Set ids) { return new SelectExecutionResult(columns, rows, rowCount, truncated, duration, - appliedMaskingPolicyIds, ids, truncatedReason, effectiveSql); + appliedMaskingPolicyIds, ids, truncatedReason, effectiveSql, resultBytes); } /** @@ -72,6 +89,30 @@ public SelectExecutionResult withRowSecurityPolicyIds(Set ids) { */ public SelectExecutionResult withEffectiveSql(String sql) { return new SelectExecutionResult(columns, rows, rowCount, truncated, duration, - appliedMaskingPolicyIds, appliedRowSecurityPolicyIds, truncatedReason, sql); + appliedMaskingPolicyIds, appliedRowSecurityPolicyIds, truncatedReason, sql, + resultBytes); + } + + /** + * Returns a copy holding only the first {@code keptRows} rows, flagged truncated for + * {@code reason}, with {@code bytes} as the delivered size — the proxy's byte trim (#942). + */ + public SelectExecutionResult truncatedTo(int keptRows, String reason, long bytes) { + return new SelectExecutionResult(columns, rows.subList(0, keptRows), keptRows, true, duration, + appliedMaskingPolicyIds, appliedRowSecurityPolicyIds, reason, effectiveSql, bytes); + } + + /** Returns a copy carrying the estimated delivered size in bytes (#942). */ + public SelectExecutionResult withResultBytes(long bytes) { + return new SelectExecutionResult(columns, rows, rowCount, truncated, duration, + appliedMaskingPolicyIds, appliedRowSecurityPolicyIds, truncatedReason, effectiveSql, + bytes); + } + + /** Returns a copy whose truncation is attributed to {@code reason}. */ + public SelectExecutionResult withTruncatedReason(String reason) { + return new SelectExecutionResult(columns, rows, rowCount, truncated, duration, + appliedMaskingPolicyIds, appliedRowSecurityPolicyIds, reason, effectiveSql, + resultBytes); } } diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/events/DataBudgetThresholdCrossedEvent.java b/backend/src/main/java/com/bablsoft/accessflow/core/events/DataBudgetThresholdCrossedEvent.java new file mode 100644 index 000000000..c8e9773f1 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/events/DataBudgetThresholdCrossedEvent.java @@ -0,0 +1,28 @@ +package com.bablsoft.accessflow.core.events; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; + +import java.util.UUID; + +/** + * A recorded read carried a user across one of their data budgets' marks (#942): its warn + * threshold ({@code exhausted=false}) or its limit ({@code exhausted=true}). Published once per + * crossing — a read that stays above a mark never re-publishes, and a read that jumps straight past + * the limit publishes only the exhausted crossing. + */ +public record DataBudgetThresholdCrossedEvent( + UUID organizationId, + UUID userId, + UUID datasourceId, + UUID budgetId, + String budgetName, + boolean exhausted, + Integer warnThresholdPercent, + int usedPercent, + Long maxRows, + Long maxBytes, + long usedRows, + long usedBytes, + int windowMinutes, + DataBudgetBreachAction breachAction) { +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/AppliesToMatcher.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/AppliesToMatcher.java new file mode 100644 index 000000000..8a339d558 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/AppliesToMatcher.java @@ -0,0 +1,46 @@ +package com.bablsoft.accessflow.core.internal; + +import java.util.Set; +import java.util.UUID; + +/** + * The {@code applies_to_roles / _group_ids / _user_ids} scope shared by row-limit policies (#934) + * and data budgets (#942): all three empty ⇒ every user; otherwise a match on any one list. + */ +final class AppliesToMatcher { + + private AppliesToMatcher() { + } + + static boolean matches(String[] roles, UUID[] groups, UUID[] users, UUID userId, + String roleName, Set groupIds) { + boolean hasRoles = roles != null && roles.length > 0; + boolean hasGroups = groups != null && groups.length > 0; + boolean hasUsers = users != null && users.length > 0; + if (!hasRoles && !hasGroups && !hasUsers) { + return true; + } + if (hasRoles && roleName != null) { + for (var allowed : roles) { + if (allowed != null && roleName.equalsIgnoreCase(allowed.trim())) { + return true; + } + } + } + if (hasUsers) { + for (var allowed : users) { + if (userId.equals(allowed)) { + return true; + } + } + } + if (hasGroups) { + for (var allowed : groups) { + if (groupIds.contains(allowed)) { + return true; + } + } + } + return false; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetAdminService.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetAdminService.java new file mode 100644 index 000000000..7b19089e6 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetAdminService.java @@ -0,0 +1,195 @@ +package com.bablsoft.accessflow.core.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetAdminService; +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetCommand; +import com.bablsoft.accessflow.core.api.DataBudgetNotFoundException; +import com.bablsoft.accessflow.core.api.DataBudgetView; +import com.bablsoft.accessflow.core.api.DatasourceNotFoundException; +import com.bablsoft.accessflow.core.api.IllegalDataBudgetException; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.RoleRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserGroupRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserRepository; +import lombok.RequiredArgsConstructor; +import org.springframework.context.MessageSource; +import org.springframework.context.i18n.LocaleContextHolder; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +@Service +@RequiredArgsConstructor +class DefaultDataBudgetAdminService implements DataBudgetAdminService { + + static final int MIN_WINDOW_MINUTES = 60; + static final int MAX_WINDOW_MINUTES = 44_640; + static final int DEFAULT_WINDOW_MINUTES = 1_440; + static final int MAX_NAME_LENGTH = 120; + + private final DataBudgetRepository dataBudgetRepository; + private final RoleRepository roleRepository; + private final DatasourceRepository datasourceRepository; + private final UserRepository userRepository; + private final UserGroupRepository userGroupRepository; + private final MessageSource messageSource; + + @Override + @Transactional(readOnly = true) + public List listForDatasource(UUID datasourceId, UUID organizationId) { + requireDatasourceInOrganization(datasourceId, organizationId); + return dataBudgetRepository + .findAllByOrganizationIdAndDatasourceIdOrderByCreatedAtAsc(organizationId, datasourceId) + .stream() + .map(DefaultDataBudgetAdminService::toView) + .toList(); + } + + @Override + @Transactional + public DataBudgetView create(UUID datasourceId, UUID organizationId, DataBudgetCommand command) { + requireDatasourceInOrganization(datasourceId, organizationId); + var entity = new DataBudgetEntity(); + entity.setId(UUID.randomUUID()); + entity.setOrganizationId(organizationId); + entity.setDatasourceId(datasourceId); + apply(entity, organizationId, command); + return toView(dataBudgetRepository.save(entity)); + } + + @Override + @Transactional + public DataBudgetView update(UUID budgetId, UUID datasourceId, UUID organizationId, + DataBudgetCommand command) { + var entity = loadInScope(budgetId, datasourceId, organizationId); + apply(entity, organizationId, command); + return toView(dataBudgetRepository.save(entity)); + } + + @Override + @Transactional + public void delete(UUID budgetId, UUID datasourceId, UUID organizationId) { + dataBudgetRepository.delete(loadInScope(budgetId, datasourceId, organizationId)); + } + + private void apply(DataBudgetEntity entity, UUID organizationId, DataBudgetCommand command) { + var name = command.name() == null ? "" : command.name().trim(); + if (name.isEmpty() || name.length() > MAX_NAME_LENGTH) { + throw new IllegalDataBudgetException(msg("error.data_budget.name_invalid")); + } + if (command.maxRows() == null && command.maxBytes() == null) { + throw new IllegalDataBudgetException(msg("error.data_budget.limit_required")); + } + if ((command.maxRows() != null && command.maxRows() < 1) + || (command.maxBytes() != null && command.maxBytes() < 1)) { + throw new IllegalDataBudgetException(msg("error.data_budget.limit_invalid")); + } + var window = command.windowMinutes() == null ? DEFAULT_WINDOW_MINUTES : command.windowMinutes(); + if (window < MIN_WINDOW_MINUTES || window > MAX_WINDOW_MINUTES) { + throw new IllegalDataBudgetException(msg("error.data_budget.window_invalid")); + } + var threshold = command.warnThresholdPercent(); + if (threshold != null && (threshold < 1 || threshold > 99)) { + throw new IllegalDataBudgetException(msg("error.data_budget.threshold_invalid")); + } + var roles = normalizeRoles(organizationId, command.appliesToRoles()); + validateAppliesToTargets(organizationId, command.appliesToUserIds(), + command.appliesToGroupIds()); + entity.setName(name); + entity.setMaxRows(command.maxRows()); + entity.setMaxBytes(command.maxBytes()); + entity.setWindowMinutes(window); + entity.setBreachAction(command.breachAction() == null + ? DataBudgetBreachAction.REQUIRE_REVIEW : command.breachAction()); + entity.setWarnThresholdPercent(threshold == null ? null : threshold.shortValue()); + entity.setAppliesToRoles(roles.isEmpty() ? null : roles.toArray(new String[0])); + entity.setAppliesToGroupIds(toUuidArray(command.appliesToGroupIds())); + entity.setAppliesToUserIds(toUuidArray(command.appliesToUserIds())); + entity.setEnabled(command.enabled() == null || command.enabled()); + } + + private DataBudgetEntity loadInScope(UUID budgetId, UUID datasourceId, UUID organizationId) { + requireDatasourceInOrganization(datasourceId, organizationId); + var entity = dataBudgetRepository.findByIdAndOrganizationId(budgetId, organizationId) + .orElseThrow(() -> new DataBudgetNotFoundException(budgetId)); + if (!entity.getDatasourceId().equals(datasourceId)) { + throw new DataBudgetNotFoundException(budgetId); + } + return entity; + } + + private void requireDatasourceInOrganization(UUID datasourceId, UUID organizationId) { + var datasource = datasourceRepository.findById(datasourceId) + .orElseThrow(() -> new DatasourceNotFoundException(datasourceId)); + if (!datasource.getOrganization().getId().equals(organizationId)) { + throw new DatasourceNotFoundException(datasourceId); + } + } + + private List normalizeRoles(UUID organizationId, List roles) { + if (roles == null || roles.isEmpty()) { + return List.of(); + } + var normalized = new ArrayList(roles.size()); + for (var role : roles) { + if (role == null || role.isBlank()) { + continue; + } + var resolved = roleRepository.findByNameInScope(organizationId, role.trim()) + .orElseThrow(() -> new IllegalDataBudgetException( + msg("error.data_budget.unknown_role", role))); + normalized.add(resolved.getName()); + } + return normalized; + } + + private void validateAppliesToTargets(UUID organizationId, List userIds, + List groupIds) { + if (userIds != null && !userIds.isEmpty()) { + var found = userRepository.findAllByOrganization_IdAndIdIn(organizationId, userIds); + if (found.size() != userIds.stream().distinct().count()) { + throw new IllegalDataBudgetException(msg("error.data_budget.user_not_in_org")); + } + } + if (groupIds != null && !groupIds.isEmpty()) { + var found = userGroupRepository.findAllByOrganization_IdAndIdIn(organizationId, + new ArrayList<>(groupIds)); + if (found.size() != groupIds.stream().distinct().count()) { + throw new IllegalDataBudgetException(msg("error.data_budget.group_not_in_org")); + } + } + } + + static DataBudgetView toView(DataBudgetEntity entity) { + return new DataBudgetView( + entity.getId(), + entity.getDatasourceId(), + entity.getName(), + entity.getMaxRows(), + entity.getMaxBytes(), + entity.getWindowMinutes(), + entity.getBreachAction(), + entity.getWarnThresholdPercent() == null + ? null : entity.getWarnThresholdPercent().intValue(), + entity.getAppliesToRoles() == null ? List.of() : List.of(entity.getAppliesToRoles()), + entity.getAppliesToGroupIds() == null ? List.of() : List.of(entity.getAppliesToGroupIds()), + entity.getAppliesToUserIds() == null ? List.of() : List.of(entity.getAppliesToUserIds()), + entity.isEnabled(), + entity.getCreatedAt(), + entity.getUpdatedAt()); + } + + private static UUID[] toUuidArray(List values) { + return values == null || values.isEmpty() + ? null : values.stream().distinct().toArray(UUID[]::new); + } + + private String msg(String key, Object... args) { + return messageSource.getMessage(key, args, LocaleContextHolder.getLocale()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetStatusService.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetStatusService.java new file mode 100644 index 000000000..0a1baa99d --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetStatusService.java @@ -0,0 +1,123 @@ +package com.bablsoft.accessflow.core.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetConsumption; +import com.bablsoft.accessflow.core.api.DataBudgetStatus; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DatasourceEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserGroupMembershipRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserRepository; +import lombok.RequiredArgsConstructor; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.time.Clock; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; + +@Service +@RequiredArgsConstructor +class DefaultDataBudgetStatusService implements DataBudgetStatusService { + + private final DataBudgetRepository dataBudgetRepository; + private final DataBudgetUsageRepository usageRepository; + private final DatasourceRepository datasourceRepository; + private final UserRepository userRepository; + private final UserGroupMembershipRepository membershipRepository; + private final Clock clock; + + @Override + @Transactional(readOnly = true) + public DataBudgetStatus statusFor(UUID datasourceId, UUID userId) { + var budgets = dataBudgetRepository.findAllByDatasourceIdAndEnabledTrue(datasourceId); + if (budgets.isEmpty()) { + return DataBudgetStatus.none(datasourceId); + } + var scope = scopeOf(userId); + var applying = budgets.stream().filter(b -> scope.matches(b, userId)).toList(); + if (applying.isEmpty()) { + return DataBudgetStatus.none(datasourceId); + } + var name = datasourceRepository.findById(datasourceId).map(DatasourceEntity::getName) + .orElse(null); + return evaluate(datasourceId, name, userId, applying); + } + + @Override + @Transactional(readOnly = true) + public List statusesForUser(UUID organizationId, UUID userId) { + var budgets = dataBudgetRepository.findAllByOrganizationIdAndEnabledTrue(organizationId); + if (budgets.isEmpty()) { + return List.of(); + } + var scope = scopeOf(userId); + var byDatasource = new LinkedHashMap>(); + budgets.stream() + .filter(b -> scope.matches(b, userId)) + .sorted(Comparator.comparing(DataBudgetEntity::getCreatedAt)) + .forEach(b -> byDatasource.computeIfAbsent(b.getDatasourceId(), k -> new ArrayList<>()) + .add(b)); + var names = new HashMap(); + datasourceRepository.findAllById(byDatasource.keySet()) + .forEach(ds -> names.put(ds.getId(), ds.getName())); + var statuses = new ArrayList(byDatasource.size()); + byDatasource.forEach((dsId, applying) -> + statuses.add(evaluate(dsId, names.get(dsId), userId, applying))); + statuses.sort(Comparator.comparing(s -> s.datasourceName() == null ? "" : s.datasourceName(), + String.CASE_INSENSITIVE_ORDER)); + return statuses; + } + + private DataBudgetStatus evaluate(UUID datasourceId, String datasourceName, UUID userId, + List applying) { + var now = clock.instant(); + // Budgets sharing a window share one ledger sum. + Map totalsByWindow = new HashMap<>(); + var consumptions = new ArrayList(applying.size()); + for (var budget : applying) { + var totals = totalsByWindow.computeIfAbsent(budget.getWindowMinutes(), + window -> usageRepository.sumSince(userId, datasourceId, + now.minus(Duration.ofMinutes(window)))); + consumptions.add(new DataBudgetConsumption( + budget.getId(), + budget.getName(), + budget.getMaxRows(), + budget.getMaxBytes(), + budget.getWindowMinutes(), + budget.getBreachAction(), + budget.getWarnThresholdPercent() == null + ? null : budget.getWarnThresholdPercent().intValue(), + valueOf(totals == null ? null : totals.getRowsRead()), + valueOf(totals == null ? null : totals.getBytesRead()))); + } + return new DataBudgetStatus(datasourceId, datasourceName, consumptions); + } + + private Scope scopeOf(UUID userId) { + var roleName = userRepository.findById(userId).map(u -> u.roleName()).orElse(null); + return new Scope(roleName, new HashSet<>(membershipRepository.findGroupIdsForUser(userId))); + } + + private static long valueOf(Long value) { + return value == null ? 0L : value; + } + + private record Scope(String roleName, Set groupIds) { + + boolean matches(DataBudgetEntity budget, UUID userId) { + return AppliesToMatcher.matches(budget.getAppliesToRoles(), budget.getAppliesToGroupIds(), + budget.getAppliesToUserIds(), userId, roleName, groupIds); + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetUsageService.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetUsageService.java new file mode 100644 index 000000000..fb7b33964 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetUsageService.java @@ -0,0 +1,106 @@ +package com.bablsoft.accessflow.core.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetConsumption; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageRecord; +import com.bablsoft.accessflow.core.api.DataBudgetUsageService; +import com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetUsageEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import lombok.RequiredArgsConstructor; +import org.springframework.context.ApplicationEventPublisher; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Propagation; +import org.springframework.transaction.annotation.Transactional; + +import java.time.Clock; +import java.util.Optional; +import java.util.UUID; + +@Service +@RequiredArgsConstructor +class DefaultDataBudgetUsageService implements DataBudgetUsageService { + + private final DataBudgetStatusService statusService; + private final DataBudgetRepository dataBudgetRepository; + private final DataBudgetUsageRepository usageRepository; + private final ApplicationEventPublisher eventPublisher; + private final Clock clock; + + @Override + @Transactional(propagation = Propagation.REQUIRES_NEW) + public void record(DataBudgetUsageRecord usage) { + // Without the lock two concurrent charges read the same `before`: a crossing that only + // their sum makes would never notify, and one both make would notify twice. + usageRepository.lockUserDatasource(lockKey(usage.userId(), usage.datasourceId())); + var before = statusService.statusFor(usage.datasourceId(), usage.userId()); + if (before.isEmpty()) { + return; + } + var organizationId = dataBudgetRepository.findById(before.budgets().getFirst().budgetId()) + .map(b -> b.getOrganizationId()) + .orElse(null); + if (organizationId == null) { + return; + } + var entry = new DataBudgetUsageEntity(); + entry.setId(UUID.randomUUID()); + entry.setOrganizationId(organizationId); + entry.setUserId(usage.userId()); + entry.setDatasourceId(usage.datasourceId()); + entry.setRowsRead(usage.rowsRead()); + entry.setBytesRead(usage.bytesRead()); + entry.setSource(usage.source()); + entry.setQueryRequestId(usage.queryRequestId()); + entry.setRequestGroupId(usage.requestGroupId()); + entry.setOccurredAt(clock.instant()); + usageRepository.save(entry); + for (var budget : before.budgets()) { + var after = budget.plus(usage.rowsRead(), usage.bytesRead()); + crossing(budget, after).ifPresent(exhausted -> eventPublisher.publishEvent( + toEvent(organizationId, usage, after, exhausted))); + } + } + + static long lockKey(UUID userId, UUID datasourceId) { + return userId.getMostSignificantBits() ^ Long.rotateLeft(userId.getLeastSignificantBits(), 17) + ^ Long.rotateLeft(datasourceId.getMostSignificantBits(), 31) + ^ datasourceId.getLeastSignificantBits(); + } + + /** Empty when no mark was crossed; true for the limit, false for the warn threshold. */ + static Optional crossing(DataBudgetConsumption before, + DataBudgetConsumption after) { + if (!before.exhausted() && after.exhausted()) { + return Optional.of(true); + } + var threshold = before.warnThresholdPercent(); + if (threshold != null && !after.exhausted() + && before.usedPercent() < threshold && after.usedPercent() >= threshold) { + return Optional.of(false); + } + return Optional.empty(); + } + + private static DataBudgetThresholdCrossedEvent toEvent(UUID organizationId, + DataBudgetUsageRecord usage, + DataBudgetConsumption after, + boolean exhausted) { + return new DataBudgetThresholdCrossedEvent( + organizationId, + usage.userId(), + usage.datasourceId(), + after.budgetId(), + after.name(), + exhausted, + after.warnThresholdPercent(), + (int) Math.min(100, Math.floor(after.usedPercent())), + after.maxRows(), + after.maxBytes(), + after.usedRows(), + after.usedBytes(), + after.windowMinutes(), + after.breachAction()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateService.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateService.java index 30788db44..7245d53a9 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateService.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateService.java @@ -79,6 +79,22 @@ public void recordBytesScannedCap(UUID queryRequestId, long limit, queryRequestRepository.save(entity); } + @Override + @Transactional + public void recordDataBudgetReviewForced(UUID queryRequestId) { + var entity = lockOrThrow(queryRequestId); + entity.setDataBudgetReviewForced(true); + queryRequestRepository.save(entity); + } + + @Override + @Transactional(readOnly = true) + public boolean isDataBudgetReviewForced(UUID queryRequestId) { + return queryRequestRepository.findById(queryRequestId) + .map(QueryRequestEntity::isDataBudgetReviewForced) + .orElse(false); + } + @Override @Transactional public RecordDecisionResult recordApprovalAndAdvance(RecordApprovalCommand command) { diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultRowLimitPolicyResolutionService.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultRowLimitPolicyResolutionService.java index 708c85ddd..10b6c4a8c 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultRowLimitPolicyResolutionService.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/DefaultRowLimitPolicyResolutionService.java @@ -77,36 +77,7 @@ static boolean matches(RowLimitPolicyEntity policy, String rawReference) { private static boolean appliesTo(RowLimitPolicyEntity policy, UUID userId, String roleName, Set groupIds) { - var roles = policy.getAppliesToRoles(); - var groups = policy.getAppliesToGroupIds(); - var users = policy.getAppliesToUserIds(); - boolean hasRoles = roles != null && roles.length > 0; - boolean hasGroups = groups != null && groups.length > 0; - boolean hasUsers = users != null && users.length > 0; - if (!hasRoles && !hasGroups && !hasUsers) { - return true; // empty scope = applies to every submitter - } - if (hasRoles && roleName != null) { - for (var allowed : roles) { - if (allowed != null && roleName.equalsIgnoreCase(allowed.trim())) { - return true; - } - } - } - if (hasUsers) { - for (var allowed : users) { - if (userId.equals(allowed)) { - return true; - } - } - } - if (hasGroups) { - for (var allowed : groups) { - if (groupIds.contains(allowed)) { - return true; - } - } - } - return false; + return AppliesToMatcher.matches(policy.getAppliesToRoles(), policy.getAppliesToGroupIds(), + policy.getAppliesToUserIds(), userId, roleName, groupIds); } } diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/config/CorePropertiesConfiguration.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/config/CorePropertiesConfiguration.java index e32a9e9e6..3bf8ddac4 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/internal/config/CorePropertiesConfiguration.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/config/CorePropertiesConfiguration.java @@ -5,6 +5,7 @@ @Configuration @EnableConfigurationProperties({EncryptionProperties.class, SecretsProperties.class, - ReviewDelegationProperties.class, PolicySimulationProperties.class}) + ReviewDelegationProperties.class, PolicySimulationProperties.class, + DataBudgetProperties.class}) class CorePropertiesConfiguration { } diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/config/DataBudgetProperties.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/config/DataBudgetProperties.java new file mode 100644 index 000000000..ecf56c44f --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/config/DataBudgetProperties.java @@ -0,0 +1,27 @@ +package com.bablsoft.accessflow.core.internal.config; + +import org.springframework.boot.context.properties.ConfigurationProperties; + +import java.time.Duration; + +/** + * Tuning for the data-budget usage ledger (#942). + * + * @param usageRetention how long ledger rows are kept. Must outlive the longest budget window + * (31 days), or a budget would under-count — shorter values are raised to + * the floor. + */ +@ConfigurationProperties("accessflow.core.data-budget") +public record DataBudgetProperties(Duration usageRetention) { + + public static final Duration MIN_RETENTION = Duration.ofDays(31).plusHours(1); + public static final Duration DEFAULT_RETENTION = Duration.ofDays(32); + + public DataBudgetProperties { + if (usageRetention == null) { + usageRetention = DEFAULT_RETENTION; + } else if (usageRetention.compareTo(MIN_RETENTION) < 0) { + usageRetention = MIN_RETENTION; + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/DataBudgetEntity.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/DataBudgetEntity.java new file mode 100644 index 000000000..974cbdc8b --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/DataBudgetEntity.java @@ -0,0 +1,91 @@ +package com.bablsoft.accessflow.core.internal.persistence.entity; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import jakarta.persistence.Access; +import jakarta.persistence.AccessType; +import jakarta.persistence.Column; +import jakarta.persistence.Entity; +import jakarta.persistence.EnumType; +import jakarta.persistence.Enumerated; +import jakarta.persistence.Id; +import jakarta.persistence.PreUpdate; +import jakarta.persistence.Table; +import jakarta.persistence.Version; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import org.hibernate.annotations.JdbcType; +import org.hibernate.annotations.JdbcTypeCode; +import org.hibernate.dialect.type.PostgreSQLEnumJdbcType; +import org.hibernate.type.SqlTypes; + +import java.time.Instant; +import java.util.UUID; + +@Entity +@Table(name = "data_budgets") +@Access(AccessType.FIELD) +@Getter +@Setter +@NoArgsConstructor +public class DataBudgetEntity { + + @Id + private UUID id; + + @Column(name = "organization_id", nullable = false) + private UUID organizationId; + + @Column(name = "datasource_id", nullable = false) + private UUID datasourceId; + + @Column(nullable = false, length = 120) + private String name; + + @Column(name = "max_rows") + private Long maxRows; + + @Column(name = "max_bytes") + private Long maxBytes; + + @Column(name = "window_minutes", nullable = false) + private int windowMinutes; + + @Enumerated(EnumType.STRING) + @JdbcType(PostgreSQLEnumJdbcType.class) + @Column(name = "breach_action", nullable = false, columnDefinition = "data_budget_breach_action") + private DataBudgetBreachAction breachAction; + + @Column(name = "warn_threshold_percent") + private Short warnThresholdPercent; + + @JdbcTypeCode(SqlTypes.ARRAY) + @Column(name = "applies_to_roles", columnDefinition = "text[]") + private String[] appliesToRoles; + + @JdbcTypeCode(SqlTypes.ARRAY) + @Column(name = "applies_to_group_ids", columnDefinition = "uuid[]") + private UUID[] appliesToGroupIds; + + @JdbcTypeCode(SqlTypes.ARRAY) + @Column(name = "applies_to_user_ids", columnDefinition = "uuid[]") + private UUID[] appliesToUserIds; + + @Column(nullable = false) + private boolean enabled = true; + + @Version + @Column(nullable = false) + private long version; + + @Column(name = "created_at", nullable = false, updatable = false) + private Instant createdAt = Instant.now(); + + @Column(name = "updated_at", nullable = false) + private Instant updatedAt = Instant.now(); + + @PreUpdate + void onUpdate() { + this.updatedAt = Instant.now(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/DataBudgetUsageEntity.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/DataBudgetUsageEntity.java new file mode 100644 index 000000000..be296160f --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/DataBudgetUsageEntity.java @@ -0,0 +1,61 @@ +package com.bablsoft.accessflow.core.internal.persistence.entity; + +import com.bablsoft.accessflow.core.api.DataBudgetUsageSource; +import jakarta.persistence.Access; +import jakarta.persistence.AccessType; +import jakarta.persistence.Column; +import jakarta.persistence.Entity; +import jakarta.persistence.EnumType; +import jakarta.persistence.Enumerated; +import jakarta.persistence.Id; +import jakarta.persistence.Table; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import org.hibernate.annotations.JdbcType; +import org.hibernate.dialect.type.PostgreSQLEnumJdbcType; + +import java.time.Instant; +import java.util.UUID; + +@Entity +@Table(name = "data_budget_usage") +@Access(AccessType.FIELD) +@Getter +@Setter +@NoArgsConstructor +public class DataBudgetUsageEntity { + + @Id + private UUID id; + + @Column(name = "organization_id", nullable = false, updatable = false) + private UUID organizationId; + + @Column(name = "user_id", nullable = false, updatable = false) + private UUID userId; + + @Column(name = "datasource_id", nullable = false, updatable = false) + private UUID datasourceId; + + @Column(name = "rows_read", nullable = false, updatable = false) + private long rowsRead; + + @Column(name = "bytes_read", nullable = false, updatable = false) + private long bytesRead; + + @Enumerated(EnumType.STRING) + @JdbcType(PostgreSQLEnumJdbcType.class) + @Column(name = "source", nullable = false, updatable = false, + columnDefinition = "data_budget_usage_source") + private DataBudgetUsageSource source; + + @Column(name = "query_request_id", updatable = false) + private UUID queryRequestId; + + @Column(name = "request_group_id", updatable = false) + private UUID requestGroupId; + + @Column(name = "occurred_at", nullable = false, updatable = false) + private Instant occurredAt; +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/QueryRequestEntity.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/QueryRequestEntity.java index 73d3c690e..99923e851 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/QueryRequestEntity.java +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/entity/QueryRequestEntity.java @@ -169,6 +169,11 @@ public class QueryRequestEntity { @Column(name = "bytes_scanned_cap_outcome", columnDefinition = "bytes_scanned_cap_outcome") private BytesScannedCapOutcome bytesScannedCapOutcome; + // #942: an exhausted REQUIRE_REVIEW data budget forced this query into review; only then may + // it run past the exhausted budget once approved. + @Column(name = "data_budget_review_forced", nullable = false) + private boolean dataBudgetReviewForced; + @Version @Column(name = "updated_at", nullable = false) private Instant updatedAt = Instant.now(); diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/repo/DataBudgetRepository.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/repo/DataBudgetRepository.java new file mode 100644 index 000000000..64711190c --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/repo/DataBudgetRepository.java @@ -0,0 +1,20 @@ +package com.bablsoft.accessflow.core.internal.persistence.repo; + +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetEntity; +import org.springframework.data.jpa.repository.JpaRepository; + +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +public interface DataBudgetRepository extends JpaRepository { + + List findAllByOrganizationIdAndDatasourceIdOrderByCreatedAtAsc( + UUID organizationId, UUID datasourceId); + + List findAllByDatasourceIdAndEnabledTrue(UUID datasourceId); + + List findAllByOrganizationIdAndEnabledTrue(UUID organizationId); + + Optional findByIdAndOrganizationId(UUID id, UUID organizationId); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/repo/DataBudgetUsageRepository.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/repo/DataBudgetUsageRepository.java new file mode 100644 index 000000000..4ef2f7831 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/persistence/repo/DataBudgetUsageRepository.java @@ -0,0 +1,43 @@ +package com.bablsoft.accessflow.core.internal.persistence.repo; + +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetUsageEntity; +import org.springframework.data.jpa.repository.JpaRepository; +import org.springframework.data.jpa.repository.Modifying; +import org.springframework.data.jpa.repository.Query; +import org.springframework.data.repository.query.Param; + +import java.time.Instant; +import java.util.UUID; + +public interface DataBudgetUsageRepository extends JpaRepository { + + /** Trailing-window totals for one user on one datasource; zeros when nothing was read. */ + @Query(""" + select coalesce(sum(u.rowsRead), 0) as rowsRead, + coalesce(sum(u.bytesRead), 0) as bytesRead + from DataBudgetUsageEntity u + where u.userId = :userId + and u.datasourceId = :datasourceId + and u.occurredAt >= :since + """) + UsageTotals sumSince(@Param("userId") UUID userId, + @Param("datasourceId") UUID datasourceId, + @Param("since") Instant since); + + /** + * Serializes charges for one (user, datasource) until the transaction ends, so the before-read + * and the insert of one charge are never interleaved with another's. + */ + @Query(value = "SELECT 1 FROM (SELECT pg_advisory_xact_lock(:key)) AS l", nativeQuery = true) + Integer lockUserDatasource(@Param("key") long key); + + @Modifying + @Query("delete from DataBudgetUsageEntity u where u.occurredAt < :cutoff") + int deleteOlderThan(@Param("cutoff") Instant cutoff); + + interface UsageTotals { + Long getRowsRead(); + + Long getBytesRead(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/core/internal/scheduled/DataBudgetUsagePruneJob.java b/backend/src/main/java/com/bablsoft/accessflow/core/internal/scheduled/DataBudgetUsagePruneJob.java new file mode 100644 index 000000000..0489bcb57 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/core/internal/scheduled/DataBudgetUsagePruneJob.java @@ -0,0 +1,39 @@ +package com.bablsoft.accessflow.core.internal.scheduled; + +import com.bablsoft.accessflow.core.internal.config.DataBudgetProperties; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import net.javacrumbs.shedlock.spring.annotation.SchedulerLock; +import org.springframework.scheduling.annotation.Scheduled; +import org.springframework.stereotype.Component; +import org.springframework.transaction.annotation.Transactional; + +import java.time.Clock; + +/** + * Deletes data-budget ledger rows (#942) older than {@code accessflow.core.data-budget + * .usage-retention}. Rows past the longest budget window can no longer count toward any budget. + */ +@Component +@RequiredArgsConstructor +@Slf4j +public class DataBudgetUsagePruneJob { + + private final DataBudgetUsageRepository usageRepository; + private final DataBudgetProperties properties; + private final Clock clock; + + @Scheduled(fixedDelayString = "${accessflow.core.data-budget.prune-interval:PT1H}") + @SchedulerLock(name = "dataBudgetUsagePruneJob", lockAtMostFor = "PT30M", lockAtLeastFor = "PT1M") + @Transactional + public void run() { + var cutoff = clock.instant().minus(properties.usageRetention()); + var deleted = usageRepository.deleteOlderThan(cutoff); + if (deleted > 0) { + log.info("Pruned {} data-budget usage rows older than {}", deleted, cutoff); + } else { + log.debug("No data-budget usage rows older than {}", cutoff); + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/api/NotificationEventType.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/api/NotificationEventType.java index 78032cd22..9b4afe183 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/api/NotificationEventType.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/api/NotificationEventType.java @@ -82,5 +82,17 @@ public enum NotificationEventType { */ SCHEMA_DRIFT_DETECTED, + /** + * A user's reads on a datasource crossed their data budget's warning threshold (#942). + * Advisory — notifies that user only; never pages, never opens a ticket. + */ + DATA_BUDGET_THRESHOLD_REACHED, + /** + * A user used up a data budget on a datasource (#942); further SELECTs are rejected or sent + * to review per the budget's breach action. Notifies the user and every + * {@code DATA_BUDGET_MANAGE} holder; never pages, never opens a ticket. + */ + DATA_BUDGET_EXHAUSTED, + TEST } diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotice.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotice.java new file mode 100644 index 000000000..9d081ea30 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotice.java @@ -0,0 +1,65 @@ +package com.bablsoft.accessflow.notifications.internal; + +import com.bablsoft.accessflow.core.api.ByteSizeFormat; +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; + +import java.util.Locale; +import java.util.UUID; + +/** + * The budget-specific part of a {@code DATA_BUDGET_*} notification (#942). A limit that is not set + * on the budget ({@code maxRows} / {@code maxBytes} null) renders no usage line at all. + */ +public record DataBudgetNotice( + UUID budgetId, + String budgetName, + boolean exhausted, + Integer warnThresholdPercent, + int usedPercent, + Long maxRows, + Long maxBytes, + long usedRows, + long usedBytes, + int windowMinutes, + DataBudgetBreachAction breachAction) { + + private static final int MINUTES_PER_HOUR = 60; + private static final int MINUTES_PER_DAY = 24 * MINUTES_PER_HOUR; + + /** {@code "1200 / 5000 rows"}, or null when the budget caps no rows. */ + public String rowsUsage() { + return maxRows == null ? null : usedRows + " / " + maxRows + " rows"; + } + + /** {@code "1.2 GB (…) / 5 GB (…)"}, or null when the budget caps no bytes. */ + public String bytesUsage() { + return maxBytes == null ? null + : ByteSizeFormat.format(usedBytes) + " / " + ByteSizeFormat.format(maxBytes); + } + + /** The largest whole unit the window divides into: {@code DAYS}, {@code HOURS} or {@code MINUTES}. */ + public String windowUnit() { + if (windowMinutes > 0 && windowMinutes % MINUTES_PER_DAY == 0) { + return "DAYS"; + } + if (windowMinutes > 0 && windowMinutes % MINUTES_PER_HOUR == 0) { + return "HOURS"; + } + return "MINUTES"; + } + + public int windowValue() { + return switch (windowUnit()) { + case "DAYS" -> windowMinutes / MINUTES_PER_DAY; + case "HOURS" -> windowMinutes / MINUTES_PER_HOUR; + default -> windowMinutes; + }; + } + + /** English window label for the chat channels, e.g. {@code "7 days"} or {@code "1 hour"}. */ + public String windowLabel() { + var value = windowValue(); + var unit = windowUnit().toLowerCase(Locale.ROOT); + return value + " " + (value == 1 ? unit.substring(0, unit.length() - 1) : unit); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotificationListener.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotificationListener.java new file mode 100644 index 000000000..c054ccc14 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotificationListener.java @@ -0,0 +1,30 @@ +package com.bablsoft.accessflow.notifications.internal; + +import com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.modulith.events.ApplicationModuleListener; +import org.springframework.stereotype.Component; + +/** + * Fans out data-budget crossings (#942). The usage service publishes inside its own + * {@code REQUIRES_NEW} transaction, so the after-commit module listener runs once the usage row is + * durable. Delivery is best-effort and never affects the read that crossed the mark. + */ +@Component +@RequiredArgsConstructor +@Slf4j +class DataBudgetNotificationListener { + + private final NotificationDispatcher dispatcher; + + @ApplicationModuleListener + void onThresholdCrossed(DataBudgetThresholdCrossedEvent event) { + try { + dispatcher.dispatchDataBudget(event); + } catch (RuntimeException ex) { + log.error("Failed to dispatch data-budget notification for budget {} and user {}", + event.budgetId(), event.userId(), ex); + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContext.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContext.java index ff245e81e..af64f95e8 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContext.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContext.java @@ -59,6 +59,9 @@ * promoter. {@code schemaChangeStatus} is the promotion status name, so a * {@code PARTIALLY_APPLIED} run can be told apart from a {@code FAILED} one, and * {@code schemaChangeErrorMessage} is the first failed statement's error. + * + *

{@code dataBudget} is only populated for the {@code DATA_BUDGET_*} events (#942) — see + * {@link #isDataBudgetEvent()}. */ public record NotificationContext( NotificationEventType eventType, @@ -114,7 +117,89 @@ public record NotificationContext( String schemaChangeSetName, String schemaChangeStatus, String schemaChangeErrorMessage, - Integer driftNewFindingCount) { + Integer driftNewFindingCount, + DataBudgetNotice dataBudget) { + + /** + * True for the #942 data-budget events. They reuse {@code datasourceId}/{@code datasourceName} + * for the budgeted datasource and the {@code submitter*} fields for the user whose budget it + * is; everything budget-specific rides in {@code dataBudget}. + */ + public boolean isDataBudgetEvent() { + return eventType == NotificationEventType.DATA_BUDGET_THRESHOLD_REACHED + || eventType == NotificationEventType.DATA_BUDGET_EXHAUSTED; + } + + /** Compatibility constructor without the #942 data-budget notice — every other path. */ + public NotificationContext( + NotificationEventType eventType, + UUID organizationId, + UUID queryRequestId, + QueryType queryType, + String fullSqlText, + String sqlPreview200, + String sqlPreview300, + RiskLevel riskLevel, + Integer riskScore, + String aiSummary, + UUID datasourceId, + String datasourceName, + UUID submittedByUserId, + String submitterEmail, + String submitterDisplayName, + String justification, + UUID reviewerUserId, + String reviewerDisplayName, + String reviewerComment, + URI reviewUrl, + List recipients, + Instant occurredAt, + String locale, + Integer approvalTimeoutHours, + UUID anomalyId, + String anomalyFeature, + Double anomalyScore, + Double anomalyObservedValue, + Double anomalyBaselineMean, + String anomalyUserLabel, + WeeklyDigestData digest, + UUID attestationCampaignId, + String attestationCampaignName, + Instant attestationDueAt, + UUID apiRequestId, + QueryStatus executionStatus, + Long executionRowsAffected, + Long executionDurationMs, + GrantResourceKind grantResourceKind, + Long grantDaysSinceLastUse, + GrantUsageRecommendation grantRecommendation, + String exportFormat, + String exportClassifications, + String exportTrigger, + UUID deploymentRequestId, + String environmentName, + String deploymentVersion, + DeploymentOutcome deploymentOutcome, + String deploymentDecisionReason, + UUID schemaChangePromotionId, + String schemaChangeSetName, + String schemaChangeStatus, + String schemaChangeErrorMessage, + Integer driftNewFindingCount) { + this(eventType, organizationId, queryRequestId, queryType, fullSqlText, sqlPreview200, + sqlPreview300, riskLevel, riskScore, aiSummary, datasourceId, datasourceName, + submittedByUserId, submitterEmail, submitterDisplayName, justification, + reviewerUserId, reviewerDisplayName, reviewerComment, reviewUrl, recipients, + occurredAt, locale, approvalTimeoutHours, anomalyId, anomalyFeature, anomalyScore, + anomalyObservedValue, anomalyBaselineMean, anomalyUserLabel, digest, + attestationCampaignId, attestationCampaignName, attestationDueAt, apiRequestId, + executionStatus, executionRowsAffected, executionDurationMs, grantResourceKind, + grantDaysSinceLastUse, grantRecommendation, exportFormat, exportClassifications, + exportTrigger, deploymentRequestId, environmentName, deploymentVersion, + deploymentOutcome, deploymentDecisionReason, schemaChangePromotionId, + schemaChangeSetName, schemaChangeStatus, schemaChangeErrorMessage, + driftNewFindingCount, null); + } /** * True for the #882 schema-change events, which render their own field set — "Datasource" diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilder.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilder.java index 0361db866..cdb24f718 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilder.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilder.java @@ -8,9 +8,11 @@ import com.bablsoft.accessflow.apigov.api.ApiRequestNotificationView; import com.bablsoft.accessflow.attestation.api.AttestationCampaignLookupService; import com.bablsoft.accessflow.compliance.events.SensitiveResultExportedEvent; +import com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent; import com.bablsoft.accessflow.core.api.AiAnalysisLookupService; import com.bablsoft.accessflow.core.api.ApproverRule; import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.core.api.DatasourceNotFoundException; import com.bablsoft.accessflow.core.api.LocalizationConfigService; import com.bablsoft.accessflow.core.api.QueryRequestLookupService; import com.bablsoft.accessflow.core.api.QueryRequestSnapshot; @@ -39,6 +41,7 @@ import java.net.URI; import java.time.Instant; import java.util.Comparator; +import java.util.LinkedHashSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Optional; @@ -191,7 +194,8 @@ private List resolveRecipients(NotificationEventType eventType, DEPLOYMENT_SUBMITTED, DEPLOYMENT_APPROVED, DEPLOYMENT_REJECTED, DEPLOYMENT_OUTCOME_FAILED, DEPLOYMENT_BREAK_GLASS_EXECUTED, SCHEMA_CHANGE_PROMOTION_SUBMITTED, SCHEMA_CHANGE_PROMOTION_APPLIED, - SCHEMA_CHANGE_PROMOTION_FAILED, SCHEMA_DRIFT_DETECTED -> List.of(); + SCHEMA_CHANGE_PROMOTION_FAILED, SCHEMA_DRIFT_DETECTED, + DATA_BUDGET_THRESHOLD_REACHED, DATA_BUDGET_EXHAUSTED -> List.of(); }; } @@ -673,6 +677,77 @@ Optional buildSchemaDrift(SchemaDriftDetectedEvent event) { event.newFindingCount())); } + /** + * Builds the context for a data-budget crossing (#942): the datasource in + * {@code datasourceId}/{@code datasourceName}, the budget's user in the {@code submitter*} + * fields and the usage in {@code dataBudget}. A warning reaches that user only; exhaustion also + * reaches every active {@code DATA_BUDGET_MANAGE} holder, deduplicated when the user is one. + * {@code reviewUrl} opens the query editor. Empty when the user is gone or inactive and nobody + * else is to be told. + */ + Optional buildDataBudget(DataBudgetThresholdCrossedEvent event) { + var eventType = event.exhausted() + ? NotificationEventType.DATA_BUDGET_EXHAUSTED + : NotificationEventType.DATA_BUDGET_THRESHOLD_REACHED; + var ids = new LinkedHashSet(); + ids.add(event.userId()); + if (event.exhausted()) { + ids.addAll(rolePermissionHolderLookupService + .findUserIdsWithPermission(event.organizationId(), Permission.DATA_BUDGET_MANAGE)); + } + var recipients = toActiveRecipients(List.copyOf(ids)); + if (recipients.isEmpty()) { + return Optional.empty(); + } + var user = userQueryService.findById(event.userId()).orElse(null); + var locale = localizationConfigService.getOrDefault(event.organizationId()).defaultLanguage(); + var notice = new DataBudgetNotice(event.budgetId(), event.budgetName(), event.exhausted(), + event.warnThresholdPercent(), event.usedPercent(), event.maxRows(), event.maxBytes(), + event.usedRows(), event.usedBytes(), event.windowMinutes(), event.breachAction()); + return Optional.of(new NotificationContext( + eventType, + event.organizationId(), + null, + null, null, null, null, + null, null, null, + event.datasourceId(), + datasourceName(event.datasourceId(), event.organizationId()), + event.userId(), + user != null ? user.email() : null, + user != null ? user.displayName() : null, + null, + null, null, null, + buildAppUrl("/editor"), + recipients, + Instant.now(), + locale, + null, + null, null, null, null, null, null, + null, + null, null, null, + null, + null, null, null, + null, null, null, + null, null, null, + null, null, null, null, null, + null, null, null, null, + null, + notice)); + } + + private String datasourceName(UUID datasourceId, UUID organizationId) { + if (datasourceId == null) { + return null; + } + try { + var datasource = datasourceAdminService.getForAdmin(datasourceId, organizationId); + return datasource != null ? datasource.name() : null; + } catch (DatasourceNotFoundException ex) { + log.debug("Datasource {} vanished before its data-budget notification", datasourceId); + return null; + } + } + private List schemaChangeRecipients(NotificationEventType eventType, SchemaChangePromotionNotificationView view) { if (eventType == NotificationEventType.SCHEMA_CHANGE_PROMOTION_SUBMITTED) { diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcher.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcher.java index 7a3e8046c..665cdf7bb 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcher.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcher.java @@ -3,6 +3,7 @@ import com.bablsoft.accessflow.notifications.api.NotificationChannelType; import com.bablsoft.accessflow.access.events.GrantStaleEvent; import com.bablsoft.accessflow.compliance.events.SensitiveResultExportedEvent; +import com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent; import com.bablsoft.accessflow.deploygov.api.DeploymentOutcome; import com.bablsoft.accessflow.notifications.api.NotificationEventType; import com.bablsoft.accessflow.schemachange.events.SchemaDriftDetectedEvent; @@ -216,6 +217,18 @@ void dispatchSchemaDrift(SchemaDriftDetectedEvent event) { deliver(NotificationEventType.SCHEMA_DRIFT_DETECTED, contextOpt.get()); } + /** Dispatch a {@code DATA_BUDGET_THRESHOLD_REACHED} / {@code DATA_BUDGET_EXHAUSTED} + * notification (#942) over all active org channels. */ + void dispatchDataBudget(DataBudgetThresholdCrossedEvent event) { + var contextOpt = contextBuilder.buildDataBudget(event); + if (contextOpt.isEmpty()) { + log.debug("Skipping data-budget notification for budget {} — no active recipient", + event.budgetId()); + return; + } + deliver(contextOpt.get().eventType(), contextOpt.get()); + } + private void deliver(NotificationEventType eventType, NotificationContext ctx) { recordInAppNotifications(ctx); var channels = resolveChannels(eventType, ctx); @@ -326,6 +339,15 @@ private String buildPayload(NotificationContext ctx) { if (ctx.datasourceName() != null) { payload.put("datasource", ctx.datasourceName()); } + if (ctx.dataBudget() != null) { + var budget = ctx.dataBudget(); + // #942: user_notifications has no datasource column — the bell reads it from here. + if (ctx.datasourceId() != null) { + payload.put("datasource_id", ctx.datasourceId().toString()); + } + payload.put("budget", budget.budgetName()); + payload.put("used_percent", budget.usedPercent()); + } if (ctx.submitterEmail() != null) { payload.put("submitter", ctx.submitterEmail()); } @@ -379,7 +401,10 @@ private List resolveChannels(NotificationEventType ev || eventType == NotificationEventType.SCHEMA_CHANGE_PROMOTION_SUBMITTED || eventType == NotificationEventType.SCHEMA_CHANGE_PROMOTION_APPLIED || eventType == NotificationEventType.SCHEMA_CHANGE_PROMOTION_FAILED - || eventType == NotificationEventType.SCHEMA_DRIFT_DETECTED) { + || eventType == NotificationEventType.SCHEMA_DRIFT_DETECTED + // #942: budgets are per user, not per review plan — advisory, org-wide. + || eventType == NotificationEventType.DATA_BUDGET_THRESHOLD_REACHED + || eventType == NotificationEventType.DATA_BUDGET_EXHAUSTED) { return channelRepository.findAllByOrganizationIdAndActiveTrue(ctx.organizationId()); } var planChannels = lookupPlanChannelIds(ctx); diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/codec/PagerDutyTrigger.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/codec/PagerDutyTrigger.java index 7fc8a3a53..8349dadaf 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/codec/PagerDutyTrigger.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/codec/PagerDutyTrigger.java @@ -27,7 +27,9 @@ * deliberately have no trigger either — routine lifecycle progress is not an incident. Nor do the * schema-change events ({@code SCHEMA_CHANGE_PROMOTION_*}, {@code SCHEMA_DRIFT_DETECTED}, #882): * a promotion's lifecycle is not an incident, and a drift finding carries no severity that could - * tell a critical divergence apart from a cosmetic one. + * tell a critical divergence apart from a cosmetic one. Nor do the data-budget events + * ({@code DATA_BUDGET_THRESHOLD_REACHED}, {@code DATA_BUDGET_EXHAUSTED}, #942): a user reaching a + * read quota is an advisory, not an incident. */ public enum PagerDutyTrigger { CRITICAL_RISK(NotificationEventType.AI_HIGH_RISK), diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetText.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetText.java new file mode 100644 index 000000000..6267aff92 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetText.java @@ -0,0 +1,52 @@ +package com.bablsoft.accessflow.notifications.internal.strategy; + +import com.bablsoft.accessflow.notifications.internal.NotificationContext; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * Shared field set for the #942 data-budget events, so Slack, Discord, Teams and Telegram show the + * same lines in the same order. Usage lines appear only for the limits the budget sets; the breach + * action only once the budget is exhausted, since that is when it takes effect. + */ +final class DataBudgetText { + + private DataBudgetText() { + } + + static List> fields(NotificationContext ctx) { + var budget = ctx.dataBudget(); + var fields = new ArrayList>(); + fields.add(Map.entry("Datasource", dash(ctx.datasourceName()))); + fields.add(Map.entry("User", dash(user(ctx)))); + if (budget == null) { + return fields; + } + fields.add(Map.entry("Budget", dash(budget.budgetName()))); + fields.add(Map.entry("Used", budget.usedPercent() + "%")); + if (budget.rowsUsage() != null) { + fields.add(Map.entry("Rows", budget.rowsUsage())); + } + if (budget.bytesUsage() != null) { + fields.add(Map.entry("Bytes", budget.bytesUsage())); + } + fields.add(Map.entry("Window", budget.windowLabel())); + if (budget.exhausted() && budget.breachAction() != null) { + fields.add(Map.entry("On breach", budget.breachAction().name())); + } + return fields; + } + + private static String user(NotificationContext ctx) { + if (ctx.submitterDisplayName() != null && ctx.submitterEmail() != null) { + return ctx.submitterDisplayName() + " (" + ctx.submitterEmail() + ")"; + } + return ctx.submitterEmail() != null ? ctx.submitterEmail() : ctx.submitterDisplayName(); + } + + private static String dash(String value) { + return value == null || value.isBlank() ? "—" : value; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/DiscordPayloadFactory.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/DiscordPayloadFactory.java index d0d4af49f..cfee2bfe7 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/DiscordPayloadFactory.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/DiscordPayloadFactory.java @@ -87,6 +87,18 @@ private static Map buildEventEmbed(NotificationContext ctx) { embed.put("fields", fields); return embed; } + // #942: data budgets — shared field set with the other chat channels. + if (ctx.isDataBudgetEvent()) { + for (var field : DataBudgetText.fields(ctx)) { + addField(fields, field.getKey(), field.getValue()); + } + if (ctx.reviewUrl() != null) { + addField(fields, "Open", ctx.reviewUrl().toString()); + embed.put("url", ctx.reviewUrl().toString()); + } + embed.put("fields", fields); + return embed; + } // #882: schema-change promotions and drift carry the pipeline in datasourceName too. if (ctx.isSchemaChangeEvent()) { if (ctx.schemaChangeSetName() != null) { @@ -213,6 +225,8 @@ private static String headerLabel(NotificationContext ctx) { case SCHEMA_CHANGE_PROMOTION_APPLIED -> "✅ Schema Change Applied"; case SCHEMA_CHANGE_PROMOTION_FAILED -> "🚨 Schema Change Failed"; case SCHEMA_DRIFT_DETECTED -> "⚠️ Schema Drift Detected"; + case DATA_BUDGET_THRESHOLD_REACHED -> "📊 Data Budget Warning Threshold Reached"; + case DATA_BUDGET_EXHAUSTED -> "🛑 Data Budget Exhausted"; }; } diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/EmailNotificationStrategy.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/EmailNotificationStrategy.java index 2c9ca6863..8567d433b 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/EmailNotificationStrategy.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/EmailNotificationStrategy.java @@ -258,6 +258,18 @@ private String renderHtml(String template, NotificationContext ctx, boolean resu context.setVariable("schemaChangeStatus", ctx.schemaChangeStatus()); context.setVariable("schemaChangeErrorMessage", ctx.schemaChangeErrorMessage()); context.setVariable("driftNewFindingCount", ctx.driftNewFindingCount()); + var budget = ctx.dataBudget(); + context.setVariable("dataBudgetName", budget != null ? budget.budgetName() : null); + context.setVariable("dataBudgetUsedPercent", budget != null ? budget.usedPercent() : null); + context.setVariable("dataBudgetWarnThresholdPercent", + budget != null ? budget.warnThresholdPercent() : null); + context.setVariable("dataBudgetUsedRows", budget != null ? budget.usedRows() : null); + context.setVariable("dataBudgetMaxRows", budget != null ? budget.maxRows() : null); + context.setVariable("dataBudgetBytesUsage", budget != null ? budget.bytesUsage() : null); + context.setVariable("dataBudgetWindowUnit", budget != null ? budget.windowUnit() : null); + context.setVariable("dataBudgetWindowValue", budget != null ? budget.windowValue() : null); + context.setVariable("dataBudgetBreachAction", + budget != null && budget.breachAction() != null ? budget.breachAction().name() : null); return templateEngine.process(template, context); } @@ -294,6 +306,9 @@ private static String templateName(NotificationEventType eventType) { case SCHEMA_CHANGE_PROMOTION_APPLIED -> "email/schema-change-promotion-applied"; case SCHEMA_CHANGE_PROMOTION_FAILED -> "email/schema-change-promotion-failed"; case SCHEMA_DRIFT_DETECTED -> "email/schema-drift-detected"; + // #942: data budgets — the user, and on exhaustion the budget managers. + case DATA_BUDGET_THRESHOLD_REACHED -> "email/data-budget-threshold-reached"; + case DATA_BUDGET_EXHAUSTED -> "email/data-budget-exhausted"; // Access (JIT) events are delivered as in-app notifications by AccessNotificationListener, // not through the channel-strategy email path — no email template. // API-request events (AF-500) deliver as in-app + chat notifications, not email. @@ -332,6 +347,11 @@ private String subject(NotificationContext ctx) { // #882: pipeline, environment, and how many findings the scan opened. case SCHEMA_DRIFT_DETECTED -> new Object[]{ctx.datasourceName(), ctx.environmentName(), ctx.driftNewFindingCount()}; + // #942: "{0} on {1}" is the budget and the datasource; the warning adds the percent. + case DATA_BUDGET_THRESHOLD_REACHED, DATA_BUDGET_EXHAUSTED -> new Object[]{ + ctx.dataBudget() != null ? ctx.dataBudget().budgetName() : null, + ctx.datasourceName(), + ctx.dataBudget() != null ? ctx.dataBudget().usedPercent() : null}; default -> new Object[]{ctx.datasourceName()}; }; return messageSource.getMessage(key, args, resolveLocale(ctx)); @@ -372,6 +392,9 @@ private static String subjectKey(NotificationEventType eventType) { case SCHEMA_CHANGE_PROMOTION_FAILED -> "notification.email.subject.schema_change_promotion_failed"; case SCHEMA_DRIFT_DETECTED -> "notification.email.subject.schema_drift_detected"; + case DATA_BUDGET_THRESHOLD_REACHED -> + "notification.email.subject.data_budget_threshold_reached"; + case DATA_BUDGET_EXHAUSTED -> "notification.email.subject.data_budget_exhausted"; // Unreachable for access events (no email template); kept for switch exhaustiveness. case TEST, ACCESS_REQUEST_SUBMITTED, ACCESS_REQUEST_APPROVED, ACCESS_REQUEST_REJECTED, ACCESS_GRANT_EXPIRED, ACCESS_GRANT_REVOKED, API_REQUEST_SUBMITTED, API_REQUEST_APPROVED, diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/MsTeamsPayloadFactory.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/MsTeamsPayloadFactory.java index 27510f720..573fe45af 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/MsTeamsPayloadFactory.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/MsTeamsPayloadFactory.java @@ -86,6 +86,22 @@ private static Map buildEventCard(NotificationContext ctx) { } return card; } + // #942: data budgets — shared field set with the other chat channels. + if (ctx.isDataBudgetEvent()) { + for (var field : DataBudgetText.fields(ctx)) { + facts.add(fact(field.getKey(), field.getValue())); + } + body.add(factSet(facts)); + card.put("body", body); + if (ctx.reviewUrl() != null) { + var action = new LinkedHashMap(); + action.put("type", "Action.OpenUrl"); + action.put("title", "Open the query editor"); + action.put("url", ctx.reviewUrl().toString()); + card.put("actions", List.of(action)); + } + return card; + } // #882: schema-change promotions and drift carry the pipeline in datasourceName too. if (ctx.isSchemaChangeEvent()) { if (ctx.schemaChangeSetName() != null) { @@ -263,6 +279,8 @@ private static String headerLabel(NotificationContext ctx) { case SCHEMA_CHANGE_PROMOTION_APPLIED -> "✅ Schema Change Applied"; case SCHEMA_CHANGE_PROMOTION_FAILED -> "🚨 Schema Change Failed"; case SCHEMA_DRIFT_DETECTED -> "⚠️ Schema Drift Detected"; + case DATA_BUDGET_THRESHOLD_REACHED -> "📊 Data Budget Warning Threshold Reached"; + case DATA_BUDGET_EXHAUSTED -> "🛑 Data Budget Exhausted"; }; } diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/PagerDutyPayloadFactory.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/PagerDutyPayloadFactory.java index 300a79b3e..bac061223 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/PagerDutyPayloadFactory.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/PagerDutyPayloadFactory.java @@ -131,6 +131,12 @@ private static Map buildCustomDetails(NotificationContext ctx) { details.put("new_finding_count", ctx.driftNewFindingCount()); } } + if (ctx.dataBudget() != null) { + var budget = ctx.dataBudget(); + details.put("budget_name", budget.budgetName()); + details.put("used_percent", budget.usedPercent()); + details.put("window_minutes", budget.windowMinutes()); + } if (ctx.anomalyId() != null) { details.put("anomaly_id", ctx.anomalyId().toString()); if (ctx.anomalyFeature() != null) { @@ -199,6 +205,11 @@ private static String summaryLine(NotificationContext ctx) { case SCHEMA_CHANGE_PROMOTION_FAILED -> "AccessFlow: schema change failed on pipeline " + datasource; case SCHEMA_DRIFT_DETECTED -> "AccessFlow: schema drift detected on pipeline " + datasource; + // #942: no PagerDutyTrigger maps the data-budget events, so neither pages — spelled + // out so one added later never falls into "for a query" below. + case DATA_BUDGET_THRESHOLD_REACHED -> + "AccessFlow: data budget warning threshold reached on " + datasource; + case DATA_BUDGET_EXHAUSTED -> "AccessFlow: data budget exhausted on " + datasource; case API_CONNECTOR_OAUTH2_TOKEN_FAILED -> "AccessFlow: OAuth2 token fetch repeatedly failing for connector " + datasource; default -> "AccessFlow: " + ctx.eventType().name() + " for a query on " + datasource; diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/SlackBlockKitFactory.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/SlackBlockKitFactory.java index 8a819b104..396249485 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/SlackBlockKitFactory.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/SlackBlockKitFactory.java @@ -99,6 +99,9 @@ private static SectionBlock summarySection(NotificationContext ctx) { if (ctx.isSchemaChangeEvent()) { return schemaChangeSection(ctx); } + if (ctx.isDataBudgetEvent()) { + return dataBudgetSection(ctx); + } var fields = new ArrayList(); fields.add(mrkdwn("*Datasource:*\n" + nullToDash(ctx.datasourceName()))); fields.add(mrkdwn("*Submitted by:*\n" + nullToDash(ctx.submitterEmail()))); @@ -175,6 +178,15 @@ private static SectionBlock schemaChangeSection(NotificationContext ctx) { return SectionBlock.builder().fields(fields).build(); } + // #942: data budgets — shared field set with the other chat channels. + private static SectionBlock dataBudgetSection(NotificationContext ctx) { + var fields = new ArrayList(); + for (var field : DataBudgetText.fields(ctx)) { + fields.add(mrkdwn("*" + field.getKey() + ":*\n" + field.getValue())); + } + return SectionBlock.builder().fields(fields).build(); + } + private static SectionBlock attestationSection(NotificationContext ctx) { var fields = new ArrayList(); fields.add(mrkdwn("*Campaign:*\n" + nullToDash(ctx.attestationCampaignName()))); @@ -280,6 +292,8 @@ private static String headerLabel(NotificationContext ctx) { case SCHEMA_CHANGE_PROMOTION_APPLIED -> "✅ Schema Change Applied"; case SCHEMA_CHANGE_PROMOTION_FAILED -> "🚨 Schema Change Failed"; case SCHEMA_DRIFT_DETECTED -> "⚠️ Schema Drift Detected"; + case DATA_BUDGET_THRESHOLD_REACHED -> "📊 Data Budget Warning Threshold Reached"; + case DATA_BUDGET_EXHAUSTED -> "🛑 Data Budget Exhausted"; }; } diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TelegramMessageFactory.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TelegramMessageFactory.java index d69244130..5ad33e411 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TelegramMessageFactory.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TelegramMessageFactory.java @@ -68,6 +68,17 @@ private static String buildEventText(NotificationContext ctx) { } return sb.toString(); } + // #942: data budgets — shared field set with the other chat channels. + if (ctx.isDataBudgetEvent()) { + for (var field : DataBudgetText.fields(ctx)) { + appendField(sb, field.getKey(), field.getValue()); + } + if (ctx.reviewUrl() != null) { + sb.append("\n[").append(escape("Open the query editor")).append("](") + .append(escapeUrl(ctx.reviewUrl().toString())).append(")"); + } + return sb.toString(); + } // #882: schema-change promotions and drift carry the pipeline in datasourceName too. if (ctx.isSchemaChangeEvent()) { if (ctx.schemaChangeSetName() != null) { @@ -183,6 +194,8 @@ private static String headerLabel(NotificationContext ctx) { case SCHEMA_CHANGE_PROMOTION_APPLIED -> "✅ Schema Change Applied"; case SCHEMA_CHANGE_PROMOTION_FAILED -> "🚨 Schema Change Failed"; case SCHEMA_DRIFT_DETECTED -> "⚠️ Schema Drift Detected"; + case DATA_BUDGET_THRESHOLD_REACHED -> "📊 Data Budget Warning Threshold Reached"; + case DATA_BUDGET_EXHAUSTED -> "🛑 Data Budget Exhausted"; }; } diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TicketDescriptionBuilder.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TicketDescriptionBuilder.java index ccaccd8eb..88f14a01e 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TicketDescriptionBuilder.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/TicketDescriptionBuilder.java @@ -31,6 +31,10 @@ static String summary(NotificationContext ctx) { case SCHEMA_CHANGE_PROMOTION_APPLIED -> "Schema change applied on pipeline " + datasource; case SCHEMA_CHANGE_PROMOTION_FAILED -> "Schema change failed on pipeline " + datasource; case SCHEMA_DRIFT_DETECTED -> "Schema drift detected on pipeline " + datasource; + // #942: no TicketingTrigger maps these either — spelled out for the same reason. + case DATA_BUDGET_THRESHOLD_REACHED -> + "Data budget warning threshold reached on " + datasource; + case DATA_BUDGET_EXHAUSTED -> "Data budget exhausted on " + datasource; default -> ctx.eventType().name() + " on " + datasource; }; var summary = "[AccessFlow] " + headline; diff --git a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/WebhookPayloadFactory.java b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/WebhookPayloadFactory.java index 395ddf61e..d2ebe2af6 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/WebhookPayloadFactory.java +++ b/backend/src/main/java/com/bablsoft/accessflow/notifications/internal/strategy/WebhookPayloadFactory.java @@ -74,6 +74,25 @@ String buildBody(NotificationContext ctx) { schemaChange.put("new_finding_count", ctx.driftNewFindingCount()); envelope.put("schema_change", schemaChange); } + // #942: data budgets — additive for the same reason. + if (ctx.dataBudget() != null) { + var budget = ctx.dataBudget(); + var dataBudget = new LinkedHashMap(); + dataBudget.put("budget_id", budget.budgetId()); + dataBudget.put("budget_name", budget.budgetName()); + dataBudget.put("datasource_id", ctx.datasourceId()); + dataBudget.put("user_id", ctx.submittedByUserId()); + dataBudget.put("exhausted", budget.exhausted()); + dataBudget.put("warn_threshold_percent", budget.warnThresholdPercent()); + dataBudget.put("used_percent", budget.usedPercent()); + dataBudget.put("max_rows", budget.maxRows()); + dataBudget.put("used_rows", budget.usedRows()); + dataBudget.put("max_bytes", budget.maxBytes()); + dataBudget.put("used_bytes", budget.usedBytes()); + dataBudget.put("window_minutes", budget.windowMinutes()); + dataBudget.put("breach_action", budget.breachAction()); + envelope.put("data_budget", dataBudget); + } return objectMapper.writeValueAsString(envelope); } diff --git a/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutor.java b/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutor.java index 07960ec71..46a298f71 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutor.java +++ b/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutor.java @@ -83,6 +83,9 @@ public QueryExecutionResult execute(QueryExecutionRequest request) { try (Observation.Scope ignored = observation.openScope()) { QueryExecutionResult result = executeInternal(request, descriptor, effectiveMaxRows, effectiveTimeout, execProps, observation); + if (result instanceof SelectExecutionResult select) { + result = measureBytes(select, request.maxResultBytesOverride()); + } observation.lowCardinalityKeyValue("outcome", "success"); return result; } catch (RuntimeException ex) { @@ -448,6 +451,25 @@ private Duration durationSince(Instant start) { return Duration.between(start, clock.instant()); } + /** + * Stamps the delivered size (#942) and, under a result-byte override — which only the data + * budget sets — trims the rows past it, attributing the cut to the budget. + * Runs over every SELECT result — JDBC, engine plugin, cache hit — so accounting is uniform. + * As in the row mapper, the first row is always kept. + */ + static SelectExecutionResult measureBytes(SelectExecutionResult result, Long maxBytes) { + long total = 0; + var rows = result.rows(); + for (int i = 0; i < rows.size(); i++) { + long next = total + ResultByteEstimator.estimateRow(rows.get(i)); + if (maxBytes != null && next > maxBytes && i > 0) { + return result.truncatedTo(i, SelectExecutionResult.TRUNCATED_DATA_BUDGET, total); + } + total = next; + } + return result.withResultBytes(total); + } + /** An override only ever lowers the cap — never above the datasource cap or global ceiling. */ private static int clampMaxRows(Integer override, int datasourceCap, int globalCap) { int candidate = override != null ? Math.min(override, datasourceCap) : datasourceCap; diff --git a/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataService.java b/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataService.java index d4d4727dd..305d178e7 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataService.java +++ b/backend/src/main/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataService.java @@ -2,6 +2,16 @@ import com.bablsoft.accessflow.core.api.AllowedTables; import com.bablsoft.accessflow.core.api.ColumnMaskDirective; +import com.bablsoft.accessflow.audit.api.AuditAction; +import com.bablsoft.accessflow.audit.api.AuditEntry; +import com.bablsoft.accessflow.audit.api.AuditLogService; +import com.bablsoft.accessflow.audit.api.AuditResourceType; +import com.bablsoft.accessflow.core.api.DataBudgetConsumption; +import com.bablsoft.accessflow.core.api.DataBudgetExhaustedException; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageRecord; +import com.bablsoft.accessflow.core.api.DataBudgetUsageService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageSource; import com.bablsoft.accessflow.core.api.DatabaseSchemaView; import com.bablsoft.accessflow.core.api.DatasourceAdminService; import com.bablsoft.accessflow.core.api.DatasourceUserPermissionLookupService; @@ -18,11 +28,13 @@ import com.bablsoft.accessflow.proxy.api.QueryExecutor; import com.bablsoft.accessflow.proxy.api.SampleDataService; import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; import org.springframework.context.MessageSource; import org.springframework.context.i18n.LocaleContextHolder; import org.springframework.security.access.AccessDeniedException; import org.springframework.stereotype.Service; +import java.util.HashMap; import java.util.List; import java.util.Objects; import java.util.Optional; @@ -32,6 +44,7 @@ @Service @RequiredArgsConstructor +@Slf4j class DefaultSampleDataService implements SampleDataService { private final DatasourceAdminService datasourceAdminService; @@ -41,6 +54,9 @@ class DefaultSampleDataService implements SampleDataService { private final RowLimitPolicyResolutionService rowLimitPolicyResolutionService; private final QueryExecutor queryExecutor; private final MessageSource messageSource; + private final DataBudgetStatusService dataBudgetStatusService; + private final DataBudgetUsageService dataBudgetUsageService; + private final AuditLogService auditLogService; @Override public SelectExecutionResult sample(UUID datasourceId, UUID organizationId, UUID userId, @@ -97,10 +113,68 @@ public SelectExecutionResult sample(UUID datasourceId, UUID organizationId, UUID rowLimitOverride = appliedRowLimit.get().tighten(rowLimitOverride); } + // #942: a preview reads real rows, so it spends the data budget like a query. It has no + // review to escalate to: an exhausted budget refuses it whatever the breach action. + var budget = dataBudgetStatusService.statusFor(datasourceId, userId); + if (budget.exhausted()) { + var deciding = budget.decidingBudget(); + auditRefusal(organizationId, datasourceId, userId, deciding, qualifiedName(target)); + throw new DataBudgetExhaustedException(messageSource.getMessage( + "error.data_budget.exhausted", new Object[]{deciding.name()}, + LocaleContextHolder.getLocale()), deciding); + } + var policyLimit = effectiveLimit(limit, rowLimitOverride); + var rowLimit = policyLimit; + if (budget.remainingRows() != null) { + rowLimit = (int) Math.min(rowLimit, Math.max(1, budget.remainingRows())); + } + // 3. Execute via the proxy executor — RLS rewrite + post-fetch masking + row cap + timeout. - return queryExecutor.sampleTable(new SampleTableRequest(datasourceId, target.schema(), + var result = queryExecutor.sampleTable(new SampleTableRequest(datasourceId, target.schema(), target.table(), restrictedColumns, columnMasks, rowSecurityPredicates, - effectiveLimit(limit, rowLimitOverride), null)); + rowLimit, null)); + if (budget.isEmpty()) { + return result; + } + var remainingBytes = budget.remainingBytes(); + var measured = DefaultQueryExecutor.measureBytes(result, + remainingBytes == null ? null : Math.max(1, remainingBytes)); + // The budget's row allowance was the binding cap: say so, as the query path does. + if (rowLimit < policyLimit && measured.truncated() + && SelectExecutionResult.TRUNCATED_ROW_LIMIT.equals(measured.truncatedReason()) + && measured.rowCount() == rowLimit) { + measured = measured.withTruncatedReason(SelectExecutionResult.TRUNCATED_DATA_BUDGET); + } + try { + dataBudgetUsageService.record(new DataBudgetUsageRecord(userId, datasourceId, + measured.rowCount(), measured.resultBytes(), DataBudgetUsageSource.SAMPLE_DATA, + null, null)); + } catch (RuntimeException ex) { + log.error("Data-budget usage write failed for a sample of datasource {}", datasourceId, + ex); + } + return measured; + } + + private void auditRefusal(UUID organizationId, UUID datasourceId, UUID userId, + DataBudgetConsumption deciding, String table) { + var metadata = new HashMap(); + metadata.put("trigger", "data_budget"); + metadata.put("stage", "sample"); + metadata.put("action", deciding.breachAction().name()); + metadata.put("data_budget_id", deciding.budgetId()); + metadata.put("used_rows", deciding.usedRows()); + metadata.put("used_bytes", deciding.usedBytes()); + metadata.put("window_minutes", deciding.windowMinutes()); + metadata.put("table", table); + try { + auditLogService.record(new AuditEntry(AuditAction.QUERY_DATA_BUDGET_ENFORCED, + AuditResourceType.DATASOURCE, datasourceId, organizationId, userId, metadata, + null, null)); + } catch (RuntimeException ex) { + log.error("Audit write failed for QUERY_DATA_BUDGET_ENFORCED on datasource {}", + datasourceId, ex); + } } /** The caller's row-limit override (#933) and row-limit policies (#934) cap the preview too, so it can't be used to get around the cap. */ diff --git a/backend/src/main/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionService.java b/backend/src/main/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionService.java index aec847e09..90ddff73d 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionService.java +++ b/backend/src/main/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionService.java @@ -12,6 +12,13 @@ import com.bablsoft.accessflow.core.api.ColumnMaskDirective; import com.bablsoft.accessflow.core.api.DatasourceLookupService; import com.bablsoft.accessflow.core.api.DatasourceUserPermissionLookupService; +import com.bablsoft.accessflow.core.api.DataBudgetExhaustedException; +import com.bablsoft.accessflow.core.api.DataBudgetStatus; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageRecord; +import com.bablsoft.accessflow.core.api.DataBudgetUsageService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageSource; +import com.bablsoft.accessflow.core.api.QueryType; import com.bablsoft.accessflow.core.api.DbType; import com.bablsoft.accessflow.core.api.MaskingPolicyResolutionService; import com.bablsoft.accessflow.core.api.QueryExecutionRequest; @@ -80,6 +87,8 @@ public class GroupExecutionService { private final MessageSource messageSource; private final QueryCostEstimateService queryCostEstimateService; private final GroupReviewDecisionRepository decisionRepository; + private final DataBudgetStatusService dataBudgetStatusService; + private final DataBudgetUsageService dataBudgetUsageService; /** Execute an APPROVED group. Idempotent: silently returns if it is not APPROVED (or not yet due). */ public void execute(UUID groupId, UUID actorUserId, String trigger) { @@ -191,7 +200,14 @@ private void runQuery(RequestGroupEntity group, RequestGroupItemEntity item) { null, restrictedColumns, columnMasks, rowSecurity, parsed.transactional(), parsed.statements(), List.of(), parsed.referencedTables()); enforceBytesScannedCap(group, item, request); + var budget = enforceDataBudget(group, item); + if (budget != null) { + request = request.withAllowance(budget.remainingRows(), budget.remainingBytes()); + } var result = queryExecutor.execute(request); + if (budget != null && result instanceof SelectExecutionResult select) { + chargeDataBudget(group, item, select); + } long rows = switch (result) { case SelectExecutionResult select -> select.rowCount(); case UpdateExecutionResult update -> update.rowsAffected(); @@ -289,6 +305,49 @@ private void enforceBytesScannedCap(RequestGroupEntity group, RequestGroupItemEn throw new BytesScannedCapExceededException(message, cap, estimated, outcome); } + /** + * The submitter's data budget (#942) for a SELECT member: {@code null} when none applies, + * otherwise the standing to cap the member to. An exhausted budget refuses the member whatever + * its action: a group's review never evaluates the budget, so an approval given while allowance + * remained must not lift it (fail closed). + */ + private DataBudgetStatus enforceDataBudget(RequestGroupEntity group, RequestGroupItemEntity item) { + if (item.getQueryType() != QueryType.SELECT) { + return null; + } + var status = dataBudgetStatusService.statusFor(item.getDatasourceId(), group.getSubmittedBy()); + if (status.isEmpty() || !status.exhausted()) { + return status.isEmpty() ? null : status; + } + var deciding = status.decidingBudget(); + var metadata = new HashMap(); + metadata.put("trigger", "data_budget"); + metadata.put("stage", "execution"); + metadata.put("item_id", item.getId().toString()); + metadata.put("action", status.breachAction().name()); + metadata.put("data_budget_id", deciding.budgetId()); + metadata.put("used_rows", deciding.usedRows()); + metadata.put("used_bytes", deciding.usedBytes()); + metadata.put("window_minutes", deciding.windowMinutes()); + audit(AuditAction.QUERY_DATA_BUDGET_ENFORCED, group, null, metadata); + throw new DataBudgetExhaustedException(messageSource.getMessage( + "error.data_budget.exhausted", new Object[]{deciding.name()}, + LocaleContextHolder.getLocale()), deciding); + } + + /** Never fails the member: the rows were already delivered. */ + private void chargeDataBudget(RequestGroupEntity group, RequestGroupItemEntity item, + SelectExecutionResult select) { + try { + dataBudgetUsageService.record(new DataBudgetUsageRecord(group.getSubmittedBy(), + item.getDatasourceId(), select.rowCount(), select.resultBytes(), + DataBudgetUsageSource.REQUEST_GROUP, null, group.getId())); + } catch (RuntimeException ex) { + log.error("Data-budget usage write failed for group {} member {}", group.getId(), + item.getId(), ex); + } + } + private void audit(AuditAction action, RequestGroupEntity group, UUID actorId, Map metadata) { try { diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/DataBudgetController.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/DataBudgetController.java new file mode 100644 index 000000000..b3eb90507 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/DataBudgetController.java @@ -0,0 +1,155 @@ +package com.bablsoft.accessflow.security.internal.web; + +import com.bablsoft.accessflow.audit.api.AuditAction; +import com.bablsoft.accessflow.audit.api.AuditEntry; +import com.bablsoft.accessflow.audit.api.AuditLogService; +import com.bablsoft.accessflow.audit.api.AuditResourceType; +import com.bablsoft.accessflow.audit.api.RequestAuditContext; +import com.bablsoft.accessflow.core.api.DataBudgetAdminService; +import com.bablsoft.accessflow.core.api.DataBudgetCommand; +import com.bablsoft.accessflow.core.api.DataBudgetView; +import com.bablsoft.accessflow.security.api.JwtClaims; +import com.bablsoft.accessflow.security.internal.web.model.DataBudgetListResponse; +import com.bablsoft.accessflow.security.internal.web.model.DataBudgetRequest; +import com.bablsoft.accessflow.security.internal.web.model.DataBudgetResponse; +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.tags.Tag; +import jakarta.validation.Valid; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.http.ResponseEntity; +import org.springframework.security.access.prepost.PreAuthorize; +import org.springframework.security.core.Authentication; +import org.springframework.web.bind.annotation.DeleteMapping; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.PutMapping; +import org.springframework.web.bind.annotation.RequestBody; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RestController; +import org.springframework.web.servlet.support.ServletUriComponentsBuilder; + +import java.util.HashMap; +import java.util.Map; +import java.util.UUID; + +@RestController +@RequestMapping("/api/v1/datasources/{datasourceId}/data-budgets") +@Tag(name = "Data budgets", + description = "Per-user data-volume budgets over a rolling window on a datasource (#942)") +@PreAuthorize("hasAuthority('PERM_DATA_BUDGET_MANAGE')") +@RequiredArgsConstructor +@Slf4j +class DataBudgetController { + + private final DataBudgetAdminService dataBudgetAdminService; + private final AuditLogService auditLogService; + + @GetMapping + @Operation(summary = "List data budgets configured on a datasource") + @ApiResponse(responseCode = "200", description = "List of data budgets") + @ApiResponse(responseCode = "404", description = "Datasource not found") + DataBudgetListResponse list(@PathVariable UUID datasourceId, Authentication authentication) { + var caller = currentClaims(authentication); + return new DataBudgetListResponse(dataBudgetAdminService + .listForDatasource(datasourceId, caller.organizationId()).stream() + .map(DataBudgetResponse::from) + .toList()); + } + + @PostMapping + @Operation(summary = "Create a data budget on a datasource") + @ApiResponse(responseCode = "201", description = "Data budget created") + @ApiResponse(responseCode = "400", description = "Bean Validation failure") + @ApiResponse(responseCode = "404", description = "Datasource not found") + @ApiResponse(responseCode = "422", description = "No limit, a limit or window out of range, or an invalid applies-to target") + ResponseEntity create(@PathVariable UUID datasourceId, + @Valid @RequestBody DataBudgetRequest request, + Authentication authentication, + RequestAuditContext auditContext) { + var caller = currentClaims(authentication); + var view = dataBudgetAdminService.create(datasourceId, caller.organizationId(), + toCommand(request)); + recordAudit(AuditAction.DATA_BUDGET_CREATED, view, caller, auditContext); + var location = ServletUriComponentsBuilder.fromCurrentRequest() + .path("/{budgetId}") + .buildAndExpand(view.id()) + .toUri(); + return ResponseEntity.created(location).body(DataBudgetResponse.from(view)); + } + + @PutMapping("/{budgetId}") + @Operation(summary = "Update a data budget") + @ApiResponse(responseCode = "200", description = "Data budget updated") + @ApiResponse(responseCode = "400", description = "Bean Validation failure") + @ApiResponse(responseCode = "404", description = "Datasource or budget not found") + @ApiResponse(responseCode = "422", description = "No limit, a limit or window out of range, or an invalid applies-to target") + DataBudgetResponse update(@PathVariable UUID datasourceId, + @PathVariable UUID budgetId, + @Valid @RequestBody DataBudgetRequest request, + Authentication authentication, + RequestAuditContext auditContext) { + var caller = currentClaims(authentication); + var view = dataBudgetAdminService.update(budgetId, datasourceId, caller.organizationId(), + toCommand(request)); + recordAudit(AuditAction.DATA_BUDGET_UPDATED, view, caller, auditContext); + return DataBudgetResponse.from(view); + } + + @DeleteMapping("/{budgetId}") + @Operation(summary = "Delete a data budget") + @ApiResponse(responseCode = "204", description = "Data budget deleted") + @ApiResponse(responseCode = "404", description = "Datasource or budget not found") + ResponseEntity delete(@PathVariable UUID datasourceId, + @PathVariable UUID budgetId, + Authentication authentication, + RequestAuditContext auditContext) { + var caller = currentClaims(authentication); + dataBudgetAdminService.delete(budgetId, datasourceId, caller.organizationId()); + var metadata = new HashMap(); + metadata.put("datasource_id", datasourceId.toString()); + recordAudit(AuditAction.DATA_BUDGET_DELETED, budgetId, caller, auditContext, metadata); + return ResponseEntity.noContent().build(); + } + + private static DataBudgetCommand toCommand(DataBudgetRequest request) { + return new DataBudgetCommand(request.name(), request.maxRows(), request.maxBytes(), + request.windowMinutes(), request.breachAction(), request.warnThresholdPercent(), + request.appliesToRoles(), request.appliesToGroupIds(), request.appliesToUserIds(), + request.enabled()); + } + + private void recordAudit(AuditAction action, DataBudgetView view, JwtClaims caller, + RequestAuditContext auditContext) { + var metadata = new HashMap(); + metadata.put("datasource_id", view.datasourceId().toString()); + metadata.put("name", view.name()); + if (view.maxRows() != null) { + metadata.put("max_rows", view.maxRows()); + } + if (view.maxBytes() != null) { + metadata.put("max_bytes", view.maxBytes()); + } + metadata.put("window_minutes", view.windowMinutes()); + metadata.put("breach_action", view.breachAction().name()); + metadata.put("enabled", view.enabled()); + recordAudit(action, view.id(), caller, auditContext, metadata); + } + + private void recordAudit(AuditAction action, UUID budgetId, JwtClaims caller, + RequestAuditContext auditContext, Map metadata) { + try { + auditLogService.record(new AuditEntry(action, AuditResourceType.DATA_BUDGET, budgetId, + caller.organizationId(), caller.userId(), new HashMap<>(metadata), + auditContext.ipAddress(), auditContext.userAgent())); + } catch (RuntimeException ex) { + log.error("Audit write failed for {} on data budget {}", action, budgetId, ex); + } + } + + private static JwtClaims currentClaims(Authentication authentication) { + return (JwtClaims) authentication.getPrincipal(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/DataBudgetStatusController.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/DataBudgetStatusController.java new file mode 100644 index 000000000..b2262db4c --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/DataBudgetStatusController.java @@ -0,0 +1,69 @@ +package com.bablsoft.accessflow.security.internal.web; + +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.core.api.Permission; +import com.bablsoft.accessflow.core.api.UserNotFoundException; +import com.bablsoft.accessflow.core.api.UserQueryService; +import com.bablsoft.accessflow.security.api.JwtClaims; +import com.bablsoft.accessflow.security.internal.web.model.DataBudgetStatusListResponse; +import com.bablsoft.accessflow.security.internal.web.model.DataBudgetStatusResponse; +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.tags.Tag; +import lombok.RequiredArgsConstructor; +import org.springframework.security.access.prepost.PreAuthorize; +import org.springframework.security.core.Authentication; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.RestController; + +import java.util.UUID; + +/** Read-only data-budget standing (#942): the caller's own, and any user's for admins. */ +@RestController +@Tag(name = "Data budgets", description = "Remaining data-volume allowance (#942)") +@RequiredArgsConstructor +class DataBudgetStatusController { + + private final DataBudgetStatusService dataBudgetStatusService; + private final DatasourceAdminService datasourceAdminService; + private final UserQueryService userQueryService; + + @GetMapping("/api/v1/datasources/{datasourceId}/data-budgets/me") + @Operation(summary = "The caller's data-budget standing on a datasource") + @ApiResponse(responseCode = "200", description = "Standing; an empty budgets list when none applies") + @ApiResponse(responseCode = "404", description = "Datasource not found or not visible to the caller") + DataBudgetStatusResponse mine(@PathVariable UUID datasourceId, Authentication authentication) { + var caller = currentClaims(authentication); + // Resolves visibility first so an invisible datasource reads 404, never an empty 200. + if (caller.has(Permission.QUERY_ADMIN) || caller.has(Permission.DATASOURCE_MANAGE)) { + datasourceAdminService.getForAdmin(datasourceId, caller.organizationId()); + } else { + datasourceAdminService.getForUser(datasourceId, caller.organizationId(), + caller.userId()); + } + return DataBudgetStatusResponse.from( + dataBudgetStatusService.statusFor(datasourceId, caller.userId())); + } + + @GetMapping("/api/v1/admin/users/{userId}/data-budget-usage") + @PreAuthorize("hasAuthority('PERM_DATA_BUDGET_MANAGE') or hasAuthority('PERM_USER_MANAGE')") + @Operation(summary = "A user's data-budget standing on every budgeted datasource") + @ApiResponse(responseCode = "200", description = "One entry per datasource where a budget applies to the user") + @ApiResponse(responseCode = "404", description = "User not found in the caller's organization") + DataBudgetStatusListResponse forUser(@PathVariable UUID userId, Authentication authentication) { + var caller = currentClaims(authentication); + userQueryService.findById(userId) + .filter(u -> caller.organizationId().equals(u.organizationId())) + .orElseThrow(() -> new UserNotFoundException(userId)); + return new DataBudgetStatusListResponse(dataBudgetStatusService + .statusesForUser(caller.organizationId(), userId).stream() + .map(DataBudgetStatusResponse::from) + .toList()); + } + + private static JwtClaims currentClaims(Authentication authentication) { + return (JwtClaims) authentication.getPrincipal(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/GlobalExceptionHandler.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/GlobalExceptionHandler.java index 668b5ca86..ae638bb17 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/GlobalExceptionHandler.java +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/GlobalExceptionHandler.java @@ -28,6 +28,9 @@ import com.bablsoft.accessflow.core.api.ExportPolicyNotFoundException; import com.bablsoft.accessflow.core.api.IllegalExportPolicyException; import com.bablsoft.accessflow.core.api.IllegalRowLimitPolicyException; +import com.bablsoft.accessflow.core.api.DataBudgetExhaustedException; +import com.bablsoft.accessflow.core.api.DataBudgetNotFoundException; +import com.bablsoft.accessflow.core.api.IllegalDataBudgetException; import com.bablsoft.accessflow.core.api.IllegalRowSecurityPolicyException; import com.bablsoft.accessflow.core.api.InvalidSimulationPeriodException; import com.bablsoft.accessflow.core.api.MaskingPolicyNotFoundException; @@ -653,6 +656,40 @@ ProblemDetail handleIllegalRowLimitPolicy(IllegalRowLimitPolicyException ex) { return pd; } + @ExceptionHandler(DataBudgetNotFoundException.class) + ProblemDetail handleDataBudgetNotFound(DataBudgetNotFoundException ex) { + var pd = ProblemDetail.forStatusAndDetail(HttpStatus.NOT_FOUND, + msg("error.data_budget.not_found")); + pd.setProperty("error", "DATA_BUDGET_NOT_FOUND"); + pd.setProperty("timestamp", Instant.now().toString()); + return pd; + } + + @ExceptionHandler(IllegalDataBudgetException.class) + ProblemDetail handleIllegalDataBudget(IllegalDataBudgetException ex) { + // Message is resolved at the throw site via MessageSource — see DefaultDataBudgetAdminService. + var pd = ProblemDetail.forStatusAndDetail(HttpStatus.UNPROCESSABLE_CONTENT, ex.getMessage()); + pd.setProperty("error", "ILLEGAL_DATA_BUDGET"); + pd.setProperty("timestamp", Instant.now().toString()); + return pd; + } + + @ExceptionHandler(DataBudgetExhaustedException.class) + ProblemDetail handleDataBudgetExhausted(DataBudgetExhaustedException ex) { + // Message is resolved at the throw site — it names the exhausted budget. + var pd = ProblemDetail.forStatusAndDetail(HttpStatus.CONFLICT, ex.getMessage()); + pd.setProperty("error", "DATA_BUDGET_EXHAUSTED"); + var budget = ex.budget(); + if (budget != null) { + pd.setProperty("budgetId", budget.budgetId()); + pd.setProperty("maxRows", budget.maxRows()); + pd.setProperty("maxBytes", budget.maxBytes()); + pd.setProperty("windowMinutes", budget.windowMinutes()); + } + pd.setProperty("timestamp", Instant.now().toString()); + return pd; + } + @ExceptionHandler(QueryExecutionTimeoutException.class) ProblemDetail handleQueryExecutionTimeout(QueryExecutionTimeoutException ex) { var pd = ProblemDetail.forStatusAndDetail(HttpStatus.GATEWAY_TIMEOUT, diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetListResponse.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetListResponse.java new file mode 100644 index 000000000..508682b43 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetListResponse.java @@ -0,0 +1,5 @@ +package com.bablsoft.accessflow.security.internal.web.model; + +import java.util.List; + +public record DataBudgetListResponse(List content) {} diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetRequest.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetRequest.java new file mode 100644 index 000000000..b9e9ecc4c --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetRequest.java @@ -0,0 +1,32 @@ +package com.bablsoft.accessflow.security.internal.web.model; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import jakarta.validation.constraints.Max; +import jakarta.validation.constraints.Min; +import jakarta.validation.constraints.NotBlank; +import jakarta.validation.constraints.Size; + +import java.util.List; +import java.util.UUID; + +/** Create/update body for a data budget (#942). Updates are total, like create. */ +public record DataBudgetRequest( + @NotBlank(message = "{validation.data_budget.name.required}") + @Size(max = 120, message = "{validation.data_budget.name.size}") + String name, + @Min(value = 1, message = "{validation.data_budget.max_rows.min}") + Long maxRows, + @Min(value = 1, message = "{validation.data_budget.max_bytes.min}") + Long maxBytes, + @Min(value = 60, message = "{validation.data_budget.window.range}") + @Max(value = 44_640, message = "{validation.data_budget.window.range}") + Integer windowMinutes, + DataBudgetBreachAction breachAction, + @Min(value = 1, message = "{validation.data_budget.threshold.range}") + @Max(value = 99, message = "{validation.data_budget.threshold.range}") + Integer warnThresholdPercent, + List appliesToRoles, + List appliesToGroupIds, + List appliesToUserIds, + Boolean enabled +) {} diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetResponse.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetResponse.java new file mode 100644 index 000000000..96b65556f --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetResponse.java @@ -0,0 +1,43 @@ +package com.bablsoft.accessflow.security.internal.web.model; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetView; + +import java.time.Instant; +import java.util.List; +import java.util.UUID; + +public record DataBudgetResponse( + UUID id, + UUID datasourceId, + String name, + Long maxRows, + Long maxBytes, + int windowMinutes, + DataBudgetBreachAction breachAction, + Integer warnThresholdPercent, + List appliesToRoles, + List appliesToGroupIds, + List appliesToUserIds, + boolean enabled, + Instant createdAt, + Instant updatedAt) { + + public static DataBudgetResponse from(DataBudgetView view) { + return new DataBudgetResponse( + view.id(), + view.datasourceId(), + view.name(), + view.maxRows(), + view.maxBytes(), + view.windowMinutes(), + view.breachAction(), + view.warnThresholdPercent(), + view.appliesToRoles(), + view.appliesToGroupIds(), + view.appliesToUserIds(), + view.enabled(), + view.createdAt(), + view.updatedAt()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetStatusListResponse.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetStatusListResponse.java new file mode 100644 index 000000000..46857ca20 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetStatusListResponse.java @@ -0,0 +1,5 @@ +package com.bablsoft.accessflow.security.internal.web.model; + +import java.util.List; + +public record DataBudgetStatusListResponse(List content) {} diff --git a/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetStatusResponse.java b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetStatusResponse.java new file mode 100644 index 000000000..c19ccac47 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/security/internal/web/model/DataBudgetStatusResponse.java @@ -0,0 +1,62 @@ +package com.bablsoft.accessflow.security.internal.web.model; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetConsumption; +import com.bablsoft.accessflow.core.api.DataBudgetStatus; + +import java.util.List; +import java.util.UUID; + +/** + * A user's data-budget standing on one datasource (#942). {@code budgets} is empty when no budget + * applies; the top-level fields are then null / false. + */ +public record DataBudgetStatusResponse( + UUID datasourceId, + String datasourceName, + boolean exhausted, + DataBudgetBreachAction breachAction, + Long remainingRows, + Long remainingBytes, + Integer usedPercent, + List budgets) { + + public record Budget( + UUID id, + String name, + Long maxRows, + Long maxBytes, + int windowMinutes, + DataBudgetBreachAction breachAction, + Integer warnThresholdPercent, + long usedRows, + long usedBytes, + Long remainingRows, + Long remainingBytes, + int usedPercent, + boolean exhausted) { + + static Budget from(DataBudgetConsumption c) { + return new Budget(c.budgetId(), c.name(), c.maxRows(), c.maxBytes(), c.windowMinutes(), + c.breachAction(), c.warnThresholdPercent(), c.usedRows(), c.usedBytes(), + c.remainingRows(), c.remainingBytes(), floor(c.usedPercent()), c.exhausted()); + } + } + + public static DataBudgetStatusResponse from(DataBudgetStatus status) { + var used = status.usedPercent(); + return new DataBudgetStatusResponse( + status.datasourceId(), + status.datasourceName(), + status.exhausted(), + status.breachAction(), + status.remainingRows(), + status.remainingBytes(), + used == null ? null : floor(used), + status.budgets().stream().map(Budget::from).toList()); + } + + private static int floor(double percent) { + return (int) Math.min(Integer.MAX_VALUE, Math.floor(percent)); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionContext.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionContext.java index a7f31d68a..109cd1076 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionContext.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionContext.java @@ -40,6 +40,10 @@ * bytes from the same row — {@code null} for every engine that reports none, and whenever the * estimate is absent or failed. * + *

{@code dataBudgetUsedPercent} (#942) is the share of the submitter's most-used data budget on + * this datasource, as a whole percentage — {@code null} when no budget applies, the statement is not + * a SELECT, and on the historical replay path (usage then is not reconstructable). + * *

{@code queryShapes} are the structural features re-derived from the SQL with the WHERE / LIMIT * signals (#940). {@code shapesAnalyzed} is {@code false} when the SQL could not be parsed or walked * (typically a non-SQL engine); the {@code query_shape} condition then fails closed. @@ -64,7 +68,8 @@ public record ConditionContext( String scanType, Set queryShapes, boolean shapesAnalyzed, - Long estimatedBytesScanned) { + Long estimatedBytesScanned, + Integer dataBudgetUsedPercent) { public ConditionContext { referencedTables = Set.copyOf(referencedTables == null ? Set.of() : referencedTables); @@ -72,6 +77,23 @@ public record ConditionContext( queryShapes = Set.copyOf(queryShapes == null ? Set.of() : queryShapes); } + /** Backward-compatible constructor without the #942 data-budget signal (defaults to absent). */ + public ConditionContext(QueryType queryType, Set referencedTables, RiskLevel riskLevel, + int riskScore, String requesterRoleName, Set requesterGroupIds, + LocalDateTime evaluatedAt, boolean hasWhereClause, + boolean hasLimitClause, boolean transactional, + String requesterIpAddress, String requesterUserAgent, + boolean ciCdOrigin, Integer minutesSinceLastApproval, + boolean anomalyActive, Long estimatedRows, String scanType, + Set queryShapes, boolean shapesAnalyzed, + Long estimatedBytesScanned) { + this(queryType, referencedTables, riskLevel, riskScore, requesterRoleName, + requesterGroupIds, evaluatedAt, hasWhereClause, hasLimitClause, transactional, + requesterIpAddress, requesterUserAgent, ciCdOrigin, minutesSinceLastApproval, + anomalyActive, estimatedRows, scanType, queryShapes, shapesAnalyzed, + estimatedBytesScanned, null); + } + /** Backward-compatible constructor without the #941 bytes estimate (defaults to absent). */ public ConditionContext(QueryType queryType, Set referencedTables, RiskLevel riskLevel, int riskScore, String requesterRoleName, Set requesterGroupIds, @@ -112,7 +134,7 @@ public ConditionContext(QueryType queryType, Set referencedTables, RiskL this(queryType, referencedTables, riskLevel, riskScore, requesterRoleName, requesterGroupIds, evaluatedAt, hasWhereClause, hasLimitClause, transactional, requesterIpAddress, requesterUserAgent, ciCdOrigin, minutesSinceLastApproval, - anomalyActive, null, null, Set.of(), false, null); + anomalyActive, null, null, Set.of(), false, null, null); } /** @return {@code true} when an AI risk level / score signal is present. */ diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionNode.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionNode.java index 1a9e3d3d2..2c650d1e4 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionNode.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/api/ConditionNode.java @@ -234,6 +234,25 @@ record EstimatedBytesScanned(ComparisonOperator operator, long value) implements } } + /** + * Compares the share of the submitter's data-volume budget already used on this datasource + * (#942 — the most-used applying budget, as a whole percentage that can exceed 100) with + * {@code value}. Fails closed: evaluates to {@code false} when no budget applies + * to the submitter, the statement is not a SELECT, or on a historical replay — so it is an + * escalation trigger, never a way to auto-approve on missing context. + */ + record DataBudgetUsedPercent(ComparisonOperator operator, int value) implements ConditionNode { + public DataBudgetUsedPercent { + if (operator == null) { + throw new IllegalArgumentException( + "DataBudgetUsedPercent condition requires an operator"); + } + if (value < 0) { + throw new IllegalArgumentException("DataBudgetUsedPercent value must be >= 0"); + } + } + } + /** * Matches when the pre-flight plan's root scan/operation type (AF-624 — e.g. {@code Seq Scan}, * {@code Index Scan}, {@code COLLSCAN}) matches any glob in {@code patterns} ({@code *} = any diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/api/QueryDecisionStepKind.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/api/QueryDecisionStepKind.java index 2b76abb70..e8d4f8015 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/api/QueryDecisionStepKind.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/api/QueryDecisionStepKind.java @@ -49,6 +49,14 @@ public enum QueryDecisionStepKind implements DecisionStepKind { */ BYTES_SCANNED_CAP, + /** + * The submitter's data-volume budget (#942): {@code DENY} when an exhausted budget rejects; + * {@code MATCH} when an exhausted budget forces human review (every auto-approve stage below is + * then suppressed); {@code ALLOW} while allowance remains; {@code NO_MATCH} when no budget + * applies or the statement is not a SELECT. + */ + DATA_BUDGET, + /** Every enabled routing policy in ascending priority order, matched and unmatched. */ ROUTING_POLICIES, diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DataBudgetCheck.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DataBudgetCheck.java new file mode 100644 index 000000000..1fe7bbe9d --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DataBudgetCheck.java @@ -0,0 +1,39 @@ +package com.bablsoft.accessflow.workflow.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetConsumption; +import com.bablsoft.accessflow.core.api.DataBudgetStatus; + +/** + * The submitter's data-volume budget standing (#942) for one SELECT — the input + * {@link QueryDecisionEvaluator} decides on, like the bytes-scanned cap. + * + * @param deciding the exhausted budget whose action applies, {@code null} while allowance remains + * @param action the strictest action among exhausted budgets, {@code null} while none is + * @param usedPercent the highest share of any applying budget used, 0–100+ + */ +record DataBudgetCheck(DataBudgetConsumption deciding, DataBudgetBreachAction action, + double usedPercent, Long remainingRows, Long remainingBytes) { + + /** {@code null} when no budget applies, so "no budget" and "within budget" stay distinct. */ + static DataBudgetCheck of(DataBudgetStatus status) { + if (status == null || status.isEmpty()) { + return null; + } + var used = status.usedPercent(); + return new DataBudgetCheck(status.decidingBudget(), status.breachAction(), + used == null ? 0d : used, status.remainingRows(), status.remainingBytes()); + } + + boolean exhausted() { + return action != null; + } + + boolean rejects() { + return action == DataBudgetBreachAction.REJECT; + } + + boolean forcesReview() { + return action == DataBudgetBreachAction.REQUIRE_REVIEW; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationService.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationService.java index 532972844..7ca06da9b 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationService.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationService.java @@ -12,6 +12,7 @@ import com.bablsoft.accessflow.core.api.Permission; import com.bablsoft.accessflow.core.api.QueryRequestSnapshot; import com.bablsoft.accessflow.core.api.QueryShape; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; import com.bablsoft.accessflow.core.api.QueryStatus; import com.bablsoft.accessflow.core.api.QueryType; import com.bablsoft.accessflow.core.api.QuotaExceededException; @@ -89,6 +90,7 @@ class DefaultAccessSimulationService implements AccessSimulationService { private final MaskingPolicyResolutionService maskingPolicyResolutionService; private final BreakGlassEligibilityService breakGlassEligibilityService; private final BytesScannedCapResolutionService bytesScannedCapResolutionService; + private final DataBudgetStatusService dataBudgetStatusService; // Time-of-day / day-of-week routing conditions evaluate in the server's local zone, so the // simulator has to use the same zone the live listener does. Deliberately NOT the injected @@ -149,9 +151,15 @@ public AccessSimulationResult simulate(UUID organizationId, AccessSimulationInpu .resolve(input.datasourceId(), input.userId()) .map(BytesCapCheck::unevaluated) .orElse(null); + // Budget usage is a live, persisted fact about the user, so it is read as it stands (#942). + var dataBudget = parsed.type() == QueryType.SELECT + ? DataBudgetCheck.of(dataBudgetStatusService.statusFor(input.datasourceId(), + input.userId())) + : null; var decision = queryDecisionEvaluator.evaluate( syntheticSnapshot(organizationId, input, parsed), input.aiOutcome(), - input.riskLevel(), input.effectiveRiskScore(), blockingRuleIds, bytesCap, clock); + input.riskLevel(), input.effectiveRiskScore(), blockingRuleIds, bytesCap, dataBudget, + clock); steps.addAll(withFullPolicyList(decision, organizationId, input.datasourceId())); steps.add(reviewerStep(input, decision.nextStatus())); diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleService.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleService.java index 1449a6bba..f33a3c091 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleService.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleService.java @@ -1,6 +1,14 @@ package com.bablsoft.accessflow.workflow.internal; import com.bablsoft.accessflow.core.api.ByteSizeFormat; +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.QueryType; +import com.bablsoft.accessflow.core.api.DataBudgetExhaustedException; +import com.bablsoft.accessflow.core.api.DataBudgetStatus; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageRecord; +import com.bablsoft.accessflow.core.api.DataBudgetUsageService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageSource; import com.bablsoft.accessflow.core.api.BytesScannedCapExceededException; import com.bablsoft.accessflow.core.api.BytesScannedCapResolutionService; import com.bablsoft.accessflow.core.api.QueryEstimateSnapshot; @@ -94,6 +102,8 @@ class DefaultQueryLifecycleService implements QueryLifecycleService { private final ApplicationEventPublisher eventPublisher; private final BytesScannedCapResolutionService bytesScannedCapResolutionService; private final QueryCostEstimateService queryCostEstimateService; + private final DataBudgetStatusService dataBudgetStatusService; + private final DataBudgetUsageService dataBudgetUsageService; private String msg(String key) { return messageSource.getMessage(key, null, LocaleContextHolder.getLocale()); @@ -288,6 +298,11 @@ private ExecutionOutcome doExecute(QueryRequestSnapshot query, UUID actorUserId, // and break-glass runs execute later or without that decision, and the cap may have // been lowered since. A refusal is recorded as a failed execution. var bytesCap = enforceBytesScannedCap(query); + // #942: the reader's data budget. Allowance left ⇒ the result is capped to it; exhausted + // ⇒ refused, unless the exhausted budget itself forced the review this query passed. + // Break-glass is counted but never capped or refused by a budget. + var budget = enforceDataBudget(query, + successAction == AuditAction.QUERY_BREAK_GLASS_EXECUTED); var permission = permissionLookupService .findFor(query.submittedByUserId(), query.datasourceId()); var restrictedColumns = permission @@ -349,10 +364,20 @@ private ExecutionOutcome doExecute(QueryRequestSnapshot query, UUID actorUserId, softDeleteFilters.stream()).toList(); var softDeletes = lifecycleDirectiveResolutionService .resolveSoftDeletes(query.organizationId(), query.datasourceId()); - var result = queryExecutor.execute(new QueryExecutionRequest( + var executionRequest = new QueryExecutionRequest( query.datasourceId(), query.sqlText(), query.queryType(), rowLimitOverride, null, restrictedColumns, columnMasks, rowSecurityPredicates, parsed.transactional(), - parsed.statements(), softDeletes, parsed.referencedTables())); + parsed.statements(), softDeletes, parsed.referencedTables()); + if (budget != null && budget.capped()) { + executionRequest = executionRequest.withAllowance( + budget.status().remainingRows(), budget.status().remainingBytes()); + } + var result = queryExecutor.execute(executionRequest); + if (budget != null && budget.capped() && result instanceof SelectExecutionResult select) { + result = attributeBudgetTruncation(select, budget.status().remainingRows(), + rowLimitOverride, descriptor.map(DatasourceConnectionDescriptor::maxRowsPerQuery) + .orElse(null)); + } var completedAt = Instant.now(); var durationMs = (int) result.duration().toMillis(); Long rowsAffected; @@ -421,6 +446,11 @@ private ExecutionOutcome doExecute(QueryRequestSnapshot query, UUID actorUserId, appliedRowLimitPolicyIds.stream() .map(UUID::toString).sorted().toList()); } + if (budget != null && result instanceof SelectExecutionResult select) { + successMetadata.put("data_budget_rows_charged", select.rowCount()); + successMetadata.put("data_budget_bytes_charged", select.resultBytes()); + chargeDataBudget(query, select); + } recordAudit(successAction, query.id(), actorUserId, query.organizationId(), successMetadata); eventPublisher.publishEvent(new QueryExecutedEvent( @@ -491,6 +521,90 @@ private BytesCapCheck enforceBytesScannedCap(QueryRequestSnapshot query) { throw new BytesScannedCapExceededException(message, cap, estimated, check.outcome()); } + /** A budget that applies to this read, and whether the result must be capped to it. */ + private record BudgetGate(DataBudgetStatus status, boolean capped) { + } + + /** + * @return {@code null} when no budget applies (or the statement is not a SELECT); otherwise the + * standing and whether to cap the result to the remaining allowance + * @throws DataBudgetExhaustedException when an exhausted budget refuses this run + */ + private BudgetGate enforceDataBudget(QueryRequestSnapshot query, boolean breakGlass) { + if (query.queryType() != QueryType.SELECT) { + return null; + } + var status = dataBudgetStatusService.statusFor(query.datasourceId(), + query.submittedByUserId()); + if (status.isEmpty()) { + return null; + } + if (breakGlass) { + return new BudgetGate(status, false); + } + if (!status.exhausted()) { + return new BudgetGate(status, true); + } + if (status.breachAction() == DataBudgetBreachAction.REQUIRE_REVIEW + && budgetForcedReview(query)) { + return new BudgetGate(status, false); + } + var deciding = status.decidingBudget(); + var metadata = new HashMap(); + metadata.put("trigger", "data_budget"); + metadata.put("stage", "execution"); + metadata.put("action", status.breachAction().name()); + metadata.put("data_budget_id", deciding.budgetId()); + metadata.put("used_rows", deciding.usedRows()); + metadata.put("used_bytes", deciding.usedBytes()); + metadata.put("window_minutes", deciding.windowMinutes()); + recordAudit(AuditAction.QUERY_DATA_BUDGET_ENFORCED, query.id(), null, + query.organizationId(), metadata); + throw new DataBudgetExhaustedException(messageSource.getMessage( + "error.data_budget.exhausted", new Object[]{deciding.name()}, + LocaleContextHolder.getLocale()), deciding); + } + + /** + * The exhausted budget itself sent this query — or, for a recurring occurrence, its series — to + * review, so its approval (by a reviewer or a synced ticket) was given knowing the budget was + * spent. An approval obtained while allowance remained never lifts the budget. + */ + private boolean budgetForcedReview(QueryRequestSnapshot query) { + return queryRequestStateService.isDataBudgetReviewForced(query.id()) + || (query.recurringParentId() != null + && queryRequestStateService.isDataBudgetReviewForced( + query.recurringParentId())); + } + + /** + * The row allowance was the binding cap when the result stopped exactly at it and it sits below + * every other row cap we know of; the global ceiling is covered by the exact-count test. + */ + static SelectExecutionResult attributeBudgetTruncation(SelectExecutionResult select, + Long budgetRows, Integer otherRowCap, + Integer datasourceRowCap) { + if (!select.truncated() || budgetRows == null + || !SelectExecutionResult.TRUNCATED_ROW_LIMIT.equals(select.truncatedReason()) + || select.rowCount() != budgetRows + || (otherRowCap != null && otherRowCap <= budgetRows) + || (datasourceRowCap != null && datasourceRowCap <= budgetRows)) { + return select; + } + return select.withTruncatedReason(SelectExecutionResult.TRUNCATED_DATA_BUDGET); + } + + /** Never fails the execution: the rows were already delivered. */ + private void chargeDataBudget(QueryRequestSnapshot query, SelectExecutionResult select) { + try { + dataBudgetUsageService.record(new DataBudgetUsageRecord(query.submittedByUserId(), + query.datasourceId(), select.rowCount(), select.resultBytes(), + DataBudgetUsageSource.QUERY, query.id(), null)); + } catch (RuntimeException ex) { + log.error("Data-budget usage write failed for query {}", query.id(), ex); + } + } + private ExecutionOutcome recordFailure(QueryRequestSnapshot query, UUID actorUserId, String trigger, Instant startedAt, RuntimeException ex) { var completedAt = Instant.now(); diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecision.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecision.java index afbd0d197..84ea6a5fa 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecision.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecision.java @@ -28,12 +28,27 @@ * @param bytesCap the bytes-scanned cap that applied (#941), or {@code null} when none * @param bytesCapChangedOutcome whether the cap refused the query or turned an automatic approval * into human review — the condition for its audit row + * @param dataBudget the submitter's data-budget standing (#942), or {@code null} when no + * budget applies + * @param dataBudgetChangedOutcome whether the exhausted budget refused the query or turned an + * automatic approval into human review — the condition for its audit row */ record QueryDecision(QueryDecisionKind kind, QueryStatus nextStatus, RoutingMatch routingMatch, Integer effectiveApprovals, UUID grantId, String grantApproverEmail, ConditionContext context, DecisionTrace trace, SqlReviewSuppression sqlReviewSuppression, BytesCapCheck bytesCap, - boolean bytesCapChangedOutcome) { + boolean bytesCapChangedOutcome, DataBudgetCheck dataBudget, + boolean dataBudgetChangedOutcome) { + + /** A decision no data budget took part in. */ + QueryDecision(QueryDecisionKind kind, QueryStatus nextStatus, RoutingMatch routingMatch, + Integer effectiveApprovals, UUID grantId, String grantApproverEmail, + ConditionContext context, DecisionTrace trace, + SqlReviewSuppression sqlReviewSuppression, BytesCapCheck bytesCap, + boolean bytesCapChangedOutcome) { + this(kind, nextStatus, routingMatch, effectiveApprovals, grantId, grantApproverEmail, context, + trace, sqlReviewSuppression, bytesCap, bytesCapChangedOutcome, null, false); + } /** A decision no bytes-scanned cap took part in. */ QueryDecision(QueryDecisionKind kind, QueryStatus nextStatus, RoutingMatch routingMatch, diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluator.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluator.java index bdfd66c54..1ef28eaec 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluator.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluator.java @@ -75,25 +75,39 @@ QueryDecision evaluate(QueryRequestSnapshot query, AiOutcome aiOutcome, RiskLeve return evaluate(query, aiOutcome, riskLevel, riskScore, blockingRuleIds, null, clock); } + /** {@link #evaluate(QueryRequestSnapshot, AiOutcome, RiskLevel, int, List, BytesCapCheck, DataBudgetCheck, Clock)} with no data budget. */ + QueryDecision evaluate(QueryRequestSnapshot query, AiOutcome aiOutcome, RiskLevel riskLevel, + int riskScore, List blockingRuleIds, BytesCapCheck bytesCap, + Clock clock) { + return evaluate(query, aiOutcome, riskLevel, riskScore, blockingRuleIds, bytesCap, null, + clock); + } + /** * @param riskScore the AI's numeric score, or {@code -1} when there is none — the same * "absent" sentinel the live completion event uses * @param blockingRuleIds the distinct SQL review rule ids that fired at {@code BLOCK} for this * request, empty when none did (#864) * @param bytesCap the bytes-scanned cap that applies (#941), {@code null} when none does + * @param dataBudget the submitter's data-budget standing (#942), {@code null} when no budget + * applies or the statement is not a SELECT */ QueryDecision evaluate(QueryRequestSnapshot query, AiOutcome aiOutcome, RiskLevel riskLevel, int riskScore, List blockingRuleIds, BytesCapCheck bytesCap, - Clock clock) { + DataBudgetCheck dataBudget, Clock clock) { var block = blockingRuleIds == null ? List.of() : List.copyOf(blockingRuleIds); - var guard = new Guard(block, bytesCap); + var guard = new Guard(block, bytesCap, dataBudget); // A hard cap is a refusal, not a routing signal: it decides before routing and before the // AI-failure fallback, so no policy or plan can approve a query the cap has refused. if (bytesCap != null && bytesCap.rejects()) { - return bytesCapRejected(block, bytesCap); + return bytesCapRejected(block, bytesCap, dataBudget); + } + // An exhausted budget under REJECT is the same kind of refusal, decided just after the cap. + if (dataBudget != null && dataBudget.rejects()) { + return dataBudgetRejected(block, bytesCap, dataBudget); } if (aiOutcome == AiOutcome.FAILED) { - return aiFailed(block, bytesCap); + return aiFailed(block, bytesCap, dataBudget); } // Only a COMPLETED analysis carries a risk signal. Normalising here rather than trusting the // caller keeps the SKIPPED branch identical to production, where the listener passes no risk @@ -106,6 +120,7 @@ QueryDecision evaluate(QueryRequestSnapshot query, AiOutcome aiOutcome, RiskLeve var steps = new ArrayList(5); steps.add(sqlReviewStep(block)); steps.add(bytesCapStep(bytesCap)); + steps.add(dataBudgetStep(dataBudget)); var match = routingPolicyEngine.evaluate(query.organizationId(), query.datasourceId(), context).orElse(null); @@ -122,7 +137,8 @@ QueryDecision evaluate(QueryRequestSnapshot query, AiOutcome aiOutcome, RiskLeve "workflow.decision.plan.skipped_grant_covered", planDetails(plan))); return new QueryDecision(QueryDecisionKind.GRANT_FAST_PATH, QueryStatus.APPROVED, null, null, grant.id(), grant.approverEmail(), context, - new DecisionTrace(steps, QueryStatus.APPROVED), null, bytesCap, false); + new DecisionTrace(steps, QueryStatus.APPROVED), null, bytesCap, false, + dataBudget, false); } return planned(query, plan, effectiveRisk, context, steps, guard, suppressed); @@ -130,24 +146,81 @@ QueryDecision evaluate(QueryRequestSnapshot query, AiOutcome aiOutcome, RiskLeve /** * What may turn an automatic approval into human review: a {@code BLOCK} SQL review finding - * (#864) and a bytes-scanned cap with no estimate to compare against (#941). The trace names - * the SQL review first when both apply. + * (#864), a bytes-scanned cap with no estimate to compare against (#941), and an exhausted data + * budget under {@code REQUIRE_REVIEW} (#942). The trace names them in that order of precedence. */ - private record Guard(List block, BytesCapCheck bytesCap) { + private record Guard(List block, BytesCapCheck bytesCap, DataBudgetCheck dataBudget) { boolean suppresses() { - return !block.isEmpty() || (bytesCap != null && bytesCap.forcesReview()); + return !block.isEmpty() || capForcesReview() || budgetForcesReview(); } boolean capForcesReview() { return bytesCap != null && bytesCap.forcesReview(); } + boolean budgetForcesReview() { + return dataBudget != null && dataBudget.forcesReview(); + } + String reasonSuffix() { - return block.isEmpty() ? "bytes_cap" : "sql_review"; + if (!block.isEmpty()) { + return "sql_review"; + } + return capForcesReview() ? "bytes_cap" : "data_budget"; } } + /** The budget step is always present so the trace keeps one entry per stage. */ + private static DecisionTraceStep dataBudgetStep(DataBudgetCheck budget) { + if (budget == null) { + return DecisionTraceStep.of(QueryDecisionStepKind.DATA_BUDGET, StepOutcome.NO_MATCH, + "workflow.decision.data_budget.none"); + } + var percent = String.valueOf((long) Math.floor(budget.usedPercent())); + var details = new LinkedHashMap(); + details.put("data_budget_used_percent", Math.floor(budget.usedPercent())); + details.put("data_budget_remaining_rows", budget.remainingRows()); + details.put("data_budget_remaining_bytes", budget.remainingBytes()); + details.put("data_budget_action", budget.action() == null ? null : budget.action().name()); + if (budget.deciding() != null) { + details.put("data_budget_id", budget.deciding().budgetId()); + details.put("data_budget_name", budget.deciding().name()); + } + if (budget.rejects()) { + return new DecisionTraceStep(QueryDecisionStepKind.DATA_BUDGET, StepOutcome.DENY, + "workflow.decision.data_budget.exhausted_rejected", + List.of(String.valueOf(budget.deciding().name())), details); + } + if (budget.forcesReview()) { + return new DecisionTraceStep(QueryDecisionStepKind.DATA_BUDGET, StepOutcome.MATCH, + "workflow.decision.data_budget.exhausted_review", + List.of(String.valueOf(budget.deciding().name())), details); + } + return new DecisionTraceStep(QueryDecisionStepKind.DATA_BUDGET, StepOutcome.ALLOW, + "workflow.decision.data_budget.within", List.of(percent), details); + } + + /** + * An exhausted budget refused the query (#942). Like the cap, nothing downstream runs. + */ + private static QueryDecision dataBudgetRejected(List block, BytesCapCheck cap, + DataBudgetCheck budget) { + var steps = List.of( + sqlReviewStep(block), + bytesCapStep(cap), + dataBudgetStep(budget), + DecisionTraceStep.of(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.SKIP, + "workflow.decision.routing.skipped_data_budget"), + DecisionTraceStep.of(QueryDecisionStepKind.GRANT_FAST_PATH, StepOutcome.SKIP, + "workflow.decision.grant.skipped_data_budget"), + DecisionTraceStep.of(QueryDecisionStepKind.REVIEW_PLAN, StepOutcome.SKIP, + "workflow.decision.plan.skipped_data_budget")); + return new QueryDecision(QueryDecisionKind.DATA_BUDGET_REJECTED, QueryStatus.REJECTED, null, + null, null, null, null, new DecisionTrace(steps, QueryStatus.REJECTED), null, cap, + false, budget, true); + } + /** The cap step is always present so the trace keeps one entry per stage. */ private static DecisionTraceStep bytesCapStep(BytesCapCheck cap) { if (cap == null) { @@ -184,10 +257,12 @@ private static DecisionTraceStep bytesCapStep(BytesCapCheck cap) { * The cap refused the query (#941). Nothing downstream runs — there is no routing decision to * record and no plan to consult — which is why the context is not even built. */ - private static QueryDecision bytesCapRejected(List block, BytesCapCheck cap) { + private static QueryDecision bytesCapRejected(List block, BytesCapCheck cap, + DataBudgetCheck budget) { var steps = List.of( sqlReviewStep(block), bytesCapStep(cap), + dataBudgetStep(budget), DecisionTraceStep.of(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.SKIP, "workflow.decision.routing.skipped_bytes_cap"), DecisionTraceStep.of(QueryDecisionStepKind.GRANT_FAST_PATH, StepOutcome.SKIP, @@ -196,7 +271,7 @@ private static QueryDecision bytesCapRejected(List block, BytesCapCheck "workflow.decision.plan.skipped_bytes_cap")); return new QueryDecision(QueryDecisionKind.BYTES_CAP_REJECTED, QueryStatus.REJECTED, null, null, null, null, null, new DecisionTrace(steps, QueryStatus.REJECTED), null, cap, - true); + true, budget, false); } /** @@ -227,10 +302,12 @@ private static SqlReviewSuppression suppression(List block, * auto-decision signal — and neither does the grant fast path or the review plan. Decided before * any lookup, mirroring the live listener, which builds no context at all. */ - private static QueryDecision aiFailed(List block, BytesCapCheck bytesCap) { + private static QueryDecision aiFailed(List block, BytesCapCheck bytesCap, + DataBudgetCheck budget) { var steps = List.of( sqlReviewStep(block), bytesCapStep(bytesCap), + dataBudgetStep(budget), DecisionTraceStep.of(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.SKIP, "workflow.decision.routing.skipped_ai_failed"), DecisionTraceStep.of(QueryDecisionStepKind.GRANT_FAST_PATH, StepOutcome.SKIP, @@ -239,7 +316,8 @@ private static QueryDecision aiFailed(List block, BytesCapCheck bytesCap "workflow.decision.plan.skipped_ai_failed")); return new QueryDecision(QueryDecisionKind.AI_FAILED_PENDING_REVIEW, QueryStatus.PENDING_REVIEW, null, null, null, null, null, - new DecisionTrace(steps, QueryStatus.PENDING_REVIEW), null, bytesCap, false); + new DecisionTrace(steps, QueryStatus.PENDING_REVIEW), null, bytesCap, false, + budget, false); } /** @@ -276,10 +354,12 @@ private QueryDecision routed(RoutingMatch match, ReviewPlanSnapshot plan, details.put("effective_min_approvals", effective); details.put("sql_review_suppressed", suppressedApprove && !block.isEmpty()); details.put("bytes_cap_suppressed", suppressedApprove && guard.capForcesReview()); + details.put("data_budget_suppressed", suppressedApprove && guard.budgetForcesReview()); steps.add(suppressedApprove ? new DecisionTraceStep(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.MATCH, block.isEmpty() - ? "workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap" + ? "workflow.decision.routing.matched_auto_approve_suppressed_" + + guard.reasonSuffix() : "workflow.decision.routing.matched_auto_approve_suppressed", List.of(String.valueOf(match.policyName())), details) : new DecisionTraceStep(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.MATCH, @@ -294,7 +374,8 @@ private QueryDecision routed(RoutingMatch match, ReviewPlanSnapshot plan, suppressedApprove && !block.isEmpty() ? new SqlReviewSuppression(block, List.of(SuppressedAutoApproval.ROUTING_AUTO_APPROVE)) : null, - guard.bytesCap(), suppressedApprove && guard.capForcesReview()); + guard.bytesCap(), suppressedApprove && guard.capForcesReview(), + guard.dataBudget(), suppressedApprove && guard.budgetForcesReview()); } /** @@ -406,6 +487,7 @@ private QueryDecision planned(QueryRequestSnapshot query, ReviewPlanSnapshot pla boolean planSuppressed = suppressed.contains(SuppressedAutoApproval.REVIEW_PLAN); details.put("sql_review_suppressed", planSuppressed && !block.isEmpty()); details.put("bytes_cap_suppressed", planSuppressed && guard.capForcesReview()); + details.put("data_budget_suppressed", planSuppressed && guard.budgetForcesReview()); steps.add(DecisionTraceStep.of(QueryDecisionStepKind.REVIEW_PLAN, nextStatus == QueryStatus.APPROVED ? StepOutcome.ALLOW : StepOutcome.DENY, reasonKey, details)); @@ -414,7 +496,8 @@ private QueryDecision planned(QueryRequestSnapshot query, ReviewPlanSnapshot pla : QueryDecisionKind.PLAN_PENDING_REVIEW; return new QueryDecision(kind, nextStatus, null, null, null, null, context, new DecisionTrace(steps, nextStatus), suppression(block, suppressed), - guard.bytesCap(), guard.capForcesReview() && !suppressed.isEmpty()); + guard.bytesCap(), guard.capForcesReview() && !suppressed.isEmpty(), + guard.dataBudget(), guard.budgetForcesReview() && !suppressed.isEmpty()); } private static LinkedHashMap consideredGrants(List grants) { diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionKind.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionKind.java index e79fd50d1..22770dbc9 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionKind.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionKind.java @@ -36,5 +36,11 @@ enum QueryDecisionKind { * The bytes-scanned cap (#941) refused the query: its estimate exceeds the cap, or it has none * and the datasource rejects on a missing estimate. Decided before routing and AI failure. */ - BYTES_CAP_REJECTED + BYTES_CAP_REJECTED, + + /** + * The submitter's data budget (#942) is exhausted and its breach action is {@code REJECT}. + * Decided right after the bytes-scanned cap, before routing and AI failure. + */ + DATA_BUDGET_REJECTED } diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachine.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachine.java index 76d68ce68..59899f726 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachine.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachine.java @@ -7,6 +7,8 @@ import com.bablsoft.accessflow.core.api.ByteSizeFormat; import com.bablsoft.accessflow.core.api.BytesScannedCapResolutionService; import com.bablsoft.accessflow.core.api.AppliedBytesCap; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.api.QueryType; import com.bablsoft.accessflow.core.api.QueryEstimateLookupService; import com.bablsoft.accessflow.core.api.QueryEstimateSnapshot; import com.bablsoft.accessflow.core.api.QueryRequestLookupService; @@ -92,6 +94,7 @@ class QueryReviewStateMachine { private final QueryCostEstimateService queryCostEstimateService; private final QueryEstimateLookupService queryEstimateLookupService; private final PlatformTransactionManager transactionManager; + private final DataBudgetStatusService dataBudgetStatusService; // Time-of-day / day-of-week routing conditions evaluate in the server's local zone. A field // (not an injected bean) so it can be overridden in tests without colliding with the proxy's @@ -180,8 +183,15 @@ private AppliedBytesCap resolveQuietly(UUID queryRequestId) { private void decide(QueryRequestSnapshot query, AiOutcome aiOutcome, RiskLevel riskLevel, int riskScore, AppliedBytesCap cap) { var bytesCap = cap == null ? null : BytesCapCheck.of(cap, estimatedBytes(query)); + var budget = dataBudget(query); var decision = queryDecisionEvaluator.evaluate(query, aiOutcome, riskLevel, riskScore, - blockingRuleIds(query), bytesCap, clock); + blockingRuleIds(query), bytesCap, budget, clock); + // #942: the reviewer is deciding on a submitter whose budget is used up — only an approval + // given here may later run the query past the exhausted budget. + if (budget != null && budget.forcesReview() + && decision.nextStatus() == QueryStatus.PENDING_REVIEW) { + queryRequestStateService.recordDataBudgetReviewForced(query.id()); + } if (bytesCap != null) { queryRequestStateService.recordBytesScannedCap(query.id(), bytesCap.limit(), bytesCap.source(), bytesCap.outcome()); @@ -197,6 +207,15 @@ private Long estimatedBytes(QueryRequestSnapshot query) { .orElse(null); } + /** The submitter's data-budget standing (#942); reads are the only thing a budget bounds. */ + private DataBudgetCheck dataBudget(QueryRequestSnapshot query) { + if (query.queryType() != QueryType.SELECT) { + return null; + } + return DataBudgetCheck.of(dataBudgetStatusService.statusFor(query.datasourceId(), + query.submittedByUserId())); + } + private List blockingRuleIds(QueryRequestSnapshot query) { return sqlReviewFindingService.blockingRuleIds(query.id()); } @@ -266,6 +285,12 @@ private void apply(QueryRequestSnapshot query, QueryDecision decision) { eventPublisher.publishEvent(new QueryAutoRejectedEvent(query.id(), null, bytesCapReason(decision.bytesCap()))); } + case DATA_BUDGET_REJECTED -> { + queryRequestStateService.transitionTo(query.id(), QueryStatus.PENDING_AI, + QueryStatus.REJECTED); + eventPublisher.publishEvent(new QueryAutoRejectedEvent(query.id(), null, + dataBudgetReason(decision.dataBudget()))); + } // A switch STATEMENT over an enum is not exhaustiveness-checked, so a new kind would // otherwise fall through silently and strand the query in PENDING_AI forever. default -> throw new IllegalStateException("Unhandled decision kind " + decision.kind()); @@ -276,6 +301,9 @@ private void apply(QueryRequestSnapshot query, QueryDecision decision) { if (decision.bytesCapChangedOutcome()) { auditBytesCapEnforced(query, decision); } + if (decision.dataBudgetChangedOutcome()) { + auditDataBudgetEnforced(query, decision); + } // Logged after the fact: a line claiming a query was auto-approved must not outlive a // persistence call that then failed. if (match != null) { @@ -344,6 +372,42 @@ private void auditBytesCapEnforced(QueryRequestSnapshot query, QueryDecision dec } } + /** + * One row per query whose outcome an exhausted data budget changed (#942) — a refusal, or an + * automatic approval turned into review. Swallow-and-log, like the rows above. + */ + private void auditDataBudgetEnforced(QueryRequestSnapshot query, QueryDecision decision) { + var budget = decision.dataBudget(); + var metadata = new LinkedHashMap(); + metadata.put("trigger", "data_budget"); + metadata.put("stage", "decision"); + metadata.put("action", budget.action().name()); + if (budget.deciding() != null) { + metadata.put("data_budget_id", budget.deciding().budgetId()); + metadata.put("used_rows", budget.deciding().usedRows()); + metadata.put("used_bytes", budget.deciding().usedBytes()); + metadata.put("window_minutes", budget.deciding().windowMinutes()); + } + if (decision.routingMatch() != null) { + metadata.put("matched_policy_id", decision.routingMatch().policyId()); + } + try { + auditLogService.record(new AuditEntry(AuditAction.QUERY_DATA_BUDGET_ENFORCED, + AuditResourceType.QUERY_REQUEST, query.id(), query.organizationId(), null, + metadata, null, null)); + } catch (RuntimeException ex) { + log.error("Audit write failed for QUERY_DATA_BUDGET_ENFORCED on query {}", query.id(), + ex); + } + } + + /** Server-default locale for the same reason as {@link #grantReason}. */ + private String dataBudgetReason(DataBudgetCheck budget) { + var name = budget.deciding() == null ? "-" : budget.deciding().name(); + return messageSource.getMessage("workflow.data_budget.rejected", new Object[]{name}, + Locale.getDefault()); + } + /** Server-default locale for the same reason as {@link #grantReason}. */ private String bytesCapReason(BytesCapCheck cap) { var limit = ByteSizeFormat.format(cap.limit()); diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactory.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactory.java index f631a013c..6abab1843 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactory.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactory.java @@ -1,10 +1,12 @@ package com.bablsoft.accessflow.workflow.internal.routing; import com.bablsoft.accessflow.ai.api.BehaviorAnomalyLookupService; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; import com.bablsoft.accessflow.core.api.QueryCorpusRow; import com.bablsoft.accessflow.core.api.QueryEstimateLookupService; import com.bablsoft.accessflow.core.api.QueryRequestLookupService; import com.bablsoft.accessflow.core.api.QueryShape; +import com.bablsoft.accessflow.core.api.QueryType; import com.bablsoft.accessflow.core.api.QueryRequestSnapshot; import com.bablsoft.accessflow.core.api.RiskLevel; import com.bablsoft.accessflow.core.api.UserGroupService; @@ -50,6 +52,7 @@ public class ConditionContextFactory { private final UserGroupService userGroupService; private final BehaviorAnomalyLookupService behaviorAnomalyLookupService; private final QueryEstimateLookupService queryEstimateLookupService; + private final DataBudgetStatusService dataBudgetStatusService; /** The live routing path: signals as they stand right now, which is also when routing runs. */ public ConditionContext forLiveQuery(QueryRequestSnapshot query, RiskLevel riskLevel, @@ -65,7 +68,17 @@ public ConditionContext forLiveQuery(QueryRequestSnapshot query, RiskLevel riskL behaviorAnomalyLookupService.hasActiveAnomaly(query.organizationId(), query.submittedByUserId(), query.datasourceId()), estimate.rows(), estimate.scanType(), parsed.shapes(), parsed.shapesAnalyzed(), - estimate.bytesScanned()); + estimate.bytesScanned(), dataBudgetUsedPercent(query)); + } + + /** Whole percentage of the submitter's most-used data budget (#942); SELECTs only. */ + private Integer dataBudgetUsedPercent(QueryRequestSnapshot query) { + if (query.queryType() != QueryType.SELECT) { + return null; + } + var used = dataBudgetStatusService.statusFor(query.datasourceId(), query.submittedByUserId()) + .usedPercent(); + return used == null ? null : (int) Math.min(Integer.MAX_VALUE, Math.floor(used)); } /** diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionNodeMixin.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionNodeMixin.java index 0befada1b..ea30e7bc1 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionNodeMixin.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionNodeMixin.java @@ -35,6 +35,7 @@ @JsonSubTypes.Type(value = ConditionNode.AnomalyDetected.class, name = "anomaly_detected"), @JsonSubTypes.Type(value = ConditionNode.EstimatedRows.class, name = "estimated_rows"), @JsonSubTypes.Type(value = ConditionNode.EstimatedBytesScanned.class, name = "estimated_bytes_scanned"), + @JsonSubTypes.Type(value = ConditionNode.DataBudgetUsedPercent.class, name = "data_budget_used_percent"), @JsonSubTypes.Type(value = ConditionNode.ScanTypeMatches.class, name = "scan_type") }) interface ConditionNodeMixin { diff --git a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluator.java b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluator.java index 69eb09520..558ce2613 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluator.java +++ b/backend/src/main/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluator.java @@ -55,6 +55,8 @@ public boolean matches(ConditionNode node, ConditionContext ctx) { ctx.hasEstimateSignal() && c.operator().test(ctx.estimatedRows(), c.value()); case ConditionNode.EstimatedBytesScanned c -> ctx.estimatedBytesScanned() != null && c.operator().test(ctx.estimatedBytesScanned(), c.value()); + case ConditionNode.DataBudgetUsedPercent c -> ctx.dataBudgetUsedPercent() != null + && c.operator().test(ctx.dataBudgetUsedPercent(), c.value()); case ConditionNode.ScanTypeMatches c -> matchesScanType(c, ctx); }; } diff --git a/backend/src/main/resources/db/migration/V193__create_data_budgets.sql b/backend/src/main/resources/db/migration/V193__create_data_budgets.sql new file mode 100644 index 000000000..7e7bccd4c --- /dev/null +++ b/backend/src/main/resources/db/migration/V193__create_data_budgets.sql @@ -0,0 +1,57 @@ +-- #942: per-user data-volume budgets. A data_budget bounds how many result rows and/or result bytes +-- EACH targeted user may read from one datasource over a rolling window (window_minutes, trailing +-- from now — no reset job, no timezone). Scoping mirrors row_limit_policy: all three applies_to_* +-- arrays empty ⇒ every user of the datasource; a group target gives every member their own +-- allowance, never a shared pool. When several budgets match, each is evaluated on its own window +-- and the most constrained one wins; REJECT beats REQUIRE_REVIEW once exhausted. + +CREATE TYPE data_budget_breach_action AS ENUM ('REJECT', 'REQUIRE_REVIEW'); +CREATE TYPE data_budget_usage_source AS ENUM ('QUERY', 'REQUEST_GROUP', 'SAMPLE_DATA'); + +CREATE TABLE data_budgets ( + id UUID PRIMARY KEY, + organization_id UUID NOT NULL REFERENCES organizations(id), + datasource_id UUID NOT NULL REFERENCES datasources(id) ON DELETE CASCADE, + name VARCHAR(120) NOT NULL, + max_rows BIGINT CHECK (max_rows > 0), + max_bytes BIGINT CHECK (max_bytes > 0), + window_minutes INTEGER NOT NULL DEFAULT 1440 + CHECK (window_minutes BETWEEN 60 AND 44640), + breach_action data_budget_breach_action NOT NULL DEFAULT 'REQUIRE_REVIEW', + warn_threshold_percent SMALLINT CHECK (warn_threshold_percent BETWEEN 1 AND 99), + applies_to_roles TEXT[], + applies_to_group_ids UUID[], + applies_to_user_ids UUID[], + enabled BOOLEAN NOT NULL DEFAULT true, + version BIGINT NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT chk_data_budgets_has_limit CHECK (max_rows IS NOT NULL OR max_bytes IS NOT NULL) +); + +CREATE INDEX idx_data_budgets_ds_enabled ON data_budgets (organization_id, datasource_id, enabled); + +-- Append-only usage ledger: one row per delivered SELECT result (query execution, request-group +-- member, sample-data read). rows_read / bytes_read are what the user actually received — after +-- row-security filtering and every cap. Pruned by DataBudgetUsagePruneJob past the longest window. +-- query_request_id / request_group_id are bare provenance UUIDs (no FK) so a later erasure or +-- retention delete of the source never blocks on the ledger. +CREATE TABLE data_budget_usage ( + id UUID PRIMARY KEY, + organization_id UUID NOT NULL, + user_id UUID NOT NULL, + datasource_id UUID NOT NULL, + rows_read BIGINT NOT NULL CHECK (rows_read >= 0), + bytes_read BIGINT NOT NULL CHECK (bytes_read >= 0), + source data_budget_usage_source NOT NULL, + query_request_id UUID, + request_group_id UUID, + occurred_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +-- Backs the trailing-window SUM (user, datasource, occurred_at >= now - window): an index range +-- scan bounded by one user's reads on one datasource inside the window. +CREATE INDEX idx_data_budget_usage_user_ds_time + ON data_budget_usage (user_id, datasource_id, occurred_at); +-- Backs the prune job's range delete. +CREATE INDEX idx_data_budget_usage_occurred_at ON data_budget_usage (occurred_at); diff --git a/backend/src/main/resources/db/migration/V194__add_data_budget_manage_permission.sql b/backend/src/main/resources/db/migration/V194__add_data_budget_manage_permission.sql new file mode 100644 index 000000000..8503ee826 --- /dev/null +++ b/backend/src/main/resources/db/migration/V194__add_data_budget_manage_permission.sql @@ -0,0 +1,7 @@ +-- #942: seeds the new DATA_BUDGET_MANAGE permission for the ADMIN system role, mirroring +-- core.api.SystemRolePermissions (ADMIN holds every catalog value via EnumSet.allOf). +-- role_permissions.permission is VARCHAR, so the new value itself needs no DDL. + +INSERT INTO role_permissions (role_id, permission) VALUES + ('c0000000-0000-0000-0000-000000000001', 'DATA_BUDGET_MANAGE') +ON CONFLICT (role_id, permission) DO NOTHING; diff --git a/backend/src/main/resources/db/migration/V195__add_query_data_budget_review_forced.sql b/backend/src/main/resources/db/migration/V195__add_query_data_budget_review_forced.sql new file mode 100644 index 000000000..288d507a0 --- /dev/null +++ b/backend/src/main/resources/db/migration/V195__add_query_data_budget_review_forced.sql @@ -0,0 +1,5 @@ +-- #942: set when an exhausted REQUIRE_REVIEW data budget forced the query into human review as it +-- left PENDING_AI. Only a query carrying this stamp may run past an exhausted budget at execution: +-- an approval given while allowance remained was never a budget escalation, so it must not lift +-- the budget once the submitter has spent the rest of it on other reads. +ALTER TABLE query_requests ADD COLUMN data_budget_review_forced BOOLEAN NOT NULL DEFAULT FALSE; diff --git a/backend/src/main/resources/i18n/messages.properties b/backend/src/main/resources/i18n/messages.properties index f28bc607e..7b9eea6fc 100644 --- a/backend/src/main/resources/i18n/messages.properties +++ b/backend/src/main/resources/i18n/messages.properties @@ -1604,3 +1604,55 @@ workflow.decision.plan.skipped_bytes_cap=The review plan is not consulted when t workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap=Routing policy "{0}" matched with action AUTO_APPROVE, suppressed because no bytes estimate is available under the bytes-scanned cap workflow.decision.grant.suppressed_bytes_cap=Covered by access grant {0}, suppressed because no bytes estimate is available under the bytes-scanned cap workflow.decision.plan.suppressed_bytes_cap=The review plan would have approved, suppressed because no bytes estimate is available under the bytes-scanned cap +# Data-volume budgets (#942) +error.data_budget.name_invalid=Budget name is required and must be at most 120 characters +error.data_budget.limit_required=A data budget needs a row limit, a byte limit, or both +error.data_budget.limit_invalid=Row and byte limits must be positive numbers +error.data_budget.window_invalid=The budget window must be between 1 hour and 31 days +error.data_budget.threshold_invalid=The warning threshold must be between 1 and 99 percent +error.data_budget.unknown_role=Unknown applies-to role: {0} +error.data_budget.user_not_in_org=One or more applies-to users do not belong to this organization +error.data_budget.group_not_in_org=One or more applies-to groups do not belong to this organization +error.data_budget.not_found=Data budget not found +error.data_budget.exhausted=Refused: your data budget "{0}" on this datasource is used up for the current window +validation.data_budget.name.required=Budget name is required +validation.data_budget.name.size=Budget name must be at most 120 characters +validation.data_budget.max_rows.min=Row limit must be at least 1 +validation.data_budget.max_bytes.min=Byte limit must be at least 1 +validation.data_budget.window.range=The budget window must be between 60 and 44,640 minutes (1 hour to 31 days) +validation.data_budget.threshold.range=The warning threshold must be between 1 and 99 percent +workflow.data_budget.rejected=Rejected: the submitter data budget "{0}" on this datasource is used up for the current window +workflow.decision.data_budget.none=No data budget applies +workflow.decision.data_budget.within={0}% of the data budget is used; the result is capped to the remaining allowance +workflow.decision.data_budget.exhausted_rejected=The data budget "{0}" is used up — the request is rejected +workflow.decision.data_budget.exhausted_review=The data budget "{0}" is used up — the request cannot auto-approve +workflow.decision.routing.skipped_data_budget=Routing policies are not evaluated when an exhausted data budget rejects the request +workflow.decision.grant.skipped_data_budget=The grant fast path is not evaluated when an exhausted data budget rejects the request +workflow.decision.plan.skipped_data_budget=The review plan is not consulted when an exhausted data budget rejects the request +workflow.decision.routing.matched_auto_approve_suppressed_data_budget=Routing policy "{0}" matched with action AUTO_APPROVE, suppressed because the data budget is used up +workflow.decision.grant.suppressed_data_budget=Covered by access grant {0}, suppressed because the data budget is used up +workflow.decision.plan.suppressed_data_budget=The review plan would have approved, suppressed because the data budget is used up + +# Data-budget notifications (#942) +notification.email.subject.data_budget_threshold_reached=[AccessFlow] Data budget "{0}" on {1} is {2}% used +notification.email.subject.data_budget_exhausted=[AccessFlow] Data budget "{0}" on {1} is used up +notification.email.data_budget_threshold_reached.title=Data budget warning +notification.email.data_budget_threshold_reached.heading=You are approaching your data budget +notification.email.data_budget_threshold_reached.body=Your reads on this datasource have crossed the warning threshold of your data budget. Once the budget is used up, further SELECT queries are rejected or sent to review until usage falls back under the limit. +notification.email.data_budget_exhausted.title=Data budget exhausted +notification.email.data_budget_exhausted.heading=A data budget has been used up +notification.email.data_budget_exhausted.body=The data budget below is used up for its current window. Further SELECT queries by this user on this datasource are handled per the breach action until usage falls back under the limit. +notification.email.data_budget.user_label=User: +notification.email.data_budget.budget_label=Budget: +notification.email.data_budget.used_label=Used: +notification.email.data_budget.rows_label=Rows: +notification.email.data_budget.bytes_label=Bytes: +notification.email.data_budget.window_label=Window: +notification.email.data_budget.threshold_label=Warning threshold: +notification.email.data_budget.breach_action_label=When used up: +notification.email.data_budget.breach_action.REJECT=Further reads are rejected +notification.email.data_budget.breach_action.REQUIRE_REVIEW=Further reads go to human review +notification.email.data_budget.window.MINUTES={0,choice,1#1 minute|1<{0} minutes} +notification.email.data_budget.window.HOURS={0,choice,1#1 hour|1<{0} hours} +notification.email.data_budget.window.DAYS={0,choice,1#1 day|1<{0} days} +notification.email.data_budget.cta=Open the query editor diff --git a/backend/src/main/resources/i18n/messages_de.properties b/backend/src/main/resources/i18n/messages_de.properties index 77c24973e..a33a9ad1f 100644 --- a/backend/src/main/resources/i18n/messages_de.properties +++ b/backend/src/main/resources/i18n/messages_de.properties @@ -1583,3 +1583,55 @@ workflow.decision.plan.skipped_bytes_cap=Der Prüfplan wird nicht herangezogen, workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap=Routing-Richtlinie "{0}" hat mit Aktion AUTO_APPROVE gegriffen, unterdrückt, weil unter dem Limit für gescannte Bytes keine Byte-Schätzung vorliegt workflow.decision.grant.suppressed_bytes_cap=Durch Zugriffsgewährung {0} abgedeckt, unterdrückt, weil unter dem Limit für gescannte Bytes keine Byte-Schätzung vorliegt workflow.decision.plan.suppressed_bytes_cap=Der Prüfplan hätte genehmigt, unterdrückt, weil unter dem Limit für gescannte Bytes keine Byte-Schätzung vorliegt +# Data-volume budgets (#942) +error.data_budget.name_invalid=Der Budgetname ist erforderlich und darf höchstens 120 Zeichen lang sein +error.data_budget.limit_required=Ein Datenbudget benötigt ein Zeilenlimit, ein Byte-Limit oder beides +error.data_budget.limit_invalid=Zeilen- und Byte-Limits müssen positive Zahlen sein +error.data_budget.window_invalid=Das Budgetfenster muss zwischen 1 Stunde und 31 Tagen liegen +error.data_budget.threshold_invalid=Die Warnschwelle muss zwischen 1 und 99 Prozent liegen +error.data_budget.unknown_role=Unbekannte Anwenden-auf-Rolle: {0} +error.data_budget.user_not_in_org=Ein oder mehrere Anwenden-auf-Benutzer gehören nicht zu dieser Organisation +error.data_budget.group_not_in_org=Eine oder mehrere Anwenden-auf-Gruppen gehören nicht zu dieser Organisation +error.data_budget.not_found=Datenbudget nicht gefunden +error.data_budget.exhausted=Abgelehnt: Ihr Datenbudget „{0}“ für diese Datenquelle ist im aktuellen Zeitfenster aufgebraucht +validation.data_budget.name.required=Der Budgetname ist erforderlich +validation.data_budget.name.size=Der Budgetname darf höchstens 120 Zeichen lang sein +validation.data_budget.max_rows.min=Das Zeilenlimit muss mindestens 1 betragen +validation.data_budget.max_bytes.min=Das Byte-Limit muss mindestens 1 betragen +validation.data_budget.window.range=Das Budgetfenster muss zwischen 60 und 44.640 Minuten (1 Stunde bis 31 Tage) liegen +validation.data_budget.threshold.range=Die Warnschwelle muss zwischen 1 und 99 Prozent liegen +workflow.data_budget.rejected=Abgelehnt: Das Datenbudget „{0}“ des Einreichenden für diese Datenquelle ist im aktuellen Zeitfenster aufgebraucht +workflow.decision.data_budget.none=Es gilt kein Datenbudget +workflow.decision.data_budget.within={0} % des Datenbudgets sind verbraucht; das Ergebnis wird auf das verbleibende Kontingent begrenzt +workflow.decision.data_budget.exhausted_rejected=Das Datenbudget „{0}“ ist aufgebraucht — die Anfrage wird abgelehnt +workflow.decision.data_budget.exhausted_review=Das Datenbudget „{0}“ ist aufgebraucht — die Anfrage kann nicht automatisch genehmigt werden +workflow.decision.routing.skipped_data_budget=Routing-Richtlinien werden nicht ausgewertet, wenn ein aufgebrauchtes Datenbudget die Anfrage ablehnt +workflow.decision.grant.skipped_data_budget=Der Grant-Schnellpfad wird nicht ausgewertet, wenn ein aufgebrauchtes Datenbudget die Anfrage ablehnt +workflow.decision.plan.skipped_data_budget=Der Prüfplan wird nicht herangezogen, wenn ein aufgebrauchtes Datenbudget die Anfrage ablehnt +workflow.decision.routing.matched_auto_approve_suppressed_data_budget=Routing-Richtlinie „{0}“ hat mit der Aktion AUTO_APPROVE gegriffen, unterdrückt, weil das Datenbudget aufgebraucht ist +workflow.decision.grant.suppressed_data_budget=Durch Zugriffsgewährung {0} abgedeckt, unterdrückt, weil das Datenbudget aufgebraucht ist +workflow.decision.plan.suppressed_data_budget=Der Prüfplan hätte genehmigt, unterdrückt, weil das Datenbudget aufgebraucht ist + +# Data-budget notifications (#942) +notification.email.subject.data_budget_threshold_reached=[AccessFlow] Datenbudget „{0}“ auf {1} zu {2} % verbraucht +notification.email.subject.data_budget_exhausted=[AccessFlow] Datenbudget „{0}“ auf {1} ist aufgebraucht +notification.email.data_budget_threshold_reached.title=Datenbudget-Warnung +notification.email.data_budget_threshold_reached.heading=Sie nähern sich Ihrem Datenbudget +notification.email.data_budget_threshold_reached.body=Ihre Lesezugriffe auf diese Datenquelle haben die Warnschwelle Ihres Datenbudgets überschritten. Ist das Budget aufgebraucht, werden weitere SELECT-Abfragen abgelehnt oder zur Prüfung geleitet, bis die Nutzung wieder unter das Limit fällt. +notification.email.data_budget_exhausted.title=Datenbudget aufgebraucht +notification.email.data_budget_exhausted.heading=Ein Datenbudget wurde aufgebraucht +notification.email.data_budget_exhausted.body=Das folgende Datenbudget ist für das aktuelle Zeitfenster aufgebraucht. Weitere SELECT-Abfragen dieses Benutzers auf dieser Datenquelle werden gemäß der Überschreitungsaktion behandelt, bis die Nutzung wieder unter das Limit fällt. +notification.email.data_budget.user_label=Benutzer: +notification.email.data_budget.budget_label=Budget: +notification.email.data_budget.used_label=Verbraucht: +notification.email.data_budget.rows_label=Zeilen: +notification.email.data_budget.bytes_label=Bytes: +notification.email.data_budget.window_label=Zeitfenster: +notification.email.data_budget.threshold_label=Warnschwelle: +notification.email.data_budget.breach_action_label=Bei Erschöpfung: +notification.email.data_budget.breach_action.REJECT=Weitere Lesezugriffe werden abgelehnt +notification.email.data_budget.breach_action.REQUIRE_REVIEW=Weitere Lesezugriffe gehen zur manuellen Prüfung +notification.email.data_budget.window.MINUTES={0,choice,1#1 Minute|1<{0} Minuten} +notification.email.data_budget.window.HOURS={0,choice,1#1 Stunde|1<{0} Stunden} +notification.email.data_budget.window.DAYS={0,choice,1#1 Tag|1<{0} Tage} +notification.email.data_budget.cta=Abfrage-Editor öffnen diff --git a/backend/src/main/resources/i18n/messages_es.properties b/backend/src/main/resources/i18n/messages_es.properties index de158e904..af22ea23e 100644 --- a/backend/src/main/resources/i18n/messages_es.properties +++ b/backend/src/main/resources/i18n/messages_es.properties @@ -1583,3 +1583,55 @@ workflow.decision.plan.skipped_bytes_cap=El plan de revisión no se consulta cua workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap=La política de enrutamiento "{0}" coincidió con la acción AUTO_APPROVE, suprimida porque no hay estimación de bytes bajo el límite de bytes escaneados workflow.decision.grant.suppressed_bytes_cap=Cubierta por la concesión de acceso {0}, suprimida porque no hay estimación de bytes bajo el límite de bytes escaneados workflow.decision.plan.suppressed_bytes_cap=El plan de revisión habría aprobado, suprimido porque no hay estimación de bytes bajo el límite de bytes escaneados +# Data-volume budgets (#942) +error.data_budget.name_invalid=El nombre del presupuesto es obligatorio y debe tener como máximo 120 caracteres +error.data_budget.limit_required=Un presupuesto de datos necesita un límite de filas, un límite de bytes o ambos +error.data_budget.limit_invalid=Los límites de filas y bytes deben ser números positivos +error.data_budget.window_invalid=La ventana del presupuesto debe estar entre 1 hora y 31 días +error.data_budget.threshold_invalid=El umbral de aviso debe estar entre 1 y 99 por ciento +error.data_budget.unknown_role=Rol de aplicación desconocido: {0} +error.data_budget.user_not_in_org=Uno o más usuarios de aplicación no pertenecen a esta organización +error.data_budget.group_not_in_org=Uno o más grupos de aplicación no pertenecen a esta organización +error.data_budget.not_found=Presupuesto de datos no encontrado +error.data_budget.exhausted=Rechazado: tu presupuesto de datos «{0}» en esta fuente de datos está agotado en la ventana actual +validation.data_budget.name.required=El nombre del presupuesto es obligatorio +validation.data_budget.name.size=El nombre del presupuesto debe tener como máximo 120 caracteres +validation.data_budget.max_rows.min=El límite de filas debe ser al menos 1 +validation.data_budget.max_bytes.min=El límite de bytes debe ser al menos 1 +validation.data_budget.window.range=La ventana del presupuesto debe estar entre 60 y 44.640 minutos (de 1 hora a 31 días) +validation.data_budget.threshold.range=El umbral de aviso debe estar entre 1 y 99 por ciento +workflow.data_budget.rejected=Rechazada: el presupuesto de datos «{0}» del solicitante en esta fuente de datos está agotado en la ventana actual +workflow.decision.data_budget.none=No se aplica ningún presupuesto de datos +workflow.decision.data_budget.within=Se ha usado el {0} % del presupuesto de datos; el resultado se limita a la cuota restante +workflow.decision.data_budget.exhausted_rejected=El presupuesto de datos «{0}» está agotado: la solicitud se rechaza +workflow.decision.data_budget.exhausted_review=El presupuesto de datos «{0}» está agotado: la solicitud no puede aprobarse automáticamente +workflow.decision.routing.skipped_data_budget=Las políticas de enrutamiento no se evalúan cuando un presupuesto de datos agotado rechaza la solicitud +workflow.decision.grant.skipped_data_budget=La vía rápida por concesión no se evalúa cuando un presupuesto de datos agotado rechaza la solicitud +workflow.decision.plan.skipped_data_budget=El plan de revisión no se consulta cuando un presupuesto de datos agotado rechaza la solicitud +workflow.decision.routing.matched_auto_approve_suppressed_data_budget=La política de enrutamiento «{0}» coincidió con la acción AUTO_APPROVE, suprimida porque el presupuesto de datos está agotado +workflow.decision.grant.suppressed_data_budget=Cubierta por la concesión de acceso {0}, suprimida porque el presupuesto de datos está agotado +workflow.decision.plan.suppressed_data_budget=El plan de revisión habría aprobado, suprimido porque el presupuesto de datos está agotado + +# Data-budget notifications (#942) +notification.email.subject.data_budget_threshold_reached=[AccessFlow] Presupuesto de datos «{0}» en {1} al {2} % de uso +notification.email.subject.data_budget_exhausted=[AccessFlow] Presupuesto de datos «{0}» en {1} agotado +notification.email.data_budget_threshold_reached.title=Aviso de presupuesto de datos +notification.email.data_budget_threshold_reached.heading=Se está acercando a su presupuesto de datos +notification.email.data_budget_threshold_reached.body=Sus lecturas en esta fuente de datos han superado el umbral de aviso de su presupuesto de datos. Cuando el presupuesto se agote, las siguientes consultas SELECT se rechazarán o se enviarán a revisión hasta que el uso vuelva a estar por debajo del límite. +notification.email.data_budget_exhausted.title=Presupuesto de datos agotado +notification.email.data_budget_exhausted.heading=Se ha agotado un presupuesto de datos +notification.email.data_budget_exhausted.body=El siguiente presupuesto de datos está agotado para su ventana actual. Las siguientes consultas SELECT de este usuario en esta fuente de datos se tratan según la acción de incumplimiento hasta que el uso vuelva a estar por debajo del límite. +notification.email.data_budget.user_label=Usuario: +notification.email.data_budget.budget_label=Presupuesto: +notification.email.data_budget.used_label=Usado: +notification.email.data_budget.rows_label=Filas: +notification.email.data_budget.bytes_label=Bytes: +notification.email.data_budget.window_label=Ventana: +notification.email.data_budget.threshold_label=Umbral de aviso: +notification.email.data_budget.breach_action_label=Al agotarse: +notification.email.data_budget.breach_action.REJECT=Las siguientes lecturas se rechazan +notification.email.data_budget.breach_action.REQUIRE_REVIEW=Las siguientes lecturas pasan a revisión manual +notification.email.data_budget.window.MINUTES={0,choice,1#1 minuto|1<{0} minutos} +notification.email.data_budget.window.HOURS={0,choice,1#1 hora|1<{0} horas} +notification.email.data_budget.window.DAYS={0,choice,1#1 día|1<{0} días} +notification.email.data_budget.cta=Abrir el editor de consultas diff --git a/backend/src/main/resources/i18n/messages_fr.properties b/backend/src/main/resources/i18n/messages_fr.properties index b072b3a59..d604f114f 100644 --- a/backend/src/main/resources/i18n/messages_fr.properties +++ b/backend/src/main/resources/i18n/messages_fr.properties @@ -1589,3 +1589,55 @@ workflow.decision.plan.skipped_bytes_cap=Le plan de revue n’est pas consulté workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap=La politique de routage "{0}" a correspondu avec l’action AUTO_APPROVE, neutralisée faute d’estimation en octets sous la limite d’octets analysés workflow.decision.grant.suppressed_bytes_cap=Couverte par l’autorisation d’accès {0}, neutralisée faute d’estimation en octets sous la limite d’octets analysés workflow.decision.plan.suppressed_bytes_cap=Le plan de revue aurait approuvé, neutralisé faute d’estimation en octets sous la limite d’octets analysés +# Data-volume budgets (#942) +error.data_budget.name_invalid=Le nom du budget est obligatoire et doit comporter au plus 120 caractères +error.data_budget.limit_required=Un budget de données nécessite une limite de lignes, une limite d’octets, ou les deux +error.data_budget.limit_invalid=Les limites de lignes et d’octets doivent être des nombres positifs +error.data_budget.window_invalid=La fenêtre du budget doit être comprise entre 1 heure et 31 jours +error.data_budget.threshold_invalid=Le seuil d’avertissement doit être compris entre 1 et 99 pour cent +error.data_budget.unknown_role=Rôle d’application inconnu : {0} +error.data_budget.user_not_in_org=Un ou plusieurs utilisateurs ciblés n’appartiennent pas à cette organisation +error.data_budget.group_not_in_org=Un ou plusieurs groupes ciblés n’appartiennent pas à cette organisation +error.data_budget.not_found=Budget de données introuvable +error.data_budget.exhausted=Refusé : votre budget de données « {0} » sur cette source de données est épuisé pour la fenêtre en cours +validation.data_budget.name.required=Le nom du budget est obligatoire +validation.data_budget.name.size=Le nom du budget doit comporter au plus 120 caractères +validation.data_budget.max_rows.min=La limite de lignes doit être d’au moins 1 +validation.data_budget.max_bytes.min=La limite d’octets doit être d’au moins 1 +validation.data_budget.window.range=La fenêtre du budget doit être comprise entre 60 et 44 640 minutes (de 1 heure à 31 jours) +validation.data_budget.threshold.range=Le seuil d’avertissement doit être compris entre 1 et 99 pour cent +workflow.data_budget.rejected=Rejetée : le budget de données « {0} » du demandeur sur cette source de données est épuisé pour la fenêtre en cours +workflow.decision.data_budget.none=Aucun budget de données ne s’applique +workflow.decision.data_budget.within={0} % du budget de données est consommé ; le résultat est plafonné au quota restant +workflow.decision.data_budget.exhausted_rejected=Le budget de données « {0} » est épuisé — la demande est rejetée +workflow.decision.data_budget.exhausted_review=Le budget de données « {0} » est épuisé — la demande ne peut pas être approuvée automatiquement +workflow.decision.routing.skipped_data_budget=Les politiques de routage ne sont pas évaluées lorsqu’un budget de données épuisé rejette la demande +workflow.decision.grant.skipped_data_budget=La voie rapide par autorisation n’est pas évaluée lorsqu’un budget de données épuisé rejette la demande +workflow.decision.plan.skipped_data_budget=Le plan de revue n’est pas consulté lorsqu’un budget de données épuisé rejette la demande +workflow.decision.routing.matched_auto_approve_suppressed_data_budget=La politique de routage « {0} » correspond avec l’action AUTO_APPROVE, suspendue car le budget de données est épuisé +workflow.decision.grant.suppressed_data_budget=Couverte par l’autorisation d’accès {0}, suspendue car le budget de données est épuisé +workflow.decision.plan.suppressed_data_budget=Le plan de revue aurait approuvé, suspendu car le budget de données est épuisé + +# Data-budget notifications (#942) +notification.email.subject.data_budget_threshold_reached=[AccessFlow] Budget de données « {0} » sur {1} utilisé à {2} % +notification.email.subject.data_budget_exhausted=[AccessFlow] Budget de données « {0} » sur {1} épuisé +notification.email.data_budget_threshold_reached.title=Alerte de budget de données +notification.email.data_budget_threshold_reached.heading=Vous approchez de votre budget de données +notification.email.data_budget_threshold_reached.body=Vos lectures sur cette source de données ont franchi le seuil d’alerte de votre budget de données. Une fois le budget épuisé, les requêtes SELECT suivantes sont rejetées ou envoyées en revue jusqu’à ce que l’utilisation repasse sous la limite. +notification.email.data_budget_exhausted.title=Budget de données épuisé +notification.email.data_budget_exhausted.heading=Un budget de données a été épuisé +notification.email.data_budget_exhausted.body=Le budget de données ci-dessous est épuisé pour sa fenêtre actuelle. Les requêtes SELECT suivantes de cet utilisateur sur cette source de données sont traitées selon l’action de dépassement jusqu’à ce que l’utilisation repasse sous la limite. +notification.email.data_budget.user_label=Utilisateur : +notification.email.data_budget.budget_label=Budget : +notification.email.data_budget.used_label=Utilisé : +notification.email.data_budget.rows_label=Lignes : +notification.email.data_budget.bytes_label=Octets : +notification.email.data_budget.window_label=Fenêtre : +notification.email.data_budget.threshold_label=Seuil d’alerte : +notification.email.data_budget.breach_action_label=Une fois épuisé : +notification.email.data_budget.breach_action.REJECT=Les lectures suivantes sont rejetées +notification.email.data_budget.breach_action.REQUIRE_REVIEW=Les lectures suivantes passent en revue manuelle +notification.email.data_budget.window.MINUTES={0,choice,1#1 minute|1<{0} minutes} +notification.email.data_budget.window.HOURS={0,choice,1#1 heure|1<{0} heures} +notification.email.data_budget.window.DAYS={0,choice,1#1 jour|1<{0} jours} +notification.email.data_budget.cta=Ouvrir l’éditeur de requêtes diff --git a/backend/src/main/resources/i18n/messages_hy.properties b/backend/src/main/resources/i18n/messages_hy.properties index deb3b475b..a6eb01256 100644 --- a/backend/src/main/resources/i18n/messages_hy.properties +++ b/backend/src/main/resources/i18n/messages_hy.properties @@ -1583,3 +1583,55 @@ workflow.decision.plan.skipped_bytes_cap=Ստուգման պլանը չի դիտ workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap=«{0}» երթուղավորման քաղաքականությունը համընկավ AUTO_APPROVE գործողությամբ, ճնշվել է, քանի որ սկանավորված բայթերի սահմանաչափի ներքո բայթերի գնահատական չկա workflow.decision.grant.suppressed_bytes_cap=Ծածկված է {0} մուտքի թույլտվությամբ, ճնշվել է, քանի որ սկանավորված բայթերի սահմանաչափի ներքո բայթերի գնահատական չկա workflow.decision.plan.suppressed_bytes_cap=Ստուգման պլանը կհաստատեր, ճնշվել է, քանի որ սկանավորված բայթերի սահմանաչափի ներքո բայթերի գնահատական չկա +# Data-volume budgets (#942) +error.data_budget.name_invalid=Բյուջեի անունը պարտադիր է և պետք է լինի առավելագույնը 120 նիշ +error.data_budget.limit_required=Տվյալների բյուջեին անհրաժեշտ է տողերի սահմանաչափ, բայթերի սահմանաչափ կամ երկուսն էլ +error.data_budget.limit_invalid=Տողերի և բայթերի սահմանաչափերը պետք է լինեն դրական թվեր +error.data_budget.window_invalid=Բյուջեի պատուհանը պետք է լինի 1 ժամից մինչև 31 օր +error.data_budget.threshold_invalid=Զգուշացման շեմը պետք է լինի 1-ից 99 տոկոս +error.data_budget.unknown_role=Կիրառման անհայտ դեր՝ {0} +error.data_budget.user_not_in_org=Կիրառման մեկ կամ մի քանի օգտատերեր չեն պատկանում այս կազմակերպությանը +error.data_budget.group_not_in_org=Կիրառման մեկ կամ մի քանի խմբեր չեն պատկանում այս կազմակերպությանը +error.data_budget.not_found=Տվյալների բյուջեն չի գտնվել +error.data_budget.exhausted=Մերժված է. այս տվյալների աղբյուրում ձեր «{0}» տվյալների բյուջեն սպառված է ընթացիկ պատուհանի համար +validation.data_budget.name.required=Բյուջեի անունը պարտադիր է +validation.data_budget.name.size=Բյուջեի անունը պետք է լինի առավելագույնը 120 նիշ +validation.data_budget.max_rows.min=Տողերի սահմանաչափը պետք է լինի առնվազն 1 +validation.data_budget.max_bytes.min=Բայթերի սահմանաչափը պետք է լինի առնվազն 1 +validation.data_budget.window.range=Բյուջեի պատուհանը պետք է լինի 60-ից 44 640 րոպե (1 ժամից մինչև 31 օր) +validation.data_budget.threshold.range=Զգուշացման շեմը պետք է լինի 1-ից 99 տոկոս +workflow.data_budget.rejected=Մերժված է. ներկայացնողի «{0}» տվյալների բյուջեն այս աղբյուրում սպառված է ընթացիկ պատուհանի համար +workflow.decision.data_budget.none=Տվյալների բյուջե չի կիրառվում +workflow.decision.data_budget.within=Տվյալների բյուջեի {0}%-ն օգտագործված է. արդյունքը սահմանափակվում է մնացած թույլատրելի ծավալով +workflow.decision.data_budget.exhausted_rejected=«{0}» տվյալների բյուջեն սպառված է — հարցումը մերժվում է +workflow.decision.data_budget.exhausted_review=«{0}» տվյալների բյուջեն սպառված է — հարցումը չի կարող ավտոմատ հաստատվել +workflow.decision.routing.skipped_data_budget=Երթուղման քաղաքականությունները չեն գնահատվում, երբ սպառված տվյալների բյուջեն մերժում է հարցումը +workflow.decision.grant.skipped_data_budget=Թույլտվության արագ ուղին չի գնահատվում, երբ սպառված տվյալների բյուջեն մերժում է հարցումը +workflow.decision.plan.skipped_data_budget=Վերանայման պլանը չի դիտարկվում, երբ սպառված տվյալների բյուջեն մերժում է հարցումը +workflow.decision.routing.matched_auto_approve_suppressed_data_budget=«{0}» երթուղման քաղաքականությունը համընկավ AUTO_APPROVE գործողությամբ, կասեցված է, քանի որ տվյալների բյուջեն սպառված է +workflow.decision.grant.suppressed_data_budget=Ծածկված է {0} մուտքի թույլտվությամբ, կասեցված է, քանի որ տվյալների բյուջեն սպառված է +workflow.decision.plan.suppressed_data_budget=Վերանայման պլանը կհաստատեր, կասեցված է, քանի որ տվյալների բյուջեն սպառված է + +# Data-budget notifications (#942) +notification.email.subject.data_budget_threshold_reached=[AccessFlow] «{0}» տվյալների բյուջեն {1}-ում օգտագործված է {2}%-ով +notification.email.subject.data_budget_exhausted=[AccessFlow] «{0}» տվյալների բյուջեն {1}-ում սպառված է +notification.email.data_budget_threshold_reached.title=Տվյալների բյուջեի զգուշացում +notification.email.data_budget_threshold_reached.heading=Դուք մոտենում եք ձեր տվյալների բյուջեին +notification.email.data_budget_threshold_reached.body=Այս տվյալների աղբյուրում ձեր ընթերցումները անցել են ձեր տվյալների բյուջեի զգուշացման շեմը։ Բյուջեն սպառվելուց հետո հաջորդ SELECT հարցումները կմերժվեն կամ կուղարկվեն վերանայման, մինչև օգտագործումը նորից իջնի սահմանաչափից։ +notification.email.data_budget_exhausted.title=Տվյալների բյուջեն սպառված է +notification.email.data_budget_exhausted.heading=Տվյալների բյուջեն սպառվել է +notification.email.data_budget_exhausted.body=Ստորև նշված տվյալների բյուջեն սպառված է ընթացիկ պատուհանի համար։ Այս օգտատիրոջ հաջորդ SELECT հարցումները այս տվյալների աղբյուրում կմշակվեն խախտման գործողության համաձայն, մինչև օգտագործումը նորից իջնի սահմանաչափից։ +notification.email.data_budget.user_label=Օգտատեր. +notification.email.data_budget.budget_label=Բյուջե. +notification.email.data_budget.used_label=Օգտագործված. +notification.email.data_budget.rows_label=Տողեր. +notification.email.data_budget.bytes_label=Բայթեր. +notification.email.data_budget.window_label=Պատուհան. +notification.email.data_budget.threshold_label=Զգուշացման շեմ. +notification.email.data_budget.breach_action_label=Սպառվելիս. +notification.email.data_budget.breach_action.REJECT=Հաջորդ ընթերցումները մերժվում են +notification.email.data_budget.breach_action.REQUIRE_REVIEW=Հաջորդ ընթերցումներն ուղարկվում են ձեռքով վերանայման +notification.email.data_budget.window.MINUTES={0} րոպե +notification.email.data_budget.window.HOURS={0} ժամ +notification.email.data_budget.window.DAYS={0} օր +notification.email.data_budget.cta=Բացել հարցումների խմբագիրը diff --git a/backend/src/main/resources/i18n/messages_ru.properties b/backend/src/main/resources/i18n/messages_ru.properties index ccadf8a29..1eed69c19 100644 --- a/backend/src/main/resources/i18n/messages_ru.properties +++ b/backend/src/main/resources/i18n/messages_ru.properties @@ -1583,3 +1583,55 @@ workflow.decision.plan.skipped_bytes_cap=План проверки не учит workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap=Политика маршрутизации "{0}" сработала с действием AUTO_APPROVE, подавлено, так как при лимите сканируемых байтов оценка в байтах недоступна workflow.decision.grant.suppressed_bytes_cap=Покрыт выдачей доступа {0}, подавлено, так как при лимите сканируемых байтов оценка в байтах недоступна workflow.decision.plan.suppressed_bytes_cap=План проверки одобрил бы запрос, подавлено, так как при лимите сканируемых байтов оценка в байтах недоступна +# Data-volume budgets (#942) +error.data_budget.name_invalid=Название бюджета обязательно и должно содержать не более 120 символов +error.data_budget.limit_required=Для бюджета данных нужен лимит строк, лимит байтов или оба +error.data_budget.limit_invalid=Лимиты строк и байтов должны быть положительными числами +error.data_budget.window_invalid=Окно бюджета должно составлять от 1 часа до 31 дня +error.data_budget.threshold_invalid=Порог предупреждения должен быть от 1 до 99 процентов +error.data_budget.unknown_role=Неизвестная роль применения: {0} +error.data_budget.user_not_in_org=Один или несколько пользователей применения не принадлежат этой организации +error.data_budget.group_not_in_org=Одна или несколько групп применения не принадлежат этой организации +error.data_budget.not_found=Бюджет данных не найден +error.data_budget.exhausted=Отклонено: ваш бюджет данных «{0}» для этого источника данных исчерпан в текущем окне +validation.data_budget.name.required=Название бюджета обязательно +validation.data_budget.name.size=Название бюджета должно содержать не более 120 символов +validation.data_budget.max_rows.min=Лимит строк должен быть не меньше 1 +validation.data_budget.max_bytes.min=Лимит байтов должен быть не меньше 1 +validation.data_budget.window.range=Окно бюджета должно составлять от 60 до 44 640 минут (от 1 часа до 31 дня) +validation.data_budget.threshold.range=Порог предупреждения должен быть от 1 до 99 процентов +workflow.data_budget.rejected=Отклонено: бюджет данных «{0}» автора запроса для этого источника данных исчерпан в текущем окне +workflow.decision.data_budget.none=Бюджет данных не применяется +workflow.decision.data_budget.within=Использовано {0}% бюджета данных; результат ограничен оставшимся объёмом +workflow.decision.data_budget.exhausted_rejected=Бюджет данных «{0}» исчерпан — запрос отклоняется +workflow.decision.data_budget.exhausted_review=Бюджет данных «{0}» исчерпан — запрос не может быть одобрен автоматически +workflow.decision.routing.skipped_data_budget=Политики маршрутизации не оцениваются, когда исчерпанный бюджет данных отклоняет запрос +workflow.decision.grant.skipped_data_budget=Быстрый путь по допуску не оценивается, когда исчерпанный бюджет данных отклоняет запрос +workflow.decision.plan.skipped_data_budget=План проверки не учитывается, когда исчерпанный бюджет данных отклоняет запрос +workflow.decision.routing.matched_auto_approve_suppressed_data_budget=Политика маршрутизации «{0}» сработала с действием AUTO_APPROVE, подавлено, так как бюджет данных исчерпан +workflow.decision.grant.suppressed_data_budget=Покрыто допуском {0}, подавлено, так как бюджет данных исчерпан +workflow.decision.plan.suppressed_data_budget=План проверки одобрил бы запрос, подавлено, так как бюджет данных исчерпан + +# Data-budget notifications (#942) +notification.email.subject.data_budget_threshold_reached=[AccessFlow] Бюджет данных «{0}» в {1} использован на {2}% +notification.email.subject.data_budget_exhausted=[AccessFlow] Бюджет данных «{0}» в {1} исчерпан +notification.email.data_budget_threshold_reached.title=Предупреждение о бюджете данных +notification.email.data_budget_threshold_reached.heading=Вы приближаетесь к своему бюджету данных +notification.email.data_budget_threshold_reached.body=Ваши чтения из этого источника данных превысили порог предупреждения вашего бюджета данных. Когда бюджет будет исчерпан, последующие запросы SELECT будут отклоняться или отправляться на проверку, пока использование не опустится ниже лимита. +notification.email.data_budget_exhausted.title=Бюджет данных исчерпан +notification.email.data_budget_exhausted.heading=Бюджет данных исчерпан +notification.email.data_budget_exhausted.body=Указанный ниже бюджет данных исчерпан для текущего окна. Последующие запросы SELECT этого пользователя к этому источнику данных обрабатываются согласно действию при превышении, пока использование не опустится ниже лимита. +notification.email.data_budget.user_label=Пользователь: +notification.email.data_budget.budget_label=Бюджет: +notification.email.data_budget.used_label=Использовано: +notification.email.data_budget.rows_label=Строки: +notification.email.data_budget.bytes_label=Байты: +notification.email.data_budget.window_label=Окно: +notification.email.data_budget.threshold_label=Порог предупреждения: +notification.email.data_budget.breach_action_label=При исчерпании: +notification.email.data_budget.breach_action.REJECT=Последующие чтения отклоняются +notification.email.data_budget.breach_action.REQUIRE_REVIEW=Последующие чтения отправляются на ручную проверку +notification.email.data_budget.window.MINUTES={0} мин. +notification.email.data_budget.window.HOURS={0} ч. +notification.email.data_budget.window.DAYS={0} дн. +notification.email.data_budget.cta=Открыть редактор запросов diff --git a/backend/src/main/resources/i18n/messages_zh_CN.properties b/backend/src/main/resources/i18n/messages_zh_CN.properties index 9fff29495..fab378876 100644 --- a/backend/src/main/resources/i18n/messages_zh_CN.properties +++ b/backend/src/main/resources/i18n/messages_zh_CN.properties @@ -1583,3 +1583,55 @@ workflow.decision.plan.skipped_bytes_cap=扫描字节上限拒绝请求时不参 workflow.decision.routing.matched_auto_approve_suppressed_bytes_cap=路由策略"{0}"以 AUTO_APPROVE 动作匹配，但因在扫描字节上限下没有可用的字节估算而被抑制 workflow.decision.grant.suppressed_bytes_cap=由访问授权 {0} 覆盖，但因在扫描字节上限下没有可用的字节估算而被抑制 workflow.decision.plan.suppressed_bytes_cap=审查计划本会批准，但因在扫描字节上限下没有可用的字节估算而被抑制 +# Data-volume budgets (#942) +error.data_budget.name_invalid=预算名称为必填项，且不能超过 120 个字符 +error.data_budget.limit_required=数据预算需要设置行数上限、字节上限或两者 +error.data_budget.limit_invalid=行数和字节上限必须为正数 +error.data_budget.window_invalid=预算窗口必须介于 1 小时到 31 天之间 +error.data_budget.threshold_invalid=预警阈值必须介于 1% 到 99% 之间 +error.data_budget.unknown_role=未知的适用角色：{0} +error.data_budget.user_not_in_org=一个或多个适用用户不属于此组织 +error.data_budget.group_not_in_org=一个或多个适用组不属于此组织 +error.data_budget.not_found=未找到数据预算 +error.data_budget.exhausted=已拒绝：你在此数据源上的数据预算“{0}”在当前窗口内已用尽 +validation.data_budget.name.required=预算名称为必填项 +validation.data_budget.name.size=预算名称不能超过 120 个字符 +validation.data_budget.max_rows.min=行数上限至少为 1 +validation.data_budget.max_bytes.min=字节上限至少为 1 +validation.data_budget.window.range=预算窗口必须介于 60 到 44,640 分钟（1 小时到 31 天）之间 +validation.data_budget.threshold.range=预警阈值必须介于 1% 到 99% 之间 +workflow.data_budget.rejected=已拒绝：提交者在此数据源上的数据预算“{0}”在当前窗口内已用尽 +workflow.decision.data_budget.none=没有适用的数据预算 +workflow.decision.data_budget.within=数据预算已使用 {0}%；结果将被限制在剩余额度内 +workflow.decision.data_budget.exhausted_rejected=数据预算“{0}”已用尽——请求被拒绝 +workflow.decision.data_budget.exhausted_review=数据预算“{0}”已用尽——请求无法自动批准 +workflow.decision.routing.skipped_data_budget=当已用尽的数据预算拒绝请求时，不评估路由策略 +workflow.decision.grant.skipped_data_budget=当已用尽的数据预算拒绝请求时，不评估授权快速通道 +workflow.decision.plan.skipped_data_budget=当已用尽的数据预算拒绝请求时，不参考审核计划 +workflow.decision.routing.matched_auto_approve_suppressed_data_budget=路由策略“{0}”以 AUTO_APPROVE 操作匹配，因数据预算已用尽而被抑制 +workflow.decision.grant.suppressed_data_budget=已被访问授权 {0} 覆盖，因数据预算已用尽而被抑制 +workflow.decision.plan.suppressed_data_budget=审核计划本会批准，因数据预算已用尽而被抑制 + +# Data-budget notifications (#942) +notification.email.subject.data_budget_threshold_reached=[AccessFlow] {1} 上的数据预算“{0}”已使用 {2}% +notification.email.subject.data_budget_exhausted=[AccessFlow] {1} 上的数据预算“{0}”已用尽 +notification.email.data_budget_threshold_reached.title=数据预算警告 +notification.email.data_budget_threshold_reached.heading=您即将用完数据预算 +notification.email.data_budget_threshold_reached.body=您在此数据源上的读取已超过数据预算的警告阈值。预算用尽后，后续的 SELECT 查询将被拒绝或送交审核，直到用量回落到限额以下。 +notification.email.data_budget_exhausted.title=数据预算已用尽 +notification.email.data_budget_exhausted.heading=一项数据预算已用尽 +notification.email.data_budget_exhausted.body=以下数据预算在当前时间窗口内已用尽。该用户在此数据源上的后续 SELECT 查询将按超限处理方式处理，直到用量回落到限额以下。 +notification.email.data_budget.user_label=用户: +notification.email.data_budget.budget_label=预算: +notification.email.data_budget.used_label=已使用: +notification.email.data_budget.rows_label=行数: +notification.email.data_budget.bytes_label=字节: +notification.email.data_budget.window_label=时间窗口: +notification.email.data_budget.threshold_label=警告阈值: +notification.email.data_budget.breach_action_label=用尽时: +notification.email.data_budget.breach_action.REJECT=后续读取将被拒绝 +notification.email.data_budget.breach_action.REQUIRE_REVIEW=后续读取将送交人工审核 +notification.email.data_budget.window.MINUTES={0} 分钟 +notification.email.data_budget.window.HOURS={0} 小时 +notification.email.data_budget.window.DAYS={0} 天 +notification.email.data_budget.cta=打开查询编辑器 diff --git a/backend/src/main/resources/templates/email/data-budget-exhausted.html b/backend/src/main/resources/templates/email/data-budget-exhausted.html new file mode 100644 index 000000000..085fccf35 --- /dev/null +++ b/backend/src/main/resources/templates/email/data-budget-exhausted.html @@ -0,0 +1,47 @@ + + + + + Data budget + + + +

Data budget

+

Data budget notice.

+

+ Datasource: + —
+ User: + — +
+ Budget: + —
+ Used: + —
+ + Rows: + —
+ + + Bytes: + —
+ + + Window: + —
+ + + When used up: + — + +

+

+ Open the query editor +

+ + diff --git a/backend/src/main/resources/templates/email/data-budget-threshold-reached.html b/backend/src/main/resources/templates/email/data-budget-threshold-reached.html new file mode 100644 index 000000000..efd59adba --- /dev/null +++ b/backend/src/main/resources/templates/email/data-budget-threshold-reached.html @@ -0,0 +1,47 @@ + + + + + Data budget + + + +

Data budget

+

Data budget notice.

+

+ Datasource: + —
+ User: + — +
+ Budget: + —
+ Used: + —
+ + Rows: + —
+ + + Bytes: + —
+ + + Window: + —
+ + + Warning threshold: + — + +

+

+ Open the query editor +

+ + diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/api/DataBudgetStatusTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/api/DataBudgetStatusTest.java new file mode 100644 index 000000000..e1d1fc6f2 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/core/api/DataBudgetStatusTest.java @@ -0,0 +1,112 @@ +package com.bablsoft.accessflow.core.api; + +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class DataBudgetStatusTest { + + private final UUID datasourceId = UUID.randomUUID(); + + @Test + void emptyStatusHasNoStanding() { + var status = DataBudgetStatus.none(datasourceId); + + assertThat(status.isEmpty()).isTrue(); + assertThat(status.exhausted()).isFalse(); + assertThat(status.breachAction()).isNull(); + assertThat(status.decidingBudget()).isNull(); + assertThat(status.remainingRows()).isNull(); + assertThat(status.remainingBytes()).isNull(); + assertThat(status.usedPercent()).isNull(); + assertThat(new DataBudgetStatus(datasourceId, null, null).budgets()).isEmpty(); + } + + @Test + void theMostConstrainedBudgetWins() { + var rows = consumption(100L, null, 40, 0, DataBudgetBreachAction.REQUIRE_REVIEW); + var bytes = consumption(null, 1_000L, 0, 900, DataBudgetBreachAction.REJECT); + var status = new DataBudgetStatus(datasourceId, "ds", List.of(rows, bytes)); + + assertThat(status.exhausted()).isFalse(); + assertThat(status.remainingRows()).isEqualTo(60L); + assertThat(status.remainingBytes()).isEqualTo(100L); + assertThat(status.usedPercent()).isEqualTo(90d); + } + + @Test + void rejectBeatsReviewAmongExhaustedBudgets() { + var review = consumption(100L, null, 100, 0, DataBudgetBreachAction.REQUIRE_REVIEW); + var reject = consumption(null, 10L, 0, 20, DataBudgetBreachAction.REJECT); + var status = new DataBudgetStatus(datasourceId, "ds", List.of(review, reject)); + + assertThat(status.exhausted()).isTrue(); + assertThat(status.breachAction()).isEqualTo(DataBudgetBreachAction.REJECT); + assertThat(status.decidingBudget()).isEqualTo(reject); + assertThat(status.remainingRows()).isZero(); + assertThat(status.remainingBytes()).isZero(); + } + + @Test + void consumptionArithmetic() { + var c = consumption(200L, 50L, 50, 100, DataBudgetBreachAction.REJECT); + + assertThat(c.usedPercent()).isEqualTo(200d); + assertThat(c.exhausted()).isTrue(); + var more = c.plus(10, 5); + assertThat(more.usedRows()).isEqualTo(60); + assertThat(more.usedBytes()).isEqualTo(105); + assertThat(consumption(null, null, 5, 5, DataBudgetBreachAction.REJECT).usedPercent()).isZero(); + } + + @Test + void strictestAction() { + assertThat(DataBudgetBreachAction.strictest(null, DataBudgetBreachAction.REQUIRE_REVIEW)) + .isEqualTo(DataBudgetBreachAction.REQUIRE_REVIEW); + assertThat(DataBudgetBreachAction.strictest(DataBudgetBreachAction.REQUIRE_REVIEW, null)) + .isEqualTo(DataBudgetBreachAction.REQUIRE_REVIEW); + assertThat(DataBudgetBreachAction.strictest(DataBudgetBreachAction.REQUIRE_REVIEW, + DataBudgetBreachAction.REJECT)).isEqualTo(DataBudgetBreachAction.REJECT); + assertThat(DataBudgetBreachAction.strictest(DataBudgetBreachAction.REQUIRE_REVIEW, + DataBudgetBreachAction.REQUIRE_REVIEW)).isEqualTo(DataBudgetBreachAction.REQUIRE_REVIEW); + } + + @Test + void usageRecordValidatesAndClamps() { + var record = new DataBudgetUsageRecord(UUID.randomUUID(), datasourceId, -5, -1, + DataBudgetUsageSource.SAMPLE_DATA, null, null); + assertThat(record.rowsRead()).isZero(); + assertThat(record.bytesRead()).isZero(); + assertThatThrownBy(() -> new DataBudgetUsageRecord(null, datasourceId, 1, 1, + DataBudgetUsageSource.QUERY, null, null)).isInstanceOf(NullPointerException.class); + } + + @Test + void viewDefensivelyCopiesTargets() { + var view = new DataBudgetView(UUID.randomUUID(), datasourceId, "n", 1L, null, 60, + DataBudgetBreachAction.REJECT, null, null, null, null, true, null, null); + assertThat(view.appliesToRoles()).isEmpty(); + assertThat(view.appliesToGroupIds()).isEmpty(); + assertThat(view.appliesToUserIds()).isEmpty(); + } + + @Test + void exhaustedExceptionCarriesTheBudget() { + var c = consumption(1L, null, 1, 0, DataBudgetBreachAction.REJECT); + var ex = new DataBudgetExhaustedException("used up", c); + assertThat(ex).hasMessage("used up"); + assertThat(ex.budget()).isEqualTo(c); + assertThat(new DataBudgetNotFoundException(c.budgetId())).hasMessageContaining("not found"); + assertThat(new IllegalDataBudgetException("bad")).hasMessage("bad"); + } + + private DataBudgetConsumption consumption(Long maxRows, Long maxBytes, long usedRows, + long usedBytes, DataBudgetBreachAction action) { + return new DataBudgetConsumption(UUID.randomUUID(), "b", maxRows, maxBytes, 60, action, 80, + usedRows, usedBytes); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/api/QueryExecutionRequestTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/api/QueryExecutionRequestTest.java index 1eb8d3d3d..6800312ea 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/core/api/QueryExecutionRequestTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/core/api/QueryExecutionRequestTest.java @@ -138,4 +138,41 @@ void negativeTimeoutOverrideIsRejected() { .isInstanceOf(IllegalArgumentException.class) .hasMessageContaining("statementTimeoutOverride must be positive"); } + + @Test + void nonPositiveResultByteOverrideIsRejected() { + assertThatThrownBy(() -> new QueryExecutionRequest(datasourceId, "SELECT 1", + QueryType.SELECT, null, null, null, null, null, false, null, null, null, 0L)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("maxResultBytesOverride must be positive"); + } + + @Test + void withAllowanceOnlyEverLowersTheCaps() { + var base = new QueryExecutionRequest(datasourceId, "SELECT 1", QueryType.SELECT, 50, + null); + + var lowered = base.withAllowance(10L, 2_048L); + assertThat(lowered.maxRowsOverride()).isEqualTo(10); + assertThat(lowered.maxResultBytesOverride()).isEqualTo(2_048L); + + var kept = base.withAllowance(500L, null); + assertThat(kept.maxRowsOverride()).isEqualTo(50); + assertThat(kept.maxResultBytesOverride()).isNull(); + + var tighterBytes = lowered.withAllowance(null, 4_096L); + assertThat(tighterBytes.maxResultBytesOverride()).isEqualTo(2_048L); + } + + @Test + void withAllowanceFloorsAtOneAndSetsAnAbsentRowCap() { + var base = new QueryExecutionRequest(datasourceId, "SELECT 1", QueryType.SELECT, null, + null); + + var floored = base.withAllowance(0L, 0L); + assertThat(floored.maxRowsOverride()).isEqualTo(1); + assertThat(floored.maxResultBytesOverride()).isEqualTo(1L); + assertThat(base.withAllowance(Long.MAX_VALUE, null).maxRowsOverride()) + .isEqualTo(Integer.MAX_VALUE); + } } diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/api/SelectExecutionResultTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/api/SelectExecutionResultTest.java index d5d0e377b..4ad56e8a7 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/core/api/SelectExecutionResultTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/core/api/SelectExecutionResultTest.java @@ -74,4 +74,31 @@ void withEffectiveSqlAndWithRowSecurityPolicyIdsPreserveEachOther() { assertThat(result.appliedRowSecurityPolicyIds()).containsExactly(id); assertThat(result.truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_ROW_LIMIT); } + + @Test + void byteAccountingCopiesPreserveEveryOtherField() { + var id = UUID.randomUUID(); + List> rows = List.of(List.of(1), List.of(2), List.of(3)); + var result = new SelectExecutionResult(List.of(), rows, 3L, false, Duration.ZERO, + Set.of(id), Set.of(), null, "SELECT 1"); + assertThat(result.resultBytes()).isZero(); + + var measured = result.withResultBytes(120L); + assertThat(measured.resultBytes()).isEqualTo(120L); + assertThat(measured.appliedMaskingPolicyIds()).containsExactly(id); + assertThat(measured.effectiveSql()).isEqualTo("SELECT 1"); + + var trimmed = measured.truncatedTo(2, SelectExecutionResult.TRUNCATED_DATA_BUDGET, 80L); + assertThat(trimmed.rows()).hasSize(2); + assertThat(trimmed.rowCount()).isEqualTo(2); + assertThat(trimmed.truncated()).isTrue(); + assertThat(trimmed.truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_DATA_BUDGET); + assertThat(trimmed.resultBytes()).isEqualTo(80L); + + var relabeled = measured.withTruncatedReason(SelectExecutionResult.TRUNCATED_ROW_LIMIT); + assertThat(relabeled.truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_ROW_LIMIT); + assertThat(relabeled.resultBytes()).isEqualTo(120L); + assertThat(relabeled.withEffectiveSql("x").resultBytes()).isEqualTo(120L); + assertThat(relabeled.withRowSecurityPolicyIds(Set.of(id)).resultBytes()).isEqualTo(120L); + } } diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/internal/AppliesToMatcherTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/internal/AppliesToMatcherTest.java new file mode 100644 index 000000000..b78bac1de --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/core/internal/AppliesToMatcherTest.java @@ -0,0 +1,39 @@ +package com.bablsoft.accessflow.core.internal; + +import org.junit.jupiter.api.Test; + +import java.util.Set; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +class AppliesToMatcherTest { + + private final UUID userId = UUID.randomUUID(); + private final UUID groupId = UUID.randomUUID(); + + @Test + void emptyScopeMatchesEveryone() { + assertThat(AppliesToMatcher.matches(null, null, null, userId, null, Set.of())).isTrue(); + assertThat(AppliesToMatcher.matches(new String[0], new UUID[0], new UUID[0], userId, null, + Set.of())).isTrue(); + } + + @Test + void matchesOnAnyOneList() { + assertThat(AppliesToMatcher.matches(new String[]{" analyst "}, null, null, userId, "ANALYST", + Set.of())).isTrue(); + assertThat(AppliesToMatcher.matches(null, null, new UUID[]{userId}, userId, null, Set.of())) + .isTrue(); + assertThat(AppliesToMatcher.matches(null, new UUID[]{groupId}, null, userId, null, + Set.of(groupId))).isTrue(); + } + + @Test + void rejectsWhenNoListMatches() { + assertThat(AppliesToMatcher.matches(new String[]{"REVIEWER", null}, new UUID[]{groupId}, + new UUID[]{UUID.randomUUID()}, userId, "ANALYST", Set.of())).isFalse(); + assertThat(AppliesToMatcher.matches(new String[]{"ANALYST"}, null, null, userId, null, + Set.of())).isFalse(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetAdminServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetAdminServiceTest.java new file mode 100644 index 000000000..1ee1efe91 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetAdminServiceTest.java @@ -0,0 +1,277 @@ +package com.bablsoft.accessflow.core.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetCommand; +import com.bablsoft.accessflow.core.api.DataBudgetNotFoundException; +import com.bablsoft.accessflow.core.api.DatasourceNotFoundException; +import com.bablsoft.accessflow.core.api.IllegalDataBudgetException; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DatasourceEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.OrganizationEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.RoleEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.UserEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.UserGroupEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.RoleRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserGroupRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserRepository; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; +import org.springframework.context.MessageSource; + +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class DefaultDataBudgetAdminServiceTest { + + @Mock DataBudgetRepository dataBudgetRepository; + @Mock RoleRepository roleRepository; + @Mock DatasourceRepository datasourceRepository; + @Mock UserRepository userRepository; + @Mock UserGroupRepository userGroupRepository; + @Mock MessageSource messageSource; + + private DefaultDataBudgetAdminService service; + + private final UUID orgId = UUID.randomUUID(); + private final UUID datasourceId = UUID.randomUUID(); + + @BeforeEach + void setUp() { + service = new DefaultDataBudgetAdminService(dataBudgetRepository, roleRepository, + datasourceRepository, userRepository, userGroupRepository, messageSource); + when(messageSource.getMessage(any(), any(), any())).thenAnswer(inv -> inv.getArgument(0)); + when(datasourceRepository.findById(datasourceId)).thenReturn(Optional.of(datasource(orgId))); + when(dataBudgetRepository.save(any())).thenAnswer(inv -> inv.getArgument(0)); + when(roleRepository.findByNameInScope(eq(orgId), any())).thenReturn(Optional.empty()); + var analyst = new RoleEntity(); + analyst.setId(UUID.randomUUID()); + analyst.setName("ANALYST"); + when(roleRepository.findByNameInScope(orgId, "analyst")).thenReturn(Optional.of(analyst)); + } + + @Test + void listMapsEntitiesToViews() { + var entity = entity(); + entity.setAppliesToRoles(new String[]{"ANALYST"}); + entity.setWarnThresholdPercent((short) 80); + when(dataBudgetRepository + .findAllByOrganizationIdAndDatasourceIdOrderByCreatedAtAsc(orgId, datasourceId)) + .thenReturn(List.of(entity)); + + var result = service.listForDatasource(datasourceId, orgId); + + assertThat(result).singleElement().satisfies(v -> { + assertThat(v.name()).isEqualTo("Analysts daily"); + assertThat(v.maxRows()).isEqualTo(100_000L); + assertThat(v.maxBytes()).isNull(); + assertThat(v.windowMinutes()).isEqualTo(1440); + assertThat(v.breachAction()).isEqualTo(DataBudgetBreachAction.REQUIRE_REVIEW); + assertThat(v.warnThresholdPercent()).isEqualTo(80); + assertThat(v.appliesToRoles()).containsExactly("ANALYST"); + assertThat(v.appliesToGroupIds()).isEmpty(); + assertThat(v.appliesToUserIds()).isEmpty(); + }); + } + + @Test + void listRejectsDatasourceOfAnotherOrganization() { + when(datasourceRepository.findById(datasourceId)) + .thenReturn(Optional.of(datasource(UUID.randomUUID()))); + + assertThatThrownBy(() -> service.listForDatasource(datasourceId, orgId)) + .isInstanceOf(DatasourceNotFoundException.class); + } + + @Test + void listRejectsUnknownDatasource() { + when(datasourceRepository.findById(datasourceId)).thenReturn(Optional.empty()); + + assertThatThrownBy(() -> service.listForDatasource(datasourceId, orgId)) + .isInstanceOf(DatasourceNotFoundException.class); + } + + @Test + void createAppliesDefaultsAndNormalizesTargets() { + var userId = UUID.randomUUID(); + var groupId = UUID.randomUUID(); + when(userRepository.findAllByOrganization_IdAndIdIn(eq(orgId), anyList())) + .thenReturn(List.of(new UserEntity())); + when(userGroupRepository.findAllByOrganization_IdAndIdIn(eq(orgId), anyList())) + .thenReturn(List.of(new UserGroupEntity())); + + var view = service.create(datasourceId, orgId, new DataBudgetCommand(" Daily cap ", + null, 5_000_000_000L, null, null, null, List.of("analyst", " "), + List.of(groupId, groupId), List.of(userId), null)); + + var captor = ArgumentCaptor.forClass(DataBudgetEntity.class); + verify(dataBudgetRepository).save(captor.capture()); + var saved = captor.getValue(); + assertThat(saved.getOrganizationId()).isEqualTo(orgId); + assertThat(saved.getDatasourceId()).isEqualTo(datasourceId); + assertThat(saved.getName()).isEqualTo("Daily cap"); + assertThat(saved.getMaxRows()).isNull(); + assertThat(saved.getMaxBytes()).isEqualTo(5_000_000_000L); + assertThat(saved.getWindowMinutes()).isEqualTo(1440); + assertThat(saved.getBreachAction()).isEqualTo(DataBudgetBreachAction.REQUIRE_REVIEW); + assertThat(saved.getWarnThresholdPercent()).isNull(); + assertThat(saved.getAppliesToRoles()).containsExactly("ANALYST"); + assertThat(saved.getAppliesToGroupIds()).containsExactly(groupId); + assertThat(saved.getAppliesToUserIds()).containsExactly(userId); + assertThat(saved.isEnabled()).isTrue(); + assertThat(view.id()).isEqualTo(saved.getId()); + } + + @Test + void createKeepsExplicitValues() { + service.create(datasourceId, orgId, new DataBudgetCommand("Strict", 1_000L, null, 60, + DataBudgetBreachAction.REJECT, 75, null, null, null, false)); + + var captor = ArgumentCaptor.forClass(DataBudgetEntity.class); + verify(dataBudgetRepository).save(captor.capture()); + var saved = captor.getValue(); + assertThat(saved.getWindowMinutes()).isEqualTo(60); + assertThat(saved.getBreachAction()).isEqualTo(DataBudgetBreachAction.REJECT); + assertThat(saved.getWarnThresholdPercent()).isEqualTo((short) 75); + assertThat(saved.getAppliesToRoles()).isNull(); + assertThat(saved.isEnabled()).isFalse(); + } + + @Test + void createRejectsInvalidInput() { + assertInvalid(new DataBudgetCommand(" ", 1L, null, null, null, null, null, null, null, null), + "error.data_budget.name_invalid"); + assertInvalid(new DataBudgetCommand(null, 1L, null, null, null, null, null, null, null, null), + "error.data_budget.name_invalid"); + assertInvalid(new DataBudgetCommand("x".repeat(121), 1L, null, null, null, null, null, null, + null, null), "error.data_budget.name_invalid"); + assertInvalid(new DataBudgetCommand("n", null, null, null, null, null, null, null, null, + null), "error.data_budget.limit_required"); + assertInvalid(new DataBudgetCommand("n", 0L, null, null, null, null, null, null, null, null), + "error.data_budget.limit_invalid"); + assertInvalid(new DataBudgetCommand("n", null, -1L, null, null, null, null, null, null, + null), "error.data_budget.limit_invalid"); + assertInvalid(new DataBudgetCommand("n", 1L, null, 59, null, null, null, null, null, null), + "error.data_budget.window_invalid"); + assertInvalid(new DataBudgetCommand("n", 1L, null, 44_641, null, null, null, null, null, + null), "error.data_budget.window_invalid"); + assertInvalid(new DataBudgetCommand("n", 1L, null, null, null, 0, null, null, null, null), + "error.data_budget.threshold_invalid"); + assertInvalid(new DataBudgetCommand("n", 1L, null, null, null, 100, null, null, null, null), + "error.data_budget.threshold_invalid"); + assertInvalid(new DataBudgetCommand("n", 1L, null, null, null, null, List.of("ghost"), null, + null, null), "error.data_budget.unknown_role"); + verify(dataBudgetRepository, never()).save(any()); + } + + @Test + void createRejectsTargetsOutsideOrganization() { + when(userRepository.findAllByOrganization_IdAndIdIn(eq(orgId), anyList())) + .thenReturn(List.of()); + when(userGroupRepository.findAllByOrganization_IdAndIdIn(eq(orgId), anyList())) + .thenReturn(List.of()); + + assertInvalid(new DataBudgetCommand("n", 1L, null, null, null, null, null, null, + List.of(UUID.randomUUID()), null), "error.data_budget.user_not_in_org"); + assertInvalid(new DataBudgetCommand("n", 1L, null, null, null, null, null, + List.of(UUID.randomUUID()), null, null), "error.data_budget.group_not_in_org"); + } + + @Test + void updateReappliesEveryField() { + var entity = entity(); + when(dataBudgetRepository.findByIdAndOrganizationId(entity.getId(), orgId)) + .thenReturn(Optional.of(entity)); + + var view = service.update(entity.getId(), datasourceId, orgId, new DataBudgetCommand( + "Renamed", null, 1_024L, 120, DataBudgetBreachAction.REJECT, 50, null, null, null, + true)); + + assertThat(view.name()).isEqualTo("Renamed"); + assertThat(view.maxRows()).isNull(); + assertThat(view.maxBytes()).isEqualTo(1_024L); + assertThat(view.windowMinutes()).isEqualTo(120); + assertThat(view.breachAction()).isEqualTo(DataBudgetBreachAction.REJECT); + assertThat(view.warnThresholdPercent()).isEqualTo(50); + } + + @Test + void updateRejectsUnknownBudget() { + var id = UUID.randomUUID(); + when(dataBudgetRepository.findByIdAndOrganizationId(id, orgId)).thenReturn(Optional.empty()); + + assertThatThrownBy(() -> service.update(id, datasourceId, orgId, + new DataBudgetCommand("n", 1L, null, null, null, null, null, null, null, null))) + .isInstanceOf(DataBudgetNotFoundException.class); + } + + @Test + void updateRejectsBudgetOfAnotherDatasource() { + var entity = entity(); + entity.setDatasourceId(UUID.randomUUID()); + when(dataBudgetRepository.findByIdAndOrganizationId(entity.getId(), orgId)) + .thenReturn(Optional.of(entity)); + + assertThatThrownBy(() -> service.update(entity.getId(), datasourceId, orgId, + new DataBudgetCommand("n", 1L, null, null, null, null, null, null, null, null))) + .isInstanceOf(DataBudgetNotFoundException.class); + } + + @Test + void deleteRemovesTheBudget() { + var entity = entity(); + when(dataBudgetRepository.findByIdAndOrganizationId(entity.getId(), orgId)) + .thenReturn(Optional.of(entity)); + + service.delete(entity.getId(), datasourceId, orgId); + + verify(dataBudgetRepository).delete(entity); + } + + private void assertInvalid(DataBudgetCommand command, String key) { + assertThatThrownBy(() -> service.create(datasourceId, orgId, command)) + .isInstanceOf(IllegalDataBudgetException.class) + .hasMessage(key); + } + + private DatasourceEntity datasource(UUID ownerOrgId) { + var org = new OrganizationEntity(); + org.setId(ownerOrgId); + var ds = new DatasourceEntity(); + ds.setId(datasourceId); + ds.setOrganization(org); + return ds; + } + + private DataBudgetEntity entity() { + var entity = new DataBudgetEntity(); + entity.setId(UUID.randomUUID()); + entity.setOrganizationId(orgId); + entity.setDatasourceId(datasourceId); + entity.setName("Analysts daily"); + entity.setMaxRows(100_000L); + entity.setWindowMinutes(1440); + entity.setBreachAction(DataBudgetBreachAction.REQUIRE_REVIEW); + entity.setEnabled(true); + return entity; + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetStatusServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetStatusServiceTest.java new file mode 100644 index 000000000..c615a454b --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetStatusServiceTest.java @@ -0,0 +1,181 @@ +package com.bablsoft.accessflow.core.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.UserRoleType; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DatasourceEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.UserEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserGroupMembershipRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserRepository; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class DefaultDataBudgetStatusServiceTest { + + private static final Instant NOW = Instant.parse("2026-09-25T12:00:00Z"); + + @Mock DataBudgetRepository dataBudgetRepository; + @Mock DataBudgetUsageRepository usageRepository; + @Mock DatasourceRepository datasourceRepository; + @Mock UserRepository userRepository; + @Mock UserGroupMembershipRepository membershipRepository; + + private DefaultDataBudgetStatusService service; + + private final UUID orgId = UUID.randomUUID(); + private final UUID datasourceId = UUID.randomUUID(); + private final UUID userId = UUID.randomUUID(); + private final UUID groupId = UUID.randomUUID(); + + @BeforeEach + void setUp() { + service = new DefaultDataBudgetStatusService(dataBudgetRepository, usageRepository, + datasourceRepository, userRepository, membershipRepository, + Clock.fixed(NOW, ZoneOffset.UTC)); + var user = new UserEntity(); + user.setId(userId); + user.setRole(UserRoleType.ANALYST); + when(userRepository.findById(userId)).thenReturn(Optional.of(user)); + when(membershipRepository.findGroupIdsForUser(userId)).thenReturn(List.of(groupId)); + var ds = new DatasourceEntity(); + ds.setId(datasourceId); + ds.setName("warehouse"); + when(datasourceRepository.findById(datasourceId)).thenReturn(Optional.of(ds)); + when(datasourceRepository.findAllById(any())).thenReturn(List.of(ds)); + when(usageRepository.sumSince(any(), any(), any())).thenReturn(totals(0, 0)); + } + + @Test + void noBudgetsReadsAsEmptyWithoutTouchingTheLedger() { + when(dataBudgetRepository.findAllByDatasourceIdAndEnabledTrue(datasourceId)) + .thenReturn(List.of()); + + var status = service.statusFor(datasourceId, userId); + + assertThat(status.isEmpty()).isTrue(); + verify(usageRepository, never()).sumSince(any(), any(), any()); + } + + @Test + void budgetsScopedToOthersReadAsEmpty() { + var budget = budget(1_000L, null, 60); + budget.setAppliesToUserIds(new UUID[]{UUID.randomUUID()}); + when(dataBudgetRepository.findAllByDatasourceIdAndEnabledTrue(datasourceId)) + .thenReturn(List.of(budget)); + + assertThat(service.statusFor(datasourceId, userId).isEmpty()).isTrue(); + } + + @Test + void sumsTheTrailingWindowOncePerDistinctWindow() { + var daily = budget(1_000L, null, 1440); + var alsoDaily = budget(null, 10_000L, 1440); + alsoDaily.setAppliesToGroupIds(new UUID[]{groupId}); + var hourly = budget(100L, null, 60); + hourly.setAppliesToRoles(new String[]{"analyst"}); + when(dataBudgetRepository.findAllByDatasourceIdAndEnabledTrue(datasourceId)) + .thenReturn(List.of(daily, alsoDaily, hourly)); + when(usageRepository.sumSince(userId, datasourceId, NOW.minusSeconds(86_400))) + .thenReturn(totals(400, 9_000)); + when(usageRepository.sumSince(userId, datasourceId, NOW.minusSeconds(3_600))) + .thenReturn(totals(100, 50)); + + var status = service.statusFor(datasourceId, userId); + + assertThat(status.datasourceName()).isEqualTo("warehouse"); + assertThat(status.budgets()).hasSize(3); + assertThat(status.remainingRows()).isZero(); + assertThat(status.remainingBytes()).isEqualTo(1_000L); + assertThat(status.exhausted()).isTrue(); + verify(usageRepository, times(2)).sumSince(eq(userId), eq(datasourceId), any()); + } + + @Test + void nullTotalsReadAsZero() { + when(dataBudgetRepository.findAllByDatasourceIdAndEnabledTrue(datasourceId)) + .thenReturn(List.of(budget(10L, null, 60))); + when(usageRepository.sumSince(any(), any(), any())).thenReturn(null); + + var status = service.statusFor(datasourceId, userId); + + assertThat(status.budgets().getFirst().usedRows()).isZero(); + assertThat(status.remainingRows()).isEqualTo(10L); + } + + @Test + void statusesForUserGroupsByDatasourceAndSkipsForeignScopes() { + var other = budget(5L, null, 60); + other.setAppliesToUserIds(new UUID[]{UUID.randomUUID()}); + when(dataBudgetRepository.findAllByOrganizationIdAndEnabledTrue(orgId)) + .thenReturn(List.of(budget(10L, null, 60), budget(20L, null, 60), other)); + + var statuses = service.statusesForUser(orgId, userId); + + assertThat(statuses).singleElement().satisfies(s -> { + assertThat(s.datasourceId()).isEqualTo(datasourceId); + assertThat(s.datasourceName()).isEqualTo("warehouse"); + assertThat(s.budgets()).hasSize(2); + }); + } + + @Test + void statusesForUserIsEmptyWithoutBudgets() { + when(dataBudgetRepository.findAllByOrganizationIdAndEnabledTrue(orgId)).thenReturn(List.of()); + + assertThat(service.statusesForUser(orgId, userId)).isEmpty(); + } + + private DataBudgetEntity budget(Long maxRows, Long maxBytes, int windowMinutes) { + var entity = new DataBudgetEntity(); + entity.setId(UUID.randomUUID()); + entity.setOrganizationId(orgId); + entity.setDatasourceId(datasourceId); + entity.setName("b-" + windowMinutes); + entity.setMaxRows(maxRows); + entity.setMaxBytes(maxBytes); + entity.setWindowMinutes(windowMinutes); + entity.setBreachAction(DataBudgetBreachAction.REQUIRE_REVIEW); + entity.setWarnThresholdPercent((short) 80); + entity.setEnabled(true); + return entity; + } + + private static DataBudgetUsageRepository.UsageTotals totals(long rows, long bytes) { + return new DataBudgetUsageRepository.UsageTotals() { + @Override + public Long getRowsRead() { + return rows; + } + + @Override + public Long getBytesRead() { + return bytes; + } + }; + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetUsageServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetUsageServiceTest.java new file mode 100644 index 000000000..8e5b4ddf7 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultDataBudgetUsageServiceTest.java @@ -0,0 +1,180 @@ +package com.bablsoft.accessflow.core.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetConsumption; +import com.bablsoft.accessflow.core.api.DataBudgetStatus; +import com.bablsoft.accessflow.core.api.DataBudgetStatusService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageRecord; +import com.bablsoft.accessflow.core.api.DataBudgetUsageSource; +import com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetUsageEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.mockito.junit.jupiter.MockitoSettings; +import org.mockito.quality.Strictness; +import org.springframework.context.ApplicationEventPublisher; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +@MockitoSettings(strictness = Strictness.LENIENT) +class DefaultDataBudgetUsageServiceTest { + + private static final Instant NOW = Instant.parse("2026-09-25T12:00:00Z"); + + @Mock DataBudgetStatusService statusService; + @Mock DataBudgetRepository dataBudgetRepository; + @Mock DataBudgetUsageRepository usageRepository; + @Mock ApplicationEventPublisher eventPublisher; + + private DefaultDataBudgetUsageService service; + + private final UUID orgId = UUID.randomUUID(); + private final UUID datasourceId = UUID.randomUUID(); + private final UUID userId = UUID.randomUUID(); + private final UUID budgetId = UUID.randomUUID(); + private final UUID queryId = UUID.randomUUID(); + + @BeforeEach + void setUp() { + service = new DefaultDataBudgetUsageService(statusService, dataBudgetRepository, + usageRepository, eventPublisher, Clock.fixed(NOW, ZoneOffset.UTC)); + var entity = new DataBudgetEntity(); + entity.setId(budgetId); + entity.setOrganizationId(orgId); + when(dataBudgetRepository.findById(budgetId)).thenReturn(Optional.of(entity)); + } + + @Test + void unbudgetedReadWritesNothing() { + when(statusService.statusFor(datasourceId, userId)) + .thenReturn(DataBudgetStatus.none(datasourceId)); + + service.record(usage(10, 100)); + + verify(usageRepository, never()).save(any()); + verify(eventPublisher, never()).publishEvent(any()); + } + + @Test + void budgetDeletedMidwayWritesNothing() { + givenStatus(consumption(100L, 0)); + when(dataBudgetRepository.findById(budgetId)).thenReturn(Optional.empty()); + + service.record(usage(10, 100)); + + verify(usageRepository, never()).save(any()); + } + + @Test + void recordsTheLedgerRowWithoutEventsBelowTheThreshold() { + givenStatus(consumption(100L, 10)); + + service.record(usage(10, 512)); + + verify(usageRepository).lockUserDatasource( + DefaultDataBudgetUsageService.lockKey(userId, datasourceId)); + var captor = ArgumentCaptor.forClass(DataBudgetUsageEntity.class); + verify(usageRepository).save(captor.capture()); + var row = captor.getValue(); + assertThat(row.getOrganizationId()).isEqualTo(orgId); + assertThat(row.getUserId()).isEqualTo(userId); + assertThat(row.getDatasourceId()).isEqualTo(datasourceId); + assertThat(row.getRowsRead()).isEqualTo(10); + assertThat(row.getBytesRead()).isEqualTo(512); + assertThat(row.getSource()).isEqualTo(DataBudgetUsageSource.QUERY); + assertThat(row.getQueryRequestId()).isEqualTo(queryId); + assertThat(row.getOccurredAt()).isEqualTo(NOW); + verify(eventPublisher, never()).publishEvent(any()); + } + + @Test + void crossingTheWarnThresholdPublishesOnce() { + givenStatus(consumption(100L, 70)); + + service.record(usage(15, 0)); + + var captor = ArgumentCaptor.forClass(DataBudgetThresholdCrossedEvent.class); + verify(eventPublisher).publishEvent(captor.capture()); + var event = captor.getValue(); + assertThat(event.exhausted()).isFalse(); + assertThat(event.usedPercent()).isEqualTo(85); + assertThat(event.organizationId()).isEqualTo(orgId); + assertThat(event.budgetId()).isEqualTo(budgetId); + assertThat(event.usedRows()).isEqualTo(85); + } + + @Test + void reachingTheLimitPublishesExhaustedOnly() { + givenStatus(consumption(100L, 70)); + + service.record(usage(200, 0)); + + var captor = ArgumentCaptor.forClass(DataBudgetThresholdCrossedEvent.class); + verify(eventPublisher).publishEvent(captor.capture()); + assertThat(captor.getValue().exhausted()).isTrue(); + assertThat(captor.getValue().usedPercent()).isEqualTo(100); + assertThat(captor.getValue().breachAction()).isEqualTo(DataBudgetBreachAction.REJECT); + } + + @Test + void alreadyExhaustedNeverRepublishes() { + givenStatus(consumption(100L, 150)); + + service.record(usage(10, 0)); + + verify(eventPublisher, never()).publishEvent(any()); + } + + @Test + void crossingIsStatelessOverBeforeAndAfter() { + var before = consumption(100L, 79); + assertThat(DefaultDataBudgetUsageService.crossing(before, before.plus(1, 0))).contains(false); + assertThat(DefaultDataBudgetUsageService.crossing(before, before.plus(0, 0))).isEmpty(); + var aboveWarn = consumption(100L, 85); + assertThat(DefaultDataBudgetUsageService.crossing(aboveWarn, aboveWarn.plus(5, 0))).isEmpty(); + var noWarn = new DataBudgetConsumption(budgetId, "b", 100L, null, 60, + DataBudgetBreachAction.REJECT, null, 10, 0); + assertThat(DefaultDataBudgetUsageService.crossing(noWarn, noWarn.plus(80, 0))).isEmpty(); + } + + @Test + void theLockKeyIsStablePerUserAndDatasource() { + assertThat(DefaultDataBudgetUsageService.lockKey(userId, datasourceId)) + .isEqualTo(DefaultDataBudgetUsageService.lockKey(userId, datasourceId)) + .isNotEqualTo(DefaultDataBudgetUsageService.lockKey(datasourceId, userId)); + } + + private void givenStatus(DataBudgetConsumption consumption) { + when(statusService.statusFor(datasourceId, userId)) + .thenReturn(new DataBudgetStatus(datasourceId, "warehouse", List.of(consumption))); + } + + private DataBudgetConsumption consumption(Long maxRows, long usedRows) { + return new DataBudgetConsumption(budgetId, "Daily", maxRows, null, 1440, + DataBudgetBreachAction.REJECT, 80, usedRows, 0); + } + + private DataBudgetUsageRecord usage(long rows, long bytes) { + return new DataBudgetUsageRecord(userId, datasourceId, rows, bytes, + DataBudgetUsageSource.QUERY, queryId, null); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateServiceTest.java index 7df5a0565..cd2d4dd84 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateServiceTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/core/internal/DefaultQueryRequestStateServiceTest.java @@ -150,6 +150,29 @@ void recordBytesScannedCapStampsTheCapWithoutTransitioning() { verify(eventPublisher, never()).publishEvent(any()); } + @Test + void recordDataBudgetReviewForcedStampsWithoutTransitioning() { + query.setStatus(QueryStatus.PENDING_AI); + when(queryRequestRepository.findByIdForUpdate(queryId)).thenReturn(Optional.of(query)); + + service.recordDataBudgetReviewForced(queryId); + + assertThat(query.isDataBudgetReviewForced()).isTrue(); + assertThat(query.getStatus()).isEqualTo(QueryStatus.PENDING_AI); + verify(queryRequestRepository).save(query); + } + + @Test + void isDataBudgetReviewForcedReadsTheStamp() { + query.setDataBudgetReviewForced(true); + when(queryRequestRepository.findById(queryId)).thenReturn(Optional.of(query)); + var other = UUID.randomUUID(); + when(queryRequestRepository.findById(other)).thenReturn(Optional.empty()); + + assertThat(service.isDataBudgetReviewForced(queryId)).isTrue(); + assertThat(service.isDataBudgetReviewForced(other)).isFalse(); + } + @Test void recordApprovalAndAdvancePromotesToApprovedAtLastStage() { query.setStatus(QueryStatus.PENDING_REVIEW); diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/internal/config/DataBudgetPropertiesTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/internal/config/DataBudgetPropertiesTest.java new file mode 100644 index 000000000..e8b50107a --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/core/internal/config/DataBudgetPropertiesTest.java @@ -0,0 +1,20 @@ +package com.bablsoft.accessflow.core.internal.config; + +import org.junit.jupiter.api.Test; + +import java.time.Duration; + +import static org.assertj.core.api.Assertions.assertThat; + +class DataBudgetPropertiesTest { + + @Test + void defaultsAndFloorsTheRetention() { + assertThat(new DataBudgetProperties(null).usageRetention()) + .isEqualTo(DataBudgetProperties.DEFAULT_RETENTION); + assertThat(new DataBudgetProperties(Duration.ofDays(1)).usageRetention()) + .isEqualTo(DataBudgetProperties.MIN_RETENTION); + assertThat(new DataBudgetProperties(Duration.ofDays(90)).usageRetention()) + .isEqualTo(Duration.ofDays(90)); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/core/internal/scheduled/DataBudgetUsagePruneJobTest.java b/backend/src/test/java/com/bablsoft/accessflow/core/internal/scheduled/DataBudgetUsagePruneJobTest.java new file mode 100644 index 000000000..28627d742 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/core/internal/scheduled/DataBudgetUsagePruneJobTest.java @@ -0,0 +1,48 @@ +package com.bablsoft.accessflow.core.internal.scheduled; + +import com.bablsoft.accessflow.core.internal.config.DataBudgetProperties; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; + +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DataBudgetUsagePruneJobTest { + + private static final Instant NOW = Instant.parse("2026-09-25T12:00:00Z"); + + @Mock DataBudgetUsageRepository usageRepository; + + @Test + void deletesRowsOlderThanTheRetention() { + var job = new DataBudgetUsagePruneJob(usageRepository, + new DataBudgetProperties(Duration.ofDays(40)), Clock.fixed(NOW, ZoneOffset.UTC)); + var cutoff = NOW.minus(Duration.ofDays(40)); + when(usageRepository.deleteOlderThan(cutoff)).thenReturn(3); + + job.run(); + + verify(usageRepository).deleteOlderThan(cutoff); + } + + @Test + void nothingToPruneIsQuiet() { + var job = new DataBudgetUsagePruneJob(usageRepository, new DataBudgetProperties(null), + Clock.fixed(NOW, ZoneOffset.UTC)); + var cutoff = NOW.minus(DataBudgetProperties.DEFAULT_RETENTION); + when(usageRepository.deleteOlderThan(cutoff)).thenReturn(0); + + job.run(); + + verify(usageRepository).deleteOlderThan(cutoff); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNoticeTest.java b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNoticeTest.java new file mode 100644 index 000000000..8ba1846b5 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNoticeTest.java @@ -0,0 +1,45 @@ +package com.bablsoft.accessflow.notifications.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; + +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +class DataBudgetNoticeTest { + + private static DataBudgetNotice notice(Long maxRows, Long maxBytes, int windowMinutes) { + return new DataBudgetNotice(UUID.randomUUID(), "b", false, 80, 82, maxRows, maxBytes, 820L, + 1_500_000L, windowMinutes, DataBudgetBreachAction.REJECT); + } + + @Test + void usageLinesOnlyForTheLimitsTheBudgetSets() { + var rowsOnly = notice(1000L, null, 60); + assertThat(rowsOnly.rowsUsage()).isEqualTo("820 / 1000 rows"); + assertThat(rowsOnly.bytesUsage()).isNull(); + + var bytesOnly = notice(null, 5_000_000L, 60); + assertThat(bytesOnly.rowsUsage()).isNull(); + assertThat(bytesOnly.bytesUsage()).isEqualTo("1.5 MB (1500000 B) / 5 MB (5000000 B)"); + } + + @ParameterizedTest + @CsvSource({ + "1440, DAYS, 1, 1 day", + "10080, DAYS, 7, 7 days", + "60, HOURS, 1, 1 hour", + "180, HOURS, 3, 3 hours", + "90, MINUTES, 90, 90 minutes", + "1, MINUTES, 1, 1 minute", + "0, MINUTES, 0, 0 minutes"}) + void windowRendersInTheLargestWholeUnit(int minutes, String unit, int value, String label) { + var n = notice(1L, null, minutes); + assertThat(n.windowUnit()).isEqualTo(unit); + assertThat(n.windowValue()).isEqualTo(value); + assertThat(n.windowLabel()).isEqualTo(label); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotificationListenerTest.java b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotificationListenerTest.java new file mode 100644 index 000000000..3ba9cc006 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/DataBudgetNotificationListenerTest.java @@ -0,0 +1,48 @@ +package com.bablsoft.accessflow.notifications.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.verify; + +@ExtendWith(MockitoExtension.class) +class DataBudgetNotificationListenerTest { + + @Mock private NotificationDispatcher dispatcher; + @InjectMocks private DataBudgetNotificationListener listener; + + private static DataBudgetThresholdCrossedEvent event(boolean exhausted) { + return new DataBudgetThresholdCrossedEvent(UUID.randomUUID(), UUID.randomUUID(), UUID.randomUUID(), + UUID.randomUUID(), "daily-reads", exhausted, 80, exhausted ? 100 : 80, 1000L, null, + 800L, 0L, 1440, DataBudgetBreachAction.REJECT); + } + + @Test + void forwardsEveryCrossingToTheDispatcher() { + var warning = event(false); + var exhausted = event(true); + + listener.onThresholdCrossed(warning); + listener.onThresholdCrossed(exhausted); + + verify(dispatcher).dispatchDataBudget(warning); + verify(dispatcher).dispatchDataBudget(exhausted); + } + + @Test + void aDispatchFailureNeverPropagates() { + var event = event(true); + doThrow(new IllegalStateException("boom")).when(dispatcher).dispatchDataBudget(event); + + assertThatCode(() -> listener.onThresholdCrossed(event)).doesNotThrowAnyException(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilderTest.java b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilderTest.java index 15840872a..99127f606 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilderTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationContextBuilderTest.java @@ -801,6 +801,68 @@ void buildSchemaDriftEmptyWhenTheTargetIsGone() { UUID.randomUUID(), orgId, UUID.randomUUID(), UUID.randomUUID(), 1))).isEmpty(); } + private com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent budgetEvent(boolean exhausted) { + return new com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent(orgId, submitterId, + datasourceId, UUID.randomUUID(), "daily-reads", exhausted, 80, exhausted ? 100 : 82, + 1000L, 5_000_000L, exhausted ? 1000L : 820L, 1_200_000L, 1440, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT); + } + + @Test + void dataBudgetWarningGoesToTheUserOnly() { + var alice = user(submitterId, "alice@example.com", UserRoleType.ANALYST); + when(userQuery.findByIds(List.of(submitterId))).thenReturn(List.of(alice)); + + var ctx = builder.buildDataBudget(budgetEvent(false)).orElseThrow(); + + assertThat(ctx.eventType()).isEqualTo(NotificationEventType.DATA_BUDGET_THRESHOLD_REACHED); + assertThat(ctx.isDataBudgetEvent()).isTrue(); + assertThat(ctx.recipients()).extracting(RecipientView::userId).containsExactly(submitterId); + assertThat(ctx.datasourceId()).isEqualTo(datasourceId); + assertThat(ctx.datasourceName()).isEqualTo("Production"); + assertThat(ctx.submitterEmail()).isEqualTo("alice@example.com"); + assertThat(ctx.dataBudget().budgetName()).isEqualTo("daily-reads"); + assertThat(ctx.dataBudget().usedPercent()).isEqualTo(82); + assertThat(ctx.reviewUrl()).hasToString("https://app.example.test/editor"); + org.mockito.Mockito.verifyNoInteractions(permissionHolderLookup); + } + + @Test + void dataBudgetExhaustionAlsoReachesBudgetManagersWithoutDuplicates() { + var alice = user(submitterId, "alice@example.com", UserRoleType.ANALYST); + var manager = user(UUID.randomUUID(), "mgr@example.com", UserRoleType.ADMIN); + when(permissionHolderLookup.findUserIdsWithPermission(orgId, + com.bablsoft.accessflow.core.api.Permission.DATA_BUDGET_MANAGE)) + .thenReturn(List.of(manager.id(), submitterId)); + when(userQuery.findByIds(List.of(submitterId, manager.id()))).thenReturn(List.of(alice, manager)); + + var ctx = builder.buildDataBudget(budgetEvent(true)).orElseThrow(); + + assertThat(ctx.eventType()).isEqualTo(NotificationEventType.DATA_BUDGET_EXHAUSTED); + assertThat(ctx.recipients()).extracting(RecipientView::userId) + .containsExactly(submitterId, manager.id()); + assertThat(ctx.dataBudget().exhausted()).isTrue(); + } + + @Test + void dataBudgetSurvivesADeletedDatasource() { + var alice = user(submitterId, "alice@example.com", UserRoleType.ANALYST); + when(userQuery.findByIds(List.of(submitterId))).thenReturn(List.of(alice)); + when(datasourceAdmin.getForAdmin(eq(datasourceId), eq(orgId))) + .thenThrow(new com.bablsoft.accessflow.core.api.DatasourceNotFoundException(datasourceId)); + + var ctx = builder.buildDataBudget(budgetEvent(false)).orElseThrow(); + + assertThat(ctx.datasourceName()).isNull(); + } + + @Test + void dataBudgetEmptyWhenNobodyActiveIsLeft() { + when(userQuery.findByIds(List.of(submitterId))).thenReturn(List.of()); + + assertThat(builder.buildDataBudget(budgetEvent(false))).isEmpty(); + } + private com.bablsoft.accessflow.schemachange.api.SchemaChangePromotionNotificationView promotionView( UUID promotionId, UUID promoterId) { return new com.bablsoft.accessflow.schemachange.api.SchemaChangePromotionNotificationView(promotionId, diff --git a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcherTest.java b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcherTest.java index 34aea6e8a..89fef86dc 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcherTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/NotificationDispatcherTest.java @@ -406,6 +406,56 @@ void unknownSchemaChangeTargetsShortCircuit() { any(), any(), any()); } + /** #942: budgets are per user, not per plan — both events fan out org-wide and record in-app. */ + @Test + void dataBudgetEventsUseAllActiveChannelsAndCarryTheDatasourceInThePayload() { + var userId = UUID.randomUUID(); + var emailCh = channel(NotificationChannelType.EMAIL); + when(channelRepository.findAllByOrganizationIdAndActiveTrue(orgId)).thenReturn(List.of(emailCh)); + for (var exhausted : List.of(false, true)) { + var event = new com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent(orgId, userId, + datasourceId, UUID.randomUUID(), "daily-reads", exhausted, 80, exhausted ? 100 : 85, + 1000L, null, 850L, 0L, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW); + var type = exhausted ? NotificationEventType.DATA_BUDGET_EXHAUSTED + : NotificationEventType.DATA_BUDGET_THRESHOLD_REACHED; + when(contextBuilder.buildDataBudget(event)).thenReturn(Optional.of(sampleDataBudgetContext(type, + userId, List.of(new RecipientView(userId, "u@x", "U"))))); + dispatcher.dispatchDataBudget(event); + } + + verify(emailStrategy, org.mockito.Mockito.times(2)).deliver(any(), eq(emailCh)); + verify(contextBuilder, never()).lookupPlanChannelIds(any()); + var payloadCaptor = ArgumentCaptor.forClass(String.class); + verify(userNotificationService).recordForUsers( + eq(NotificationEventType.DATA_BUDGET_EXHAUSTED), + eq(Set.of(userId)), + eq(orgId), + isNull(), + isNull(), + isNull(), + isNull(), + payloadCaptor.capture()); + assertThat(payloadCaptor.getValue()) + .contains("\"datasource_id\":\"" + datasourceId + "\"") + .contains("\"datasource\":\"warehouse\"") + .contains("\"budget\":\"daily-reads\"") + .contains("\"used_percent\":100"); + } + + @Test + void dataBudgetWithNoRecipientShortCircuits() { + when(contextBuilder.buildDataBudget(any())).thenReturn(Optional.empty()); + + dispatcher.dispatchDataBudget(new com.bablsoft.accessflow.core.events.DataBudgetThresholdCrossedEvent( + orgId, UUID.randomUUID(), datasourceId, UUID.randomUUID(), "b", false, 80, 80, 10L, null, + 8L, 0L, 60, com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT)); + + verify(channelRepository, never()).findAllByOrganizationIdAndActiveTrue(any()); + verify(userNotificationService, never()).recordForUsers(any(), any(), any(), any(), any(), + any(), any(), any()); + } + @Test void unknownDeploymentRequestShortCircuits() { when(contextBuilder.buildDeployment(any(), any(), any(), any())) @@ -598,6 +648,34 @@ private NotificationContext sampleSchemaChangeContext(NotificationEventType type newFindingCount); } + private NotificationContext sampleDataBudgetContext(NotificationEventType type, UUID userId, + List recipients) { + var exhausted = type == NotificationEventType.DATA_BUDGET_EXHAUSTED; + return new NotificationContext( + type, orgId, null, + null, null, null, null, + null, null, null, + datasourceId, "warehouse", + userId, "u@x", "U", + null, + null, null, null, + null, + recipients, Instant.now(), "en", null, + null, null, null, null, null, null, + null, + null, null, null, + null, + null, null, null, + null, null, null, + null, null, null, + null, null, null, null, null, + null, null, null, null, + null, + new DataBudgetNotice(UUID.randomUUID(), "daily-reads", exhausted, 80, exhausted ? 100 : 85, + 1000L, null, 850L, 0L, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW)); + } + private NotificationContext sampleExecutedContext(List recipients) { return new NotificationContext( NotificationEventType.QUERY_EXECUTED, diff --git a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetNotificationTest.java b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetNotificationTest.java new file mode 100644 index 000000000..944886ff2 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetNotificationTest.java @@ -0,0 +1,267 @@ +package com.bablsoft.accessflow.notifications.internal.strategy; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.notifications.api.NotificationEventType; +import com.bablsoft.accessflow.notifications.internal.DataBudgetNotice; +import com.bablsoft.accessflow.notifications.internal.NotificationContext; +import com.bablsoft.accessflow.notifications.internal.RecipientView; +import com.bablsoft.accessflow.notifications.internal.codec.DiscordChannelConfig; +import com.bablsoft.accessflow.notifications.internal.codec.PagerDutyChannelConfig; +import com.bablsoft.accessflow.notifications.internal.codec.PagerDutySeverity; +import com.bablsoft.accessflow.notifications.internal.codec.PagerDutyTrigger; +import com.bablsoft.accessflow.notifications.internal.codec.TicketingTrigger; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.EnumSource; +import org.junit.jupiter.params.provider.ValueSource; +import org.springframework.context.support.ReloadableResourceBundleMessageSource; +import org.thymeleaf.context.Context; +import org.thymeleaf.messageresolver.StandardMessageResolver; +import org.thymeleaf.spring6.SpringTemplateEngine; +import org.thymeleaf.spring6.messageresolver.SpringMessageResolver; +import org.thymeleaf.templatemode.TemplateMode; +import org.thymeleaf.templateresolver.ClassLoaderTemplateResolver; +import tools.jackson.databind.json.JsonMapper; + +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.time.Instant; +import java.util.EnumSet; +import java.util.List; +import java.util.Locale; +import java.util.Set; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Covers the #942 data-budget fan-out on every channel: both events render a type-specific title + * and the budget field set, never the generic query fields or a default branch — PagerDuty's + * summary, both silent email switches and the ticket headline included. + */ +class DataBudgetNotificationTest { + + private static final String ALL = "DATA_BUDGET_.*"; + + private static final List TEMPLATES = List.of( + "email/data-budget-threshold-reached", + "email/data-budget-exhausted"); + + private final JsonMapper json = JsonMapper.builder().build(); + + /** A fully populated context for one data-budget event, as the context builder shapes it. */ + static NotificationContext dataBudgetCtx(NotificationEventType type) { + var exhausted = type == NotificationEventType.DATA_BUDGET_EXHAUSTED; + return new NotificationContext( + type, UUID.randomUUID(), null, + null, null, null, null, + null, null, null, + UUID.randomUUID(), "warehouse", + UUID.randomUUID(), "ana@example.com", "Ana Analyst", + null, + null, null, null, + URI.create("https://app.example.test/editor"), + List.of(new RecipientView(UUID.randomUUID(), "rcpt@example.com", "R")), + Instant.now(), "en", null, + null, null, null, null, null, null, + null, + null, null, null, + null, + null, null, null, + null, null, null, + null, null, null, + null, null, null, null, null, + null, null, null, null, + null, + new DataBudgetNotice(UUID.randomUUID(), "daily-reads", exhausted, 80, + exhausted ? 100 : 85, 1000L, 5_000_000_000L, exhausted ? 1000L : 850L, + 1_200_000_000L, 1440, DataBudgetBreachAction.REQUIRE_REVIEW)); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void slackRendersADataBudgetTitleAndFieldSet(NotificationEventType type) { + var payload = new SlackBlockKitFactory().buildEventPayload(dataBudgetCtx(type), null); + var text = payload.getText() + "\n" + payload.getBlocks().toString(); + + assertThat(text).contains("Data Budget").contains("warehouse").contains("daily-reads") + .contains("ana@example.com").doesNotContain("Submitted by").doesNotContain(type.name()); + assertExtras(type, text); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void discordRendersADataBudgetTitleAndFieldSet(NotificationEventType type) { + var body = new DiscordPayloadFactory(json).buildEventBody(dataBudgetCtx(type), + new DiscordChannelConfig(URI.create("https://discord.example/hook"), null, null)); + + assertThat(body).contains("Data Budget").contains("warehouse").contains("daily-reads") + .contains("https://app.example.test/editor").doesNotContain("\"Submitted by\""); + assertExtras(type, body); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void teamsRendersADataBudgetTitleAndFieldSet(NotificationEventType type) { + var body = new MsTeamsPayloadFactory(json).buildEventBody(dataBudgetCtx(type)); + + assertThat(body).contains("Data Budget").contains("warehouse").contains("daily-reads") + .contains("Action.OpenUrl").doesNotContain("\"Submitted by\""); + assertExtras(type, body); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void telegramRendersADataBudgetTitleAndFieldSet(NotificationEventType type) { + var body = new TelegramMessageFactory(json).buildEventBody(dataBudgetCtx(type), "42").replace("\\\\", ""); + + assertThat(body).contains("Data Budget").contains("warehouse").contains("daily-reads") + .contains("Open the query editor").doesNotContain("Submitted by"); + assertExtras(type, body); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void pagerDutyHasASpecificSummaryEvenThoughNothingPages(NotificationEventType type) { + var body = new PagerDutyPayloadFactory(json).buildEventBody(dataBudgetCtx(type), + new PagerDutyChannelConfig("KEY", PagerDutySeverity.WARNING, EnumSet.allOf(PagerDutyTrigger.class))); + + assertThat(body).contains("data budget").contains("on warehouse") + .contains("\"budget_name\":\"daily-reads\"").doesNotContain("for a query"); + assertThat(PagerDutyTrigger.forEvent(type)).as("%s must not page", type).isEmpty(); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void webhookCarriesTheAdditiveDataBudgetBlock(NotificationEventType type) { + var ctx = dataBudgetCtx(type); + var tree = json.readTree(new WebhookPayloadFactory(json).buildBody(ctx)); + + assertThat(tree.path("event").asString()).isEqualTo(type.name()); + var block = tree.path("data_budget"); + assertThat(block.isObject()).isTrue(); + assertThat(block.path("budget_name").asString()).isEqualTo("daily-reads"); + assertThat(block.path("datasource_id").asString()).isEqualTo(ctx.datasourceId().toString()); + assertThat(block.path("user_id").asString()).isEqualTo(ctx.submittedByUserId().toString()); + assertThat(block.path("exhausted").asBoolean()) + .isEqualTo(type == NotificationEventType.DATA_BUDGET_EXHAUSTED); + assertThat(block.path("window_minutes").asInt()).isEqualTo(1440); + assertThat(block.path("breach_action").asString()).isEqualTo("REQUIRE_REVIEW"); + assertThat(tree.has("query_request")).isTrue(); + assertThat(tree.has("schema_change")).isFalse(); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void ticketHeadlineIsSpecificAndNothingTickets(NotificationEventType type) { + assertThat(TicketDescriptionBuilder.summary(dataBudgetCtx(type))) + .contains("Data budget").contains("on warehouse") + .doesNotContain(type.name()); + assertThat(TicketingTrigger.forEvent(type)).isEmpty(); + } + + @ParameterizedTest + @EnumSource(value = NotificationEventType.class, mode = EnumSource.Mode.MATCH_ANY, names = ALL) + void everyEventRoutesToAnEmailTemplate(NotificationEventType type) { + assertThat(EmailNotificationStrategy.hasTemplateFor(type)).isTrue(); + assertThat(EmailNotificationStrategy.hasTemplateFor(dataBudgetCtx(type))).isTrue(); + } + + @Test + void thresholdTemplateShowsUsageThresholdAndCta() { + var html = buildEngine().process("email/data-budget-threshold-reached", templateContext(Locale.ENGLISH)); + + assertThat(html).contains("warehouse").contains("daily-reads").contains("85%") + .contains("850 / 1000").contains("1.2 GB").contains("1 day").contains("80%") + .contains("Ana Analyst").contains("https://app.example.test/editor") + .doesNotContain("When used up"); + } + + @Test + void exhaustedTemplateShowsTheBreachActionAndOmitsUnsetLimits() { + var ctx = templateContext(Locale.ENGLISH); + ctx.setVariable("dataBudgetMaxRows", null); + ctx.setVariable("dataBudgetWindowUnit", "HOURS"); + ctx.setVariable("dataBudgetWindowValue", 3); + ctx.setVariable("reviewUrl", null); + + var html = buildEngine().process("email/data-budget-exhausted", ctx); + + assertThat(html).contains("Further reads go to human review").contains("3 hours") + .doesNotContain("Rows:").doesNotContain("href=\"null\""); + } + + @ParameterizedTest + @ValueSource(strings = {"en", "de", "es", "fr", "hy", "ru", "zh-CN"}) + void everyTemplateRendersInEveryShippedLocale(String locale) { + var engine = buildEngine(); + for (var template : TEMPLATES) { + assertThat(engine.process(template, templateContext(Locale.forLanguageTag(locale)))) + .as("template %s in locale %s", template, locale) + .doesNotContain("??notification.email"); + } + } + + @ParameterizedTest + @ValueSource(strings = {"en", "de", "es", "fr", "hy", "ru", "zh-CN"}) + void subjectsFillEveryPlaceholderInEveryShippedLocale(String locale) { + var messages = messageSource(); + var loc = Locale.forLanguageTag(locale); + var args = new Object[]{"daily-reads", "warehouse", 85}; + assertThat(messages.getMessage("notification.email.subject.data_budget_threshold_reached", args, loc)) + .contains("daily-reads").contains("warehouse").contains("85").doesNotContain("{"); + assertThat(messages.getMessage("notification.email.subject.data_budget_exhausted", args, loc)) + .contains("daily-reads").contains("warehouse").doesNotContain("{"); + } + + private static void assertExtras(NotificationEventType type, String rendered) { + assertThat(rendered).contains("850 / 1000 rows".replace("850", type == NotificationEventType.DATA_BUDGET_EXHAUSTED + ? "1000" : "850")).contains("1 day"); + if (type == NotificationEventType.DATA_BUDGET_EXHAUSTED) { + assertThat(rendered).contains("REQUIRE_REVIEW").contains("100%"); + } else { + assertThat(rendered).doesNotContain("REQUIRE_REVIEW").contains("85%"); + } + } + + private static Context templateContext(Locale locale) { + var ctx = new Context(locale); + ctx.setVariable("datasourceName", "warehouse"); + ctx.setVariable("submitterEmail", "ana@example.com"); + ctx.setVariable("submitterDisplayName", "Ana Analyst"); + ctx.setVariable("dataBudgetName", "daily-reads"); + ctx.setVariable("dataBudgetUsedPercent", 85); + ctx.setVariable("dataBudgetWarnThresholdPercent", 80); + ctx.setVariable("dataBudgetUsedRows", 850L); + ctx.setVariable("dataBudgetMaxRows", 1000L); + ctx.setVariable("dataBudgetBytesUsage", "1.2 GB (1200000000 B) / 5 GB (5000000000 B)"); + ctx.setVariable("dataBudgetWindowUnit", "DAYS"); + ctx.setVariable("dataBudgetWindowValue", 1); + ctx.setVariable("dataBudgetBreachAction", "REQUIRE_REVIEW"); + ctx.setVariable("reviewUrl", "https://app.example.test/editor"); + return ctx; + } + + private static ReloadableResourceBundleMessageSource messageSource() { + var messages = new ReloadableResourceBundleMessageSource(); + messages.setBasename("classpath:i18n/messages"); + messages.setDefaultEncoding(StandardCharsets.UTF_8.name()); + messages.setFallbackToSystemLocale(false); + return messages; + } + + private static SpringTemplateEngine buildEngine() { + var resolver = new ClassLoaderTemplateResolver(); + resolver.setPrefix("templates/"); + resolver.setSuffix(".html"); + resolver.setTemplateMode(TemplateMode.HTML); + resolver.setCharacterEncoding(StandardCharsets.UTF_8.name()); + resolver.setCacheable(false); + var springResolver = new SpringMessageResolver(); + springResolver.setMessageSource(messageSource()); + var engine = new SpringTemplateEngine(); + engine.setTemplateResolver(resolver); + engine.setMessageResolvers(Set.of(springResolver, new StandardMessageResolver())); + return engine; + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetTextTest.java b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetTextTest.java new file mode 100644 index 000000000..7dc95f7a7 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/notifications/internal/strategy/DataBudgetTextTest.java @@ -0,0 +1,42 @@ +package com.bablsoft.accessflow.notifications.internal.strategy; + +import com.bablsoft.accessflow.notifications.api.NotificationEventType; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class DataBudgetTextTest { + + @Test + void warningListsUsageButNotTheBreachAction() { + var fields = DataBudgetText.fields( + DataBudgetNotificationTest.dataBudgetCtx(NotificationEventType.DATA_BUDGET_THRESHOLD_REACHED)); + + assertThat(fields).extracting(Map.Entry::getKey) + .containsExactly("Datasource", "User", "Budget", "Used", "Rows", "Bytes", "Window"); + assertThat(fields).contains(Map.entry("User", "Ana Analyst (ana@example.com)"), + Map.entry("Used", "85%"), Map.entry("Window", "1 day")); + } + + @Test + void exhaustionAddsTheBreachAction() { + var fields = DataBudgetText.fields( + DataBudgetNotificationTest.dataBudgetCtx(NotificationEventType.DATA_BUDGET_EXHAUSTED)); + + assertThat(fields).contains(Map.entry("On breach", "REQUIRE_REVIEW")); + } + + @Test + void missingNamesRenderAsDashes() { + var ctx = DataBudgetNotificationTest.dataBudgetCtx(NotificationEventType.DATA_BUDGET_EXHAUSTED); + var bare = new com.bablsoft.accessflow.notifications.internal.NotificationContext( + ctx.eventType(), ctx.organizationId(), null, null, null, null, null, null, null, null, + ctx.datasourceId(), null, null, null, null, null, null, null, null, null, + ctx.recipients(), ctx.occurredAt(), "en", null); + + assertThat(DataBudgetText.fields(bare)) + .containsExactly(Map.entry("Datasource", "—"), Map.entry("User", "—")); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutorTest.java b/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutorTest.java index 5de6382d4..124929a66 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutorTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultQueryExecutorTest.java @@ -912,4 +912,42 @@ private static ResultSet emptyResultSet() throws SQLException { when(rs.next()).thenReturn(false); return rs; } + + @Test + void measureBytesStampsTheDeliveredSizeWithoutAnOverride() { + List> rows = List.of(List.of("abc"), List.of("defg")); + var result = new SelectExecutionResult(List.of(), rows, 2, false, Duration.ZERO); + + var measured = DefaultQueryExecutor.measureBytes(result, null); + + assertThat(measured.resultBytes()).isEqualTo(ResultByteEstimator.estimateRow(rows.get(0)) + + ResultByteEstimator.estimateRow(rows.get(1))); + assertThat(measured.truncated()).isFalse(); + assertThat(measured.rowCount()).isEqualTo(2); + } + + @Test + void measureBytesTrimsPastTheOverrideAndAttributesItToTheBudget() { + List> rows = List.of(List.of("aaaa"), List.of("bbbb"), List.of("cccc")); + var one = ResultByteEstimator.estimateRow(rows.get(0)); + var result = new SelectExecutionResult(List.of(), rows, 3, false, Duration.ZERO); + + var trimmed = DefaultQueryExecutor.measureBytes(result, one + 1); + + assertThat(trimmed.rowCount()).isEqualTo(1); + assertThat(trimmed.truncated()).isTrue(); + assertThat(trimmed.truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_DATA_BUDGET); + assertThat(trimmed.resultBytes()).isEqualTo(one); + } + + @Test + void measureBytesAlwaysKeepsTheFirstRow() { + List> rows = List.of(List.of("a very long value indeed")); + var result = new SelectExecutionResult(List.of(), rows, 1, false, Duration.ZERO); + + var kept = DefaultQueryExecutor.measureBytes(result, 1L); + + assertThat(kept.rowCount()).isEqualTo(1); + assertThat(kept.truncated()).isFalse(); + } } diff --git a/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataServiceTest.java index 2549c01b0..5a79ad67e 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataServiceTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/proxy/internal/DefaultSampleDataServiceTest.java @@ -56,6 +56,13 @@ class DefaultSampleDataServiceTest { @Mock private org.springframework.context.MessageSource messageSource; + @Mock + private com.bablsoft.accessflow.core.api.DataBudgetStatusService dataBudgetStatusService; + @Mock + private com.bablsoft.accessflow.core.api.DataBudgetUsageService dataBudgetUsageService; + @Mock + private com.bablsoft.accessflow.audit.api.AuditLogService auditLogService; + @InjectMocks private DefaultSampleDataService service; @@ -68,6 +75,9 @@ class DefaultSampleDataServiceTest { @BeforeEach void setUp() { + lenient().when(dataBudgetStatusService.statusFor(any(), any())) + .thenAnswer(inv -> com.bablsoft.accessflow.core.api.DataBudgetStatus.none( + inv.getArgument(0))); var schemaView = new DatabaseSchemaView(List.of( new DatabaseSchemaView.Schema("public", List.of( new DatabaseSchemaView.Table("Users", List.of( @@ -475,4 +485,114 @@ private DatasourceUserPermissionView denying(List allowedSchemas, false, false, false, allowedSchemas, List.of(), List.of(), null, deniedSchemas, deniedTables, List.of(), null, null); } + + @Test + void exhaustedBudgetRefusesThePreviewWhateverTheAction() { + when(permissionLookupService.findFor(userId, datasourceId)).thenReturn(Optional.empty()); + var budget = new com.bablsoft.accessflow.core.api.DataBudgetConsumption(UUID.randomUUID(), + "Daily", 10L, null, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, null, 10, 0); + when(dataBudgetStatusService.statusFor(datasourceId, userId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", + List.of(budget))); + when(messageSource.getMessage(eq("error.data_budget.exhausted"), any(), any())) + .thenReturn("used up"); + + assertThatThrownBy(() -> service.sample(datasourceId, organizationId, userId, true, + "public", "users", 50)) + .isInstanceOf(com.bablsoft.accessflow.core.api.DataBudgetExhaustedException.class) + .hasMessage("used up"); + verify(queryExecutor, never()).sampleTable(any()); + var audit = ArgumentCaptor.forClass(com.bablsoft.accessflow.audit.api.AuditEntry.class); + verify(auditLogService).record(audit.capture()); + assertThat(audit.getValue().action()) + .isEqualTo(com.bablsoft.accessflow.audit.api.AuditAction.QUERY_DATA_BUDGET_ENFORCED); + assertThat(audit.getValue().metadata()).containsEntry("stage", "sample") + .containsEntry("table", "public.users"); + } + + @Test + void remainingAllowanceCapsThePreviewAndIsCharged() { + when(permissionLookupService.findFor(userId, datasourceId)).thenReturn(Optional.empty()); + var budget = new com.bablsoft.accessflow.core.api.DataBudgetConsumption(UUID.randomUUID(), + "Daily", 10L, 1_000_000L, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, null, 7, 0); + when(dataBudgetStatusService.statusFor(datasourceId, userId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", + List.of(budget))); + List> rows = List.of(List.of("a"), List.of("b")); + when(queryExecutor.sampleTable(any())).thenReturn( + new SelectExecutionResult(List.of(), rows, 2, false, Duration.ZERO)); + + var out = service.sample(datasourceId, organizationId, userId, true, "public", "users", 50); + + var captor = ArgumentCaptor.forClass(SampleTableRequest.class); + verify(queryExecutor).sampleTable(captor.capture()); + assertThat(captor.getValue().maxRowsOverride()).isEqualTo(3); + assertThat(out.resultBytes()).isPositive(); + var usage = ArgumentCaptor.forClass(com.bablsoft.accessflow.core.api.DataBudgetUsageRecord.class); + verify(dataBudgetUsageService).record(usage.capture()); + assertThat(usage.getValue().rowsRead()).isEqualTo(2); + assertThat(usage.getValue().bytesRead()).isEqualTo(out.resultBytes()); + assertThat(usage.getValue().source()) + .isEqualTo(com.bablsoft.accessflow.core.api.DataBudgetUsageSource.SAMPLE_DATA); + } + + @Test + void aPreviewCutAtTheBudgetRowAllowanceIsAttributedToTheBudget() { + when(permissionLookupService.findFor(userId, datasourceId)).thenReturn(Optional.empty()); + var budget = new com.bablsoft.accessflow.core.api.DataBudgetConsumption(UUID.randomUUID(), + "Daily", 10L, null, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, null, 8, 0); + when(dataBudgetStatusService.statusFor(datasourceId, userId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", + List.of(budget))); + List> rows = List.of(List.of("a"), List.of("b")); + when(queryExecutor.sampleTable(any())).thenReturn(new SelectExecutionResult(List.of(), rows, + 2, true, Duration.ZERO, java.util.Set.of(), java.util.Set.of(), + SelectExecutionResult.TRUNCATED_ROW_LIMIT)); + + var out = service.sample(datasourceId, organizationId, userId, true, "public", "users", 50); + + assertThat(out.truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_DATA_BUDGET); + } + + @Test + void aPreviewCutByItsOwnLimitKeepsTheRowLimitReason() { + when(permissionLookupService.findFor(userId, datasourceId)).thenReturn(Optional.empty()); + var budget = new com.bablsoft.accessflow.core.api.DataBudgetConsumption(UUID.randomUUID(), + "Daily", 1_000L, null, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, null, 0, 0); + when(dataBudgetStatusService.statusFor(datasourceId, userId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", + List.of(budget))); + List> rows = List.of(List.of("a"), List.of("b")); + when(queryExecutor.sampleTable(any())).thenReturn(new SelectExecutionResult(List.of(), rows, + 2, true, Duration.ZERO, java.util.Set.of(), java.util.Set.of(), + SelectExecutionResult.TRUNCATED_ROW_LIMIT)); + + var out = service.sample(datasourceId, organizationId, userId, true, "public", "users", 2); + + assertThat(out.truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_ROW_LIMIT); + } + + @Test + void aFailedUsageWriteNeverFailsThePreview() { + when(permissionLookupService.findFor(userId, datasourceId)).thenReturn(Optional.empty()); + var budget = new com.bablsoft.accessflow.core.api.DataBudgetConsumption(UUID.randomUUID(), + "Daily", null, 1_000_000L, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, null, 0, 0); + when(dataBudgetStatusService.statusFor(datasourceId, userId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", + List.of(budget))); + org.mockito.Mockito.doThrow(new IllegalStateException("db down")) + .when(dataBudgetUsageService).record(any()); + + var out = service.sample(datasourceId, organizationId, userId, true, "public", "users", 50); + + assertThat(out.rowCount()).isZero(); + var captor = ArgumentCaptor.forClass(SampleTableRequest.class); + verify(queryExecutor).sampleTable(captor.capture()); + assertThat(captor.getValue().maxRowsOverride()).isEqualTo(50); + } } diff --git a/backend/src/test/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionServiceTest.java index d3a70b70b..094cf590c 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionServiceTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/requestgroups/internal/GroupExecutionServiceTest.java @@ -69,6 +69,10 @@ class GroupExecutionServiceTest { private com.bablsoft.accessflow.proxy.api.QueryCostEstimateService queryCostEstimateService; @Mock private com.bablsoft.accessflow.requestgroups.internal.persistence.repo.GroupReviewDecisionRepository decisionRepository; + @Mock + private com.bablsoft.accessflow.core.api.DataBudgetStatusService dataBudgetStatusService; + @Mock + private com.bablsoft.accessflow.core.api.DataBudgetUsageService dataBudgetUsageService; @InjectMocks private GroupExecutionService service; @@ -76,6 +80,9 @@ class GroupExecutionServiceTest { @BeforeEach void setUp() { + org.mockito.Mockito.lenient().when(dataBudgetStatusService.statusFor(any(), any())) + .thenAnswer(inv -> com.bablsoft.accessflow.core.api.DataBudgetStatus.none( + inv.getArgument(0))); group = new RequestGroupEntity(); group.setId(UUID.randomUUID()); group.setOrganizationId(UUID.randomUUID()); @@ -259,6 +266,94 @@ void aQueryMemberWithoutAnEstimateIsRefusedWhenNoPersonApprovedTheGroup() { verify(queryExecutor, org.mockito.Mockito.never()).execute(any()); } + private void givenBudget(RequestGroupItemEntity item, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction action, + long usedRows) { + when(dataBudgetStatusService.statusFor(item.getDatasourceId(), group.getSubmittedBy())) + .thenReturn(new com.bablsoft.accessflow.core.api.DataBudgetStatus( + item.getDatasourceId(), "ds", List.of( + new com.bablsoft.accessflow.core.api.DataBudgetConsumption( + UUID.randomUUID(), "Daily", 100L, null, 60, action, null, + usedRows, 0)))); + } + + @Test + void aQueryMemberIsCappedToTheRemainingAllowanceAndCharged() { + var item = queryItem(); + givenBudget(item, com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 95); + when(queryExecutor.execute(any())).thenReturn( + new com.bablsoft.accessflow.core.api.SelectExecutionResult( + List.of(), List.of(), 5L, true, java.time.Duration.ofMillis(3))); + + service.execute(group.getId(), null, "manual"); + + assertThat(item.getStatus()).isEqualTo(RequestGroupItemStatus.EXECUTED); + var request = org.mockito.ArgumentCaptor.forClass( + com.bablsoft.accessflow.core.api.QueryExecutionRequest.class); + verify(queryExecutor).execute(request.capture()); + assertThat(request.getValue().maxRowsOverride()).isEqualTo(5); + var usage = org.mockito.ArgumentCaptor.forClass( + com.bablsoft.accessflow.core.api.DataBudgetUsageRecord.class); + verify(dataBudgetUsageService).record(usage.capture()); + assertThat(usage.getValue().requestGroupId()).isEqualTo(group.getId()); + assertThat(usage.getValue().source()) + .isEqualTo(com.bablsoft.accessflow.core.api.DataBudgetUsageSource.REQUEST_GROUP); + } + + @Test + void anExhaustedRejectBudgetFailsTheMemberWithoutRunning() { + var item = queryItem(); + givenBudget(item, com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100); + when(messageSource.getMessage(org.mockito.ArgumentMatchers.eq("error.data_budget.exhausted"), + any(), any())).thenReturn("used up"); + + service.execute(group.getId(), null, "manual"); + + assertThat(item.getStatus()).isEqualTo(RequestGroupItemStatus.FAILED); + assertThat(item.getErrorMessage()).isEqualTo("used up"); + verify(queryExecutor, org.mockito.Mockito.never()).execute(any()); + var audit = org.mockito.ArgumentCaptor.forClass( + com.bablsoft.accessflow.audit.api.AuditEntry.class); + verify(auditLogService, org.mockito.Mockito.atLeastOnce()).record(audit.capture()); + assertThat(audit.getAllValues()).anySatisfy(entry -> { + assertThat(entry.action()) + .isEqualTo(com.bablsoft.accessflow.audit.api.AuditAction.QUERY_DATA_BUDGET_ENFORCED); + assertThat(entry.metadata()).containsEntry("item_id", item.getId().toString()) + .containsEntry("window_minutes", 60); + }); + } + + @Test + void anExhaustedReviewBudgetFailsTheMemberEvenOnceAPersonApprovedTheGroup() { + var item = queryItem(); + givenBudget(item, com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100); + // A person approved the group — which, fail-closed, must not lift an exhausted budget. + org.mockito.Mockito.lenient().when(decisionRepository.existsByRequestGroupIdAndDecision(group.getId(), + com.bablsoft.accessflow.core.api.DecisionType.APPROVED)).thenReturn(true); + when(messageSource.getMessage(org.mockito.ArgumentMatchers.eq("error.data_budget.exhausted"), + any(), any())).thenReturn("used up"); + + service.execute(group.getId(), null, "manual"); + + assertThat(item.getStatus()).isEqualTo(RequestGroupItemStatus.FAILED); + verify(queryExecutor, org.mockito.Mockito.never()).execute(any()); + } + + @Test + void aFailedUsageWriteNeverFailsTheMember() { + var item = queryItem(); + givenBudget(item, com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 0); + when(queryExecutor.execute(any())).thenReturn( + new com.bablsoft.accessflow.core.api.SelectExecutionResult( + List.of(), List.of(), 1L, false, java.time.Duration.ofMillis(3))); + org.mockito.Mockito.doThrow(new IllegalStateException("ledger down")) + .when(dataBudgetUsageService).record(any()); + + service.execute(group.getId(), null, "manual"); + + assertThat(item.getStatus()).isEqualTo(RequestGroupItemStatus.EXECUTED); + } + @Test void anUncappedQueryMemberIsNeverDryRun() { queryItem(); diff --git a/backend/src/test/java/com/bablsoft/accessflow/security/internal/web/DataBudgetControllerIntegrationTest.java b/backend/src/test/java/com/bablsoft/accessflow/security/internal/web/DataBudgetControllerIntegrationTest.java new file mode 100644 index 000000000..722b55b27 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/security/internal/web/DataBudgetControllerIntegrationTest.java @@ -0,0 +1,375 @@ +package com.bablsoft.accessflow.security.internal.web; + +import com.bablsoft.accessflow.TestcontainersConfig; +import com.bablsoft.accessflow.core.api.AuthProviderType; +import com.bablsoft.accessflow.core.api.CredentialEncryptionService; +import com.bablsoft.accessflow.core.api.DataBudgetUsageSource; +import com.bablsoft.accessflow.core.api.DbType; +import com.bablsoft.accessflow.core.api.SslMode; +import com.bablsoft.accessflow.core.api.UserRoleType; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetUsageEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DatasourceEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DatasourceUserPermissionEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.OrganizationEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.UserEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceUserPermissionRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.OrganizationRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserRepository; +import com.bablsoft.accessflow.security.internal.jwt.JwtService; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.testcontainers.context.ImportTestcontainers; +import org.springframework.http.HttpHeaders; +import org.springframework.http.MediaType; +import org.springframework.security.crypto.password.PasswordEncoder; +import org.springframework.test.web.servlet.assertj.MockMvcTester; +import org.springframework.web.context.WebApplicationContext; + +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.springframework.security.test.web.servlet.setup.SecurityMockMvcConfigurers.springSecurity; + +/** + * #942: admin CRUD, the caller's own standing and the admin per-user view, end to end over the + * real ledger. Identifiers are randomized and cleanup is scoped to what this class created, so it + * never trips over another class's rows in the shared container. + */ +@SpringBootTest +@ImportTestcontainers(TestcontainersConfig.class) +class DataBudgetControllerIntegrationTest { + + @Autowired WebApplicationContext context; + @Autowired UserRepository userRepository; + @Autowired OrganizationRepository organizationRepository; + @Autowired DatasourceRepository datasourceRepository; + @Autowired DatasourceUserPermissionRepository permissionRepository; + @Autowired DataBudgetRepository dataBudgetRepository; + @Autowired DataBudgetUsageRepository usageRepository; + @Autowired PasswordEncoder passwordEncoder; + @Autowired JwtService jwtService; + @Autowired CredentialEncryptionService encryptionService; + + private final String suffix = UUID.randomUUID().toString().substring(0, 8); + private final List createdPermissions = new ArrayList<>(); + private MockMvcTester mvc; + private OrganizationEntity primaryOrg; + private OrganizationEntity otherOrg; + private UserEntity admin; + private UserEntity analyst; + private UserEntity stranger; + private DatasourceEntity datasource; + private DatasourceEntity hidden; + private String adminToken; + private String analystToken; + + @BeforeEach + void setUp() { + mvc = MockMvcTester.from(context, builder -> builder.apply(springSecurity()).build()); + primaryOrg = saveOrg("Primary", "primary-db-" + suffix); + otherOrg = saveOrg("Other", "other-db-" + suffix); + admin = saveUser(primaryOrg, "admin-db-" + suffix + "@example.com", UserRoleType.ADMIN); + analyst = saveUser(primaryOrg, "analyst-db-" + suffix + "@example.com", + UserRoleType.ANALYST); + stranger = saveUser(otherOrg, "stranger-db-" + suffix + "@example.com", + UserRoleType.ANALYST); + datasource = saveDatasource(primaryOrg, "DB-DS-" + suffix); + hidden = saveDatasource(primaryOrg, "DB-HIDDEN-" + suffix); + grantRead(analyst, datasource); + adminToken = token(admin); + analystToken = token(analyst); + } + + @AfterEach + void cleanup() { + var datasourceIds = List.of(datasource.getId(), hidden.getId()); + usageRepository.deleteAll(usageRepository.findAll().stream() + .filter(u -> datasourceIds.contains(u.getDatasourceId())).toList()); + dataBudgetRepository.deleteAll(dataBudgetRepository.findAll().stream() + .filter(b -> datasourceIds.contains(b.getDatasourceId())).toList()); + permissionRepository.deleteAllById(createdPermissions); + datasourceRepository.deleteAllById(datasourceIds); + userRepository.deleteAllById(List.of(admin.getId(), analyst.getId(), stranger.getId())); + organizationRepository.deleteAllById(List.of(primaryOrg.getId(), otherOrg.getId())); + } + + private String base() { + return "/api/v1/datasources/" + datasource.getId() + "/data-budgets"; + } + + @Test + void createReturns201WithDefaults() { + var result = mvc.post().uri(base()) + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content(""" + {"name":"Analysts daily","max_rows":1000,"applies_to_roles":["ANALYST"], + "enabled":true} + """) + .exchange(); + + assertThat(result).hasStatus(201); + assertThat(result.getResponse().getHeader(HttpHeaders.LOCATION)).contains("/data-budgets/"); + assertThat(result).bodyJson().extractingPath("$.window_minutes").asNumber() + .isEqualTo(1440); + assertThat(result).bodyJson().extractingPath("$.breach_action").asString() + .isEqualTo("REQUIRE_REVIEW"); + assertThat(result).bodyJson().extractingPath("$.applies_to_roles").asArray() + .containsExactly("ANALYST"); + } + + @Test + void listUpdateAndDeleteRoundTrip() { + var id = createBudget("{\"name\":\"B\",\"max_rows\":10,\"enabled\":true}"); + + var list = mvc.get().uri(base()) + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken).exchange(); + assertThat(list).hasStatus(200); + assertThat(list).bodyJson().extractingPath("$.content[*].name").asArray() + .containsExactly("B"); + + var update = mvc.put().uri(base() + "/" + id) + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content(""" + {"name":"B2","max_bytes":2048,"window_minutes":60, + "breach_action":"REJECT","warn_threshold_percent":75,"enabled":false} + """) + .exchange(); + assertThat(update).hasStatus(200); + assertThat(update).bodyJson().extractingPath("$.breach_action").asString() + .isEqualTo("REJECT"); + assertThat(update).bodyJson().extractingPath("$.warn_threshold_percent").asNumber() + .isEqualTo(75); + + var delete = mvc.delete().uri(base() + "/" + id) + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken).exchange(); + assertThat(delete).hasStatus(204); + assertThat(dataBudgetRepository.findById(UUID.fromString(id))).isEmpty(); + } + + @Test + void anAnalystCannotManageBudgets() { + var result = mvc.post().uri(base()) + .header(HttpHeaders.AUTHORIZATION, "Bearer " + analystToken) + .contentType(MediaType.APPLICATION_JSON) + .content("{\"name\":\"B\",\"max_rows\":10}") + .exchange(); + + assertThat(result).hasStatus(403); + } + + @Test + void beanValidationFailuresReturn400() { + assertThat(post("{\"name\":\"\",\"max_rows\":10}")).hasStatus(400); + assertThat(post("{\"name\":\"B\",\"max_rows\":0}")).hasStatus(400); + assertThat(post("{\"name\":\"B\",\"max_rows\":10,\"window_minutes\":30}")).hasStatus(400); + assertThat(post("{\"name\":\"B\",\"max_rows\":10,\"warn_threshold_percent\":100}")) + .hasStatus(400); + } + + @Test + void aBudgetWithoutAnyLimitReturns422() { + var result = post("{\"name\":\"B\"}"); + + assertThat(result).hasStatus(422); + assertThat(result).bodyJson().extractingPath("$.error").asString() + .isEqualTo("ILLEGAL_DATA_BUDGET"); + } + + @Test + void anAppliesToUserOfAnotherOrgReturns422() { + var result = post("{\"name\":\"B\",\"max_rows\":10,\"applies_to_user_ids\":[\"" + + stranger.getId() + "\"]}"); + + assertThat(result).hasStatus(422); + } + + @Test + void anUnknownBudgetReturns404() { + var result = mvc.put().uri(base() + "/" + UUID.randomUUID()) + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content("{\"name\":\"B\",\"max_rows\":10}") + .exchange(); + + assertThat(result).hasStatus(404); + assertThat(result).bodyJson().extractingPath("$.error").asString() + .isEqualTo("DATA_BUDGET_NOT_FOUND"); + } + + @Test + void myStandingReflectsTheLedgerOverTheWindow() { + createBudget("{\"name\":\"Daily\",\"max_rows\":100,\"window_minutes\":60," + + "\"breach_action\":\"REJECT\",\"enabled\":true}"); + ledger(analyst, 30, Instant.now().minusSeconds(60)); + // Outside the one-hour window: must not count. + ledger(analyst, 50, Instant.now().minusSeconds(7_200)); + + var result = mvc.get().uri(base() + "/me") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + analystToken).exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.remaining_rows").asNumber().isEqualTo(70); + assertThat(result).bodyJson().extractingPath("$.used_percent").asNumber().isEqualTo(30); + assertThat(result).bodyJson().extractingPath("$.exhausted").asBoolean().isFalse(); + assertThat(result).bodyJson().extractingPath("$.budgets[0].used_rows").asNumber() + .isEqualTo(30); + } + + @Test + void myStandingIsEmptyWithoutABudget() { + var result = mvc.get().uri(base() + "/me") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + analystToken).exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.budgets").asArray().isEmpty(); + } + + @Test + void myStandingOnAnInvisibleDatasourceIs404() { + var result = mvc.get() + .uri("/api/v1/datasources/" + hidden.getId() + "/data-budgets/me") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + analystToken).exchange(); + + assertThat(result).hasStatus(404); + } + + @Test + void anAdminSeesAUsersStandingAcrossDatasources() { + createBudget("{\"name\":\"Daily\",\"max_rows\":10,\"breach_action\":\"REJECT\"," + + "\"enabled\":true}"); + ledger(analyst, 10, Instant.now().minusSeconds(60)); + + var result = mvc.get() + .uri("/api/v1/admin/users/" + analyst.getId() + "/data-budget-usage") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken).exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.content[0].datasource_name").asString() + .isEqualTo(datasource.getName()); + assertThat(result).bodyJson().extractingPath("$.content[0].exhausted").asBoolean() + .isTrue(); + assertThat(result).bodyJson().extractingPath("$.content[0].breach_action").asString() + .isEqualTo("REJECT"); + } + + @Test + void theAdminViewHidesUsersOfOtherOrganizationsAndAnalysts() { + var foreign = mvc.get() + .uri("/api/v1/admin/users/" + stranger.getId() + "/data-budget-usage") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken).exchange(); + var forbidden = mvc.get() + .uri("/api/v1/admin/users/" + analyst.getId() + "/data-budget-usage") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + analystToken).exchange(); + + assertThat(foreign).hasStatus(404); + assertThat(forbidden).hasStatus(403); + } + + private org.springframework.test.web.servlet.assertj.MvcTestResult post(String body) { + return mvc.post().uri(base()) + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content(body) + .exchange(); + } + + private String createBudget(String body) { + var result = post(body); + assertThat(result).hasStatus(201); + return dataBudgetRepository.findAllByOrganizationIdAndDatasourceIdOrderByCreatedAtAsc( + primaryOrg.getId(), datasource.getId()).getLast().getId().toString(); + } + + private void ledger(UserEntity user, long rows, Instant at) { + var entry = new DataBudgetUsageEntity(); + entry.setId(UUID.randomUUID()); + entry.setOrganizationId(primaryOrg.getId()); + entry.setUserId(user.getId()); + entry.setDatasourceId(datasource.getId()); + entry.setRowsRead(rows); + entry.setBytesRead(rows * 10); + entry.setSource(DataBudgetUsageSource.QUERY); + entry.setOccurredAt(at); + usageRepository.save(entry); + } + + private void grantRead(UserEntity user, DatasourceEntity ds) { + var permission = new DatasourceUserPermissionEntity(); + permission.setId(UUID.randomUUID()); + permission.setDatasource(ds); + permission.setUser(user); + permission.setCanRead(true); + permission.setCreatedBy(admin); + createdPermissions.add(permissionRepository.save(permission).getId()); + } + + private OrganizationEntity saveOrg(String name, String slug) { + var org = new OrganizationEntity(); + org.setId(UUID.randomUUID()); + org.setName(name + " " + suffix); + org.setSlug(slug); + return organizationRepository.save(org); + } + + private UserEntity saveUser(OrganizationEntity org, String email, UserRoleType role) { + var user = new UserEntity(); + user.setId(UUID.randomUUID()); + user.setEmail(email); + user.setDisplayName(email); + user.setPasswordHash(passwordEncoder.encode("Password123!")); + user.setRole(role); + user.setAuthProvider(AuthProviderType.LOCAL); + user.setActive(true); + user.setOrganization(org); + return userRepository.save(user); + } + + private DatasourceEntity saveDatasource(OrganizationEntity org, String name) { + var ds = new DatasourceEntity(); + ds.setId(UUID.randomUUID()); + ds.setOrganization(org); + ds.setName(name); + ds.setDbType(DbType.POSTGRESQL); + ds.setHost("nope.invalid"); + ds.setPort(65000); + ds.setDatabaseName("appdb"); + ds.setUsername("svc"); + ds.setPasswordEncrypted(encryptionService.encrypt("seed-password")); + ds.setSslMode(SslMode.DISABLE); + ds.setConnectionPoolSize(10); + ds.setMaxRowsPerQuery(1000); + ds.setRequireReviewReads(false); + ds.setRequireReviewWrites(true); + ds.setAiAnalysisEnabled(false); + ds.setActive(true); + return datasourceRepository.save(ds); + } + + private String token(UserEntity entity) { + var view = new com.bablsoft.accessflow.core.api.UserView( + entity.getId(), + entity.getEmail(), + entity.getDisplayName(), + entity.getRole(), + entity.getOrganization().getId(), + entity.isActive(), + entity.getAuthProvider(), + entity.getPasswordHash(), + entity.getLastLoginAt(), + entity.getPreferredLanguage(), + entity.isTotpEnabled(), + entity.getCreatedAt()); + return jwtService.generateAccessToken(view); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/api/ConditionNodeTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/api/ConditionNodeTest.java index f0b917889..e2c8f8aed 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/api/ConditionNodeTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/api/ConditionNodeTest.java @@ -181,6 +181,27 @@ void estimatedBytesScannedRejectsANullOperatorAndANegativeValue() { assertThat(node.value()).isEqualTo(1_000_000L); } + @Test + void dataBudgetUsedPercentRejectsANullOperatorAndANegativeValue() { + assertThatThrownBy(() -> new ConditionNode.DataBudgetUsedPercent(null, 5)) + .isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> new ConditionNode.DataBudgetUsedPercent(ComparisonOperator.GT, -1)) + .isInstanceOf(IllegalArgumentException.class); + var node = new ConditionNode.DataBudgetUsedPercent(ComparisonOperator.GTE, 80); + assertThat(node.operator()).isEqualTo(ComparisonOperator.GTE); + assertThat(node.value()).isEqualTo(80); + } + + @Test + void theBytesCompatibleContextConstructorLeavesTheBudgetSignalAbsent() { + var ctx = new ConditionContext(com.bablsoft.accessflow.core.api.QueryType.SELECT, + java.util.Set.of(), null, -1, null, java.util.Set.of(), + java.time.LocalDateTime.of(2026, 6, 3, 14, 30), false, false, false, null, null, + false, null, false, null, null, java.util.Set.of(), false, 5L); + assertThat(ctx.estimatedBytesScanned()).isEqualTo(5L); + assertThat(ctx.dataBudgetUsedPercent()).isNull(); + } + @Test void theShapeCompatibleContextConstructorLeavesTheBytesEstimateAbsent() { var context = new ConditionContext(QueryType.SELECT, Set.of(), RiskLevel.LOW, 1, "ANALYST", diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DataBudgetCheckTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DataBudgetCheckTest.java new file mode 100644 index 000000000..95ede5888 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DataBudgetCheckTest.java @@ -0,0 +1,53 @@ +package com.bablsoft.accessflow.workflow.internal; + +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetConsumption; +import com.bablsoft.accessflow.core.api.DataBudgetStatus; +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +class DataBudgetCheckTest { + + private final UUID datasourceId = UUID.randomUUID(); + + @Test + void noStandingMeansNoCheck() { + assertThat(DataBudgetCheck.of(null)).isNull(); + assertThat(DataBudgetCheck.of(DataBudgetStatus.none(datasourceId))).isNull(); + } + + @Test + void allowanceLeftDecidesNothing() { + var check = DataBudgetCheck.of(status(DataBudgetBreachAction.REJECT, 40)); + + assertThat(check.exhausted()).isFalse(); + assertThat(check.rejects()).isFalse(); + assertThat(check.forcesReview()).isFalse(); + assertThat(check.deciding()).isNull(); + assertThat(check.usedPercent()).isEqualTo(40d); + assertThat(check.remainingRows()).isEqualTo(60L); + assertThat(check.remainingBytes()).isNull(); + } + + @Test + void exhaustedBudgetsDecideByAction() { + var reject = DataBudgetCheck.of(status(DataBudgetBreachAction.REJECT, 100)); + assertThat(reject.exhausted()).isTrue(); + assertThat(reject.rejects()).isTrue(); + assertThat(reject.forcesReview()).isFalse(); + assertThat(reject.deciding()).isNotNull(); + + var review = DataBudgetCheck.of(status(DataBudgetBreachAction.REQUIRE_REVIEW, 120)); + assertThat(review.rejects()).isFalse(); + assertThat(review.forcesReview()).isTrue(); + } + + private DataBudgetStatus status(DataBudgetBreachAction action, long usedRows) { + return new DataBudgetStatus(datasourceId, "ds", List.of(new DataBudgetConsumption( + UUID.randomUUID(), "b", 100L, null, 60, action, null, usedRows, 0))); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DataBudgetEnforcementIntegrationTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DataBudgetEnforcementIntegrationTest.java new file mode 100644 index 000000000..f83cf4994 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DataBudgetEnforcementIntegrationTest.java @@ -0,0 +1,378 @@ +package com.bablsoft.accessflow.workflow.internal; + +import com.bablsoft.accessflow.TestcontainersConfig; +import com.bablsoft.accessflow.core.api.AuthProviderType; +import com.bablsoft.accessflow.core.api.CredentialEncryptionService; +import com.bablsoft.accessflow.core.api.DataBudgetBreachAction; +import com.bablsoft.accessflow.core.api.DataBudgetUsageSource; +import com.bablsoft.accessflow.core.api.DbType; +import com.bablsoft.accessflow.core.api.QueryDryRunResult; +import com.bablsoft.accessflow.core.api.QueryExecutionRequest; +import com.bablsoft.accessflow.core.api.QueryStatus; +import com.bablsoft.accessflow.core.api.QueryType; +import com.bablsoft.accessflow.core.api.ResultColumn; +import com.bablsoft.accessflow.core.api.SelectExecutionResult; +import com.bablsoft.accessflow.core.api.SqlParseResult; +import com.bablsoft.accessflow.core.api.SslMode; +import com.bablsoft.accessflow.core.api.UserRoleType; +import com.bablsoft.accessflow.core.events.AiAnalysisSkippedEvent; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DataBudgetUsageEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.DatasourceEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.OrganizationEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.QueryRequestEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.ReviewPlanApproverEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.ReviewPlanEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.UserEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataBudgetUsageRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.OrganizationRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.QueryRequestRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.ReviewPlanApproverRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.ReviewPlanRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserRepository; +import com.bablsoft.accessflow.proxy.api.QueryExecutor; +import com.bablsoft.accessflow.proxy.api.QueryParser; +import com.bablsoft.accessflow.workflow.api.ComparisonOperator; +import com.bablsoft.accessflow.workflow.api.ConditionNode; +import com.bablsoft.accessflow.workflow.api.QueryLifecycleService; +import com.bablsoft.accessflow.workflow.api.QueryLifecycleService.ExecuteQueryCommand; +import com.bablsoft.accessflow.workflow.api.RoutingAction; +import com.bablsoft.accessflow.workflow.internal.persistence.entity.RoutingPolicyEntity; +import com.bablsoft.accessflow.workflow.internal.persistence.repo.RoutingDecisionRepository; +import com.bablsoft.accessflow.workflow.internal.persistence.repo.RoutingPolicyRepository; +import com.bablsoft.accessflow.workflow.internal.routing.RoutingConditionCodec; +import org.awaitility.Awaitility; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.testcontainers.context.ImportTestcontainers; +import org.springframework.context.ApplicationEventPublisher; +import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.test.context.bean.override.mockito.MockitoBean; +import org.springframework.transaction.PlatformTransactionManager; +import org.springframework.transaction.support.TransactionTemplate; + +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import java.util.stream.IntStream; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * #942 end to end over the real ledger: an exhausted budget refuses or escalates a SELECT when it + * leaves {@code PENDING_AI}, a budget with allowance left caps the execution and is charged for + * what was delivered, and a {@code data_budget_used_percent} routing policy sees the live share. + * No customer database is contacted — {@link QueryExecutor} is mocked. + */ +@SpringBootTest +@ImportTestcontainers(TestcontainersConfig.class) +class DataBudgetEnforcementIntegrationTest { + + private static final String SQL = "SELECT * FROM events"; + + @MockitoBean QueryExecutor queryExecutor; + @MockitoBean QueryParser queryParser; + + @Autowired ApplicationEventPublisher eventPublisher; + @Autowired PlatformTransactionManager transactionManager; + @Autowired QueryLifecycleService queryLifecycleService; + @Autowired OrganizationRepository organizationRepository; + @Autowired UserRepository userRepository; + @Autowired DatasourceRepository datasourceRepository; + @Autowired QueryRequestRepository queryRequestRepository; + @Autowired ReviewPlanRepository reviewPlanRepository; + @Autowired ReviewPlanApproverRepository reviewPlanApproverRepository; + @Autowired RoutingPolicyRepository routingPolicyRepository; + @Autowired RoutingDecisionRepository routingDecisionRepository; + @Autowired RoutingConditionCodec routingConditionCodec; + @Autowired DataBudgetRepository dataBudgetRepository; + @Autowired DataBudgetUsageRepository usageRepository; + @Autowired CredentialEncryptionService encryptionService; + @Autowired JdbcTemplate jdbcTemplate; + + private OrganizationEntity organization; + private UserEntity submitter; + private DatasourceEntity datasource; + private ReviewPlanEntity plan; + + @BeforeEach + void setUp() { + organization = new OrganizationEntity(); + organization.setId(UUID.randomUUID()); + organization.setName("Budget org"); + organization.setSlug("budget-" + UUID.randomUUID()); + organizationRepository.save(organization); + submitter = persistUser(); + plan = persistPlan(); + datasource = persistDatasource(plan); + when(queryParser.parse(any(), any())).thenReturn(new SqlParseResult(QueryType.SELECT, SQL)); + when(queryExecutor.dryRun(any())).thenReturn(QueryDryRunResult.unsupported("postgresql")); + } + + @AfterEach + void cleanup() { + var orgId = organization.getId(); + var dsId = datasource.getId(); + jdbcTemplate.update("DELETE FROM audit_log WHERE organization_id = ?", orgId); + jdbcTemplate.update("DELETE FROM user_notifications WHERE user_id = ?", submitter.getId()); + jdbcTemplate.update("DELETE FROM routing_decision WHERE query_request_id IN " + + "(SELECT id FROM query_requests WHERE datasource_id = ?)", dsId); + jdbcTemplate.update("DELETE FROM routing_policy WHERE organization_id = ?", orgId); + jdbcTemplate.update("UPDATE query_requests SET query_estimate_id = NULL WHERE datasource_id = ?", + dsId); + jdbcTemplate.update("DELETE FROM query_estimates WHERE query_request_id IN " + + "(SELECT id FROM query_requests WHERE datasource_id = ?)", dsId); + jdbcTemplate.update("DELETE FROM query_requests WHERE datasource_id = ?", dsId); + jdbcTemplate.update("DELETE FROM data_budget_usage WHERE datasource_id = ?", dsId); + jdbcTemplate.update("DELETE FROM data_budgets WHERE datasource_id = ?", dsId); + datasourceRepository.deleteById(dsId); + jdbcTemplate.update("DELETE FROM review_plan_approvers WHERE review_plan_id = ?", plan.getId()); + reviewPlanRepository.deleteById(plan.getId()); + userRepository.deleteById(submitter.getId()); + organizationRepository.deleteById(orgId); + } + + @Test + void anExhaustedRejectBudgetRejectsTheQueryWhenItLeavesPendingAi() { + persistBudget(DataBudgetBreachAction.REJECT, 100L); + spend(100); + var query = persistPendingAiQuery(); + + publish(new AiAnalysisSkippedEvent(query.getId(), "ai_analysis_enabled=false")); + + awaitStatus(query.getId(), QueryStatus.REJECTED); + assertThat(budgetAuditMetadata(query.getId())) + .contains("\"stage\": \"decision\"") + .contains("\"action\": \"REJECT\""); + } + + @Test + void anExhaustedReviewBudgetHoldsAnAutoApprovalForAPerson() { + persistBudget(DataBudgetBreachAction.REQUIRE_REVIEW, 100L); + spend(250); + var query = persistPendingAiQuery(); + + publish(new AiAnalysisSkippedEvent(query.getId(), "ai_analysis_enabled=false")); + + awaitStatus(query.getId(), QueryStatus.PENDING_REVIEW); + assertThat(budgetAuditMetadata(query.getId())).contains("REQUIRE_REVIEW"); + assertThat(queryRequestRepository.findById(query.getId()).orElseThrow() + .isDataBudgetReviewForced()).isTrue(); + } + + @Test + void theRemainingAllowanceCapsTheExecutionAndTheDeliveredRowsAreCharged() { + persistBudget(DataBudgetBreachAction.REJECT, 100L); + spend(97); + var query = persistPendingAiQuery(); + publish(new AiAnalysisSkippedEvent(query.getId(), "ai_analysis_enabled=false")); + awaitStatus(query.getId(), QueryStatus.APPROVED); + List> rows = IntStream.range(0, 3).>mapToObj(List::of).toList(); + when(queryExecutor.execute(any())).thenReturn(new SelectExecutionResult( + List.of(new ResultColumn("id", 4, "int4")), rows, 3L, true, Duration.ofMillis(3), + Set.of(), Set.of(), SelectExecutionResult.TRUNCATED_ROW_LIMIT, null, 120L)); + + var outcome = queryLifecycleService.execute(new ExecuteQueryCommand(query.getId(), + submitter.getId(), organization.getId(), false)); + + assertThat(outcome.status()).isEqualTo(QueryStatus.EXECUTED); + var request = ArgumentCaptor.forClass(QueryExecutionRequest.class); + verify(queryExecutor).execute(request.capture()); + assertThat(request.getValue().maxRowsOverride()).isEqualTo(3); + assertThat(jdbcTemplate.queryForObject("SELECT truncated_reason FROM query_request_results " + + "WHERE query_request_id = ?", String.class, query.getId())) + .isEqualTo(SelectExecutionResult.TRUNCATED_DATA_BUDGET); + var charged = usageRepository.findAll().stream() + .filter(u -> query.getId().equals(u.getQueryRequestId())).toList(); + assertThat(charged).singleElement().satisfies(u -> { + assertThat(u.getRowsRead()).isEqualTo(3); + assertThat(u.getBytesRead()).isEqualTo(120); + assertThat(u.getSource()).isEqualTo(DataBudgetUsageSource.QUERY); + }); + } + + @Test + void aBudgetExhaustedAfterApprovalFailsTheExecution() { + persistBudget(DataBudgetBreachAction.REJECT, 100L); + var query = persistPendingAiQuery(); + publish(new AiAnalysisSkippedEvent(query.getId(), "ai_analysis_enabled=false")); + awaitStatus(query.getId(), QueryStatus.APPROVED); + spend(100); + + var outcome = queryLifecycleService.execute(new ExecuteQueryCommand(query.getId(), + submitter.getId(), organization.getId(), false)); + + assertThat(outcome.status()).isEqualTo(QueryStatus.FAILED); + verify(queryExecutor, never()).execute(any()); + assertThat(budgetAuditMetadata(query.getId())).contains("\"stage\": \"execution\""); + } + + @Test + void aUsedPercentPolicyEscalatesAHeavyReader() { + persistBudget(DataBudgetBreachAction.REJECT, 100L); + spend(85); + var policy = persistPolicy(new ConditionNode.DataBudgetUsedPercent(ComparisonOperator.GTE, + 80)); + var query = persistPendingAiQuery(); + + publish(new AiAnalysisSkippedEvent(query.getId(), "ai_analysis_enabled=false")); + + awaitStatus(query.getId(), QueryStatus.PENDING_REVIEW); + assertThat(routingDecisionRepository.findByQueryRequestId(query.getId()).orElseThrow() + .getMatchedPolicyId()).isEqualTo(policy.getId()); + } + + @Test + void noBudgetLeavesBehaviourUnchangedAndWritesNoLedgerRow() { + var query = persistPendingAiQuery(); + publish(new AiAnalysisSkippedEvent(query.getId(), "ai_analysis_enabled=false")); + awaitStatus(query.getId(), QueryStatus.APPROVED); + when(queryExecutor.execute(any())).thenReturn(new SelectExecutionResult(List.of(), + List.of(), 0L, false, Duration.ofMillis(3))); + + queryLifecycleService.execute(new ExecuteQueryCommand(query.getId(), submitter.getId(), + organization.getId(), false)); + + var request = ArgumentCaptor.forClass(QueryExecutionRequest.class); + verify(queryExecutor).execute(request.capture()); + assertThat(request.getValue().maxResultBytesOverride()).isNull(); + assertThat(jdbcTemplate.queryForObject("SELECT count(*) FROM data_budget_usage " + + "WHERE datasource_id = ?", Integer.class, datasource.getId())).isZero(); + } + + // ── Fixtures ────────────────────────────────────────────────────────────── + + private void publish(Object event) { + new TransactionTemplate(transactionManager) + .executeWithoutResult(status -> eventPublisher.publishEvent(event)); + } + + private void awaitStatus(UUID queryId, QueryStatus expected) { + Awaitility.await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> + assertThat(queryRequestRepository.findById(queryId).orElseThrow().getStatus()) + .isEqualTo(expected)); + } + + private String budgetAuditMetadata(UUID queryId) { + var rows = jdbcTemplate.queryForList("SELECT metadata::text FROM audit_log WHERE " + + "resource_id = ? AND action = 'QUERY_DATA_BUDGET_ENFORCED'", String.class, + queryId); + assertThat(rows).hasSize(1); + return rows.get(0); + } + + private void spend(long rows) { + var entry = new DataBudgetUsageEntity(); + entry.setId(UUID.randomUUID()); + entry.setOrganizationId(organization.getId()); + entry.setUserId(submitter.getId()); + entry.setDatasourceId(datasource.getId()); + entry.setRowsRead(rows); + entry.setBytesRead(rows); + entry.setSource(DataBudgetUsageSource.QUERY); + entry.setOccurredAt(Instant.now().minusSeconds(30)); + usageRepository.save(entry); + } + + private void persistBudget(DataBudgetBreachAction action, Long maxRows) { + var budget = new DataBudgetEntity(); + budget.setId(UUID.randomUUID()); + budget.setOrganizationId(organization.getId()); + budget.setDatasourceId(datasource.getId()); + budget.setName("Daily"); + budget.setMaxRows(maxRows); + budget.setWindowMinutes(1440); + budget.setBreachAction(action); + budget.setEnabled(true); + dataBudgetRepository.save(budget); + } + + private UserEntity persistUser() { + var user = new UserEntity(); + user.setId(UUID.randomUUID()); + user.setEmail("budget-" + UUID.randomUUID() + "@example.com"); + user.setDisplayName("submitter"); + user.setPasswordHash("hash"); + user.setRole(UserRoleType.ANALYST); + user.setAuthProvider(AuthProviderType.LOCAL); + user.setActive(true); + user.setOrganization(organization); + return userRepository.save(user); + } + + private ReviewPlanEntity persistPlan() { + var reviewPlan = new ReviewPlanEntity(); + reviewPlan.setId(UUID.randomUUID()); + reviewPlan.setOrganization(organization); + reviewPlan.setName("plan-" + UUID.randomUUID()); + reviewPlan.setRequiresAiReview(false); + reviewPlan.setRequiresHumanApproval(false); + reviewPlan.setMinApprovalsRequired(1); + reviewPlan.setApprovalTimeoutHours(24); + reviewPlan.setAutoApproveReads(false); + reviewPlanRepository.save(reviewPlan); + var rule = new ReviewPlanApproverEntity(); + rule.setId(UUID.randomUUID()); + rule.setReviewPlan(reviewPlan); + rule.setRole("REVIEWER"); + rule.setStage(1); + reviewPlanApproverRepository.save(rule); + return reviewPlan; + } + + private DatasourceEntity persistDatasource(ReviewPlanEntity reviewPlan) { + var ds = new DatasourceEntity(); + ds.setId(UUID.randomUUID()); + ds.setOrganization(organization); + ds.setName("BUDGET-" + UUID.randomUUID()); + ds.setDbType(DbType.POSTGRESQL); + ds.setHost("nope.invalid"); + ds.setPort(65000); + ds.setDatabaseName("appdb"); + ds.setUsername("svc"); + ds.setPasswordEncrypted(encryptionService.encrypt("seed-password")); + ds.setSslMode(SslMode.DISABLE); + ds.setConnectionPoolSize(5); + ds.setMaxRowsPerQuery(1000); + ds.setReviewPlan(reviewPlan); + ds.setAiAnalysisEnabled(false); + ds.setActive(true); + return datasourceRepository.save(ds); + } + + private QueryRequestEntity persistPendingAiQuery() { + var query = new QueryRequestEntity(); + query.setId(UUID.randomUUID()); + query.setDatasource(datasource); + query.setSubmittedBy(submitter); + query.setSqlText(SQL); + query.setQueryType(QueryType.SELECT); + query.setStatus(QueryStatus.PENDING_AI); + return queryRequestRepository.save(query); + } + + private RoutingPolicyEntity persistPolicy(ConditionNode condition) { + var policy = new RoutingPolicyEntity(); + policy.setId(UUID.randomUUID()); + policy.setOrganizationId(organization.getId()); + policy.setName("policy-" + UUID.randomUUID()); + policy.setPriority(1); + policy.setEnabled(true); + policy.setAction(RoutingAction.ESCALATE); + policy.setRequiredApprovals(1); + policy.setConditionJson(routingConditionCodec.encode(condition)); + return routingPolicyRepository.save(policy); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationServiceTest.java index 99719f337..dde8bdc70 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationServiceTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultAccessSimulationServiceTest.java @@ -77,6 +77,7 @@ import static org.mockito.ArgumentMatchers.eq; import static org.mockito.Mockito.never; import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.lenient; import static org.mockito.Mockito.when; @ExtendWith(MockitoExtension.class) @@ -99,6 +100,7 @@ class DefaultAccessSimulationServiceTest { @Mock MaskingPolicyResolutionService maskingPolicyResolutionService; @Mock BreakGlassEligibilityService breakGlassEligibilityService; @Mock com.bablsoft.accessflow.core.api.BytesScannedCapResolutionService bytesScannedCapResolutionService; + @Mock com.bablsoft.accessflow.core.api.DataBudgetStatusService dataBudgetStatusService; private DefaultAccessSimulationService service; @@ -115,8 +117,11 @@ void buildService() { routingPolicyEngine, reviewPlanLookupService, reviewerEligibilityService, rowSecurityResolutionService, rowSecurityClassificationService, maskingPolicyResolutionService, - breakGlassEligibilityService, bytesScannedCapResolutionService); + breakGlassEligibilityService, bytesScannedCapResolutionService, + dataBudgetStatusService); service.setClock(clock); + lenient().when(dataBudgetStatusService.statusFor(any(), any())) + .thenAnswer(inv -> com.bablsoft.accessflow.core.api.DataBudgetStatus.none(inv.getArgument(0))); } @BeforeEach @@ -136,7 +141,7 @@ void stubHappyPath() { when(permissionLookupService.findFor(userId, datasourceId)) .thenReturn(Optional.of(permission(true, false, false, List.of("public")))); when(queryDecisionEvaluator.evaluate(any(), any(), any(), org.mockito.ArgumentMatchers.anyInt(), - any(), any(), any())).thenReturn(planDecision(QueryStatus.PENDING_REVIEW)); + any(), any(), any(), any())).thenReturn(planDecision(QueryStatus.PENDING_REVIEW)); when(sqlReviewService.evaluate(eq(organizationId), eq(datasourceId), any())) .thenReturn(SqlReviewResult.clean()); when(routingPolicyEngine.enabledFor(organizationId, datasourceId)).thenReturn(List.of()); @@ -176,7 +181,8 @@ void aSimulationNeverAsksTheEvaluatorToApplyAnything() { service.simulate(organizationId, input(AiOutcome.COMPLETED, RiskLevel.LOW, 5)); verify(queryDecisionEvaluator).evaluate(any(), eq(AiOutcome.COMPLETED), eq(RiskLevel.LOW), - eq(5), eq(List.of()), org.mockito.ArgumentMatchers.isNull(), eq(clock)); + eq(5), eq(List.of()), org.mockito.ArgumentMatchers.isNull(), + org.mockito.ArgumentMatchers.isNull(), eq(clock)); verify(datasourceAdminService, never()).update(any(), any(), any()); } @@ -197,7 +203,8 @@ void theSimulatorHandsTheEvaluatorTheSameBlockingRuleIdsTheLivePathWouldRead() { verify(queryDecisionEvaluator).evaluate(any(), eq(AiOutcome.COMPLETED), eq(RiskLevel.LOW), eq(5), eq(List.of("cross_join", "select_star")), - org.mockito.ArgumentMatchers.isNull(), eq(clock)); + org.mockito.ArgumentMatchers.isNull(), org.mockito.ArgumentMatchers.isNull(), + eq(clock)); verify(sqlReviewService).evaluate(organizationId, datasourceId, "SELECT card_number FROM public.payments"); } @@ -210,7 +217,8 @@ void aNonRelationalDatasourceContributesNoBlockingRuleIds() { service.simulate(organizationId, input(AiOutcome.SKIPPED, null, null)); verify(queryDecisionEvaluator).evaluate(any(), eq(AiOutcome.SKIPPED), eq(null), eq(-1), - eq(List.of()), org.mockito.ArgumentMatchers.isNull(), eq(clock)); + eq(List.of()), org.mockito.ArgumentMatchers.isNull(), + org.mockito.ArgumentMatchers.isNull(), eq(clock)); } @Test @@ -224,12 +232,42 @@ void aBytesCapIsHandedToTheEvaluatorUnevaluatedBecauseASimulationHasNoEstimate() var passed = org.mockito.ArgumentCaptor.forClass(BytesCapCheck.class); verify(queryDecisionEvaluator).evaluate(any(), any(), any(), - org.mockito.ArgumentMatchers.anyInt(), any(), passed.capture(), any()); + org.mockito.ArgumentMatchers.anyInt(), any(), passed.capture(), any(), any()); assertThat(passed.getValue().limit()).isEqualTo(500L); assertThat(passed.getValue().outcome()).isNull(); assertThat(passed.getValue().rejects()).isFalse(); } + @Test + void theLiveDataBudgetStandingIsHandedToTheEvaluatorForASelect() { + when(dataBudgetStatusService.statusFor(datasourceId, userId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", List.of( + new com.bablsoft.accessflow.core.api.DataBudgetConsumption(UUID.randomUUID(), + "Daily", 10L, null, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, null, + 10, 0)))); + + service.simulate(organizationId, input(AiOutcome.COMPLETED, RiskLevel.LOW, 5)); + + var passed = org.mockito.ArgumentCaptor.forClass(DataBudgetCheck.class); + verify(queryDecisionEvaluator).evaluate(any(), any(), any(), + org.mockito.ArgumentMatchers.anyInt(), any(), any(), passed.capture(), any()); + assertThat(passed.getValue().rejects()).isTrue(); + } + + @Test + void aWriteIsNeverCheckedAgainstADataBudget() { + when(queryParser.parse(any(), any())).thenReturn( + new SqlParseResult(QueryType.UPDATE, false, List.of("UPDATE t SET a = 1"), + Set.of("public.payments"), true, false)); + when(permissionLookupService.findFor(userId, datasourceId)) + .thenReturn(Optional.of(permission(true, true, false, List.of("public")))); + + service.simulate(organizationId, input(AiOutcome.COMPLETED, RiskLevel.LOW, 5)); + + verify(dataBudgetStatusService, never()).statusFor(any(), any()); + } + // ── Shape ───────────────────────────────────────────────────────────────── @Test @@ -533,7 +571,7 @@ void theRoutingStepCarriesEveryPolicyNotJustTheWinner() { @Test void reviewersAreSkippedWhenTheRequestWouldNotReachReview() { when(queryDecisionEvaluator.evaluate(any(), any(), any(), - org.mockito.ArgumentMatchers.anyInt(), any(), any(), any())) + org.mockito.ArgumentMatchers.anyInt(), any(), any(), any(), any())) .thenReturn(planDecision(QueryStatus.APPROVED)); var result = service.simulate(organizationId, input(AiOutcome.COMPLETED, RiskLevel.LOW, 5)); @@ -553,13 +591,14 @@ void theSimulatorEvaluatesInTheSameZoneTheLiveListenerDoes() { routingPolicyEngine, reviewPlanLookupService, reviewerEligibilityService, rowSecurityResolutionService, rowSecurityClassificationService, maskingPolicyResolutionService, - breakGlassEligibilityService, bytesScannedCapResolutionService); + breakGlassEligibilityService, bytesScannedCapResolutionService, + dataBudgetStatusService); fresh.simulate(organizationId, input(AiOutcome.COMPLETED, RiskLevel.LOW, 5)); var passed = org.mockito.ArgumentCaptor.forClass(Clock.class); verify(queryDecisionEvaluator).evaluate(any(), any(), any(), - org.mockito.ArgumentMatchers.anyInt(), any(), any(), passed.capture()); + org.mockito.ArgumentMatchers.anyInt(), any(), any(), any(), passed.capture()); assertThat(passed.getValue().getZone()).isEqualTo(ZoneId.systemDefault()); } @@ -602,7 +641,7 @@ void theDecisivePolicyIsTheOneTheDecisionNamesNotASecondEvaluationsOwnFirstMatch when(routingPolicyEngine.evaluateAll(any(), any())) .thenReturn(List.of(alsoMatched, decided)); when(queryDecisionEvaluator.evaluate(any(), any(), any(), - org.mockito.ArgumentMatchers.anyInt(), any(), any(), any())) + org.mockito.ArgumentMatchers.anyInt(), any(), any(), any(), any())) .thenReturn(routedDecision(decidedId)); var result = service.simulate(organizationId, input(AiOutcome.COMPLETED, RiskLevel.LOW, 5)); @@ -744,6 +783,7 @@ private QueryDecision planDecision(QueryStatus status) { var steps = List.of( DecisionTraceStep.of(QueryDecisionStepKind.SQL_REVIEW, StepOutcome.NO_MATCH, "k"), DecisionTraceStep.of(QueryDecisionStepKind.BYTES_SCANNED_CAP, StepOutcome.NO_MATCH, "k"), + DecisionTraceStep.of(QueryDecisionStepKind.DATA_BUDGET, StepOutcome.NO_MATCH, "k"), DecisionTraceStep.of(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.NO_MATCH, "k"), DecisionTraceStep.of(QueryDecisionStepKind.GRANT_FAST_PATH, StepOutcome.NO_MATCH, "k"), DecisionTraceStep.of(QueryDecisionStepKind.REVIEW_PLAN, StepOutcome.DENY, "k")); @@ -764,6 +804,7 @@ private QueryDecision routedDecision(UUID policyId) { var steps = List.of( DecisionTraceStep.of(QueryDecisionStepKind.SQL_REVIEW, StepOutcome.NO_MATCH, "k"), DecisionTraceStep.of(QueryDecisionStepKind.BYTES_SCANNED_CAP, StepOutcome.NO_MATCH, "k"), + DecisionTraceStep.of(QueryDecisionStepKind.DATA_BUDGET, StepOutcome.NO_MATCH, "k"), DecisionTraceStep.of(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.MATCH, "k"), DecisionTraceStep.of(QueryDecisionStepKind.GRANT_FAST_PATH, StepOutcome.SKIP, "k"), DecisionTraceStep.of(QueryDecisionStepKind.REVIEW_PLAN, StepOutcome.SKIP, "k")); diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleServiceTest.java index 19d52f703..eb8ef7c20 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleServiceTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/DefaultQueryLifecycleServiceTest.java @@ -97,6 +97,8 @@ class DefaultQueryLifecycleServiceTest { @Mock ApplicationEventPublisher eventPublisher; @Mock com.bablsoft.accessflow.core.api.BytesScannedCapResolutionService bytesScannedCapResolutionService; @Mock com.bablsoft.accessflow.proxy.api.QueryCostEstimateService queryCostEstimateService; + @Mock com.bablsoft.accessflow.core.api.DataBudgetStatusService dataBudgetStatusService; + @Mock com.bablsoft.accessflow.core.api.DataBudgetUsageService dataBudgetUsageService; DefaultQueryLifecycleService service; @@ -136,7 +138,12 @@ void setUp() { messageSource, eventPublisher, bytesScannedCapResolutionService, - queryCostEstimateService); + queryCostEstimateService, + dataBudgetStatusService, + dataBudgetUsageService); + when(dataBudgetStatusService.statusFor(any(), any())) + .thenAnswer(inv -> com.bablsoft.accessflow.core.api.DataBudgetStatus.none(inv.getArgument(0))); + when(queryParser.parse(anyString(), any())).thenAnswer(inv -> { String sql = inv.getArgument(0); return new SqlParseResult(QueryType.SELECT, sql); @@ -1425,4 +1432,213 @@ void executeRecurringOccurrenceHaltsWhenSubmitterInactive() { verify(queryRequestPersistenceService).clearRecurrenceNextRun(eq(queryId), anyString()); verify(queryExecutor, never()).execute(any()); } + + // ── Data budget (#942) ──────────────────────────────────────────────────── + + private final UUID budgetId = UUID.randomUUID(); + + private void givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction action, + long usedRows) { + when(dataBudgetStatusService.statusFor(datasourceId, submitterId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", List.of( + new com.bablsoft.accessflow.core.api.DataBudgetConsumption(budgetId, + "Daily", 100L, 10_000L, 1440, action, 80, usedRows, 0)))); + } + + private SelectExecutionResult tenRows(boolean truncated, String reason) { + List> rows = java.util.stream.IntStream.range(0, 10) + .>mapToObj(List::of).toList(); + return new SelectExecutionResult(List.of(new ResultColumn("id", 4, "int4")), rows, 10L, + truncated, Duration.ofMillis(5), java.util.Set.of(), java.util.Set.of(), reason, + null, 480L); + } + + @Test + void executeCapsTheResultToTheRemainingAllowanceAndChargesIt() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.SELECT))); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 90); + when(queryExecutor.execute(any())).thenReturn(tenRows(true, + SelectExecutionResult.TRUNCATED_ROW_LIMIT)); + + var outcome = service.execute(new ExecuteQueryCommand(queryId, submitterId, organizationId, + false)); + + assertThat(outcome.status()).isEqualTo(QueryStatus.EXECUTED); + var request = ArgumentCaptor.forClass(QueryExecutionRequest.class); + verify(queryExecutor).execute(request.capture()); + assertThat(request.getValue().maxRowsOverride()).isEqualTo(10); + assertThat(request.getValue().maxResultBytesOverride()).isEqualTo(10_000L); + var usage = ArgumentCaptor.forClass(com.bablsoft.accessflow.core.api.DataBudgetUsageRecord.class); + verify(dataBudgetUsageService).record(usage.capture()); + assertThat(usage.getValue().rowsRead()).isEqualTo(10); + assertThat(usage.getValue().bytesRead()).isEqualTo(480); + assertThat(usage.getValue().queryRequestId()).isEqualTo(queryId); + assertThat(usage.getValue().source()) + .isEqualTo(com.bablsoft.accessflow.core.api.DataBudgetUsageSource.QUERY); + var audit = ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService).record(audit.capture()); + assertThat(audit.getValue().metadata()).containsEntry("data_budget_rows_charged", 10L) + .containsEntry("data_budget_bytes_charged", 480L); + } + + @Test + void executeFailsBeforeTheExecutorWhenARejectBudgetIsExhausted() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.SELECT))); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100); + when(messageSource.getMessage(eq("error.data_budget.exhausted"), any(), any())) + .thenReturn("budget used up"); + + var outcome = service.execute(new ExecuteQueryCommand(queryId, submitterId, organizationId, + false)); + + assertThat(outcome.status()).isEqualTo(QueryStatus.FAILED); + verify(queryExecutor, never()).execute(any()); + var exec = ArgumentCaptor.forClass(RecordExecutionCommand.class); + verify(queryRequestStateService).recordExecutionOutcome(exec.capture()); + assertThat(exec.getValue().errorMessage()).isEqualTo("budget used up"); + var audit = ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService, org.mockito.Mockito.times(2)).record(audit.capture()); + assertThat(audit.getAllValues()).extracting(AuditEntry::action).containsExactly( + AuditAction.QUERY_DATA_BUDGET_ENFORCED, AuditAction.QUERY_FAILED); + assertThat(audit.getAllValues().get(0).metadata()) + .containsEntry("stage", "execution") + .containsEntry("action", "REJECT") + .containsEntry("data_budget_id", budgetId); + verify(dataBudgetUsageService, never()).record(any()); + } + + @Test + void executeFailsAnUnreviewedQueryWhenAReviewBudgetIsExhausted() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.SELECT))); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100); + // Approved while allowance remained: the approval was never a budget escalation. + when(queryRequestStateService.isDataBudgetReviewForced(queryId)).thenReturn(false); + + var outcome = service.execute(new ExecuteQueryCommand(queryId, submitterId, organizationId, + false)); + + assertThat(outcome.status()).isEqualTo(QueryStatus.FAILED); + verify(queryExecutor, never()).execute(any()); + } + + @Test + void executeRunsABudgetEscalatedQueryUncappedWhenAReviewBudgetIsExhausted() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.SELECT))); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100); + when(queryRequestStateService.isDataBudgetReviewForced(queryId)).thenReturn(true); + when(queryExecutor.execute(any())).thenReturn(tenRows(false, null)); + + var outcome = service.execute(new ExecuteQueryCommand(queryId, submitterId, organizationId, + false)); + + assertThat(outcome.status()).isEqualTo(QueryStatus.EXECUTED); + var request = ArgumentCaptor.forClass(QueryExecutionRequest.class); + verify(queryExecutor).execute(request.capture()); + assertThat(request.getValue().maxRowsOverride()).isNull(); + assertThat(request.getValue().maxResultBytesOverride()).isNull(); + verify(dataBudgetUsageService).record(any()); + } + + @Test + void aRecurringOccurrenceInheritsItsSeriesApproval() { + var parentId = UUID.randomUUID(); + var occurrence = new QueryRequestSnapshot(queryId, datasourceId, organizationId, + submitterId, "SELECT 1", QueryType.SELECT, false, QueryStatus.APPROVED, + java.time.Instant.EPOCH, null, null, false, null, null, null, parentId); + when(queryRequestLookupService.findById(queryId)).thenReturn(Optional.of(occurrence)); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100); + when(queryRequestStateService.isDataBudgetReviewForced(queryId)).thenReturn(false); + when(queryRequestStateService.isDataBudgetReviewForced(parentId)).thenReturn(true); + when(queryExecutor.execute(any())).thenReturn(tenRows(false, null)); + + service.executeScheduled(queryId); + + verify(queryExecutor).execute(any()); + verify(dataBudgetUsageService).record(any()); + } + + @Test + void breakGlassRunsDespiteAnExhaustedRejectBudgetAndStillCharges() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.SELECT))); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100); + when(queryExecutor.execute(any())).thenReturn(tenRows(false, null)); + + var outcome = service.executeBreakGlass(queryId, submitterId); + + assertThat(outcome.status()).isEqualTo(QueryStatus.EXECUTED); + verify(dataBudgetUsageService).record(any()); + } + + @Test + void breakGlassIsNeverCappedByARemainingAllowance() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.SELECT))); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 99); + when(queryExecutor.execute(any())).thenReturn(tenRows(false, null)); + + service.executeBreakGlass(queryId, submitterId); + + var request = ArgumentCaptor.forClass(QueryExecutionRequest.class); + verify(queryExecutor).execute(request.capture()); + assertThat(request.getValue().maxRowsOverride()).isNull(); + assertThat(request.getValue().maxResultBytesOverride()).isNull(); + verify(dataBudgetUsageService).record(any()); + } + + @Test + void aFailedUsageWriteNeverFailsTheExecution() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.SELECT))); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 0); + when(queryExecutor.execute(any())).thenReturn(tenRows(false, null)); + org.mockito.Mockito.doThrow(new IllegalStateException("ledger down")) + .when(dataBudgetUsageService).record(any()); + + var outcome = service.execute(new ExecuteQueryCommand(queryId, submitterId, organizationId, + false)); + + assertThat(outcome.status()).isEqualTo(QueryStatus.EXECUTED); + } + + @Test + void aWriteNeverConsultsTheBudget() { + when(queryRequestLookupService.findById(queryId)) + .thenReturn(Optional.of(snapshot(QueryStatus.APPROVED, QueryType.UPDATE))); + when(queryParser.parse(anyString(), any())).thenReturn( + new SqlParseResult(QueryType.UPDATE, "UPDATE t SET a = 1")); + when(queryExecutor.execute(any())).thenReturn( + new com.bablsoft.accessflow.core.api.UpdateExecutionResult(3, Duration.ofMillis(5))); + + service.execute(new ExecuteQueryCommand(queryId, submitterId, organizationId, false)); + + verify(dataBudgetStatusService, never()).statusFor(any(), any()); + verify(dataBudgetUsageService, never()).record(any()); + } + + @Test + void budgetTruncationIsAttributedOnlyWhenTheAllowanceWasTheBindingRowCap() { + var cut = tenRows(true, SelectExecutionResult.TRUNCATED_ROW_LIMIT); + + assertThat(DefaultQueryLifecycleService.attributeBudgetTruncation(cut, 10L, null, 1_000) + .truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_DATA_BUDGET); + assertThat(DefaultQueryLifecycleService.attributeBudgetTruncation(cut, 10L, 10, null) + .truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_ROW_LIMIT); + assertThat(DefaultQueryLifecycleService.attributeBudgetTruncation(cut, 10L, null, 10) + .truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_ROW_LIMIT); + assertThat(DefaultQueryLifecycleService.attributeBudgetTruncation(cut, 50L, null, null) + .truncatedReason()).isEqualTo(SelectExecutionResult.TRUNCATED_ROW_LIMIT); + assertThat(DefaultQueryLifecycleService.attributeBudgetTruncation(cut, null, null, null)) + .isSameAs(cut); + var notCut = tenRows(false, null); + assertThat(DefaultQueryLifecycleService.attributeBudgetTruncation(notCut, 10L, null, null)) + .isSameAs(notCut); + var byteCut = tenRows(true, SelectExecutionResult.TRUNCATED_BYTE_LIMIT); + assertThat(DefaultQueryLifecycleService.attributeBudgetTruncation(byteCut, 10L, null, null)) + .isSameAs(byteCut); + } } diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluatorTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluatorTest.java index 5945435eb..c6070c2c3 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluatorTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryDecisionEvaluatorTest.java @@ -65,6 +65,7 @@ class QueryDecisionEvaluatorTest { @Mock BehaviorAnomalyLookupService behaviorAnomalyLookupService; @Mock AccessGrantLookupService accessGrantLookupService; @Mock QueryEstimateLookupService queryEstimateLookupService; + @Mock com.bablsoft.accessflow.core.api.DataBudgetStatusService dataBudgetStatusService; private QueryDecisionEvaluator evaluator; @@ -80,7 +81,9 @@ class QueryDecisionEvaluatorTest { void buildEvaluator() { var contextFactory = new ConditionContextFactory(queryRequestLookupService, sqlParserService, userQueryService, userGroupService, behaviorAnomalyLookupService, - queryEstimateLookupService); + queryEstimateLookupService, dataBudgetStatusService); + lenient().when(dataBudgetStatusService.statusFor(any(), any())) + .thenAnswer(inv -> com.bablsoft.accessflow.core.api.DataBudgetStatus.none(inv.getArgument(0))); evaluator = new QueryDecisionEvaluator(reviewPlanLookupService, contextFactory, sqlParserService, routingPolicyEngine, accessGrantLookupService); } @@ -116,6 +119,8 @@ void aiFailureTraceSkipsEveryDecisionStage() { StepOutcome.NO_MATCH), org.assertj.core.groups.Tuple.tuple(QueryDecisionStepKind.BYTES_SCANNED_CAP, StepOutcome.NO_MATCH), + org.assertj.core.groups.Tuple.tuple(QueryDecisionStepKind.DATA_BUDGET, + StepOutcome.NO_MATCH), org.assertj.core.groups.Tuple.tuple(QueryDecisionStepKind.ROUTING_POLICIES, StepOutcome.SKIP), org.assertj.core.groups.Tuple.tuple(QueryDecisionStepKind.GRANT_FAST_PATH, @@ -435,6 +440,7 @@ void thePlanFallThroughTraceCoversAllThreeStages() { assertThat(trace.steps()).extracting("step").containsExactly( QueryDecisionStepKind.SQL_REVIEW, QueryDecisionStepKind.BYTES_SCANNED_CAP, + QueryDecisionStepKind.DATA_BUDGET, QueryDecisionStepKind.ROUTING_POLICIES, QueryDecisionStepKind.GRANT_FAST_PATH, QueryDecisionStepKind.REVIEW_PLAN); assertThat(step(trace, QueryDecisionStepKind.ROUTING_POLICIES).outcome()) @@ -788,6 +794,165 @@ void theSqlReviewNamesTheSuppressionWhenBothGuardsApply() { .isEqualTo("workflow.decision.plan.suppressed_sql_review"); } + // ── Data budget (#942) ──────────────────────────────────────────────────── + + private static DataBudgetCheck budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction action, + long usedRows) { + var consumption = new com.bablsoft.accessflow.core.api.DataBudgetConsumption( + UUID.randomUUID(), "Daily", 100L, null, 1440, + action == null ? com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT : action, + 80, usedRows, 0); + return DataBudgetCheck.of(new com.bablsoft.accessflow.core.api.DataBudgetStatus( + UUID.randomUUID(), "ds", List.of(consumption))); + } + + @Test + void anExhaustedRejectBudgetRejectsBeforeRoutingIsConsulted() { + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.COMPLETED, + RiskLevel.LOW, 5, List.of(), null, + budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100), clock); + + assertThat(decision.kind()).isEqualTo(QueryDecisionKind.DATA_BUDGET_REJECTED); + assertThat(decision.nextStatus()).isEqualTo(QueryStatus.REJECTED); + assertThat(decision.dataBudgetChangedOutcome()).isTrue(); + assertThat(decision.bytesCapChangedOutcome()).isFalse(); + var budgetStep = step(decision.trace(), QueryDecisionStepKind.DATA_BUDGET); + assertThat(budgetStep.outcome()).isEqualTo(StepOutcome.DENY); + assertThat(budgetStep.reasonKey()).isEqualTo("workflow.decision.data_budget.exhausted_rejected"); + assertThat(budgetStep.reasonArgs()).containsExactly("Daily"); + assertThat(budgetStep.details()).containsEntry("data_budget_name", "Daily") + .containsEntry("data_budget_action", "REJECT"); + assertThat(step(decision.trace(), QueryDecisionStepKind.ROUTING_POLICIES).reasonKey()) + .isEqualTo("workflow.decision.routing.skipped_data_budget"); + assertThat(step(decision.trace(), QueryDecisionStepKind.GRANT_FAST_PATH).reasonKey()) + .isEqualTo("workflow.decision.grant.skipped_data_budget"); + assertThat(step(decision.trace(), QueryDecisionStepKind.REVIEW_PLAN).reasonKey()) + .isEqualTo("workflow.decision.plan.skipped_data_budget"); + verify(routingPolicyEngine, never()).evaluate(any(), any(), any()); + } + + @Test + void theBytesCapRefusalWinsOverAnExhaustedBudget() { + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.COMPLETED, + RiskLevel.LOW, 5, List.of(), + cap(2_000_000_000_000L, BytesScannedCapOutcome.EXCEEDED), + budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100), clock); + + assertThat(decision.kind()).isEqualTo(QueryDecisionKind.BYTES_CAP_REJECTED); + assertThat(decision.dataBudget()).isNotNull(); + assertThat(decision.dataBudgetChangedOutcome()).isFalse(); + } + + @Test + void anExhaustedRejectBudgetRejectsEvenWhenAiFailed() { + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.FAILED, null, -1, + List.of(), null, + budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100), clock); + + assertThat(decision.kind()).isEqualTo(QueryDecisionKind.DATA_BUDGET_REJECTED); + } + + @Test + void theAiFailedPathRecordsTheBudgetStanding() { + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.FAILED, null, -1, + List.of(), null, budget(null, 10), clock); + + assertThat(decision.kind()).isEqualTo(QueryDecisionKind.AI_FAILED_PENDING_REVIEW); + assertThat(decision.dataBudget()).isNotNull(); + var budgetStep = step(decision.trace(), QueryDecisionStepKind.DATA_BUDGET); + assertThat(budgetStep.outcome()).isEqualTo(StepOutcome.ALLOW); + assertThat(budgetStep.reasonKey()).isEqualTo("workflow.decision.data_budget.within"); + assertThat(budgetStep.reasonArgs()).containsExactly("10"); + } + + @Test + void anExhaustedReviewBudgetSuppressesRoutingAutoApprove() { + givenPlan(false, true); + givenPolicyMatch(RoutingAction.AUTO_APPROVE, null); + + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.COMPLETED, + RiskLevel.LOW, 5, List.of(), null, + budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100), + clock); + + assertThat(decision.kind()).isEqualTo(QueryDecisionKind.ROUTING_AUTO_APPROVE_SUPPRESSED); + assertThat(decision.nextStatus()).isEqualTo(QueryStatus.PENDING_REVIEW); + assertThat(decision.dataBudgetChangedOutcome()).isTrue(); + assertThat(decision.bytesCapChangedOutcome()).isFalse(); + var routing = step(decision.trace(), QueryDecisionStepKind.ROUTING_POLICIES); + assertThat(routing.reasonKey()).isEqualTo( + "workflow.decision.routing.matched_auto_approve_suppressed_data_budget"); + assertThat(routing.details()).containsEntry("data_budget_suppressed", true) + .containsEntry("bytes_cap_suppressed", false); + var budgetStep = step(decision.trace(), QueryDecisionStepKind.DATA_BUDGET); + assertThat(budgetStep.outcome()).isEqualTo(StepOutcome.MATCH); + assertThat(budgetStep.reasonKey()).isEqualTo("workflow.decision.data_budget.exhausted_review"); + } + + @Test + void anExhaustedReviewBudgetSuppressesTheGrantAndThePlan() { + givenPlan(false, false); + givenNoPolicyMatch(); + givenActiveGrant(grant(true, false, false, List.of(), List.of())); + + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.SKIPPED, null, -1, + List.of(), null, + budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100), + clock); + + assertThat(decision.nextStatus()).isEqualTo(QueryStatus.PENDING_REVIEW); + assertThat(decision.dataBudgetChangedOutcome()).isTrue(); + assertThat(step(decision.trace(), QueryDecisionStepKind.GRANT_FAST_PATH).reasonKey()) + .isEqualTo("workflow.decision.grant.suppressed_data_budget"); + var plan = step(decision.trace(), QueryDecisionStepKind.REVIEW_PLAN); + assertThat(plan.reasonKey()).isEqualTo("workflow.decision.plan.suppressed_data_budget"); + assertThat(plan.details()).containsEntry("data_budget_suppressed", true); + } + + @Test + void aBudgetWithAllowanceLeftChangesNothing() { + givenPlan(false, false); + givenNoPolicyMatch(); + givenNoGrants(); + + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.SKIPPED, null, -1, + List.of(), null, budget(null, 30), clock); + + assertThat(decision.kind()).isEqualTo(QueryDecisionKind.PLAN_APPROVED); + assertThat(decision.dataBudgetChangedOutcome()).isFalse(); + } + + @Test + void anExhaustedReviewBudgetNeverSoftensAnAutoReject() { + givenPlan(false, true); + givenPolicyMatch(RoutingAction.AUTO_REJECT, null); + + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.COMPLETED, + RiskLevel.LOW, 5, List.of(), null, + budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100), + clock); + + assertThat(decision.kind()).isEqualTo(QueryDecisionKind.ROUTING_AUTO_REJECT); + assertThat(decision.dataBudgetChangedOutcome()).isFalse(); + } + + @Test + void theBytesCapNamesTheSuppressionAheadOfTheBudget() { + givenPlan(false, false); + givenNoPolicyMatch(); + givenNoGrants(); + + var decision = evaluator.evaluate(query(QueryType.SELECT), AiOutcome.SKIPPED, null, -1, + List.of(), cap(null, BytesScannedCapOutcome.NO_ESTIMATE_REVIEW), + budget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 100), + clock); + + assertThat(decision.bytesCapChangedOutcome()).isTrue(); + assertThat(decision.dataBudgetChangedOutcome()).isTrue(); + assertThat(step(decision.trace(), QueryDecisionStepKind.REVIEW_PLAN).reasonKey()) + .isEqualTo("workflow.decision.plan.suppressed_bytes_cap"); + } + // ── Fixtures ────────────────────────────────────────────────────────────── private static com.bablsoft.accessflow.core.api.DecisionTraceStep step( diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachineTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachineTest.java index 5afc2b887..6b11c7852 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachineTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/QueryReviewStateMachineTest.java @@ -75,6 +75,7 @@ class QueryReviewStateMachineTest { @Mock com.bablsoft.accessflow.core.api.BytesScannedCapResolutionService bytesScannedCapResolutionService; @Mock com.bablsoft.accessflow.proxy.api.QueryCostEstimateService queryCostEstimateService; @Mock org.springframework.transaction.PlatformTransactionManager transactionManager; + @Mock com.bablsoft.accessflow.core.api.DataBudgetStatusService dataBudgetStatusService; // A real ConditionContextFactory over the same mocks, not a mock of it: the context builder is // what turns these signals into routing input, and mocking it away would stop testing that. @@ -93,7 +94,7 @@ void buildStateMachine() { var contextFactory = new com.bablsoft.accessflow.workflow.internal.routing .ConditionContextFactory(queryRequestLookupService, sqlParserService, userQueryService, userGroupService, behaviorAnomalyLookupService, - queryEstimateLookupService); + queryEstimateLookupService, dataBudgetStatusService); // A real QueryDecisionEvaluator too, for the same reason: the assertions below are about the // chain's behaviour, and mocking the decision away would leave only the switch under test. var evaluator = new QueryDecisionEvaluator(reviewPlanLookupService, contextFactory, @@ -101,7 +102,10 @@ void buildStateMachine() { stateMachine = new QueryReviewStateMachine(queryRequestLookupService, evaluator, queryRequestStateService, routingDecisionService, sqlReviewFindingService, auditLogService, messageSource, eventPublisher, bytesScannedCapResolutionService, - queryCostEstimateService, queryEstimateLookupService, transactionManager); + queryCostEstimateService, queryEstimateLookupService, transactionManager, + dataBudgetStatusService); + lenient().when(dataBudgetStatusService.statusFor(any(), any())) + .thenAnswer(inv -> com.bablsoft.accessflow.core.api.DataBudgetStatus.none(inv.getArgument(0))); } @BeforeEach @@ -865,6 +869,89 @@ void aFailingCapAuditNeverUndoesTheRejection() { QueryStatus.REJECTED); } + // ── Data budget (#942) ──────────────────────────────────────────────────── + + private void givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction action, + long usedRows) { + when(dataBudgetStatusService.statusFor(datasourceId, submitterId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", List.of( + new com.bablsoft.accessflow.core.api.DataBudgetConsumption(budgetId, + "Daily", 100L, null, 1440, action, 80, usedRows, 0)))); + } + + private final UUID budgetId = UUID.randomUUID(); + + @Test + void anExhaustedRejectBudgetRejectsAndAudits() { + givenPendingAiQuery(QueryType.SELECT); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100); + when(messageSource.getMessage(eq("workflow.data_budget.rejected"), any(), any())) + .thenReturn("budget used up"); + + stateMachine.onAiCompleted(new AiAnalysisCompletedEvent(queryId, aiAnalysisId, + RiskLevel.LOW)); + + verify(queryRequestStateService).transitionTo(queryId, QueryStatus.PENDING_AI, + QueryStatus.REJECTED); + verify(queryRequestStateService, never()).recordDataBudgetReviewForced(any()); + verify(eventPublisher).publishEvent(new QueryAutoRejectedEvent(queryId, null, + "budget used up")); + var audit = org.mockito.ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService).record(audit.capture()); + assertThat(audit.getValue().action()).isEqualTo(AuditAction.QUERY_DATA_BUDGET_ENFORCED); + assertThat(audit.getValue().metadata()) + .containsEntry("trigger", "data_budget") + .containsEntry("stage", "decision") + .containsEntry("action", "REJECT") + .containsEntry("data_budget_id", budgetId) + .containsEntry("used_rows", 100L) + .containsEntry("window_minutes", 1440); + verify(routingPolicyEngine, never()).evaluate(any(), any(), any()); + } + + @Test + void anExhaustedReviewBudgetHoldsAnAutoApprovalAndAuditsIt() { + givenPendingAiQuery(QueryType.SELECT); + givenPlan(false, false, RiskLevel.LOW); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REQUIRE_REVIEW, 150); + + stateMachine.onAiCompleted(new AiAnalysisCompletedEvent(queryId, aiAnalysisId, + RiskLevel.LOW)); + + verify(queryRequestStateService).transitionTo(queryId, QueryStatus.PENDING_AI, + QueryStatus.PENDING_REVIEW); + verify(queryRequestStateService).recordDataBudgetReviewForced(queryId); + var audit = org.mockito.ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService).record(audit.capture()); + assertThat(audit.getValue().action()).isEqualTo(AuditAction.QUERY_DATA_BUDGET_ENFORCED); + assertThat(audit.getValue().metadata()).containsEntry("action", "REQUIRE_REVIEW"); + } + + @Test + void aWriteIsNeverCheckedAgainstTheBudget() { + givenPendingAiQuery(QueryType.UPDATE); + givenPlan(false, false, RiskLevel.LOW); + + stateMachine.onAiCompleted(new AiAnalysisCompletedEvent(queryId, aiAnalysisId, + RiskLevel.LOW)); + + verify(dataBudgetStatusService, never()).statusFor(any(), any()); + } + + @Test + void aFailingBudgetAuditNeverUndoesTheRejection() { + givenPendingAiQuery(QueryType.SELECT); + givenBudget(com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, 100); + org.mockito.Mockito.doThrow(new IllegalStateException("audit down")) + .when(auditLogService).record(any()); + + stateMachine.onAiCompleted(new AiAnalysisCompletedEvent(queryId, aiAnalysisId, + RiskLevel.LOW)); + + verify(queryRequestStateService).transitionTo(queryId, QueryStatus.PENDING_AI, + QueryStatus.REJECTED); + } + private void givenActiveGrant(AccessGrantView grant) { when(accessGrantLookupService.findActivePreApprovedGrants(organizationId, submitterId, datasourceId)).thenReturn(List.of(grant)); diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactoryTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactoryTest.java index a4f61cb8b..bac048dee 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactoryTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/ConditionContextFactoryTest.java @@ -31,6 +31,7 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.lenient; import static org.mockito.Mockito.when; @ExtendWith(MockitoExtension.class) @@ -43,6 +44,7 @@ class ConditionContextFactoryTest { @Mock UserGroupService userGroupService; @Mock BehaviorAnomalyLookupService behaviorAnomalyLookupService; @Mock QueryEstimateLookupService queryEstimateLookupService; + @Mock com.bablsoft.accessflow.core.api.DataBudgetStatusService dataBudgetStatusService; private ConditionContextFactory factory; @@ -56,7 +58,9 @@ class ConditionContextFactoryTest { void setUp() { factory = new ConditionContextFactory(queryRequestLookupService, sqlParserService, userQueryService, userGroupService, behaviorAnomalyLookupService, - queryEstimateLookupService); + queryEstimateLookupService, dataBudgetStatusService); + lenient().when(dataBudgetStatusService.statusFor(any(), any())) + .thenAnswer(inv -> com.bablsoft.accessflow.core.api.DataBudgetStatus.none(inv.getArgument(0))); when(sqlParserService.parse(any())).thenReturn(new SqlParseResult(QueryType.SELECT, false, List.of("SELECT 1"), Set.of("public.orders"), true, false)); when(userQueryService.findById(submitterId)).thenReturn(Optional.empty()); @@ -237,4 +241,50 @@ void historicalRowCarriesTheParsedQueryShapes() { assertThat(context.queryShapes()).containsExactlyInAnyOrder(QueryShape.JOIN, QueryShape.AGGREGATE); assertThat(context.shapesAnalyzed()).isTrue(); } + + private com.bablsoft.accessflow.core.api.QueryRequestSnapshot live(QueryType type) { + return new com.bablsoft.accessflow.core.api.QueryRequestSnapshot(queryId, datasourceId, + orgId, submitterId, "SELECT 1", type, false, + com.bablsoft.accessflow.core.api.QueryStatus.PENDING_AI, null, null, null, false); + } + + @Test + void aLiveSelectCarriesTheSubmittersMostUsedBudgetShare() { + when(dataBudgetStatusService.statusFor(datasourceId, submitterId)).thenReturn( + new com.bablsoft.accessflow.core.api.DataBudgetStatus(datasourceId, "ds", List.of( + new com.bablsoft.accessflow.core.api.DataBudgetConsumption( + java.util.UUID.randomUUID(), "b", 200L, null, 60, + com.bablsoft.accessflow.core.api.DataBudgetBreachAction.REJECT, + null, 171, 0)))); + + var context = factory.forLiveQuery(live(QueryType.SELECT), RiskLevel.LOW, 10, + java.time.Clock.systemUTC()); + + assertThat(context.dataBudgetUsedPercent()).isEqualTo(85); + } + + @Test + void aLiveSelectWithoutABudgetHasNoBudgetSignal() { + var context = factory.forLiveQuery(live(QueryType.SELECT), RiskLevel.LOW, 10, + java.time.Clock.systemUTC()); + + assertThat(context.dataBudgetUsedPercent()).isNull(); + } + + @Test + void aLiveWriteNeverReadsTheBudget() { + var context = factory.forLiveQuery(live(QueryType.UPDATE), RiskLevel.LOW, 10, + java.time.Clock.systemUTC()); + + assertThat(context.dataBudgetUsedPercent()).isNull(); + org.mockito.Mockito.verify(dataBudgetStatusService, org.mockito.Mockito.never()) + .statusFor(any(), any()); + } + + @Test + void theHistoricalReplayNeverCarriesTheBudgetSignal() { + var context = factory.forHistoricalRow(row(RiskLevel.LOW, 10), ZoneId.of("UTC")); + + assertThat(context.dataBudgetUsedPercent()).isNull(); + } } diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionCodecTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionCodecTest.java index 6ffdd7627..34248c5b4 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionCodecTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionCodecTest.java @@ -53,6 +53,7 @@ void roundTripsEveryLeafAndCombinator() { new ConditionNode.CiCdOrigin(true), new ConditionNode.EstimatedRows(ComparisonOperator.GT, 100_000L), new ConditionNode.EstimatedBytesScanned(ComparisonOperator.GT, 1_000_000_000L), + new ConditionNode.DataBudgetUsedPercent(ComparisonOperator.GTE, 80), new ConditionNode.ScanTypeMatches(List.of("Seq*", "COLLSCAN")))); var json = codec.encode(tree); @@ -82,6 +83,16 @@ void estimatedBytesScannedUsesItsSnakeCaseDiscriminator() { .isEqualTo(new ConditionNode.EstimatedBytesScanned(ComparisonOperator.GT, 5L)); } + @Test + void dataBudgetUsedPercentUsesItsSnakeCaseDiscriminator() { + var json = codec.encode(new ConditionNode.DataBudgetUsedPercent(ComparisonOperator.GTE, 80)); + + assertThat(json).contains("\"type\":\"data_budget_used_percent\""); + assertThat(codec.decode( + "{\"type\":\"data_budget_used_percent\",\"operator\":\"GT\",\"value\":50}")) + .isEqualTo(new ConditionNode.DataBudgetUsedPercent(ComparisonOperator.GT, 50)); + } + @Test void queryShapeUsesItsDiscriminatorAndUpperCaseShapeNames() { var json = codec.encode(new ConditionNode.QueryShapeIn(Set.of(QueryShape.GROUP_BY))); diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluatorTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluatorTest.java index 709a845cc..2a7b15fa2 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluatorTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/routing/RoutingConditionEvaluatorTest.java @@ -358,6 +358,28 @@ void estimatedBytesScannedFailsClosedWhenNoEstimate() { ComparisonOperator.GTE, 0), bytesContext(null))).isFalse(); } + @Test + void dataBudgetUsedPercent() { + var ctx = budgetContext(85); + assertThat(evaluator.matches(new ConditionNode.DataBudgetUsedPercent( + ComparisonOperator.GTE, 80), ctx)).isTrue(); + assertThat(evaluator.matches(new ConditionNode.DataBudgetUsedPercent( + ComparisonOperator.GTE, 90), ctx)).isFalse(); + } + + @Test + void dataBudgetUsedPercentFailsClosedWhenNoBudgetApplies() { + assertThat(evaluator.matches(new ConditionNode.DataBudgetUsedPercent( + ComparisonOperator.GTE, 0), budgetContext(null))).isFalse(); + } + + private ConditionContext budgetContext(Integer usedPercent) { + return new ConditionContext(QueryType.SELECT, Set.of("ds.events"), RiskLevel.LOW, 10, + "ANALYST", Set.of(groupId), LocalDateTime.of(2026, 6, 3, 14, 30), + false, false, false, null, null, false, null, false, 10L, null, Set.of(), false, + null, usedPercent); + } + private ConditionContext bytesContext(Long bytes) { return new ConditionContext(QueryType.SELECT, Set.of("ds.events"), RiskLevel.LOW, 10, "ANALYST", Set.of(groupId), LocalDateTime.of(2026, 6, 3, 14, 30), diff --git a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/web/AdminAccessSimulationControllerIntegrationTest.java b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/web/AdminAccessSimulationControllerIntegrationTest.java index f7e7d56be..171caac77 100644 --- a/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/web/AdminAccessSimulationControllerIntegrationTest.java +++ b/backend/src/test/java/com/bablsoft/accessflow/workflow/internal/web/AdminAccessSimulationControllerIntegrationTest.java @@ -119,7 +119,7 @@ void anAdminGetsTheFullOrderedTrace() { .exchange(); assertThat(result).hasStatus(200); - assertThat(result).bodyJson().extractingPath("$.steps.length()").asNumber().isEqualTo(13); + assertThat(result).bodyJson().extractingPath("$.steps.length()").asNumber().isEqualTo(14); assertThat(result).bodyJson().extractingPath("$.steps[0].step").asString() .isEqualTo("DATASOURCE_GATES"); // #864: the SQL review verdict sits between the permission gate and routing. @@ -130,7 +130,12 @@ void anAdminGetsTheFullOrderedTrace() { // #941: the bytes-scanned cap follows it; no cap configured here. assertThat(result).bodyJson().extractingPath("$.steps[5].step").asString() .isEqualTo("BYTES_SCANNED_CAP"); - assertThat(result).bodyJson().extractingPath("$.steps[12].step").asString() + // #942: the data budget follows the cap; no budget configured here. + assertThat(result).bodyJson().extractingPath("$.steps[6].step").asString() + .isEqualTo("DATA_BUDGET"); + assertThat(result).bodyJson().extractingPath("$.steps[6].outcome").asString() + .isEqualTo("NO_MATCH"); + assertThat(result).bodyJson().extractingPath("$.steps[13].step").asString() .isEqualTo("BREAK_GLASS"); assertThat(result).bodyJson().extractingPath("$.resulting_status").asString() .isEqualTo("PENDING_REVIEW"); @@ -295,11 +300,11 @@ void aDetailWithNoValueIsOmittedRatherThanSentAsNull() { .content(body(analyst.getId(), datasource.getId(), "SELECT id FROM orders")) .exchange(); - assertThat(result).bodyJson().extractingPath("$.steps[8].step").asString() + assertThat(result).bodyJson().extractingPath("$.steps[9].step").asString() .isEqualTo("REVIEW_PLAN"); - assertThat(result).bodyJson().doesNotHavePath("$.steps[8].details.review_plan_id"); - assertThat(result).bodyJson().doesNotHavePath("$.steps[8].details.min_approvals_required"); - assertThat(result).bodyJson().extractingPath("$.steps[8].details.requires_human_approval") + assertThat(result).bodyJson().doesNotHavePath("$.steps[9].details.review_plan_id"); + assertThat(result).bodyJson().doesNotHavePath("$.steps[9].details.min_approvals_required"); + assertThat(result).bodyJson().extractingPath("$.steps[9].details.requires_human_approval") .asBoolean().isFalse(); } diff --git a/docs/03-data-model.md b/docs/03-data-model.md index bb6d52c54..10a91e581 100644 --- a/docs/03-data-model.md +++ b/docs/03-data-model.md @@ -99,7 +99,7 @@ roles these rows are display/catalog data only (runtime resolution answers from Catalog values added after `V114` are seeded for the system roles that hold them by their own one-file migration (`V134`, `V146`, `V148`, `V151`, `V171`, `V174`, `V179` — the last three seed `SQL_REVIEW_MANAGE` (#861), `SERVICE_ACCOUNT_MANAGE` (#868) and `SCHEMA_CHANGE_MANAGE` (#878) -for `ADMIN`); +for `ADMIN`; later additions follow the same pattern, e.g. `V194` seeds `DATA_BUDGET_MANAGE`, #942); `SystemRoleSeedParityIntegrationTest` fails when a value lands without its seed. --- @@ -489,6 +489,71 @@ row. The table preview (`GET /datasources/{id}/sample-rows`) is capped the same --- +## data_budgets + +Per-user **data-volume budgets** (#942, `V193__create_data_budgets.sql`). A budget bounds how many +result rows and/or result bytes **each** targeted user may read from one datasource over a rolling +window. Per-query caps (`max_rows_per_query`, `row_limit_policy`, the byte cap) bound one read; a +budget bounds the sum of many — the slow-exfiltration control. Owned by `core`; see +[05-backend.md → Per-user data-volume budgets](05-backend.md#per-user-data-volume-budgets-942). + +| Column | Type / Notes | +|--------|-------------| +| `id` | UUID PK | +| `organization_id` | FK → `organizations` | +| `datasource_id` | FK → `datasources` `ON DELETE CASCADE` | +| `name` | VARCHAR(120) NOT NULL — shown to the user in the editor, the notifications and the refusal message | +| `max_rows` | BIGINT nullable, `CHECK (max_rows > 0)` — rows the user may read in the window | +| `max_bytes` | BIGINT nullable, `CHECK (max_bytes > 0)` — result bytes the user may read in the window (the proxy's estimated in-memory result size, not wire bytes) | +| `window_minutes` | INTEGER NOT NULL DEFAULT `1440`, `CHECK BETWEEN 60 AND 44640` — the **rolling** window, trailing back from now: 1 hour to 31 days. No calendar day, no timezone, no reset job | +| `breach_action` | ENUM `data_budget_breach_action`: `REJECT` \| `REQUIRE_REVIEW`, NOT NULL DEFAULT `REQUIRE_REVIEW` — what happens to a SELECT once the budget is used up | +| `warn_threshold_percent` | SMALLINT nullable, `CHECK BETWEEN 1 AND 99` — crossing it sends `DATA_BUDGET_THRESHOLD_REACHED` to the user; NULL = no warning | +| `applies_to_roles` | TEXT[] nullable — role names the budget applies to | +| `applies_to_group_ids` | UUID[] nullable — user-group ids | +| `applies_to_user_ids` | UUID[] nullable — individual user ids | +| `enabled` | BOOLEAN NOT NULL DEFAULT true — a disabled budget neither counts nor bounds | +| `version` | BIGINT — optimistic lock | +| `created_at` / `updated_at` | TIMESTAMPTZ | + +`CONSTRAINT chk_data_budgets_has_limit CHECK (max_rows IS NOT NULL OR max_bytes IS NOT NULL)` — a +budget sets at least one limit. Indexed by `(organization_id, datasource_id, enabled)`. + +**Scope** follows `row_limit_policy`: all three `applies_to_*` empty ⇒ every user of the datasource, +admins included; otherwise users whose role / group / id matches. **Each targeted user gets their own +allowance** — a group target is not a shared pool. When several budgets apply to one user, each is +evaluated on its own window and the most constrained wins: the smallest remaining rows and remaining +bytes, exhausted when any one is, and `REJECT` beats `REQUIRE_REVIEW` among exhausted budgets. + +--- + +## data_budget_usage + +Append-only **usage ledger** behind data budgets (#942, V193). One row per delivered SELECT result for +a user on a datasource where at least one budget applies to them; a read with no applying budget writes +nothing. Usage is therefore charged from the moment a budget exists — reads before it are not counted. + +| Column | Type / Notes | +|--------|-------------| +| `id` | UUID PK | +| `organization_id` | UUID NOT NULL | +| `user_id` | UUID NOT NULL — the reader (a query's submitter, a group's submitter, the previewing user) | +| `datasource_id` | UUID NOT NULL | +| `rows_read` | BIGINT NOT NULL, `CHECK >= 0` — rows actually delivered: after row security, with masked rows still counted, after every cap and truncation | +| `bytes_read` | BIGINT NOT NULL, `CHECK >= 0` — the delivered result's estimated in-memory size, measured host-side for JDBC and engine-plugin results alike | +| `source` | ENUM `data_budget_usage_source`: `QUERY` \| `REQUEST_GROUP` \| `SAMPLE_DATA` | +| `query_request_id` | UUID nullable — provenance for `QUERY` rows (no FK) | +| `request_group_id` | UUID nullable — provenance for `REQUEST_GROUP` rows (no FK) | +| `occurred_at` | TIMESTAMPTZ NOT NULL DEFAULT now() | + +Deliberately no foreign keys: the provenance ids are bare UUIDs so an erasure or retention delete of +the source never blocks on the ledger. `idx_data_budget_usage_user_ds_time (user_id, datasource_id, +occurred_at)` backs the trailing-window `SUM`; `idx_data_budget_usage_occurred_at` backs +`DataBudgetUsagePruneJob`, which deletes rows older than `accessflow.core.data-budget.usage-retention` +(default `P32D`, never below 31 days + 1 hour, so a row always outlives the longest window). Writes are +never charged. + +--- + ## export_policy Per-datasource **result-export governance / DLP** policies (#626). Each row governs how a query's @@ -832,6 +897,7 @@ The condition is a polymorphic, `"type"`-discriminated tree (snake_case, no exte | `cicd_origin` (AF-446) | `expected: bool` | whether the request came from a CI/CD pipeline (submitted via an API key or with the `X-AccessFlow-CI` header) equals `expected`. Deterministic — the flag defaults to `false` | | `anomaly_detected` (AF-383) | `expected: bool` | whether the submitter currently has an `OPEN` `behavior_anomaly` on the target datasource equals `expected`. The UBA detector is a periodic batch over **past** data, so this signal escalates the flagged user's **next** query — pair it with `ESCALATE`. Deterministic — false when the user has no open anomaly there | | `estimated_rows` (AF-624) | `operator` (`LT`/`LTE`/`GT`/`GTE`/`EQ`), `value` | the query's pre-flight estimated row impact (the exact affected-row count for UPDATE/DELETE when available, else the EXPLAIN estimate) satisfies the comparison. Read live from `query_estimates` at routing time. **Fails closed**: false when no estimate signal exists (not yet computed, unsupported engine, or failed) | +| `data_budget_used_percent` (#942) | `operator` (`LT`/`LTE`/`GT`/`GTE`/`EQ`), `value` (whole percent, ≥ 0) | the share of the submitter's most-used applying data budget on the datasource, floored to a whole percent (may exceed 100), satisfies the comparison. An advisory escalation trigger — the hard limit is the budget itself. **Fails closed**: false when no budget applies, for a non-SELECT, and on the policy simulator's historical replay | | `estimated_bytes_scanned` (#941) | `operator` (`LT`/`LTE`/`GT`/`GTE`/`EQ`), `value` (raw bytes) | the warehouse's pre-flight bytes-scanned estimate (`query_estimates.estimated_bytes_scanned`) satisfies the comparison. An advisory escalation trigger — the hard cap is `datasources.max_bytes_scanned_per_query`. **Fails closed**: false when no bytes estimate exists (every engine other than BigQuery / Snowflake / Databricks, and a failed or unsupported estimate) | | `scan_type` (AF-624) | `patterns: [string]` | the pre-flight plan's root operation (e.g. `Seq Scan`, `COLLSCAN`) matches any glob (`*` wildcard, case-insensitive). **Fails closed**: false when no plan was captured | @@ -1064,6 +1130,7 @@ The central entity. Represents a single SQL submission through the platform. | `bytes_scanned_cap` | BIGINT nullable (#941, V192) — the bytes-scanned cap that bound the submitter when the query left `PENDING_AI`: the smaller of the datasource cap and the merged grant override. Stamped by `QueryReviewStateMachine` before the decision is applied, whatever the outcome; NULL when no cap applied. Surfaced as `bytes_scanned_cap` on `GET /queries/{id}` | | `bytes_scanned_cap_source` | ENUM `bytes_scanned_cap_source`: `DATASOURCE` \| `GRANT`, nullable (#941) — which setting supplied the cap; on a tie the datasource. NULL exactly when `bytes_scanned_cap` is NULL | | `bytes_scanned_cap_outcome` | ENUM `bytes_scanned_cap_outcome`: `WITHIN` \| `EXCEEDED` \| `NO_ESTIMATE_REVIEW` \| `NO_ESTIMATE_REJECTED`, nullable (#941) — how the persisted estimate compared. `EXCEEDED` and `NO_ESTIMATE_REJECTED` accompany a `PENDING_AI → REJECTED` transition. The re-check just before execution does not update it: a refusal there is recorded as the `FAILED` row's `error_message` | +| `data_budget_review_forced` | BOOLEAN NOT NULL DEFAULT FALSE (#942, V195) — set when an exhausted `REQUIRE_REVIEW` data budget sent the query to `PENDING_REVIEW` as it left `PENDING_AI`. Only a query carrying it (or, for a recurring occurrence, whose series parent carries it) may run past an exhausted budget once approved | | `approved_by_grant_id` | UUID nullable (#582, `V112`) — id of the `access_grant_request` whose pre-approval fast-path auto-approved this query. Bare UUID (no FK, mirroring `granted_permission_id`): the grant's lifecycle (expiry, revocation) is independent of the query's audit trail. Stamped atomically with the `PENDING_AI → APPROVED` transition by `QueryRequestStateService.approveByAccessGrant`; surfaced as `approved_by_grant` on `GET /queries/{id}`. | | `created_at` | TIMESTAMPTZ | | `updated_at` | TIMESTAMPTZ | @@ -1081,6 +1148,10 @@ PENDING_AI → PENDING_REVIEW → APPROVED → EXECUTED bytes_cap_missing_estimate=REJECT; decided before routing) ↘ PENDING_REVIEW (bytes-scanned cap with no estimate under REQUIRE_REVIEW — every auto-approve path is held for a person, like a SQL review BLOCK) + ↘ REJECTED (data budget, #942 — a SELECT whose submitter has used up a budget + with breach_action=REJECT; decided right after the bytes-scanned cap) + ↘ PENDING_REVIEW (data budget used up under REQUIRE_REVIEW — every auto-approve path + is held for a person, like a SQL review BLOCK) PENDING_REVIEW → CANCELLED (by submitter) APPROVED → CANCELLED (submitter, when scheduled_for is set and run hasn't fired yet; for a recurring series — recurrence_rule set, #627 — also any @@ -1089,7 +1160,8 @@ APPROVED → EXECUTED (ScheduledQueryRunJob at scheduled_for ≤ now()) APPROVED → EXECUTED / FAILED (recurring occurrence rows — created directly in APPROVED by RecurringQueryRunJob with submission_reason=RECURRING and executed in the same tick; the series parent stays APPROVED) -APPROVED → FAILED (on execution error) +APPROVED → FAILED (on execution error; also a bytes-scanned cap or an exhausted data + budget re-checked just before execution — #941, #942) ``` **Recurring occurrence rows (#627)** are *inserted* in `APPROVED` (an insert, not a transition — no @@ -1113,7 +1185,7 @@ JSONB snapshot of the **last** SELECT execution for a query request (migration ` | `rows` | JSONB NOT NULL — array of row arrays | | `row_count` | BIGINT NOT NULL | | `truncated` | BOOLEAN NOT NULL DEFAULT FALSE — result was cut short by a cap | -| `truncated_reason` | TEXT NULL (`V116`, #49) — `ROW_LIMIT` \| `BYTE_LIMIT`; NULL when not truncated or persisted pre-V116 | +| `truncated_reason` | TEXT NULL (`V116`, #49) — `ROW_LIMIT` \| `BYTE_LIMIT` \| `DATA_BUDGET` (#942 — the reader's remaining data-budget allowance was the binding cap); NULL when not truncated or persisted pre-V116 | | `duration_ms` | INTEGER NOT NULL | | `recorded_at` | TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP | @@ -1776,10 +1848,12 @@ The hash chain (added in V26) is per organization. Inserts are serialized by a P | `SERVICE_ACCOUNT_DELEGATION_GRANTED` / `SERVICE_ACCOUNT_DELEGATION_REVOKED` | A human (`channel=self_service`, via `/me/service-account-delegations`) or an admin (`channel=admin`) let a service account act on a human's behalf, or revoked that grant (#874). Resource: `service_account`. Metadata: `delegation_id`, `principal_user_id`, `expires_at` (when set), `channel`. | | `SERVICE_ACCOUNT_KEY_ISSUED` / `SERVICE_ACCOUNT_KEY_ROTATED` / `SERVICE_ACCOUNT_KEY_REVOKED` | Admin issues / rotates / revokes an API key on behalf of a service account (#871). Resource: `service_account`. Metadata: `api_key_id`, `name` (issue); `api_key_id`, `superseded_key_id`, `name`, `superseded_expires_at` (rotate — the old key's new expiry, i.e. the end of the grace window); `api_key_id` (revoke). The raw key is never logged. | | `SQL_REVIEW_BLOCKED` | A `BLOCK` SQL review finding suppressed an auto-approve path and forced the request to human review (#864). System-attributed: `actor_id` is NULL. Resource: `query_request` (written by `QueryReviewStateMachine` after the transition) or `request_group` (written by `GroupAiAnalysisListener`). Metadata: `trigger: "sql_review"`, `blocking_rule_ids` (distinct, sorted), `suppressed_paths` — one or more of `ROUTING_AUTO_APPROVE`, `GRANT_FAST_PATH`, `REVIEW_PLAN` for a query, `GROUP_REVIEW_PLAN` for a group — plus `matched_policy_id` when routing was the suppressed path and `blocking_item_ids` for a group. Written **only when the guard changed the outcome**: never for `WARN`, never on a routing `AUTO_REJECT` (the rejection stands), never on the AI-failed path or a plan that already required review (the findings are still on the detail), and never for break-glass, which records findings but is not gated. | -| `QUERY_BYTES_SCANNED_CAP_ENFORCED` | The bytes-scanned cap (#941) changed a query's outcome. System-attributed: `actor_id` is NULL. Resource: `query_request` — or `request_group` when a group member was refused at execution (metadata then carries `item_id`). Metadata: `trigger: "bytes_scanned_cap"`, `stage` (`decision` — the query left `PENDING_AI` rejected, or an automatic approval was held for review; `execution` — refused just before running, the query is then `FAILED`), `limit`, `source` (`DATASOURCE` \| `GRANT`), `outcome`, and `estimated_bytes` when an estimate existed. Never written when the estimate was within the cap; a successful `QUERY_EXECUTED` row under a cap instead carries `bytes_scanned_cap`, `bytes_scanned_cap_source` and `bytes_scanned_estimate` | +| `QUERY_BYTES_SCANNED_CAP_ENFORCED` | The bytes-scanned cap (#941) changed a query's outcome. System-attributed: `actor_id` is NULL. Resource: `query_request` — or `request_group` when a group member was refused at execution (metadata then carries `item_id`), or `datasource` when a table preview was refused (`stage: "sample"`, actor = the caller, metadata adds `table`). Metadata: `trigger: "bytes_scanned_cap"`, `stage` (`decision` — the query left `PENDING_AI` rejected, or an automatic approval was held for review; `execution` — refused just before running, the query is then `FAILED`), `limit`, `source` (`DATASOURCE` \| `GRANT`), `outcome`, and `estimated_bytes` when an estimate existed. Never written when the estimate was within the cap; a successful `QUERY_EXECUTED` row under a cap instead carries `bytes_scanned_cap`, `bytes_scanned_cap_source` and `bytes_scanned_estimate` | +| `QUERY_DATA_BUDGET_ENFORCED` | An exhausted data budget (#942) changed a query's outcome. System-attributed: `actor_id` is NULL. Resource: `query_request` — or `request_group` when a group member was refused at execution (metadata then carries `item_id`). Metadata: `trigger: "data_budget"`, `stage` (`decision` — the query left `PENDING_AI` rejected, or an automatic approval was held for review; `execution` — refused just before running, the query is then `FAILED`), `action` (`REJECT` \| `REQUIRE_REVIEW`), `data_budget_id`, `used_rows`, `used_bytes`, `window_minutes`, and `matched_policy_id` when a routing `AUTO_APPROVE` was suppressed. Never written when allowance remained; a successful `QUERY_EXECUTED` row under a budget instead carries `data_budget_rows_charged` and `data_budget_bytes_charged` | | `MASKING_POLICY_CREATED` / `MASKING_POLICY_UPDATED` / `MASKING_POLICY_DELETED` | Admin creates / updates / deletes a masking policy via the `/datasources/{id}/masking-policies` CRUD endpoints. Resource: `masking_policy`. | | `ROW_SECURITY_POLICY_CREATED` / `ROW_SECURITY_POLICY_UPDATED` / `ROW_SECURITY_POLICY_DELETED` | Admin creates / updates / deletes a row-security policy via the `/datasources/{id}/row-security-policies` CRUD endpoints (AF-380). Resource: `row_security_policy`. Applied row-security policy ids at execute time ride on `QUERY_EXECUTED` metadata (`applied_row_security_policy_ids`), not a separate action. | | `ROW_LIMIT_POLICY_CREATED` / `ROW_LIMIT_POLICY_UPDATED` / `ROW_LIMIT_POLICY_DELETED` | Admin creates / updates / deletes a per-table row-limit policy via the `/datasources/{id}/row-limit-policies` CRUD endpoints (#934). Resource: `row_limit_policy`. Metadata carries `datasource_id`, `schema_name`, `table_name`, `max_rows`, `enabled`. The lowest-cap matching policies of a SELECT ride on `QUERY_EXECUTED` metadata (`applied_row_limit_policy_ids`) when their cap was the binding one. | +| `DATA_BUDGET_CREATED` / `DATA_BUDGET_UPDATED` / `DATA_BUDGET_DELETED` | Admin creates / updates / deletes a data budget via the `/datasources/{id}/data-budgets` CRUD endpoints (#942). Resource: `data_budget`. Metadata carries `datasource_id`, and on create/update `name`, `max_rows` / `max_bytes` (whichever is set), `window_minutes`, `breach_action`, `enabled`. | | `DATA_CLASSIFICATION_TAG_ADDED` / `DATA_CLASSIFICATION_TAG_REMOVED` | Admin tags / untags a datasource table or column via the `/datasources/{id}/classification-tags` endpoints (AF-447). Resource: `data_classification_tag`. Metadata records the table, column, classification, and (on add) whether masking was auto-applied. | | `DISCOVERY_SCAN_COMPLETED` | A sensitive-data discovery scan finished (AF-623) — scheduled (`actor_id` NULL) or on-demand (the triggering admin). Resource: `datasource`. Metadata: tables scanned/skipped/failed, findings created/refreshed/revived/aged/expired, `expiredAuditTruncated`, AI suggestions, duration, `partial` flag, and the error summary when the scan failed. | | `DISCOVERY_FINDING_CONFIRMED` / `DISCOVERY_FINDING_DISMISSED` | Admin confirms (tag applied via the AF-447 service, masking derived) or dismisses (permanently suppressed) a discovery finding via `/datasources/{id}/discovery/findings/bulk-decision`. Resource: `discovery_finding`. Metadata: table, column, classification, detector, confidence, and `tagConflict` when the tag already existed. | @@ -1815,7 +1889,7 @@ Bootstrap reuses the existing `*_CREATED` / `*_UPDATED` actions for `DATASOURCE` ### Audit Resource Types -`resource_type` is the snake_case form of one of the values in `AuditResourceType`: `query_request`, `datasource`, `user`, `api_key`, `permission`, `review_plan`, `review_delegation`, `notification_channel`, `ai_config`, `knowledge_document`, `custom_jdbc_driver`, `system_smtp`, `user_invitation`, `organization`, `oauth2_config`, `saml_config`, `langfuse_config`, `help_agent_config`, `audit_log`, `user_group`, `role`, `datasource_reviewer`, `query_template`, `slack_app_config`, `access_grant_request`, `masking_policy`, `routing_policy`, `sql_review_ruleset`, `service_account`, `row_security_policy`, `row_limit_policy`, `connector`, `query_comment`, `data_classification_tag`, `compliance_report`, `behavior_anomaly`, `break_glass_event`, `dashboard_summary`, `attestation_campaign`, `attestation_item`, `grant_usage_summary`, `api_connector`, `api_request`, `retention_policy`, `deletion_request`, `request_group`, `query_ticket`, `discovery_finding`, `scim_config`, `scim_token`, `export_policy`, `audit_sink`, `deployment_pipeline`, `deployment_request`, `deployment_rollback_review`, `schema_change_promotion`, `schema_drift_scan`, `schema_drift_finding`, `schema_drift_config`. +`resource_type` is the snake_case form of one of the values in `AuditResourceType`: `query_request`, `datasource`, `user`, `api_key`, `permission`, `review_plan`, `review_delegation`, `notification_channel`, `ai_config`, `knowledge_document`, `custom_jdbc_driver`, `system_smtp`, `user_invitation`, `organization`, `oauth2_config`, `saml_config`, `langfuse_config`, `help_agent_config`, `audit_log`, `user_group`, `role`, `datasource_reviewer`, `query_template`, `slack_app_config`, `access_grant_request`, `masking_policy`, `routing_policy`, `sql_review_ruleset`, `service_account`, `row_security_policy`, `row_limit_policy`, `data_budget`, `connector`, `query_comment`, `data_classification_tag`, `compliance_report`, `behavior_anomaly`, `break_glass_event`, `dashboard_summary`, `attestation_campaign`, `attestation_item`, `grant_usage_summary`, `api_connector`, `api_request`, `retention_policy`, `deletion_request`, `request_group`, `query_ticket`, `discovery_finding`, `scim_config`, `scim_token`, `export_policy`, `audit_sink`, `deployment_pipeline`, `deployment_request`, `deployment_rollback_review`, `schema_change_promotion`, `schema_drift_scan`, `schema_drift_finding`, `schema_drift_config`. SCIM-driven mutations (#621) audit as `SCIM_USER_PROVISIONED` / `SCIM_USER_UPDATED` / `SCIM_USER_DEACTIVATED` / `SCIM_GROUP_SYNCED` / `SCIM_GROUP_DELETED` with `actor_id = NULL` (the actor is the IdP's provisioning engine) and `metadata.scim_token_id` / `metadata.scim_token_name` carrying the token identity; admin-side changes audit as `SCIM_CONFIG_UPDATED` / `SCIM_TOKEN_CREATED` / `SCIM_TOKEN_REVOKED` with the caller as actor. @@ -2043,7 +2117,8 @@ notifications module dispatches also writes one row per recipient here so the be inbox can show history, unread counts, and act on individual entries. The `event_type` mirrors the backend `NotificationEventType` enum — query, review, access, anomaly, API-request (`API_REQUEST_*`, AF-500), deployment (`DEPLOYMENT_*`, #695) and -schema-change (`SCHEMA_CHANGE_PROMOTION_*`, `SCHEMA_DRIFT_DETECTED`, #882) events; `TEST` +schema-change (`SCHEMA_CHANGE_PROMOTION_*`, `SCHEMA_DRIFT_DETECTED`, #882) and data-budget +(`DATA_BUDGET_THRESHOLD_REACHED`, `DATA_BUDGET_EXHAUSTED`, #942) events; `TEST` events are skipped. | Column | Type / Notes | @@ -2056,7 +2131,7 @@ events are skipped. | `api_request_id` | FK → `api_requests` ON DELETE CASCADE, nullable (V109, AF-529) | | `deployment_request_id` | FK → `deployment_requests` ON DELETE CASCADE, nullable (V155, #695) | | `schema_change_promotion_id` | FK → `schema_change_set_promotions` ON DELETE CASCADE, nullable (V182, #882). Set on the `SCHEMA_CHANGE_PROMOTION_*` rows; a `SCHEMA_DRIFT_DETECTED` row names no target | -| `payload` | JSONB — denormalised render context (datasource/pipeline name, submitter, risk_level, reviewer comment; deployment rows add `deployment_id`, `environment`, `version`, `outcome`; schema-change rows add `schema_change_promotion_id`, `change_set`, `environment`, `promotion_status`, and `new_finding_count` for drift) | +| `payload` | JSONB — denormalised render context (datasource/pipeline name, submitter, risk_level, reviewer comment; deployment rows add `deployment_id`, `environment`, `version`, `outcome`; schema-change rows add `schema_change_promotion_id`, `change_set`, `environment`, `promotion_status`, and `new_finding_count` for drift; data-budget rows add `datasource_id`, `budget`, `used_percent` and name no target column) | | `is_read` | BOOLEAN DEFAULT false | | `created_at` | TIMESTAMPTZ DEFAULT now() | | `read_at` | TIMESTAMPTZ, nullable | diff --git a/docs/04-api-spec.md b/docs/04-api-spec.md index 312aea6a8..2accca7d6 100644 --- a/docs/04-api-spec.md +++ b/docs/04-api-spec.md @@ -625,12 +625,13 @@ GET /api/v1/datasources/{id}/sample-rows?schema=public&table=users&limit=50 } ``` -`restricted: true` flags a column the backend masked (via a masking policy or `restricted_columns`); its cell values are the masked output only. `truncated` is `true` when the sample hit the row cap or the result byte cap (#49); `truncated_reason` says which (`"ROW_LIMIT"` | `"BYTE_LIMIT"`, `null` when not truncated). +`restricted: true` flags a column the backend masked (via a masking policy or `restricted_columns`); its cell values are the masked output only. `truncated` is `true` when the sample hit the row cap or the result byte cap (#49); `truncated_reason` says which (`"ROW_LIMIT"` | `"BYTE_LIMIT"` | `"DATA_BUDGET"` — the caller's remaining data-budget allowance, #942 — `null` when not truncated). A preview spends the caller's data budget like a query: it is capped to the remaining rows and bytes and recorded as usage. ADMINs may sample any datasource in their organization; non-ADMINs need a permission row with `can_read` and the target within their `allowed_schemas` / `allowed_tables` and outside their `denied_schemas` / `denied_tables` (#939). **Response 400:** `limit` is out of the `1`–`200` range. `error: VALIDATION_ERROR`. **Response 404:** Datasource not accessible, or the table is absent from the introspected schema / outside the caller's allow-list / on their table or schema deny-list. `error: DATASOURCE_NOT_FOUND` or `TABLE_NOT_FOUND`. +**Response 409:** A data budget that applies to the caller on this datasource is used up (#942), whatever its breach action — a preview has no review to fall back on. `error: DATA_BUDGET_EXHAUSTED`, with `budgetId`, `maxRows`, `maxBytes` (a limit the budget does not set is `null`) and `windowMinutes`; `detail` names the budget. **Response 422:** Sampling failed (e.g. customer database unreachable). `error: DATASOURCE_CONNECTION_TEST_FAILED`. ### GET /datasources/{id}/permissions — Response 200 @@ -1046,6 +1047,164 @@ Create, update and delete write `ROW_LIMIT_POLICY_CREATED` / `_UPDATED` / `_DELE --- +### Data budgets (#942) + +Per-user **data-volume budgets** on a datasource: how many result rows and/or result bytes **each** +targeted user may read from it over a rolling window. CRUD requires the `DATA_BUDGET_MANAGE` +permission (ADMIN by default); reading your own standing requires only authentication. A budget +bounds the *sum* of a user's reads; the per-query caps (`max_rows_per_query`, row-limit policies, the +byte cap) still bound each one. See +[05-backend.md → Per-user data-volume budgets](05-backend.md#per-user-data-volume-budgets-942) for +the enforcement semantics. + +- **What counts.** Every delivered SELECT result: interactive, scheduled, recurring, break-glass, + request-group members and table previews (`GET /datasources/{id}/sample-rows`). Rows are what the + user actually received — after row security and every cap, masked rows included. Bytes are the + proxy's estimated in-memory result size, measured the same way for every engine — an estimate, not + wire bytes. Writes never count. Usage is recorded only while a budget applies to the user, so it is + charged from the moment a budget exists. +- **Window.** `window_minutes` is a rolling window trailing back from now (60–44,640 minutes, 1 hour to + 31 days); usage ages out continuously — there is no reset time. +- **Scope.** The `applies_to_*` model of row-limit policies: all three empty ⇒ every user of the + datasource. Each user gets their own allowance; a group target is not a shared pool. Several + applying budgets combine to the most constrained (smallest remaining rows / bytes; `REJECT` beats + `REQUIRE_REVIEW` once exhausted). +- **Enforcement.** While allowance remains, a SELECT's result is capped to it (`truncated_reason: + "DATA_BUDGET"` when the budget was the binding cap). Once a budget is used up, a SELECT is rejected + (`breach_action: REJECT` — status `REJECTED` at submission, `FAILED` at execution) or held for a + person (`REQUIRE_REVIEW` — every automatic approval is suppressed, and the run executes once a human + approved it — only an approval given for a review the exhausted budget forced lifts it). Break-glass is counted but never capped or blocked by a budget. A table preview has no review to + fall back on: an exhausted budget answers **409** `DATA_BUDGET_EXHAUSTED` whatever the action. + +#### POST /datasources/{datasourceId}/data-budgets — Request Body + +```json +{ + "name": "Analyst daily read budget", + "max_rows": 500000, + "max_bytes": 2000000000, + "window_minutes": 1440, + "breach_action": "REQUIRE_REVIEW", + "warn_threshold_percent": 80, + "applies_to_roles": ["ANALYST"], + "applies_to_group_ids": [], + "applies_to_user_ids": [], + "enabled": true +} +``` + +`name` is required (non-blank, ≤ 120 chars). `max_rows` and `max_bytes` are optional (`≥ 1`) but at +least one must be set. `window_minutes` defaults to `1440` (60–44,640). `breach_action` is `REJECT` or +`REQUIRE_REVIEW` (default). `warn_threshold_percent` is optional (1–99); crossing it sends the user a +`DATA_BUDGET_THRESHOLD_REACHED` notification, and reaching the limit sends `DATA_BUDGET_EXHAUSTED` to +the user and every `DATA_BUDGET_MANAGE` holder (see [08-notifications.md](08-notifications.md)). +`enabled` defaults to `true`. Applies-to targets must belong to the caller's organization. + +**Response 201:** Data budget object. `Location` header points to +`/api/v1/datasources/{datasourceId}/data-budgets/{budgetId}`. +**Response 400:** Bean Validation failure (blank or over-long `name`, a limit below 1, a window or +threshold out of range). +**Response 404:** Datasource does not exist in the caller's organization. `error: DATASOURCE_NOT_FOUND`. +**Response 422:** No limit set, a value out of range, an unknown applies-to role, or an applies-to +user/group outside the organization. `error: ILLEGAL_DATA_BUDGET`. + +#### GET /datasources/{datasourceId}/data-budgets — Response 200 + +```json +{ + "content": [ + { + "id": "uuid", + "datasource_id": "uuid", + "name": "Analyst daily read budget", + "max_rows": 500000, + "max_bytes": 2000000000, + "window_minutes": 1440, + "breach_action": "REQUIRE_REVIEW", + "warn_threshold_percent": 80, + "applies_to_roles": ["ANALYST"], + "applies_to_group_ids": [], + "applies_to_user_ids": [], + "enabled": true, + "created_at": "2026-09-25T10:00:00Z", + "updated_at": "2026-09-25T10:00:00Z" + } + ] +} +``` + +A limit the budget does not set, and an unset `warn_threshold_percent`, are omitted. + +#### PUT /datasources/{datasourceId}/data-budgets/{budgetId} + +Same body and validation as `POST`. The update is a full replacement: an omitted limit, threshold or +applies-to list is cleared. **Response 200:** the updated budget object. **Response 404:** +`DATASOURCE_NOT_FOUND` or `DATA_BUDGET_NOT_FOUND`. **Response 422:** `ILLEGAL_DATA_BUDGET`. + +#### DELETE /datasources/{datasourceId}/data-budgets/{budgetId} + +**Response 204:** No content. **Response 404:** `DATASOURCE_NOT_FOUND` or `DATA_BUDGET_NOT_FOUND`. +Recorded usage is kept until it ages out of the ledger. + +Create, update and delete write `DATA_BUDGET_CREATED` / `_UPDATED` / `_DELETED` audit rows +(resource `data_budget`). + +#### GET /datasources/{datasourceId}/data-budgets/me — Response 200 + +The caller's own standing on the datasource — what the query editor shows. Any authenticated user; +the datasource is resolved first (`QUERY_ADMIN` / `DATASOURCE_MANAGE` holders see any datasource in the +organization, everyone else needs a permission on it), so an invisible datasource is **404**, never an +empty 200. + +```json +{ + "datasource_id": "uuid", + "datasource_name": "Production DB", + "exhausted": false, + "remaining_rows": 120000, + "remaining_bytes": 850000000, + "used_percent": 76, + "budgets": [ + { + "id": "uuid", + "name": "Analyst daily read budget", + "max_rows": 500000, + "max_bytes": 2000000000, + "window_minutes": 1440, + "breach_action": "REQUIRE_REVIEW", + "warn_threshold_percent": 80, + "used_rows": 380000, + "used_bytes": 1150000000, + "remaining_rows": 120000, + "remaining_bytes": 850000000, + "used_percent": 76, + "exhausted": false + } + ] +} +``` + +The top-level fields are the effective (most constrained) standing across `budgets`: `used_percent` +is the highest share any budget has used (floored, may exceed 100), `remaining_rows` / +`remaining_bytes` the smallest remaining allowance per metric (omitted when no applying budget sets +that limit), and `breach_action` — present only while `exhausted` — the strictest action among +exhausted budgets. When no budget applies, `budgets` is `[]`, `exhausted` is `false` and the other +effective fields are omitted. **Response 404:** `DATASOURCE_NOT_FOUND`. + +#### GET /admin/users/{userId}/data-budget-usage — Response 200 + +A user's standing on every datasource where a budget applies to them — the users page's *Data usage* +drawer. Requires `DATA_BUDGET_MANAGE` or `USER_MANAGE`. + +```json +{ "content": [ { "datasource_id": "uuid", "datasource_name": "Production DB", "exhausted": false, "budgets": [ … ] } ] } +``` + +Each entry has the shape of `GET /datasources/{datasourceId}/data-budgets/me`. **Response 404:** the +user does not exist in the caller's organization. `error: USER_NOT_FOUND`. + +--- + ### Result-export policies (#626) Admin-only, organization-scoped result-export governance (DLP) per datasource. A policy governs how a @@ -2060,7 +2219,7 @@ The replay is **distinctly audited**: a `QUERY_SUBMITTED` audit row is written o } ``` -`truncated_reason` is `"ROW_LIMIT"` when the stored result hit the row cap, `"BYTE_LIMIT"` when it hit a per-result byte cap — `ACCESSFLOW_PROXY_EXECUTION_MAX_RESULT_BYTES` on the relational JDBC path (#49), or an engine's own equivalent, currently only `ACCESSFLOW_PROXY_ENGINES_DATABRICKS_MAX_RESULT_BYTES` (#633), and `null` when the result was not truncated (or was persisted before the field existed). +`truncated_reason` is `"ROW_LIMIT"` when the stored result hit the row cap, `"BYTE_LIMIT"` when it hit a per-result byte cap — `ACCESSFLOW_PROXY_EXECUTION_MAX_RESULT_BYTES` on the relational JDBC path (#49), or an engine's own equivalent, currently only `ACCESSFLOW_PROXY_ENGINES_DATABRICKS_MAX_RESULT_BYTES` (#633), `"DATA_BUDGET"` when the submitter's remaining data-budget allowance (#942) was the binding cap on rows or bytes, and `null` when the result was not truncated (or was persisted before the field existed). `columns[].restricted` is `true` when the column matched a `restricted_columns` entry on the caller's `(user_id, datasource_id)` permission row. The matcher (in priority order: `schema.table.column` → `table.column` → bare `column`) flags the column at proxy-result-set time, and the value in `rows` is replaced with `"***"` before persistence — the raw sensitive value is never written to `query_request_results.rows`. Frontends should render restricted columns with a visual marker (lock icon, muted styling) so the user understands the value was redacted. @@ -4035,7 +4194,7 @@ All endpoints require `role=ADMIN` and operate within the caller's organization. } ``` -`name`, `condition`, and `action` are **required**. `datasource_id` is optional (null = org-wide). `priority` must be unique within the organization. `required_approvals` is required (and only meaningful) for `action: REQUIRE_APPROVALS` (absolute minimum approvers) and `action: ESCALATE` (delta added to the review-plan minimum, default 1); it must be null for `AUTO_APPROVE` / `AUTO_REJECT`. The `condition` is the typed `"type"`-discriminated tree documented in the data model — including the AF-446 client-context operands `source_ip` (CIDR allow-list; deny via `not`), `user_agent`, `time_since_last_approval`, and `cicd_origin`, which **fail closed** when their signal is absent. A malformed CIDR in a `source_ip` leaf is rejected with **422** `ROUTING_POLICY_INVALID`. The `query_shape` operand (#940) — `{"type": "query_shape", "any_of": ["JOIN", "SUBQUERY"]}` — matches a query that has any listed shape (`JOIN`, `UNION`, `SUBQUERY`, `CTE`, `GROUP_BY`, `HAVING`, `AGGREGATE`, `WINDOW_FUNCTION`) anywhere in the statement; an empty `any_of` is rejected with **422** `ROUTING_POLICY_INVALID`, and the leaf fails closed when the SQL cannot be parsed for its shape. Routing re-parses the stored SQL text with JSqlParser whatever the engine, so on a plugin datasource the leaf matches only when that text happens to be standard SQL JSqlParser can read (often true for a warehouse, never for a MongoDB or Redis command). Pair it with `ESCALATE` or `REQUIRE_APPROVALS` to send, say, every joined query to a second reviewer, or with `AUTO_REJECT` to refuse it outright; a grant's `denied_shapes` refuses at submission instead. The `estimated_bytes_scanned` operand (#941) — `{"type": "estimated_bytes_scanned", "operator": "GT", "value": 1000000000000}` — compares the warehouse's pre-flight bytes-scanned estimate (raw bytes, `value ≥ 0`) and, like `estimated_rows`, **fails closed**: it is `false` whenever no bytes estimate exists, which is every engine except BigQuery, Snowflake and Databricks. It is the advisory counterpart of the datasource and grant bytes-scanned cap. +`name`, `condition`, and `action` are **required**. `datasource_id` is optional (null = org-wide). `priority` must be unique within the organization. `required_approvals` is required (and only meaningful) for `action: REQUIRE_APPROVALS` (absolute minimum approvers) and `action: ESCALATE` (delta added to the review-plan minimum, default 1); it must be null for `AUTO_APPROVE` / `AUTO_REJECT`. The `condition` is the typed `"type"`-discriminated tree documented in the data model — including the AF-446 client-context operands `source_ip` (CIDR allow-list; deny via `not`), `user_agent`, `time_since_last_approval`, and `cicd_origin`, which **fail closed** when their signal is absent. A malformed CIDR in a `source_ip` leaf is rejected with **422** `ROUTING_POLICY_INVALID`. The `query_shape` operand (#940) — `{"type": "query_shape", "any_of": ["JOIN", "SUBQUERY"]}` — matches a query that has any listed shape (`JOIN`, `UNION`, `SUBQUERY`, `CTE`, `GROUP_BY`, `HAVING`, `AGGREGATE`, `WINDOW_FUNCTION`) anywhere in the statement; an empty `any_of` is rejected with **422** `ROUTING_POLICY_INVALID`, and the leaf fails closed when the SQL cannot be parsed for its shape. Routing re-parses the stored SQL text with JSqlParser whatever the engine, so on a plugin datasource the leaf matches only when that text happens to be standard SQL JSqlParser can read (often true for a warehouse, never for a MongoDB or Redis command). Pair it with `ESCALATE` or `REQUIRE_APPROVALS` to send, say, every joined query to a second reviewer, or with `AUTO_REJECT` to refuse it outright; a grant's `denied_shapes` refuses at submission instead. The `estimated_bytes_scanned` operand (#941) — `{"type": "estimated_bytes_scanned", "operator": "GT", "value": 1000000000000}` — compares the warehouse's pre-flight bytes-scanned estimate (raw bytes, `value ≥ 0`) and, like `estimated_rows`, **fails closed**: it is `false` whenever no bytes estimate exists, which is every engine except BigQuery, Snowflake and Databricks. It is the advisory counterpart of the datasource and grant bytes-scanned cap. The `data_budget_used_percent` operand (#942) — `{"type": "data_budget_used_percent", "operator": "GTE", "value": 80}` — compares the share of the submitter's most-used applying data budget on the datasource, as a whole percent (`value ≥ 0`; usage may exceed 100). It **fails closed**: `false` when no budget applies, for a non-SELECT, and in the policy simulator's historical replay (past usage is not reconstructed). Pair it with `ESCALATE` to send a heavy reader's queries to a second reviewer before the budget itself stops them. **Response 201:** Full routing-policy object (see the list shape below). `Location` header points to `/api/v1/admin/routing-policies/{id}`. **Response 400:** Bean Validation failure on the request body. `error: VALIDATION_ERROR`. @@ -4642,7 +4801,8 @@ follow up with one simulation per user of interest. It is deliberately not an N- "scan_type": null, "query_shapes": [], "shapes_analyzed": true, - "estimated_bytes_scanned": null + "estimated_bytes_scanned": null, + "data_budget_used_percent": null }, "caveats": ["CLIENT_CONTEXT_ABSENT", "COST_ESTIMATE_ABSENT"] } @@ -4658,7 +4818,7 @@ it. Every later step is still present with `outcome: "SKIP"`. all. Those are exactly the traces worth reading closely, so treat its absence as information — it means no routing condition was evaluated, not that the signals were empty. -**`steps` is always all twelve, in this fixed order**, so a client can render a stable checklist: a step +**`steps` is always all fourteen, in this fixed order**, so a client can render a stable checklist: a step that did not apply is reported with `outcome: "SKIP"` rather than omitted. `outcome` is one of `ALLOW`, `DENY`, `MATCH`, `NO_MATCH`, `SKIP`. `reason` is localized to the request's `Accept-Language`. @@ -4673,11 +4833,12 @@ that did not apply is reported with `outcome: "SKIP"` rather than omitted. `outc | `EFFECTIVE_PERMISSION` | `query_admin_short_circuit`, `contributing_grants[]`, `rejected_tables`, `denied_tables` (#939 — the referenced tables a `denied_schemas` / `denied_tables` entry reaches; checked after the allow-list, a non-empty list denies with `workflow.access_simulation.permission.table_denied`), `rejected_columns` (#935 — the denied entries the query reaches; a non-empty list denies with `workflow.access_simulation.permission.column_denied`), `denied_shapes` (#940 — the grant's denied shapes the query has, in declaration order; checked last, a non-empty list denies with `workflow.access_simulation.permission.shape_denied`), `expires_at` | always (`expires_at` omitted when the permission is standing or the caller is a `QUERY_ADMIN` holder) | | `SQL_REVIEW` | `blocking_rule_ids[]`, `blocking_count` | `MATCH` only — a deterministic SQL review rule fired at `BLOCK` (#864); `{}` on `NO_MATCH` | | `BYTES_SCANNED_CAP` | `bytes_scanned_cap`, `bytes_scanned_cap_source`, `estimated_bytes_scanned`, `bytes_scanned_cap_outcome` | whenever a cap applies (#941); `{}` on `NO_MATCH` (no cap). In a simulation the step is always `SKIP` with no estimate — the cap is named, never compared. On a live trace: `ALLOW` within the cap, `DENY` over it or with no estimate under `REJECT` (the trace then ends `REJECTED` and every later decision stage is `SKIP`), `MATCH` with no estimate under `REQUIRE_REVIEW` — every auto-approve stage below then reports `bytes_cap_suppressed: true` | +| `DATA_BUDGET` | `data_budget_used_percent`, `data_budget_remaining_rows`, `data_budget_remaining_bytes`, `data_budget_action`, `data_budget_id`, `data_budget_name` | whenever a budget applies to the user on a SELECT (#942); `{}` on `NO_MATCH` (no budget, or not a SELECT). Read live in both a simulation and a real decision — usage is a persisted fact about the user. `ALLOW` while allowance remains; `DENY` when an exhausted budget has `breach_action: REJECT` (the trace then ends `REJECTED` and every later decision stage is `SKIP`); `MATCH` when it has `REQUIRE_REVIEW` — every auto-approve stage below then reports `data_budget_suppressed: true`. `data_budget_id` / `_name` name the exhausted budget that decided | | `ROUTING_POLICIES` | `policies[]` | always (`[]` when the org has none) | -| | `matched_policy_id`, `matched_policy_name`, `action`, `effective_min_approvals`, `sql_review_suppressed`, `bytes_cap_suppressed` | `MATCH` only | +| | `matched_policy_id`, `matched_policy_name`, `action`, `effective_min_approvals`, `sql_review_suppressed`, `bytes_cap_suppressed`, `data_budget_suppressed` | `MATCH` only | | `GRANT_FAST_PATH` | `considered_grant_ids` | whenever grants were looked up | | | `grant_id`, `approver_email` | `MATCH`, and `NO_MATCH` when a covering grant was suppressed by a `BLOCK` finding | -| `REVIEW_PLAN` | `requires_human_approval`, `auto_approve_reads`, `sql_review_suppressed`, `bytes_cap_suppressed` | always (both `*_suppressed` keys absent on `SKIP`) | +| `REVIEW_PLAN` | `requires_human_approval`, `auto_approve_reads`, `sql_review_suppressed`, `bytes_cap_suppressed`, `data_budget_suppressed` | always (the `*_suppressed` keys absent on `SKIP`) | | | `review_plan_id`, `min_approvals_required` | only when the datasource has a review plan | | `ELIGIBLE_REVIEWERS` | `submitter_excluded` | always | | | `reviewers[]` (`user_id`, `email`, `display_name`) | when the datasource has its own reviewer assignment | @@ -7871,15 +8032,19 @@ deployment events (`DEPLOYMENT_SUBMITTED` \| `DEPLOYMENT_APPROVED` \| `DEPLOYMENT_REJECTED` \| `DEPLOYMENT_OUTCOME_FAILED` \| `DEPLOYMENT_BREAK_GLASS_EXECUTED` — #695), and the schema-change events (`SCHEMA_CHANGE_PROMOTION_SUBMITTED` \| `SCHEMA_CHANGE_PROMOTION_APPLIED` \| -`SCHEMA_CHANGE_PROMOTION_FAILED` \| `SCHEMA_DRIFT_DETECTED` — #882). At most one of +`SCHEMA_CHANGE_PROMOTION_FAILED` \| `SCHEMA_DRIFT_DETECTED` — #882), and the data-budget +events (`DATA_BUDGET_THRESHOLD_REACHED` \| `DATA_BUDGET_EXHAUSTED` — #942). At most one of `query_request_id` \| `api_request_id` \| `deployment_request_id` \| -`schema_change_promotion_id` is set, naming the row's target (a drift row names none). The +`schema_change_promotion_id` is set, naming the row's target (a drift or data-budget row names +none). The `payload` keys are best-effort context for the client to render a human-readable message and link — UIs must treat individual keys as optional; deployment rows add `deployment_id`, `environment`, `version`, and `outcome`, with the pipeline name riding the `datasource` key; schema-change rows add `schema_change_promotion_id`, `change_set`, `environment` and `promotion_status` (`FAILED` or `PARTIALLY_APPLIED` on a failure), and a -drift row adds `new_finding_count` — again with the pipeline name in `datasource`. +drift row adds `new_finding_count` — again with the pipeline name in `datasource`. Data-budget +rows add `datasource_id`, `budget` and `used_percent`, with the budget's user in `submitter` / +`submitter_name`. ### GET /notifications/unread-count — Response 200 @@ -9207,6 +9372,9 @@ The following codes are returned in addition to the per-endpoint codes documente | `ON_BEHALF_OF_NOT_PERMITTED` | 403 | *(written by `ApiKeyRequestFilter`, no exception type)* | `X-AccessFlow-On-Behalf-Of` on a JWT session (`reason=not_api_key`), or naming a human the calling service account may not act for (`reason=not_permitted` — one opaque code for unknown / other organization / inactive / not human / no live grant) (#874). | | `ON_BEHALF_OF_REVIEW_FORBIDDEN` | 403 | *(written by `ApiKeyRequestFilter`, no exception type)* | `X-AccessFlow-On-Behalf-Of` sent to a review / decision endpoint — an agent may submit *for* a human, never vote *as* one (#874). | | `REQUEST_GROUP_ON_BEHALF_OF_CONFLICT` | 409 | `IllegalRequestGroupStateException.OnBehalfOfConflict` | A request-group draft already naming a different on-behalf-of principal was submitted with another (#874). | +| `DATA_BUDGET_NOT_FOUND` | 404 | `DataBudgetNotFoundException` | Unknown data-budget id, or the budget belongs to another datasource / organization (#942). | +| `ILLEGAL_DATA_BUDGET` | 422 | `IllegalDataBudgetException` | A data budget with no limit, a value out of range, an unknown applies-to role, or an applies-to user / group outside the organization (#942). | +| `DATA_BUDGET_EXHAUSTED` | 409 | `DataBudgetExhaustedException` | A table preview (`GET /datasources/{id}/sample-rows`) by a user whose data budget on the datasource is used up (#942). Body includes `budgetId`, `maxRows`, `maxBytes`, `windowMinutes`. Query execution records `FAILED` with the same message instead of returning it. | | `ROUTING_POLICY_NOT_FOUND` | 404 | `RoutingPolicyNotFoundException` | Unknown routing-policy id, or the policy is in another organization. | | `ROUTING_POLICY_PRIORITY_CONFLICT` | 409 | `RoutingPolicyPriorityConflictException` | Another routing policy in the organization already uses that priority. | | `ROUTING_POLICY_INVALID` | 422 | `RoutingPolicyInvalidException` | Malformed condition tree, action/`required_approvals` mismatch, or a reorder set that doesn't match the org's policies. | diff --git a/docs/05-backend.md b/docs/05-backend.md index e91f6ad69..44268bf0f 100644 --- a/docs/05-backend.md +++ b/docs/05-backend.md @@ -320,7 +320,7 @@ Inside a `BEGIN…COMMIT` envelope, `BatchInsertPlanner` (`proxy/internal/`) gro Implemented in `proxy/internal/`: - `QueryExecutor` (public API in `proxy/api/`) — single method `QueryExecutionResult execute(QueryExecutionRequest)`. Pure execution primitive: input is `(datasourceId, sql, queryType, maxRowsOverride?, statementTimeoutOverride?)`; output is a sealed `QueryExecutionResult` (`SelectExecutionResult` | `UpdateExecutionResult`). Status transitions and `query_requests` writes live in the workflow orchestrator that consumes this service. -- `DefaultQueryExecutor` — `@Service`. Resolves the datasource descriptor, computes `effectiveMaxRows = min(override ?? datasource.maxRowsPerQuery, datasource.maxRowsPerQuery, accessflow.proxy.execution.max-rows)` (an override only ever lowers the cap — #933) and `effectiveTimeout = override ?? accessflow.proxy.execution.statement-timeout`, then branches on the request's `transactional` flag (below). The row override comes from the submitter's effective `row_limit_override`: `DefaultQueryLifecycleService.doExecute` (direct, scheduled, recurring and break-glass runs) and `GroupExecutionService` (grouped members) read it from the same `DatasourceUserPermissionLookupService.findFor` call that supplies `restrictedColumns`, where the merge takes the smallest non-null override across the direct and group grants. Both callers then lower it further by the per-table row-limit policies (#934): `core.api.RowLimitPolicyResolutionService.resolve(org, datasource, submitter, parsed.referencedTables())` returns the lowest `max_rows` among the enabled policies that apply to the submitter and name a referenced table (lenient, case-insensitive suffix matching — an unqualified reference or a schema-less policy matches the table in any schema, and a database-prefixed `db.schema.table` still matches a `schema.table` policy; an empty `referencedTables` matches nothing), folded in as `min(grantOverride, policyCap)`. The executor itself is unchanged: `clampMaxRows` still clamps to the datasource and global caps, so a policy can only lower the limit. `doExecute` records the lowest-cap policy ids as `applied_row_limit_policy_ids` on the success audit row, for SELECTs only and only when that cap is at or below both the grant override and the descriptor's `maxRowsPerQuery`; `GroupExecutionService` enforces but does not audit per member. +- `DefaultQueryExecutor` — `@Service`. Resolves the datasource descriptor, computes `effectiveMaxRows = min(override ?? datasource.maxRowsPerQuery, datasource.maxRowsPerQuery, accessflow.proxy.execution.max-rows)` (an override only ever lowers the cap — #933) and `effectiveTimeout = override ?? accessflow.proxy.execution.statement-timeout`, then branches on the request's `transactional` flag (below). The row override comes from the submitter's effective `row_limit_override`: `DefaultQueryLifecycleService.doExecute` (direct, scheduled, recurring and break-glass runs) and `GroupExecutionService` (grouped members) read it from the same `DatasourceUserPermissionLookupService.findFor` call that supplies `restrictedColumns`, where the merge takes the smallest non-null override across the direct and group grants. Both callers then lower it further by the per-table row-limit policies (#934): `core.api.RowLimitPolicyResolutionService.resolve(org, datasource, submitter, parsed.referencedTables())` returns the lowest `max_rows` among the enabled policies that apply to the submitter and name a referenced table (lenient, case-insensitive suffix matching — an unqualified reference or a schema-less policy matches the table in any schema, and a database-prefixed `db.schema.table` still matches a `schema.table` policy; an empty `referencedTables` matches nothing), folded in as `min(grantOverride, policyCap)`. The executor itself is unchanged: `clampMaxRows` still clamps to the datasource and global caps, so a policy can only lower the limit. `doExecute` records the lowest-cap policy ids as `applied_row_limit_policy_ids` on the success audit row, for SELECTs only and only when that cap is at or below both the grant override and the descriptor's `maxRowsPerQuery`; `GroupExecutionService` enforces but does not audit per member. Both callers lower the row cap once more to the submitter's remaining **data-budget** allowance (#942) and pass the remaining bytes as `QueryExecutionRequest.maxResultBytesOverride`; after every SELECT the executor's `measureBytes` stamps `SelectExecutionResult.resultBytes` (the `ResultByteEstimator` sum over the delivered rows — JDBC, engine-plugin and cache-hit results alike) and, under a byte override, trims the rows past it with `truncated_reason=DATA_BUDGET` (the first row is always kept). See [Per-user data-volume budgets](#per-user-data-volume-budgets-942). - Non-transactional (default): ``` Connection.setReadOnly(queryType == SELECT) @@ -1153,12 +1153,14 @@ The result is returned via `DatabaseSchemaView` (immutable nested records: `Sche 1. **Authorization + allow-list.** `DefaultSampleDataService` calls `DatasourceAdminService.introspectSchema(...)` (which enforces org + permission-row access) and validates the requested `schema`/`table` against the returned `DatabaseSchemaView`. Non-ADMINs additionally need `can_read` and the target inside their `allowed_schemas`/`allowed_tables`, matched by `core.api.AllowedTables` (`normalize` + `coveringEntry`) — the query gate's matcher (`DatasourcePermissionChecker.rejectedTables`), so a `schema.table` or `allowed_schemas` grant covers the preview exactly as it covers a `SELECT` (#1089). A **bare** `allowed_tables` entry is the one place the two differ: the gate applies it to an *unqualified* reference, which the database resolves, while the preview reads a concrete `schema.table`. The preview admits it only when no other schema in the database has a table of that name — counted over the unfiltered catalog (`introspectSchemaForSystem`, fetched only for this fallback), since the caller's #936-filtered view may already hide the twin — and otherwise fails closed, matching `core.internal.SchemaViewPermissionFilter`. So `allowed_tables=[orders]` with both `public.orders` and `archive.orders` previews neither until the admin qualifies the entry, and a target whose schema reports no name is covered by a bare entry only. The fallback is deliberately looser than the gate in one case: with `orders` only in `archive` and `archive` off the search path, the preview reads `archive.orders` while an unqualified `SELECT * FROM orders` fails to resolve — the same table the schema tree already shows. The view's catalog-suffix rule (`mydb.dbo.orders` showing `dbo.orders`) is not honoured here, so such a table is listed but its preview returns 404. A target on the caller's `denied_schemas` / `denied_tables` (#939, `DeniedTables.deniesTable`) is refused before the allow-list is consulted. A miss raises `TableNotFoundException` (HTTP 404) — existence is never leaked. 2. **Directive resolution.** Restricted columns (from the permission), `ColumnMaskDirective`s (`MaskingPolicyResolutionService`), and `RowSecurityDirective`s (`RowSecurityResolutionService`) are resolved for the caller. -3. **Execution.** `QueryExecutor.sampleTable(SampleTableRequest)` enforces the row cap (`maxRowsOverride` — the requested limit, lowered to the caller's effective `row_limit_override` when one applies (#933) and to any row-limit policy on the sampled table (#934) — clamped to the datasource + global `ACCESSFLOW_PROXY_EXECUTION_MAX_ROWS`) and statement timeout, then: +3. **Execution.** `QueryExecutor.sampleTable(SampleTableRequest)` enforces the row cap (`maxRowsOverride` — the requested limit, lowered to the caller's effective `row_limit_override` when one applies (#933), to any row-limit policy on the sampled table (#934) and to the caller's remaining data-budget rows (#942) — clamped to the datasource + global `ACCESSFLOW_PROXY_EXECUTION_MAX_ROWS`) and statement timeout, then: - **Relational** datasources: builds `SELECT * FROM ` (via `IdentifierQuoter`, never raw input) and runs the existing JDBC path — `RowSecurityRewriter` injects RLS, `JdbcResultRowMapper` + `ColumnMasker` mask post-fetch, JDBC `setMaxRows` caps without a dialect-specific `LIMIT`. - **Engine-managed** (NoSQL) datasources: delegates to the engine's `QueryEngine.sampleTable(QueryEngineSampleRequest)` (see [Engine SDK](15-engine-sdk.md)), which issues its native "read all rows from this table, capped at N" and funnels it through the same parse → row-security → mask pipeline as `execute`. Mongo `find({}).limit(N)`, Couchbase/Cassandra/DynamoDB `SELECT * FROM `, Elasticsearch `match_all`, Neo4j `MATCH (n:Label) RETURN n`. **Redis fails closed** — a key-value prefix has no per-row security meaning, so any matching `RowSecurityDirective` denies with an empty result; otherwise it SCANs the prefix and fetches values, with field masking still applied. The result is a `SelectExecutionResult` mapped to `SampleRowsResponse` for `GET /api/v1/datasources/{id}/sample-rows` — masked columns carry the masked value only. +**Data budget (#942).** A preview reads real rows, so it spends the caller's data budget. `DefaultSampleDataService` reads the standing first: an exhausted budget throws `DataBudgetExhaustedException` (409 `DATA_BUDGET_EXHAUSTED`) whatever its breach action — a preview has no review to escalate to; otherwise the row limit is lowered to the remaining rows (a cut at that allowance reports `truncated_reason=DATA_BUDGET`), the result is measured and trimmed to the remaining bytes (`DefaultQueryExecutor.measureBytes`), and a `SAMPLE_DATA` ledger row is written. A refused preview writes a `QUERY_DATA_BUDGET_ENFORCED` audit row against the datasource (`stage=sample`, actor = the caller). The MCP sample tool goes through the same service. + ### Dry-run / EXPLAIN path (AF-445) `proxy.api.QueryDryRunService` returns a **non-committing execution plan + best-effort estimated row impact** for a query — the playground/sandbox a user reaches for before formal submission (`POST /api/v1/queries/dry-run`). Like the sample path it is an **ad-hoc read that bypasses review but not governance**, creates no `query_request`, and never mutates data — every engine plans the statement (relational `EXPLAIN`, Mongo `explain`, …) but never executes it. @@ -1247,6 +1249,91 @@ It is not billing, chargeback or reconciliation against actual spend: estimates (`BYTES_SCANNED_CAP` → `SKIP`) and never compares it, under the `COST_ESTIMATE_ABSENT` caveat. - No configuration knob: everything is per datasource and per grant. +### Per-user data-volume budgets (#942) + +Every other cap bounds **one** read: `max_rows_per_query`, row-limit policies, the result byte cap. +A user who stays under all of them can still pull a table out a few thousand rows at a time. A **data +budget** bounds the sum — rows and/or result bytes per user, per datasource, over a rolling window — +which makes it the slow-exfiltration control. It is a consumption guardrail, not an access boundary: +it never widens anything, and no budget configured leaves behaviour unchanged. + +- **Model.** `data_budgets` (`core`, admin CRUD through `core.api.DataBudgetAdminService` / + `DefaultDataBudgetAdminService`, controller `security.internal.web.DataBudgetController`, + `DATA_BUDGET_MANAGE`): `max_rows` and/or `max_bytes`, `window_minutes` (60–44,640, default 1,440), + `breach_action` `REJECT` | `REQUIRE_REVIEW` (default), optional `warn_threshold_percent`, and the + row-limit-policy `applies_to_*` scope (all empty ⇒ every user of the datasource, matched by the + shared `core.internal.AppliesToMatcher`). Each targeted user has their own allowance — a group + target is never a shared pool. +- **Accounting.** `core.api.DataBudgetUsageService` appends one `data_budget_usage` row per delivered + SELECT result — `QUERY` (`DefaultQueryLifecycleService.doExecute`: interactive, scheduled, + recurring occurrence, break-glass), `REQUEST_GROUP` (`GroupExecutionService`, per `QUERY` member) + or `SAMPLE_DATA` (`DefaultSampleDataService`, which also serves the MCP sample tool). Rows are what + the user received — after row security, with masked rows counted, after every cap and truncation; + bytes are `SelectExecutionResult.resultBytes`, the proxy's `ResultByteEstimator` sum measured + host-side for every engine, an estimate of the in-memory result rather than wire bytes. Writes are + never charged, and a read on a datasource where no budget applies to the user writes nothing, so + usage counts from the moment a budget exists. A ledger write failure is logged and swallowed: the + rows were already delivered. +- **Standing.** `core.api.DataBudgetStatusService.statusFor(datasource, user)` sums each applying + budget's own trailing window (`idx_data_budget_usage_user_ds_time`) — no counter, no reset job, no + calendar day or timezone. The effective standing (`DataBudgetStatus`) is the most constrained: + smallest remaining rows / bytes per metric, exhausted when any budget is, and the strictest action + among exhausted budgets (`REJECT` beats `REQUIRE_REVIEW`). +- **At the decision.** For a SELECT, `QueryReviewStateMachine` reads the standing at every entry point + and hands a `DataBudgetCheck` to `QueryDecisionEvaluator` — the explicit-input idiom of the SQL + review block and the bytes-scanned cap. Exhausted under `REJECT` decides right after the cap and + before routing and the AI-failed path: `PENDING_AI → REJECTED` (`QueryDecisionKind.DATA_BUDGET_REJECTED`, + no `routing_decision` row), published as `QueryAutoRejectedEvent` with a null policy id and a reason + (`workflow.data_budget.rejected`) naming the budget, so the existing notification fan-out applies. + Exhausted under `REQUIRE_REVIEW` joins the review guard: it suppresses a routing `AUTO_APPROVE` + (`ROUTING_AUTO_APPROVE_SUPPRESSED`, the policy still recorded), the grant fast path and the plan's + own approvals, and never softens an `AUTO_REJECT`. The decision trace gains a `DATA_BUDGET` step + between `BYTES_SCANNED_CAP` and `ROUTING_POLICIES`. The access explainer reads the same live standing. +- **At execution.** `doExecute` resolves the standing next to the bytes-cap re-check. With allowance + left, the row cap is lowered to the remaining rows (`maxRowsOverride`) and + `QueryExecutionRequest.maxResultBytesOverride` is set to the remaining bytes; when the budget was the + binding cap the result's `truncated_reason` is `DATA_BUDGET` (`attributeBudgetTruncation` for rows, + the executor's byte trim for bytes). Exhausted under `REJECT` ⇒ `APPROVED → FAILED` + (`DataBudgetExhaustedException`, localized `error_message` naming the budget). Exhausted under + `REQUIRE_REVIEW` ⇒ the run goes ahead, uncapped by the budget, only when **the exhausted budget + itself forced the review** — `query_requests.data_budget_review_forced`, stamped by the state + machine when a `REQUIRE_REVIEW` budget was exhausted as the query left `PENDING_AI` and it went to + `PENDING_REVIEW` (for a recurring occurrence, the series parent's stamp). Whoever approved it — + a reviewer or a synced external ticket (AF-453, which writes no `review_decisions` row) — did so + knowing the budget was spent. An approval obtained while allowance remained never lifts the + budget, so spending the rest on small reads cannot unlock an earlier-approved large one; such a + run, and an auto-approved scheduled run whose submitter ran out meanwhile, fails. + `GroupExecutionService` refuses a `QUERY` member under an exhausted budget **whatever its action**: + group review never evaluates the budget, so a group approval is not a budget escalation (fail + closed); `continue_on_error` decides the rest. **Break-glass is counted but never capped or + refused by a budget** (per-query caps still apply) — its compensating control is the mandatory + retro-review. +- **Concurrency.** Standing is read before a run and charged after it, so concurrent reads by one + user can each overshoot the remaining allowance — up to one query's worth per concurrent read. + Accepted: the next read sees the overshoot. Charging itself is serialized per (user, datasource) + by a transaction-scoped `pg_advisory_xact_lock` in `DefaultDataBudgetUsageService.record`, so the + before/after crossing check never misses or double-counts a threshold. +- **Notifications.** `DefaultDataBudgetUsageService` compares the standing before and after each + charge and publishes `core.events.DataBudgetThresholdCrossedEvent` on a crossing — stateless, once + per crossing. `notifications` turns it into `DATA_BUDGET_THRESHOLD_REACHED` (the user) or + `DATA_BUDGET_EXHAUSTED` (the user and every `DATA_BUDGET_MANAGE` holder); neither pages nor tickets. + See [08-notifications.md](08-notifications.md). +- **Audit.** One `QUERY_DATA_BUDGET_ENFORCED` row (null actor, `trigger=data_budget`, + `stage=decision|execution`, `action`, `data_budget_id`, `used_rows`, `used_bytes`, `window_minutes`) + whenever the budget changed the outcome; a successful `QUERY_EXECUTED` under a budget carries + `data_budget_rows_charged` and `data_budget_bytes_charged`. +- **The advisory half.** The `data_budget_used_percent` routing condition escalates on the same + standing without refusing anything — the submitter's most-used applying budget as a whole percent, + possibly above 100. It fails closed (null) when no budget applies, for a non-SELECT, and on the + policy simulator's historical replay, where past usage is not reconstructed. +- **Retention.** `DataBudgetUsagePruneJob` (`core/internal/scheduled/`) deletes ledger rows older than + `accessflow.core.data-budget.usage-retention` (default `P32D`; raised to 31 days + 1 hour when set + lower, so a row always outlives the longest window) every `accessflow.core.data-budget.prune-interval` + (default `PT1H`). +- **Read surfaces.** `GET /datasources/{id}/data-budgets/me` (the editor indicator, 404-never-403) and + `GET /admin/users/{id}/data-budget-usage` (`DATA_BUDGET_MANAGE` or `USER_MANAGE`), both in + `security.internal.web.DataBudgetStatusController`. + ### Data classification & derivation (AF-447) `data_classification_tag` rows (see [docs/03-data-model.md](03-data-model.md)) tag tables/columns with @@ -1690,6 +1777,8 @@ Decision rules: | Datasource has no review plan | `PENDING_REVIEW` (safe default) | | **Bytes-scanned cap refuses the query (#941)** — the estimate exceeds the cap, or there is none and the datasource says `REJECT`. Decided first, before routing and the AI-failed path | `REJECTED` — see [Bytes-scanned cost caps](#bytes-scanned-cost-caps-941) | | **Bytes-scanned cap with no estimate under `REQUIRE_REVIEW` (#941)** | `PENDING_REVIEW` — suppresses the same three `APPROVED` rows as a SQL review `BLOCK` | +| **Data budget used up under `REJECT` (#942)** — a SELECT whose submitter has exhausted an applying budget. Decided right after the bytes-scanned cap, before routing and the AI-failed path | `REJECTED` — see [Per-user data-volume budgets](#per-user-data-volume-budgets-942) | +| **Data budget used up under `REQUIRE_REVIEW` (#942)** | `PENDING_REVIEW` — suppresses the same three `APPROVED` rows as a SQL review `BLOCK` | | **Any `BLOCK` SQL review finding recorded at submission (#864)** — checked at every entry point, before all three rows above can approve | `PENDING_REVIEW` — each of the three `APPROVED` rows is suppressed (a routing `AUTO_APPROVE` too; `AUTO_REJECT` is untouched). Audited once as `SQL_REVIEW_BLOCKED` when it changed the outcome. See the "Submission enforcement (#864)" paragraph of [Deterministic SQL review rules](#deterministic-sql-review-rules-sqlreview-862) | `AiAnalysisFailedEvent` **always** transitions to `PENDING_REVIEW`, regardless of plan flags. Auto-approve is a positive-signal shortcut; failure is a missing signal — they aren't symmetric, so an AI provider error never short-circuits human review. The AI module persists a sentinel `CRITICAL` analysis row on failure with `failed=true` and `error_message=` (added in AF-249) so the reviewer can render an "AI analysis failed" surface on `QueryDetailPage` instead of seeing a fake CRITICAL verdict. Reviewers and admins can call [`POST /queries/{id}/reanalyze`](04-api-spec.md#post-queriesidreanalyze--response-202) to re-run analysis on the failed row — the workflow service deletes the sentinel and publishes `AiReanalysisRequestedEvent`, which the AI module's listener consumes by invoking the normal `analyzeSubmittedQuery` pipeline. A `QUERY_AI_REANALYZE_REQUESTED` audit row is written from the controller on each call. @@ -1762,7 +1851,7 @@ caller's language. #### The simulation `workflow.internal.DefaultAccessSimulationService` reconstructs the whole journey of a hypothetical -request as twelve ordered steps, delegating stages 5–8 (`SQL_REVIEW` through `REVIEW_PLAN`) to the +request as fourteen ordered steps, delegating stages 5–10 (`SQL_REVIEW` through `REVIEW_PLAN`) to the evaluator wholesale and owning the rest. A stage that did not apply is reported as `SKIP`, never omitted, so a client renders a stable checklist and a missing stage is always a bug. @@ -2275,6 +2364,7 @@ This makes horizontal scaling safe: when the AccessFlow backend runs as multiple | `QuerySuggestionAggregationJob` | workflow | `querySuggestionAggregationJob` | `accessflow.workflow.query-suggestions.aggregation-poll-interval` | `PT6H` | | `SchemaDriftJob` | schemachange | `schemaDriftJob` | `accessflow.schemachange.drift-poll-interval` | `PT6H` | | `JobExecutionRetentionJob` | scheduling | `jobExecutionRetentionJob` | `accessflow.scheduling.executions.retention-poll-interval` | `PT6H` | +| `DataBudgetUsagePruneJob` | core | `dataBudgetUsagePruneJob` | `accessflow.core.data-budget.prune-interval` | `PT1H` | `WeeklyDigestJob` implements the opt-in weekly dashboard digest (AF-498): it scans `dashboard_digest_subscription` for `enabled = true` rows whose `last_sent_at` is null or older than `accessflow.dashboard.weekly-digest.period` (default `P7D`, a partial index backs the scan) and, per row, builds that user's weekly summary, publishes a `dashboard.events.WeeklyDigestReadyEvent`, and stamps `last_sent_at`. The per-row build+publish+stamp runs inside `WeeklyDigestDispatchService.publishDigest` (`@Transactional`) so the event is published within a committed transaction — otherwise the notifications module's AFTER_COMMIT `@ApplicationModuleListener` would silently drop it. Per-row `RuntimeException`s are swallowed (`log.error`) so one bad subscription cannot abort the batch. The `notifications` module consumes the event and fans the summary out over the user's email + chat channels (`WEEKLY_DIGEST`); PagerDuty treats it as not-applicable (never pages). diff --git a/docs/06-frontend.md b/docs/06-frontend.md index 2f3c28806..89ddeee7e 100644 --- a/docs/06-frontend.md +++ b/docs/06-frontend.md @@ -228,6 +228,7 @@ Features: - **AI Hint Panel** — displays AI analysis after the user clicks the **Analyze** button. When the SQL is edited (including via **Apply as draft**) the analysis is **kept on screen but marked stale** — a "stale" badge plus a warning banner with a **Re-run analysis** button — so the user can still read the risks and apply the remaining optimization suggestions; Submit re-gates until the query is re-analyzed (staleness is derived by comparing the live SQL against the snapshot the analysis ran against). - **Analyze button** — explicit user action that calls `POST /queries/analyze`. Rendered only when the selected datasource has `ai_analysis_enabled=true` and a non-null `ai_config_id`. - **Live SQL review** (#865, epic #860) — no button: `hooks/useSqlReviewLint.ts` evaluates the draft the author has paused on (400 ms trailing debounce via `useDebouncedValue`) through the read-only `POST /sql-review/evaluate`, a TanStack `useQuery` keyed `['sqlReview', 'evaluation', datasourceId, sql]` (`sqlReviewKeys.evaluation`) and `enabled` only for a selected datasource, a non-blank draft under the 100 000-character limit, and a relational engine (`utils/sqlReview.ts` → `isSqlReviewSupported`; the server's `applicable:false` also turns the surface off). `placeholderData: keepPreviousData` keeps the previous findings on screen while the next evaluation is in flight — CodeMirror maps them through the author's edits. A 422 `INVALID_SQL` is an expected mid-keystroke state rendered as a quiet "Can't parse this yet" hint (`isInvalidSqlError`), never a toast; any other failure yields no findings, silently. Results render twice: as CodeMirror **diagnostics** in `SqlEditor` (see the SQL Editor section) and in the **findings strip** under the editor (`components/editor/SqlReviewFindingsStrip.tsx` → shared `components/review/SqlReviewFindingList.tsx`: severity pill via `riskColors.sqlReviewSeverityColor`, `L{n}` or "Statement N" for envelope members the parser gave no position, and the backend-localized message). The group builder's member drawer inherits both through the shared panel. **Submit stays enabled on a BLOCK** — blocking escalates to a human, it never rejects — and the Submit tooltip states how many blocking findings will require human approval. +- **Data-budget indicator** (#942, `components/editor/DataBudgetIndicator.tsx`) — above the review-plan preview, a compact card fed by `GET /datasources/{id}/data-budgets/me` (`dataBudgetKeys.mine`): one progress bar per applying budget with the remaining rows / bytes (`formatBytes`) and the window (`formatWindow`), a warning alert once a budget passes its warning threshold (`budgetNearlyUsed`), and an error alert when one is used up that says whether new reads are refused (`REJECT`) or need a reviewer (`REQUIRE_REVIEW`). It renders nothing when no budget applies, so an unbudgeted datasource looks exactly as before. - **Justification field** — required text area for the reason behind the query - **Scheduled execution picker** (AF-345) — optional Ant Design `DatePicker` with `showTime` that records `scheduled_for` on the submission payload. Past dates are disabled and a "scheduled time must be in the future" hint disables Submit when the user picks an already-elapsed instant. Leave empty for the default immediate-review flow. - **Recurrence picker** (#627, `components/editor/RecurrencePicker.tsx`) — a mode `Select` (Does not repeat / Fixed interval / Cron schedule (UTC)) plus the rule input for the chosen mode (interval presets 15m–7d composing ISO-8601 durations, or a free-text 6-field Spring cron with a UTC hint) and a mandatory series-end `DatePicker`. Mutually exclusive with the scheduled-execution picker (setting one clears the other); Submit is gated until the rule is present, the cron has 6 fields, and the end time is in the future — mirroring the backend validation. Records `recurrence_rule` + `recurrence_until` on the submission payload. @@ -388,6 +389,7 @@ immediately. - **Masking** tab (`components/datasources/MaskingTab.tsx`, AF-381) — a table of dynamic data masking policies (column, strategy, reveal-to summary, enabled) with a create/edit modal. The modal picks a column via an `AutoComplete` from the introspected schema, a strategy `Select` driven by `enumOptions(MASKING_STRATEGIES, maskingStrategyLabel, t)`, a conditional `visible_suffix` field shown only for `PARTIAL`, and reveal-to multi-selects for roles (`enumOptions`), groups, and users. A **live preview** renders the masked output of an editable sample value through `src/utils/maskingPreview.ts` (a pure client-side mirror of the backend `ColumnMasker` strategies; `HASH` shows an illustrative fixed digest since the real SHA-256 is computed server-side). CRUD calls `src/api/maskingPolicies.ts`; validation parity matches the backend DTO (required column ≤ 512 chars, required strategy, `visible_suffix` 1–256). - **Row security** tab (`components/datasources/RowSecurityTab.tsx`, AF-380) — a table of row-level security policies (table, `column operator value` predicate, applies-to summary, enabled) with a create/edit modal. The structured form picks a table and column via `AutoComplete`s from the introspected schema, an operator `Select` (`enumOptions(ROW_SECURITY_OPERATORS, rowSecurityOperatorLabel, t)`), a value-source `Select` (`VARIABLE` | `LITERAL`), and a value field that switches to a `:user.*` variable `AutoComplete` (offering the `:user.id` / `:user.email` / `:user.role` / `:user.groups` built-ins) when the source is `VARIABLE`. Applies-to multi-selects target roles, groups, and users (empty = everyone). CRUD calls `src/api/rowSecurityPolicies.ts`; validation parity matches the backend DTO (required table/column/value ≤ 512 chars, required operator, required value source). - **Row limits** tab (`components/datasources/RowLimitTab.tsx`, #934) — a table of per-table row-limit policies (`schema.table`, max rows, applies-to summary, enabled) with a create/edit modal. The modal picks an optional schema and a table through `AutoComplete`s fed by the introspected schema (the table list narrows to the chosen schema; an empty schema means "any schema"), a `max_rows` `InputNumber` (1–1,000,000) and applies-to multi-selects for roles, groups and users (empty = everyone, admins included). CRUD calls `src/api/rowLimitPolicies.ts`; validation parity matches the backend DTO (required table ≤ 255 chars, optional schema ≤ 255 chars, required `max_rows` 1–1,000,000). +- **Data budgets** tab (`components/datasources/DataBudgetTab.tsx`, #942) — labelled "Data budgets · N", right after *Row limits*. A table of per-user data-volume budgets (name, limits, window, breach action, applies-to summary, enabled) with a create/edit modal: `name` (required, ≤ 120), a row limit (`InputNumber`, ≥ 1) and a result-size limit (the shared `components/common/BytesInput.tsx`, ≥ 1 byte) with a form-level rule that at least one is set, a rolling window entered as an amount plus an hours/days unit (`splitWindow` / `joinWindow` in `src/utils/dataBudget.ts`, 60–44,640 minutes), a breach-action `Select` (`REQUIRE_REVIEW` / `REJECT`), an optional warning threshold (1–99 %), and applies-to multi-selects for roles, groups and users (empty = every user of the datasource). CRUD calls `src/api/dataBudgets.ts` (`dataBudgetKeys`); validation parity matches the backend `DataBudgetRequest` bounds (`DATA_BUDGET_*` constants in `src/utils/dataBudget.ts`). - **Exports** tab (`components/datasources/ExportPolicyTab.tsx`, #626) — a table of result-export governance policies (mode, details — row cap / deny classifications, applies-to summary, enabled) with a create/edit modal. The modal picks a mode `Select` (`enumOptions(EXPORT_POLICY_MODES, exportPolicyModeLabel, t)`), a conditional `row_cap` field shown only for `ROW_CAP` (1–1,000,000), a conditional deny-classifications multi-select shown only for `DENY_CLASSIFIED` (empty = any classification), and applies-to multi-selects for roles, groups, and users (empty = everyone, no admin bypass). A **live watermark preview** renders the exact header/footer stamp through `src/utils/watermarkPreview.ts` (a pure client-side mirror of the backend `ResultExportWatermark` templates) for the `WATERMARK`/`ROW_CAP` modes. CRUD calls `src/api/exportPolicies.ts`; validation parity matches the backend DTO (required mode, `row_cap` 1–1,000,000). - **Discovery** tab (`components/datasources/DiscoveryTab.tsx`, AF-623) — the sensitive-data discovery worklist. A settings card edits the per-datasource config (`GET`/`PUT /datasources/{id}/discovery/config`: enable switch, sample size 10–1000, interval 1–720 h, AI-pass toggle — validation parity with the backend DTO) with a **Scan now** button (`POST …/discovery/scan`, 202) and the last-scan time / error. Below, a paginated findings table (`GET …/discovery/findings`, status `Segmented` filter defaulting to `PENDING`, including the AF-659 `Stale` segment whose tag carries a hover hint explaining why the proposal aged out) shows column, proposed classification, detector (AI rationale as tooltip), confidence with match ratio, redacted sample, and status. `STALE` rows stay selectable alongside `PENDING` ones, since bulk-dismissing aged proposals is the point of retiring them. Row selection surfaces a sticky bulk bar (the `AttestationWorklistPage` pattern) whose **Confirm/Dismiss selected** call `POST …/discovery/findings/bulk-decision` with partial-success handling — failed rows stay selected with a per-row status tag. Confirming invalidates the classification + masking query keys so the sibling tabs refresh (the tag + derived policy appear there). API module `src/api/discovery.ts` (`discoveryKeys`); the tab badge on `DatasourceSettingsPage` shows the PENDING-findings count. - Review plan assignment and row limit configuration @@ -398,6 +400,7 @@ immediately. - Searchable, filterable table of all audit events - Filters: date range picker, user selector, action type multi-select - Row click opens `AuditDetailDrawer` with full metadata JSON +- The action filter includes the data-budget actions (#942): `QUERY_DATA_BUDGET_ENFORCED` and `DATA_BUDGET_CREATED` / `_UPDATED` / `_DELETED` - **Verify chain** button in the page header calls `GET /api/v1/admin/audit-log/verify` and renders an inline dismissible alert with the outcome (`Chain valid` + rows-checked count on success; `Chain invalid` with `first_bad_row_id` / `first_bad_reason` when tampering is detected) ### AuditorDashboardPage *(AUDITOR or ADMIN)* — AF-459 @@ -1438,6 +1441,8 @@ The primary action button is now a `Dropdown.Button` — the default click sends The **Edit user** modal includes an **Attributes** key/value editor (AF-380, a `Form.List`) bound to `users.attributes`. Current values are loaded via `GET /admin/users/{id}/attributes` (`getUserAttributes`) and saved through the existing `PUT /admin/users/{id}` with the `attributes` field. These attributes resolve in row-security predicates as `:user.` (up to 50 entries; key ≤ 128, value ≤ 512 chars — validation parity with the backend). +Every user row's action menu has a **Data usage** item (#942) that opens `components/admin/UserDataBudgetDrawer.tsx` — there is no user detail page. It lists, per datasource where a budget applies to that user, each budget's rows / bytes used against its limits, the window, and a *Used up* tag, from `GET /admin/users/{id}/data-budget-usage` (`dataBudgetKeys.forUser`); an empty state says no budget applies. + ### User groups and reviewer scope (AF-353) `GroupsListPage` (`/admin/groups`, lazy, admin-only) is a paginated `` of the org's user @@ -1483,6 +1488,13 @@ returns a non-null `matched_policy`, and a **bytes-scanned cap** alert (`data-te the title — an error for `EXCEEDED` / `NO_ESTIMATE_REJECTED`, a warning for `NO_ESTIMATE_REVIEW`, info for `WITHIN` — and the estimate, the cap and its source in the body. The decision trace labels the `BYTES_SCANNED_CAP` step and formats its byte details through `formatBytes`. +The builder also offers **Data budget used (%)** (`data_budget_used_percent`, #942): a comparison +operator and a whole-percent value (≥ 0, above 100 allowed), mapped in `routingPolicyForm.ts`. The +decision trace labels the `DATA_BUDGET` step and its `data_budget_*` details (including +`data_budget_suppressed` on the routing and review-plan steps). A result cut short by the +submitter's remaining allowance shows its own truncation notice — `QueryResultsTable` and the +table-preview `SampleDataPreview` map `truncated_reason: "DATA_BUDGET"` to a dedicated message beside +the `ROW_LIMIT` / `BYTE_LIMIT` ones. ### Policy simulator (AF-630) diff --git a/docs/07-security.md b/docs/07-security.md index 69b2690e4..250364384 100644 --- a/docs/07-security.md +++ b/docs/07-security.md @@ -537,6 +537,15 @@ CRUD (`/api/v1/datasources/{id}/row-limit-policies`, `@PreAuthorize("hasAuthority('PERM_ROW_LIMIT_POLICY_MANAGE')")`). It sits in the `DATA_POLICIES` group and is held by `ADMIN` (seeded by `V185`). +**Per-user data budgets (#942):** `DATA_BUDGET_MANAGE` gates the per-datasource data-budget CRUD +(`/api/v1/datasources/{id}/data-budgets`, `@PreAuthorize("hasAuthority('PERM_DATA_BUDGET_MANAGE')")`) +and receives the `DATA_BUDGET_EXHAUSTED` notification. It sits in the `DATA_POLICIES` group and is +held by `ADMIN` (seeded by `V194`). Two read endpoints are not admin surfaces: a user's own standing +(`GET /datasources/{id}/data-budgets/me`) needs only authentication and resolves the datasource the +way the rest of the API does, so an invisible datasource is 404, never an empty 200; another user's +usage (`GET /admin/users/{id}/data-budget-usage`) needs `DATA_BUDGET_MANAGE` **or** `USER_MANAGE` +and is org-scoped (a foreign user is 404). + **Result-export governance (#626):** `EXPORT_POLICY_MANAGE` gates the per-datasource export-policy CRUD (`/api/v1/datasources/{id}/export-policies`, `@PreAuthorize("hasAuthority('PERM_EXPORT_POLICY_MANAGE')")`); it sits in the `DATA_POLICIES` group @@ -696,6 +705,9 @@ A deployment hosts one or more `organizations`, each a fully isolated tenant. Is `max_queries_per_day` (a rolling trailing-24h count over `query_requests` — no counter table, no reset job). A breach throws and the API responds `409 Conflict` with `error: "QUOTA_EXCEEDED"` and a localized `detail` naming the limit. Quotas bound consumption; they are not an access boundary. + They count **queries per organization**; the per-user, data-volume counterpart is the + [data budget](#per-user-data-volume-budgets-942), which bounds how many rows and bytes one person + reads from one datasource over a rolling window. - **Multi-org login routing is future work.** Per-org login pages / SSO routing across multiple orgs are explicitly out of scope for AF-456. Unauthenticated provider discovery degrades gracefully when more than one org exists (it never discloses per-org identity — see @@ -1048,6 +1060,43 @@ an over-estimate refuses a legitimate query, an under-estimate lets an expensive - **Audited.** `QUERY_BYTES_SCANNED_CAP_ENFORCED` (null actor) records the limit, its source, the estimate and the outcome whenever the cap changed an outcome. +### Per-user data-volume budgets (#942) + +A **slow-exfiltration control.** Every per-query cap — `max_rows_per_query`, row-limit policies, the +result byte cap — bounds one read, so a user who stays under all of them can still copy a table out a +few thousand rows at a time. A data budget bounds the **sum**: how many result rows and/or bytes each +targeted user may read from one datasource over a rolling window (1 hour to 31 days). Like the org +quotas it bounds consumption rather than granting or denying access, and with no budget configured +nothing changes. + +- **Per person, never pooled.** A budget scoped to a role or group gives every member their own + allowance; one heavy reader cannot exhaust a colleague's. Several applying budgets combine to the + most constrained, and `REJECT` beats `REQUIRE_REVIEW`. +- **Counts what was delivered.** Rows after row security and every cap, masked rows included, and + the result's estimated in-memory size, measured the same way for every engine. Every read path + charges it: interactive, scheduled, recurring, grouped, table previews (including through MCP) and + break-glass. Writes are never charged. Usage is charged from the moment a budget exists. +- **Enforced twice.** When the query leaves `PENDING_AI` — an exhausted `REJECT` budget rejects before + routing, an exhausted `REQUIRE_REVIEW` budget suppresses every automatic approval (a routing + `AUTO_APPROVE`, the JIT grant fast path, the plan's own auto-approval) so a person decides — and + again just before execution, where a run is capped to the allowance left, and an exhausted budget + fails the run unless a person approved it under `REQUIRE_REVIEW`. That second check covers scheduled + and recurring runs approved while allowance remained. `QUERY_ADMIN` holders are not exempt unless + the budget's scope leaves them out. +- **Only a budget-forced review lifts the budget.** Under `REQUIRE_REVIEW` an approved query runs + past an exhausted budget only if the exhausted budget itself sent it to review + (`data_budget_review_forced`); an approval given while allowance remained never does, so a user + cannot pre-approve a large read and unlock it by spending the rest on small ones. Request-group + members are refused under any exhausted budget (group review is not budget-aware — fail closed). +- **Break-glass is counted, never capped or blocked.** An emergency read is not cut short or refused + by a budget (the per-query row and byte caps still apply) and is still counted. Its control is the + mandatory retro-review, not the budget. +- **Known overshoot.** Standing is read before a run and charged after it, so concurrent reads by one + user can each overshoot by up to one query's worth. An estimate, not billing. +- **Audited and notified.** `QUERY_DATA_BUDGET_ENFORCED` (null actor) whenever a budget changed an + outcome, `DATA_BUDGET_CREATED/_UPDATED/_DELETED` for configuration; the user is warned at the + optional threshold, and the user plus every `DATA_BUDGET_MANAGE` holder is told when it runs out. + ### Dynamic data masking policies (AF-381) `masking_policy` rows extend the static masking above with **per-column strategies** and a diff --git a/docs/08-notifications.md b/docs/08-notifications.md index c84a6caa7..e64f79133 100644 --- a/docs/08-notifications.md +++ b/docs/08-notifications.md @@ -36,6 +36,8 @@ The dispatcher runs on virtual-thread executors and consumes events using Spring | `SCHEMA_CHANGE_PROMOTION_APPLIED` | Every statement of a promotion executed on its target environment (`APPLIED`, #882) | The promoter. Org-wide channel fanout; never pages, never opens a ticket. | implemented | | `SCHEMA_CHANGE_PROMOTION_FAILED` | A promotion's run failed — `FAILED`, or `PARTIALLY_APPLIED` (the run stops on the first error, so a partial run means a statement failed; the in-app payload's `promotion_status` (the webhook block's `status`) says which) (#882) | The promoter. Org-wide channel fanout; never pages, never opens a ticket. | implemented | | `SCHEMA_DRIFT_DETECTED` | A schema-drift scan of one environment **opened** at least one finding (#882) — a newly created finding, or a `RESOLVED` one that came back. An acknowledged finding whose values changed counts too — nobody accepted the new difference. A finding merely re-seen on a later scan while already open never notifies, so a persistent finding does not alert every scan. One notification per environment scan, carrying the count | Every active holder of `SCHEMA_CHANGE_MANAGE` (system and custom roles). Org-wide channel fanout; never pages and never opens a ticket — findings carry no severity, so there is nothing to key a paging trigger on. | implemented | +| `DATA_BUDGET_THRESHOLD_REACHED` | A recorded read carried a user across the warning threshold of one of their data budgets on a datasource (#942). Fired once per crossing — a read that stays above the mark never re-fires, and a read that jumps straight past the limit fires only `DATA_BUDGET_EXHAUSTED` | That user only. Org-wide channel fanout; never pages and never opens a ticket — a read quota is advisory, not an incident. The in-app row has no target column; its payload carries `datasource_id`, `budget` and `used_percent`, and the bell opens the query editor (`/editor`). | implemented | +| `DATA_BUDGET_EXHAUSTED` | A recorded read used up one of a user's data budgets on a datasource (#942); further SELECTs are rejected or sent to review per the budget's breach action until usage falls back under the limit | That user **and** every active holder of `DATA_BUDGET_MANAGE` (system and custom roles), de-duplicated when the user is one. Org-wide channel fanout; never pages, never opens a ticket. Same in-app payload and `/editor` link as the warning. | implemented | | `QUERY_CHANGES_REQUESTED` | Reviewer requests changes | Query submitter | deferred — no event published yet | | `QUERY_EXECUTED` | A **recurring occurrence** completes (#627) — fired for both `EXECUTED` and `FAILED` occurrence outcomes; one-off executions do not notify | Query submitter, via the review plan's channels. Email carries the occurrence results as a `results.csv` attachment (successful SELECT occurrences only; post-mask values; capped at 10,000 rows with a truncation note row); chat channels get a summary with a link to the occurrence. PagerDuty and ticketing not-applicable (no trigger mapping). | implemented | | `QUERY_FAILED` | Execution error | Query submitter + all ADMIN users | deferred — proxy executor not implemented | @@ -96,6 +98,8 @@ Email bodies are rendered using **Thymeleaf** HTML templates located in `resourc - `email/schema-change-promotion-applied.html` — `SCHEMA_CHANGE_PROMOTION_APPLIED` (#882; green accent) - `email/schema-change-promotion-failed.html` — `SCHEMA_CHANGE_PROMOTION_FAILED` (#882; copy branches on `FAILED` vs `PARTIALLY_APPLIED`, red banner) - `email/schema-drift-detected.html` — `SCHEMA_DRIFT_DETECTED` (#882; pipeline, environment, and the number of newly opened findings; its *Open in AccessFlow* button links to `/schema-drift`, #883) +- `email/data-budget-threshold-reached.html` — `DATA_BUDGET_THRESHOLD_REACHED` (#942; datasource, user, budget, percent used, rows / bytes used against each limit the budget sets, the window, and the warning threshold; links to `/editor`) +- `email/data-budget-exhausted.html` — `DATA_BUDGET_EXHAUSTED` (#942; the same fields plus the breach action, red accent) Templates include: - Query summary (datasource, query type, SQL preview — first 200 chars) @@ -236,7 +240,11 @@ environment, version, outcome, decision_reason }` — since the `query_request` the pipeline name for them. Schema-change events (`SCHEMA_CHANGE_PROMOTION_*`, `SCHEMA_DRIFT_DETECTED`, #882) add a sibling `schema_change` object — `{ promotion_id, change_set_name, pipeline_id, pipeline_name, environment, status, new_finding_count }`, with the -promotion fields null for drift and `new_finding_count` null for promotions. All three blocks are **additive**: existing event shapes are unchanged, so +promotion fields null for drift and `new_finding_count` null for promotions. Data-budget events +(`DATA_BUDGET_THRESHOLD_REACHED`, `DATA_BUDGET_EXHAUSTED`, #942) add a sibling `data_budget` +object — `{ budget_id, budget_name, datasource_id, user_id, exhausted, warn_threshold_percent, +used_percent, max_rows, used_rows, max_bytes, used_bytes, window_minutes, breach_action }`, with +`max_rows` / `max_bytes` null for a limit the budget does not set. All four blocks are **additive**: existing event shapes are unchanged, so subscribers' HMAC-verified payloads are unaffected. **Request headers:** @@ -336,7 +344,7 @@ Pages an on-call responder via the [PagerDuty Events API v2](https://developer.p - `CRITICAL_RISK` → the `AI_HIGH_RISK` event (raised only when the AI analysis returns `CRITICAL` risk). - `REVIEW_TIMEOUT` → the `REVIEW_TIMEOUT` event (a query auto-rejected past its `approval_timeout_hours`). - `ANOMALY` → the `ANOMALY_DETECTED` event (a behavioural anomaly flagged by `BehaviorAnomalyDetectionJob`, UBA, AF-383). - - `BREAK_GLASS` → the `BREAK_GLASS_EXECUTED` event (an emergency-access query executed, bypassing review, AF-385) **and** the `DEPLOYMENT_BREAK_GLASS_EXECUTED` event (#695) — one operator knob covers break-glass queries and break-glass deployments. The routine deployment lifecycle events (`DEPLOYMENT_SUBMITTED`/`_APPROVED`/`_REJECTED`/`_OUTCOME_FAILED`) deliberately have no trigger: lifecycle progress is not an incident. Neither do the schema-change events (`SCHEMA_CHANGE_PROMOTION_*`, `SCHEMA_DRIFT_DETECTED`, #882): a promotion's lifecycle is not an incident, and a drift finding carries no severity to separate a critical divergence from a cosmetic one. + - `BREAK_GLASS` → the `BREAK_GLASS_EXECUTED` event (an emergency-access query executed, bypassing review, AF-385) **and** the `DEPLOYMENT_BREAK_GLASS_EXECUTED` event (#695) — one operator knob covers break-glass queries and break-glass deployments. The routine deployment lifecycle events (`DEPLOYMENT_SUBMITTED`/`_APPROVED`/`_REJECTED`/`_OUTCOME_FAILED`) deliberately have no trigger: lifecycle progress is not an incident. Neither do the schema-change events (`SCHEMA_CHANGE_PROMOTION_*`, `SCHEMA_DRIFT_DETECTED`, #882): a promotion's lifecycle is not an incident, and a drift finding carries no severity to separate a critical divergence from a cosmetic one. Nor do the data-budget events (`DATA_BUDGET_THRESHOLD_REACHED`, `DATA_BUDGET_EXHAUSTED`, #942): a user reaching a read quota is an advisory, not an incident. - `ESCALATION` → the `QUERY_ESCALATED` event (a routing policy escalated the query, AF-453). - `REVIEW_STALLED` → the `REVIEW_ESCALATED` event (nobody decided within the plan's `escalation_after_hours`, #622). There is deliberately **no** trigger for `REVIEW_NUDGE` — a reminder is not an incident and must never page. Events with no matching trigger (and every other event type, e.g. `QUERY_SUBMITTED`) are dropped silently. @@ -540,6 +548,8 @@ in `NotificationContextBuilder`: | `SCHEMA_CHANGE_PROMOTION_SUBMITTED` | The target datasource's eligible reviewers (every plan approver rule ∪ datasource reviewer assignments, else `REVIEW_OVERRIDE` holders), excluding the promoter | | `SCHEMA_CHANGE_PROMOTION_APPLIED` / `SCHEMA_CHANGE_PROMOTION_FAILED` | The promoter | | `SCHEMA_DRIFT_DETECTED` | Every active `SCHEMA_CHANGE_MANAGE` holder | +| `DATA_BUDGET_THRESHOLD_REACHED` | The user whose budget it is | +| `DATA_BUDGET_EXHAUSTED` | The user whose budget it is ∪ every active `DATA_BUDGET_MANAGE` holder, de-duplicated | | `TEST` | Skipped — never persisted to the inbox | **Persistence flow.** `NotificationDispatcher` first calls `userNotificationService.recordForUsers(...)` diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 739412ecb..19a30a422 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -1304,6 +1304,8 @@ Tuning for approval-outcome prediction: a per-organization logistic model retrai | Variable | Required | Default | Description | |----------|---------|---------|-------------| | `ACCESSFLOW_CORE_REVIEW_DELEGATION_MAX_OPEN_PER_DELEGATOR` | Optional | `10` | Maximum out-of-office review delegations one user may have open at once (#622). Creating another returns HTTP 422 `ILLEGAL_REVIEW_DELEGATION`. Bounds the per-identity OR-tree the API-review queue builds, so a pathological delegation set cannot produce an unbounded query. | +| `ACCESSFLOW_CORE_DATA_BUDGET_PRUNE_INTERVAL` | Optional | `PT1H` | ISO-8601 duration. Cadence of `DataBudgetUsagePruneJob` (#942), which deletes data-budget usage-ledger rows older than the retention below. Clustered-safe via ShedLock (`dataBudgetUsagePruneJob`). | +| `ACCESSFLOW_CORE_DATA_BUDGET_USAGE_RETENTION` | Optional | `P32D` | ISO-8601 duration. How long per-user data-budget usage rows (#942) are kept. Must outlive the longest budget window (31 days), so a shorter value is raised to 31 days + 1 hour; otherwise a budget would under-count. Budgets themselves are configured per datasource, not by env var. | | `ACCESSFLOW_WORKFLOW_ESCALATION_POLL_INTERVAL` | Optional | `PT5M` | ISO-8601 duration. Cadence at which `ReviewEscalationJob` (#622) escalates `PENDING_REVIEW` queries past their plan's `escalation_after_hours` and re-nudges undecided reviewers on `nudge_interval_hours`. Notify-only — it never changes who may approve. ShedLock makes this safe under horizontal scaling. | | `ACCESSFLOW_SCHEDULING_ENABLED` | Optional | `true` | Master switch for Spring's `@EnableScheduling`. When `false`, **no `@Scheduled` job in any module runs** — timeouts, escalations, scheduled and recurring query runs, retention, erasure, attestation and deployment jobs all stop. ShedLock's `@SchedulerLock` advice stays wired either way (`SchedulerLockConfiguration` is separate and unconditional). Spring Modulith Moments is disabled in `application.yml` (`spring.modulith.moments.enabled: false`) because its auto-configuration carries its own `@EnableScheduling` that would otherwise override this switch — do not re-enable it. Leave it `true` in every real deployment; it exists so the integration suite, which shares one long-lived Spring context across ~90 test classes, does not have those jobs mutating the shared test database underneath assertions. | | `ACCESSFLOW_SCHEDULING_EXECUTIONS_ENABLED` | Optional | `true` | Record every `@Scheduled` run that actually held its ShedLock lock in `job_executions` (#923), shown on the platform-admin **Scheduled jobs** page. Lock-skipped ticks are never written, and recording is fail-soft — it never changes a job's outcome. `false` stops recording; the job registry still works. | diff --git a/docs/13-mcp.md b/docs/13-mcp.md index 34f859dff..b6cfc7b3e 100644 --- a/docs/13-mcp.md +++ b/docs/13-mcp.md @@ -180,7 +180,9 @@ names both `application/json` and `text/event-stream`; real MCP clients send bot also refuses the human a query was submitted *on behalf of*. - `get_column_samples` runs through the same governed read path as the schema explorer (AF-443): it applies the caller's row-level security predicates and column masks and enforces `canRead` + - the schema/table allow-list, so a masked column never returns a raw value. + the schema/table allow-list, so a masked column never returns a raw value. It also spends the + caller's data-volume budget (#942): the sample is capped at the allowance left, and an exhausted + budget refuses it with a tool error, since a preview has no review to escalate to. - `get_audit_log` is always scoped to the caller (`actorId` is forced to the calling user) and the caller's organisation — it never returns another user's activity, even for an admin key. - `validate_sql` only parses; it never executes or runs AI analysis. It needs the datasource to be diff --git a/e2e/helpers/datasources.ts b/e2e/helpers/datasources.ts index a2bf8f934..de981ed30 100644 --- a/e2e/helpers/datasources.ts +++ b/e2e/helpers/datasources.ts @@ -842,6 +842,52 @@ export async function createRowLimitPolicyViaApi( return (await res.json()) as CreatedRowLimitPolicy; } +export interface CreatedDataBudget { + id: string; + name: string; + max_rows?: number | null; + max_bytes?: number | null; + window_minutes: number; + breach_action: 'REJECT' | 'REQUIRE_REVIEW'; +} + +// POST /api/v1/datasources/{id}/data-budgets (#942) — bounds the rows / result bytes each +// targeted user may read from the datasource over a rolling window. Requires an ADMIN token. +// Empty applies-to lists apply the budget to every user of the datasource. +export async function createDataBudgetViaApi( + request: APIRequestContext, + adminAccessToken: string, + datasourceId: string, + opts: { + name: string; + maxRows?: number; + maxBytes?: number; + windowMinutes?: number; + breachAction?: 'REJECT' | 'REQUIRE_REVIEW'; + appliesToUserIds?: string[]; + }, +): Promise { + const res = await request.post(`${apiBase()}/api/v1/datasources/${datasourceId}/data-budgets`, { + headers: { Authorization: `Bearer ${adminAccessToken}` }, + data: { + name: opts.name, + max_rows: opts.maxRows ?? null, + max_bytes: opts.maxBytes ?? null, + window_minutes: opts.windowMinutes ?? 1440, + breach_action: opts.breachAction ?? 'REQUIRE_REVIEW', + warn_threshold_percent: null, + applies_to_roles: [], + applies_to_group_ids: [], + applies_to_user_ids: opts.appliesToUserIds ?? [], + enabled: true, + }, + }); + if (!res.ok()) { + throw new Error(`Create data budget failed: ${res.status()} ${await res.text()}`); + } + return (await res.json()) as CreatedDataBudget; +} + // PUT /api/v1/admin/users/{id} (AF-380) — sets the admin-editable attribute map // resolvable in row-security predicates as `:user.`. Requires an ADMIN token. export async function setUserAttributesViaApi( diff --git a/e2e/tests/data-budgets.spec.ts b/e2e/tests/data-budgets.spec.ts new file mode 100644 index 000000000..5c39972b0 --- /dev/null +++ b/e2e/tests/data-budgets.spec.ts @@ -0,0 +1,227 @@ +import { randomUUID } from 'node:crypto'; +import { expect, test, type APIRequestContext, type Page } from '@playwright/test'; +import { + acceptInvitationViaApi, + apiBase, + createDataBudgetViaApi, + createPostgresDatasource, + createReviewPlanViaApi, + deleteDatasource, + executeQueryViaApi, + findUserByEmailViaApi, + grantPermissionViaApi, + inviteUserViaApi, + loginViaApi, + submitQueryViaApi, + waitForInviteToken, + waitForQueryStatus, + type CreatedDatasource, + type InvitedUser, +} from '../helpers/datasources'; +import { login } from '../helpers/login'; +import { clickTab, findRowAcrossPages } from '../helpers/ui'; + +const ADMIN_EMAIL = 'e2e@accessflow.test'; +const ADMIN_PASSWORD = 'E2ePassword!123'; +const ANALYST_PASSWORD = 'Analyst-Pwd!123'; + +// The e2e datasource points at AccessFlow's own database: `role_permissions` holds one row per +// ADMIN permission, comfortably more than the three-row budget used below. +const SELECT_PERMISSIONS = 'SELECT permission FROM role_permissions ORDER BY permission'; + +interface ResultPage { + row_count: number; + truncated: boolean; + truncated_reason: string | null; +} + +interface BudgetStatus { + exhausted: boolean; + remaining_rows?: number | null; + budgets: { used_rows: number }[]; +} + +async function provisionAnalyst( + request: APIRequestContext, + adminToken: string, + label: string, +): Promise<{ user: InvitedUser; email: string; token: string }> { + const email = `${label}-${randomUUID()}@e2e.local`; + await inviteUserViaApi(request, adminToken, email, `Budget ${label}`, 'ANALYST'); + const inviteToken = await waitForInviteToken(request, email); + await acceptInvitationViaApi(request, inviteToken, ANALYST_PASSWORD, `Budget ${label}`); + const token = await loginViaApi(request, email, ANALYST_PASSWORD); + const user = await findUserByEmailViaApi(request, adminToken, email); + return { user, email, token }; +} + +async function myStanding( + request: APIRequestContext, + token: string, + datasourceId: string, +): Promise { + const res = await request.get(`${apiBase()}/api/v1/datasources/${datasourceId}/data-budgets/me`, { + headers: { Authorization: `Bearer ${token}` }, + }); + if (!res.ok()) throw new Error(`Budget standing failed: ${res.status()} ${await res.text()}`); + return (await res.json()) as BudgetStatus; +} + +async function selectDatasource(page: Page, name: string): Promise { + await page.goto('/editor'); + await page.getByRole('combobox').first().click(); + await page.locator('.ant-select-item-option').filter({ hasText: name }).click(); +} + +test.describe.configure({ timeout: 90_000 }); + +test.describe.serial('per-user data-volume budgets (#942)', () => { + let adminToken = ''; + let datasource: CreatedDatasource | null = null; + let reviewAnalyst: { user: InvitedUser; email: string; token: string }; + let rejectAnalyst: { user: InvitedUser; email: string; token: string }; + + test.beforeAll(async ({ request }) => { + adminToken = await loginViaApi(request, ADMIN_EMAIL, ADMIN_PASSWORD); + reviewAnalyst = await provisionAnalyst(request, adminToken, 'budget-review'); + rejectAnalyst = await provisionAnalyst(request, adminToken, 'budget-reject'); + + // No human approval: an in-budget SELECT auto-approves, so any review below is the budget's. + const plan = await createReviewPlanViaApi(request, adminToken, { + name: `E2E Budget Plan ${Date.now()}`, + requiresHumanApproval: false, + }); + datasource = await createPostgresDatasource(request, adminToken, { + name: `Postgres E2E Budget ${Date.now()}`, + reviewPlanId: plan.id, + }); + for (const analyst of [reviewAnalyst, rejectAnalyst]) { + await grantPermissionViaApi(request, adminToken, datasource.id, analyst.user.id, { + canRead: true, + }); + } + await createDataBudgetViaApi(request, adminToken, datasource.id, { + name: 'Three rows, then review', + maxRows: 3, + breachAction: 'REQUIRE_REVIEW', + appliesToUserIds: [reviewAnalyst.user.id], + }); + await createDataBudgetViaApi(request, adminToken, datasource.id, { + name: 'Three rows, then reject', + maxRows: 3, + breachAction: 'REJECT', + appliesToUserIds: [rejectAnalyst.user.id], + }); + }); + + test.afterAll(async ({ request }) => { + if (datasource) { + await deleteDatasource(request, adminToken, datasource.id); + } + }); + + test('a read is capped at the allowance left, then the next one goes to review', async ({ + request, + }) => { + if (!datasource) throw new Error('datasource not created in beforeAll'); + const first = await submitQueryViaApi(request, reviewAnalyst.token, datasource.id, + SELECT_PERMISSIONS, '#942 data budget — first read'); + await waitForQueryStatus(request, reviewAnalyst.token, first.id, 'APPROVED'); + const outcome = await executeQueryViaApi(request, reviewAnalyst.token, first.id); + expect(outcome.status).toBe('EXECUTED'); + const res = await request.get(`${apiBase()}/api/v1/queries/${first.id}/results`, { + headers: { Authorization: `Bearer ${reviewAnalyst.token}` }, + }); + expect(res.ok()).toBe(true); + const page = (await res.json()) as ResultPage; + expect(page.row_count).toBe(3); + expect(page.truncated_reason).toBe('DATA_BUDGET'); + + const standing = await myStanding(request, reviewAnalyst.token, datasource.id); + expect(standing.exhausted).toBe(true); + expect(standing.budgets[0]?.used_rows).toBe(3); + + const second = await submitQueryViaApi(request, reviewAnalyst.token, datasource.id, + SELECT_PERMISSIONS, '#942 data budget — over budget'); + await waitForQueryStatus(request, reviewAnalyst.token, second.id, 'PENDING_REVIEW'); + }); + + test('a rejecting budget refuses the read once used up', async ({ request }) => { + if (!datasource) throw new Error('datasource not created in beforeAll'); + const first = await submitQueryViaApi(request, rejectAnalyst.token, datasource.id, + SELECT_PERMISSIONS, '#942 data budget — first read'); + await waitForQueryStatus(request, rejectAnalyst.token, first.id, 'APPROVED'); + await executeQueryViaApi(request, rejectAnalyst.token, first.id); + + const second = await submitQueryViaApi(request, rejectAnalyst.token, datasource.id, + SELECT_PERMISSIONS, '#942 data budget — refused'); + await waitForQueryStatus(request, rejectAnalyst.token, second.id, 'REJECTED'); + }); + + test('the analyst sees their used-up budget in the editor', async ({ page }) => { + if (!datasource) throw new Error('datasource not created in beforeAll'); + await login(page, reviewAnalyst.email, ANALYST_PASSWORD); + await selectDatasource(page, datasource.name); + + const indicator = page.getByTestId('data-budget-indicator'); + await expect(indicator).toBeVisible({ timeout: 15_000 }); + await expect(indicator.getByText('Data budget used up')).toBeVisible(); + await expect(indicator.getByText('Three rows, then review')).toBeVisible(); + }); + + test('admin creates a data budget via the Data budgets tab', async ({ page }) => { + if (!datasource) throw new Error('datasource not created in beforeAll'); + await login(page, ADMIN_EMAIL, ADMIN_PASSWORD); + await page.goto(`/datasources/${datasource.id}/settings`); + await clickTab(page, /Data budgets/); + + // The settings page renders tab content beside , not inside its tabpanel. + await expect(page.getByText('Three rows, then review')).toBeVisible({ timeout: 15_000 }); + + await page.getByRole('button', { name: 'Add budget' }).click(); + const dialog = page.getByRole('dialog'); + await dialog.getByLabel('Name').fill('Weekly analyst cap'); + await dialog.getByLabel('Row limit').fill('50000'); + await dialog.getByRole('button', { name: 'Save' }).click(); + + await expect(page.getByText('Data budget saved')).toBeVisible({ timeout: 10_000 }); + await expect(page.getByRole('cell', { name: 'Weekly analyst cap', exact: true })).toBeVisible({ + timeout: 10_000, + }); + }); + + test('admin sees a user\'s data usage from the users page', async ({ page }) => { + if (!datasource) throw new Error('datasource not created in beforeAll'); + await login(page, ADMIN_EMAIL, ADMIN_PASSWORD); + // Collapse the onboarding checklist: it is sticky and would overlay the row actions. + await page.evaluate(() => { + const key = 'af-preferences'; + const raw = localStorage.getItem(key); + const stored = raw ? JSON.parse(raw) : { state: {}, version: 0 }; + stored.state = { ...(stored.state ?? {}), setupProgressCollapsed: true }; + localStorage.setItem(key, JSON.stringify(stored)); + }); + await page.goto('/admin/users'); + + // The users table has no server-side search; page through it (parallel specs add users). + // The pending-invitations table lists the same email as an ACCEPTED invitation; skip it. + const row = page + .getByRole('row') + .filter({ hasText: reviewAnalyst.email }) + .filter({ hasNotText: 'ACCEPTED' }); + await findRowAcrossPages(page, row); + // The users table is wider than the viewport, so its action column can sit off-screen where a + // physical click cannot land; dispatch the events instead (the clickTab trick). + await expect(async () => { + await row.getByRole('button', { name: 'Edit' }).dispatchEvent('click'); + await page.getByRole('menuitem', { name: /Data usage/ }).dispatchEvent('click', undefined, { + timeout: 5_000, + }); + }).toPass({ timeout: 30_000 }); + + const drawer = page.getByRole('dialog').filter({ hasText: 'Data usage —' }); + await expect(drawer.getByText(datasource.name)).toBeVisible({ timeout: 15_000 }); + await expect(drawer.getByText('Three rows, then review')).toBeVisible(); + await expect(drawer.getByText('Used up')).toBeVisible(); + }); +}); diff --git a/e2e/tests/discovery.spec.ts b/e2e/tests/discovery.spec.ts index 885f35916..c5e54e2d8 100644 --- a/e2e/tests/discovery.spec.ts +++ b/e2e/tests/discovery.spec.ts @@ -18,7 +18,7 @@ import { type CreatedReviewPlan, } from '../helpers/datasources'; import { login } from '../helpers/login'; -import { findRowAcrossPages } from '../helpers/ui'; +import { clickTab, findRowAcrossPages } from '../helpers/ui'; const ADMIN_EMAIL = 'e2e@accessflow.test'; const ADMIN_PASSWORD = 'E2ePassword!123'; @@ -155,7 +155,7 @@ test.describe.serial('sensitive-data discovery (AF-623)', () => { await login(page, ADMIN_EMAIL, ADMIN_PASSWORD); await page.goto(`/datasources/${datasource.id}/settings`); - await page.getByRole('tab', { name: /Discovery/ }).click(); + await clickTab(page, /Discovery/); // Enable scheduled discovery and save. await expect(page.getByText('Discovery settings')).toBeVisible({ timeout: 15_000 }); @@ -172,7 +172,7 @@ test.describe.serial('sensitive-data discovery (AF-623)', () => { // The worklist shows the finding after a reload. await page.goto(`/datasources/${datasource.id}/settings`); - await page.getByRole('tab', { name: /Discovery/ }).click(); + await clickTab(page, /Discovery/); await expect( page.getByText(`public.${TABLE}.customer_email`).first(), ).toBeVisible({ timeout: 15_000 }); @@ -186,7 +186,7 @@ test.describe.serial('sensitive-data discovery (AF-623)', () => { await login(page, ADMIN_EMAIL, ADMIN_PASSWORD); await page.goto(`/datasources/${datasource.id}/settings`); - await page.getByRole('tab', { name: /Discovery/ }).click(); + await clickTab(page, /Discovery/); const row = page.getByRole('row', { name: new RegExp(`public\\.${TABLE}\\.customer_email`) }); await expect(row).toBeVisible({ timeout: 15_000 }); @@ -201,7 +201,7 @@ test.describe.serial('sensitive-data discovery (AF-623)', () => { ).toHaveCount(0); // Confirming applied the AF-447 tag… - await page.getByRole('tab', { name: /Classification/ }).click(); + await clickTab(page, /Classification/); await expect( page.getByText(`public.${TABLE}.customer_email`).first(), ).toBeVisible({ timeout: 15_000 }); @@ -244,7 +244,7 @@ test.describe.serial('sensitive-data discovery (AF-623)', () => { await login(page, ADMIN_EMAIL, ADMIN_PASSWORD); await page.goto(`/datasources/${datasource.id}/settings`); - await page.getByRole('tab', { name: /Discovery/ }).click(); + await clickTab(page, /Discovery/); await page.getByRole('button', { name: 'Scan now' }).click(); await expect(page.getByText('Discovery scan started')).toBeVisible({ timeout: 15_000 }); @@ -277,7 +277,7 @@ test.describe.serial('sensitive-data discovery (AF-623)', () => { // In the UI it is reachable under the Stale filter, where it stays selectable so an admin can // bulk-dismiss it — the behaviour the whole change exists to provide. await page.reload(); - await page.getByRole('tab', { name: /Discovery/ }).click(); + await clickTab(page, /Discovery/); await page.getByTitle('Stale', { exact: true }).click(); const row = page.getByRole('row', { name: new RegExp(`public\\.${TABLE}\\.secondary_email`) }); // The scan covers every table in the shared e2e database, so this datasource's Stale list can diff --git a/frontend/src/api/dataBudgets.test.ts b/frontend/src/api/dataBudgets.test.ts new file mode 100644 index 000000000..cb5f96371 --- /dev/null +++ b/frontend/src/api/dataBudgets.test.ts @@ -0,0 +1,84 @@ +import { describe, expect, it, vi, beforeEach } from 'vitest'; + +const { get, post, put, del } = vi.hoisted(() => ({ + get: vi.fn(), + post: vi.fn(), + put: vi.fn(), + del: vi.fn(), +})); + +vi.mock('./client', () => ({ + apiClient: { get, post, put, delete: del }, +})); + +import * as api from './dataBudgets'; +import { dataBudgetKeys } from './dataBudgets'; +import type { DataBudgetInput } from '@/types/api'; + +const budget = { + id: 'b-1', + datasource_id: 'ds-1', + name: 'Daily', + max_rows: 1000, + window_minutes: 1440, + breach_action: 'REQUIRE_REVIEW' as const, + applies_to_roles: [], + applies_to_group_ids: [], + applies_to_user_ids: [], + enabled: true, + created_at: '2026-09-01T10:00:00Z', + updated_at: '2026-09-01T10:00:00Z', +}; + +const input: DataBudgetInput = { + name: 'Daily', + max_rows: 1000, + window_minutes: 1440, + breach_action: 'REQUIRE_REVIEW', + enabled: true, +}; + +describe('api/dataBudgets', () => { + beforeEach(() => { + get.mockReset(); + post.mockReset(); + put.mockReset(); + del.mockReset(); + }); + + it('builds query keys', () => { + expect(dataBudgetKeys.list('ds-1')).toEqual(['data-budgets', 'list', 'ds-1']); + expect(dataBudgetKeys.mine('ds-1')).toEqual(['data-budgets', 'me', 'ds-1']); + expect(dataBudgetKeys.forUser('u-1')).toEqual(['data-budgets', 'user', 'u-1']); + }); + + it('lists budgets for a datasource', async () => { + get.mockResolvedValue({ data: { content: [budget] } }); + await expect(api.listDataBudgets('ds-1')).resolves.toEqual([budget]); + expect(get).toHaveBeenCalledWith('/api/v1/datasources/ds-1/data-budgets'); + }); + + it('creates, updates and deletes a budget', async () => { + post.mockResolvedValue({ data: budget }); + put.mockResolvedValue({ data: budget }); + del.mockResolvedValue({}); + + await expect(api.createDataBudget('ds-1', input)).resolves.toEqual(budget); + expect(post).toHaveBeenCalledWith('/api/v1/datasources/ds-1/data-budgets', input); + await expect(api.updateDataBudget('ds-1', 'b-1', input)).resolves.toEqual(budget); + expect(put).toHaveBeenCalledWith('/api/v1/datasources/ds-1/data-budgets/b-1', input); + await api.deleteDataBudget('ds-1', 'b-1'); + expect(del).toHaveBeenCalledWith('/api/v1/datasources/ds-1/data-budgets/b-1'); + }); + + it('reads the caller standing and a user standing', async () => { + const status = { datasource_id: 'ds-1', exhausted: false, budgets: [] }; + get.mockResolvedValueOnce({ data: status }); + await expect(api.getMyDataBudgetStatus('ds-1')).resolves.toEqual(status); + expect(get).toHaveBeenCalledWith('/api/v1/datasources/ds-1/data-budgets/me'); + + get.mockResolvedValueOnce({ data: { content: [status] } }); + await expect(api.getUserDataBudgetUsage('u-1')).resolves.toEqual([status]); + expect(get).toHaveBeenCalledWith('/api/v1/admin/users/u-1/data-budget-usage'); + }); +}); diff --git a/frontend/src/api/dataBudgets.ts b/frontend/src/api/dataBudgets.ts new file mode 100644 index 000000000..d7274f31b --- /dev/null +++ b/frontend/src/api/dataBudgets.ts @@ -0,0 +1,49 @@ +import { apiClient } from './client'; +import type { DataBudget, DataBudgetInput, DataBudgetStatus } from '@/types/api'; + +const base = (datasourceId: string) => `/api/v1/datasources/${datasourceId}/data-budgets`; + +export const dataBudgetKeys = { + all: ['data-budgets'] as const, + list: (datasourceId: string) => ['data-budgets', 'list', datasourceId] as const, + mine: (datasourceId: string) => ['data-budgets', 'me', datasourceId] as const, + forUser: (userId: string) => ['data-budgets', 'user', userId] as const, +}; + +export async function listDataBudgets(datasourceId: string): Promise { + const { data } = await apiClient.get<{ content: DataBudget[] }>(base(datasourceId)); + return data.content; +} + +export async function createDataBudget( + datasourceId: string, + input: DataBudgetInput, +): Promise { + const { data } = await apiClient.post(base(datasourceId), input); + return data; +} + +export async function updateDataBudget( + datasourceId: string, + budgetId: string, + input: DataBudgetInput, +): Promise { + const { data } = await apiClient.put(`${base(datasourceId)}/${budgetId}`, input); + return data; +} + +export async function deleteDataBudget(datasourceId: string, budgetId: string): Promise { + await apiClient.delete(`${base(datasourceId)}/${budgetId}`); +} + +export async function getMyDataBudgetStatus(datasourceId: string): Promise { + const { data } = await apiClient.get(`${base(datasourceId)}/me`); + return data; +} + +export async function getUserDataBudgetUsage(userId: string): Promise { + const { data } = await apiClient.get<{ content: DataBudgetStatus[] }>( + `/api/v1/admin/users/${userId}/data-budget-usage`, + ); + return data.content; +} diff --git a/frontend/src/components/admin/UserDataBudgetDrawer.test.tsx b/frontend/src/components/admin/UserDataBudgetDrawer.test.tsx new file mode 100644 index 000000000..3a65fb97f --- /dev/null +++ b/frontend/src/components/admin/UserDataBudgetDrawer.test.tsx @@ -0,0 +1,95 @@ +import { describe, expect, it, vi, beforeEach } from 'vitest'; +import { render, screen, waitFor } from '@testing-library/react'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import '@/i18n'; +import type { DataBudgetStatus, User } from '@/types/api'; + +const getUserDataBudgetUsage = vi.fn(); + +vi.mock('@/api/dataBudgets', async (importOriginal) => { + const original = await importOriginal(); + return { + dataBudgetKeys: original.dataBudgetKeys, + getUserDataBudgetUsage: (...args: unknown[]) => getUserDataBudgetUsage(...args), + }; +}); + +const { UserDataBudgetDrawer } = await import('./UserDataBudgetDrawer'); + +const user = { + id: 'u-1', + email: 'ana@example.com', + display_name: 'Ana Analyst', + active: true, +} as User; + +const status: DataBudgetStatus = { + datasource_id: 'ds-1', + datasource_name: 'Warehouse', + exhausted: true, + breach_action: 'REJECT', + budgets: [ + { + id: 'b-1', + name: 'Daily rows', + max_rows: 1000, + max_bytes: 2_000_000_000, + window_minutes: 1440, + breach_action: 'REJECT', + used_rows: 1000, + used_bytes: 500_000_000, + remaining_rows: 0, + remaining_bytes: 1_500_000_000, + used_percent: 100, + exhausted: true, + }, + ], +}; + +function renderDrawer(u: User | null) { + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return render( + + undefined} /> + , + ); +} + +describe('UserDataBudgetDrawer', () => { + beforeEach(() => { + getUserDataBudgetUsage.mockReset(); + }); + + it('does not load anything while closed', () => { + renderDrawer(null); + expect(getUserDataBudgetUsage).not.toHaveBeenCalled(); + }); + + it('shows usage per datasource and budget', async () => { + getUserDataBudgetUsage.mockResolvedValue([status]); + renderDrawer(user); + + expect(await screen.findByText('Warehouse')).toBeInTheDocument(); + expect(screen.getByText('Data usage — Ana Analyst')).toBeInTheDocument(); + expect(screen.getByText('Used up')).toBeInTheDocument(); + expect(screen.getByText('Daily rows')).toBeInTheDocument(); + expect(screen.getByText(/1,000 of 1,000 rows/)).toBeInTheDocument(); + expect(getUserDataBudgetUsage).toHaveBeenCalledWith('u-1'); + }); + + it('says when no budget applies', async () => { + getUserDataBudgetUsage.mockResolvedValue([]); + renderDrawer(user); + + expect(await screen.findByText('No data budget applies to this user.')).toBeInTheDocument(); + }); + + it('reports a load failure', async () => { + getUserDataBudgetUsage.mockRejectedValue(new Error('boom')); + renderDrawer(user); + + await waitFor(() => + expect(screen.getByText('Could not load this user’s data usage')).toBeInTheDocument(), + ); + }); +}); diff --git a/frontend/src/components/admin/UserDataBudgetDrawer.tsx b/frontend/src/components/admin/UserDataBudgetDrawer.tsx new file mode 100644 index 000000000..1fb4838e0 --- /dev/null +++ b/frontend/src/components/admin/UserDataBudgetDrawer.tsx @@ -0,0 +1,119 @@ +import { Alert, Drawer, Empty, Progress, Skeleton, Tag } from 'antd'; +import { useQuery } from '@tanstack/react-query'; +import { useTranslation } from 'react-i18next'; +import { dataBudgetKeys, getUserDataBudgetUsage } from '@/api/dataBudgets'; +import type { DataBudgetConsumption, DataBudgetStatus, User } from '@/types/api'; +import { dataBudgetBreachActionLabel } from '@/utils/enumLabels'; +import { formatWindow } from '@/utils/dataBudget'; +import { formatBytes } from '@/utils/queryPlan'; +import { userDisplay } from '@/utils/userDisplay'; + +interface Props { + user: User | null; + onClose: () => void; +} + +/** + * An admin's view of one user's data-volume usage (#942) against every budget that applies to them, + * grouped by datasource. There is no user detail page, so the drawer opens from the users list. + */ +export function UserDataBudgetDrawer({ user, onClose }: Props) { + const { t } = useTranslation(); + const usageQuery = useQuery({ + queryKey: user ? dataBudgetKeys.forUser(user.id) : ['data-budgets', 'user', 'idle'], + queryFn: () => getUserDataBudgetUsage(user!.id), + enabled: !!user, + }); + + return ( + +
+ {t('dataBudgets.usage.description')} +
+ {usageQuery.isLoading ? ( + + ) : usageQuery.isError ? ( + + ) : (usageQuery.data ?? []).length === 0 ? ( + + ) : ( + (usageQuery.data ?? []).map((status) => ( + + )) + )} + + ); +} + +function DatasourceUsage({ status }: { status: DataBudgetStatus }) { + const { t } = useTranslation(); + return ( +
+
+ {status.datasource_name ?? status.datasource_id} + {status.exhausted && {t('dataBudgets.usage.exhausted')}} +
+ {status.budgets.map((budget) => ( + + ))} +
+ ); +} + +function BudgetUsage({ budget }: { budget: DataBudgetConsumption }) { + const { t } = useTranslation(); + const parts: string[] = []; + if (budget.max_rows != null) { + parts.push( + t('dataBudgets.usage.rows_used', { + used: budget.used_rows.toLocaleString(), + limit: budget.max_rows.toLocaleString(), + }), + ); + } + if (budget.max_bytes != null) { + parts.push( + t('dataBudgets.usage.bytes_used', { + used: formatBytes(budget.used_bytes), + limit: formatBytes(budget.max_bytes) ?? '', + }), + ); + } + const percent = Math.min(100, budget.used_percent); + return ( +
+
+ {budget.name} + + {t('dataBudgets.indicator.per_window', { window: formatWindow(t, budget.window_minutes) })} + {' · '} + {dataBudgetBreachActionLabel(t, budget.breach_action)} + +
+ +
{parts.join(' · ')}
+
+ ); +} diff --git a/frontend/src/components/common/NotificationBell.tsx b/frontend/src/components/common/NotificationBell.tsx index 6cb96c76b..00715a398 100644 --- a/frontend/src/components/common/NotificationBell.tsx +++ b/frontend/src/components/common/NotificationBell.tsx @@ -308,6 +308,19 @@ function renderMessage( pipeline: datasource, environment: payload.environment ?? '—', }); + // #942 — the budget's user rides in `submitter` (the admin copy of an exhaustion names them). + case 'DATA_BUDGET_THRESHOLD_REACHED': + return t('notifications.events.DATA_BUDGET_THRESHOLD_REACHED', { + percent: payload.used_percent ?? '—', + budget: payload.budget ?? '—', + datasource, + }); + case 'DATA_BUDGET_EXHAUSTED': + return t('notifications.events.DATA_BUDGET_EXHAUSTED', { + budget: payload.budget ?? '—', + user: payload.submitter_name ?? payload.submitter ?? '—', + datasource, + }); default: return t('notifications.events.fallback'); } @@ -386,6 +399,14 @@ export function routeForNotification(item: UserNotification): string | null { if (item.event_type === 'SCHEMA_DRIFT_DETECTED') { return '/schema-drift'; } + // #942: a data budget constrains reads, so the recipient lands where reads are written. The + // editor takes its datasource from router state, not the URL, so the route carries no id. + if ( + item.event_type === 'DATA_BUDGET_THRESHOLD_REACHED' || + item.event_type === 'DATA_BUDGET_EXHAUSTED' + ) { + return '/editor'; + } return item.query_request_id ? `/queries/${item.query_request_id}` : null; } diff --git a/frontend/src/components/common/__tests__/NotificationBell.test.tsx b/frontend/src/components/common/__tests__/NotificationBell.test.tsx index 7b2d4ea05..31c441312 100644 --- a/frontend/src/components/common/__tests__/NotificationBell.test.tsx +++ b/frontend/src/components/common/__tests__/NotificationBell.test.tsx @@ -677,6 +677,54 @@ describe('NotificationBell', () => { expect(navigateMock).toHaveBeenCalledWith('/schema-drift'); }); + it.each([ + [ + 'DATA_BUDGET_THRESHOLD_REACHED' as const, + { datasource: 'warehouse', datasource_id: 'ds-1', budget: 'daily-reads', used_percent: 80 }, + 'You have used 80% of your data budget daily-reads on warehouse', + ], + [ + 'DATA_BUDGET_EXHAUSTED' as const, + { + datasource: 'warehouse', + datasource_id: 'ds-1', + budget: 'daily-reads', + used_percent: 100, + submitter: 'ana@example.com', + submitter_name: 'Ana', + }, + 'Data budget daily-reads for Ana on warehouse is used up', + ], + ])('renders %s and opens the query editor (#942)', async (eventType, payload, expected) => { + fetchUnreadCountMock.mockResolvedValue({ count: 1 }); + markNotificationReadMock.mockResolvedValue(undefined); + listNotificationsMock.mockResolvedValue( + page([ + { + id: 'budget1', + event_type: eventType, + query_request_id: null, + api_request_id: null, + deployment_request_id: null, + payload, + read: false, + created_at: new Date().toISOString(), + read_at: null, + }, + ]), + ); + + render(wrap()); + fireEvent.click(screen.getByLabelText('Notifications')); + const text = await screen.findByText(expected); + const row = text.closest('.ant-list-item'); + if (!row) throw new Error('list row not found'); + fireEvent.click(row); + + await waitFor(() => expect(markNotificationReadMock).toHaveBeenCalledWith('budget1')); + expect(navigateMock).toHaveBeenCalledWith('/editor'); + }); + it('routes a plain FAILED outcome to the deployment, not the rollback worklist', async () => { // A rollback review only exists for ROLLED_BACK; a FAILED deploy would never appear there. fetchUnreadCountMock.mockResolvedValue({ count: 1 }); diff --git a/frontend/src/components/datasources/DataBudgetTab.test.tsx b/frontend/src/components/datasources/DataBudgetTab.test.tsx new file mode 100644 index 000000000..170c4acf4 --- /dev/null +++ b/frontend/src/components/datasources/DataBudgetTab.test.tsx @@ -0,0 +1,213 @@ +import { describe, expect, it, vi, beforeEach } from 'vitest'; +import { fireEvent, render, screen, waitFor, within } from '@testing-library/react'; +import { App as AntdApp } from 'antd'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import '@/i18n'; +import type { DataBudget } from '@/types/api'; + +const listDataBudgets = vi.fn(); +const createDataBudget = vi.fn(); +const updateDataBudget = vi.fn(); +const deleteDataBudget = vi.fn(); + +vi.mock('@/api/dataBudgets', async (importOriginal) => { + const original = await importOriginal(); + return { + dataBudgetKeys: original.dataBudgetKeys, + listDataBudgets: (...args: unknown[]) => listDataBudgets(...args), + createDataBudget: (...args: unknown[]) => createDataBudget(...args), + updateDataBudget: (...args: unknown[]) => updateDataBudget(...args), + deleteDataBudget: (...args: unknown[]) => deleteDataBudget(...args), + }; +}); + +vi.mock('@/api/admin', async (importOriginal) => { + const original = await importOriginal(); + return { + ...original, + listUsers: () => + Promise.resolve({ content: [], page: 0, size: 100, total_elements: 0, total_pages: 0 }), + }; +}); + +vi.mock('@/api/groups', async (importOriginal) => { + const original = await importOriginal(); + return { ...original, listAllGroups: () => Promise.resolve([]) }; +}); + +vi.mock('@/api/roles', async (importOriginal) => { + const original = await importOriginal(); + return { ...original, listRoles: () => Promise.resolve([]) }; +}); + +const { DataBudgetTab } = await import('./DataBudgetTab'); + +const budget: DataBudget = { + id: 'b-1', + datasource_id: 'ds-1', + name: 'Analysts daily', + max_rows: 100000, + max_bytes: 5_000_000_000, + window_minutes: 1440, + breach_action: 'REQUIRE_REVIEW', + warn_threshold_percent: 80, + applies_to_roles: ['ANALYST'], + applies_to_group_ids: [], + applies_to_user_ids: ['u-1', 'u-2'], + enabled: true, + created_at: '2026-09-01T10:00:00Z', + updated_at: '2026-09-01T10:00:00Z', +}; + +function renderTab() { + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return render( + + + + + , + ); +} + +async function openModal(trigger: HTMLElement) { + fireEvent.click(trigger); + return waitFor(() => { + const el = document.querySelector('.ant-modal') as HTMLElement; + expect(el).toBeTruthy(); + return el; + }); +} + +describe('DataBudgetTab', () => { + beforeEach(() => { + listDataBudgets.mockReset().mockResolvedValue([budget]); + createDataBudget.mockReset().mockResolvedValue(budget); + updateDataBudget.mockReset().mockResolvedValue(budget); + deleteDataBudget.mockReset().mockResolvedValue(undefined); + }); + + it('renders the empty state when there are no budgets', async () => { + listDataBudgets.mockResolvedValue([]); + renderTab(); + + expect(await screen.findByText('No data budgets')).toBeInTheDocument(); + }); + + it('lists budgets with limits, window, action and scope', async () => { + listDataBudgets.mockResolvedValue([ + budget, + { ...budget, id: 'b-2', name: 'Weekly', max_bytes: null, window_minutes: 10080, + breach_action: 'REJECT', applies_to_roles: [], applies_to_user_ids: [], enabled: false }, + ]); + renderTab(); + + expect(await screen.findByText('Analysts daily')).toBeInTheDocument(); + expect(screen.getByText('1 day')).toBeInTheDocument(); + expect(screen.getByText('7 days')).toBeInTheDocument(); + expect(screen.getByText('Send to human review')).toBeInTheDocument(); + expect(screen.getByText('Reject')).toBeInTheDocument(); + expect(screen.getByText('1 role · 2 users')).toBeInTheDocument(); + expect(screen.getByText('Everyone')).toBeInTheDocument(); + }); + + it('creates a row budget with the default window, action and threshold', async () => { + listDataBudgets.mockResolvedValue([]); + renderTab(); + const modal = await openModal(await screen.findByText('Add budget')); + + fireEvent.change(within(modal).getByLabelText('Name'), { target: { value: 'Daily cap' } }); + fireEvent.change(within(modal).getByLabelText('Row limit'), { target: { value: '5000' } }); + fireEvent.click(within(modal).getByText('Save')); + + await waitFor(() => expect(createDataBudget).toHaveBeenCalled()); + expect(createDataBudget.mock.calls[0]?.[0]).toBe('ds-1'); + expect(createDataBudget.mock.calls[0]?.[1]).toEqual({ + name: 'Daily cap', + max_rows: 5000, + max_bytes: null, + window_minutes: 1440, + breach_action: 'REQUIRE_REVIEW', + warn_threshold_percent: 80, + applies_to_roles: [], + applies_to_group_ids: [], + applies_to_user_ids: [], + enabled: true, + }); + }); + + it('requires a name and at least one limit', async () => { + listDataBudgets.mockResolvedValue([]); + renderTab(); + const modal = await openModal(await screen.findByText('Add budget')); + + fireEvent.click(within(modal).getByText('Save')); + + expect(await within(modal).findByText('Enter a name')).toBeInTheDocument(); + expect( + (await within(modal).findAllByText('Set a row limit or a result-size limit')).length, + ).toBeGreaterThan(0); + expect(createDataBudget).not.toHaveBeenCalled(); + }); + + it('rejects a window outside one hour to 31 days', async () => { + listDataBudgets.mockResolvedValue([]); + renderTab(); + const modal = await openModal(await screen.findByText('Add budget')); + + fireEvent.change(within(modal).getByLabelText('Name'), { target: { value: 'Too long' } }); + fireEvent.change(within(modal).getByLabelText('Row limit'), { target: { value: '10' } }); + fireEvent.change(within(modal).getByLabelText('Rolling window'), { target: { value: '40' } }); + fireEvent.click(within(modal).getByText('Save')); + + expect( + await within(modal).findByText('The window must be between 1 hour and 31 days'), + ).toBeInTheDocument(); + expect(createDataBudget).not.toHaveBeenCalled(); + }); + + it('edits an existing budget through update, keeping its window', async () => { + renderTab(); + const modal = await openModal(await screen.findByLabelText('Edit budget')); + await waitFor(() => + expect(within(modal).getByLabelText('Name')).toHaveValue('Analysts daily'), + ); + + fireEvent.change(within(modal).getByLabelText('Name'), { target: { value: 'Renamed' } }); + fireEvent.click(within(modal).getByText('Save')); + + await waitFor(() => expect(updateDataBudget).toHaveBeenCalled()); + expect(updateDataBudget.mock.calls[0]?.[1]).toBe('b-1'); + expect(updateDataBudget.mock.calls[0]?.[2]).toMatchObject({ + name: 'Renamed', + max_rows: 100000, + max_bytes: 5_000_000_000, + window_minutes: 1440, + applies_to_roles: ['ANALYST'], + applies_to_user_ids: ['u-1', 'u-2'], + }); + }); + + it('deletes a budget after confirmation', async () => { + renderTab(); + fireEvent.click(await screen.findByLabelText('Delete budget')); + const [confirm] = await screen.findAllByText('Delete this data budget?'); + const confirmDialog = confirm?.closest('.ant-modal') as HTMLElement; + fireEvent.click(within(confirmDialog).getByRole('button', { name: 'Delete budget' })); + + await waitFor(() => expect(deleteDataBudget).toHaveBeenCalled()); + expect(deleteDataBudget.mock.calls[0]).toEqual(['ds-1', 'b-1']); + }); + + it('surfaces a save failure', async () => { + listDataBudgets.mockResolvedValue([]); + createDataBudget.mockRejectedValue(new Error('Budget rejected')); + renderTab(); + const modal = await openModal(await screen.findByText('Add budget')); + fireEvent.change(within(modal).getByLabelText('Name'), { target: { value: 'X' } }); + fireEvent.change(within(modal).getByLabelText('Row limit'), { target: { value: '1' } }); + fireEvent.click(within(modal).getByText('Save')); + + expect(await screen.findByText('Budget rejected')).toBeInTheDocument(); + }); +}); diff --git a/frontend/src/components/datasources/DataBudgetTab.tsx b/frontend/src/components/datasources/DataBudgetTab.tsx new file mode 100644 index 000000000..910e5e94d --- /dev/null +++ b/frontend/src/components/datasources/DataBudgetTab.tsx @@ -0,0 +1,554 @@ +import { useEffect, useMemo, useState } from 'react'; +import { + App, + Button, + Form, + Input, + InputNumber, + Modal, + Select, + Space, + Switch, + Table, + Tooltip, +} from 'antd'; +import { + CheckCircleOutlined, + DeleteOutlined, + EditOutlined, + MinusCircleOutlined, + PlusOutlined, +} from '@ant-design/icons'; +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; +import { useTranslation } from 'react-i18next'; +import { EmptyState } from '@/components/common/EmptyState'; +import { BytesInput } from '@/components/common/BytesInput'; +import { renderUserOption } from '@/components/common/renderUserOption'; +import { + createDataBudget, + dataBudgetKeys, + deleteDataBudget, + listDataBudgets, + updateDataBudget, +} from '@/api/dataBudgets'; +import { listUsers, userKeys } from '@/api/admin'; +import { groupKeys, listAllGroups } from '@/api/groups'; +import { listRoles, roleKeys } from '@/api/roles'; +import { roleSelectOptions } from '@/utils/roleOptions'; +import { userDisplay } from '@/utils/userDisplay'; +import { apiErrorMessage } from '@/utils/apiErrors'; +import { showApiError } from '@/utils/showApiError'; +import { formatBytes } from '@/utils/queryPlan'; +import { + DATA_BUDGET_BREACH_ACTIONS, + dataBudgetBreachActionLabel, + enumOptions, +} from '@/utils/enumLabels'; +import { + DATA_BUDGET_DEFAULT_WINDOW_MINUTES, + DATA_BUDGET_NAME_MAX, + DATA_BUDGET_THRESHOLD_MAX, + DATA_BUDGET_THRESHOLD_MIN, + DATA_BUDGET_WINDOW_MAX_MINUTES, + DATA_BUDGET_WINDOW_MIN_MINUTES, + DATA_BUDGET_WINDOW_UNITS, + formatWindow, + joinWindow, + splitWindow, + type DataBudgetWindowUnit, +} from '@/utils/dataBudget'; +import type { DataBudget, DataBudgetBreachAction, DataBudgetInput, User } from '@/types/api'; + +export function DataBudgetTab({ dsId }: { dsId: string }) { + const { t } = useTranslation(); + const { message, modal } = App.useApp(); + const queryClient = useQueryClient(); + const [editing, setEditing] = useState(null); + const [modalOpen, setModalOpen] = useState(false); + + const budgetsQuery = useQuery({ + queryKey: dataBudgetKeys.list(dsId), + queryFn: () => listDataBudgets(dsId), + }); + const budgets = budgetsQuery.data ?? []; + + const deleteMutation = useMutation({ + mutationFn: (budgetId: string) => deleteDataBudget(dsId, budgetId), + onSuccess: () => { + void queryClient.invalidateQueries({ queryKey: dataBudgetKeys.all }); + message.success(t('dataBudgets.tab.delete_success')); + }, + onError: (err) => { + showApiError(message, err, (e) => + apiErrorMessage(e, () => t('dataBudgets.tab.delete_error')), + ); + }, + }); + + const onAdd = () => { + setEditing(null); + setModalOpen(true); + }; + + const onDelete = (budget: DataBudget) => { + modal.confirm({ + title: t('dataBudgets.tab.delete_confirm_title'), + content: t('dataBudgets.tab.delete_confirm_body'), + okType: 'danger', + okText: t('dataBudgets.tab.delete'), + cancelText: t('common.cancel'), + onOk: () => deleteMutation.mutateAsync(budget.id), + }); + }; + + return ( +
+
+
+
{t('dataBudgets.tab.title')}
+
+ {t('dataBudgets.tab.description')} +
+
+ +
+ + {!budgetsQuery.isLoading && budgets.length === 0 ? ( + + ) : ( + + rowKey="id" + size="middle" + loading={budgetsQuery.isLoading} + dataSource={budgets} + pagination={false} + scroll={{ x: 'max-content' }} + columns={[ + { title: t('dataBudgets.tab.col_name'), dataIndex: 'name' }, + { + title: t('dataBudgets.tab.col_limits'), + render: (_v, b) => , + }, + { + title: t('dataBudgets.tab.col_window'), + width: 110, + render: (_v, b) => formatWindow(t, b.window_minutes), + }, + { + title: t('dataBudgets.tab.col_action'), + width: 180, + render: (_v, b) => dataBudgetBreachActionLabel(t, b.breach_action), + }, + { + title: t('dataBudgets.tab.col_applies_to'), + render: (_v, b) => , + }, + { + title: t('dataBudgets.tab.col_enabled'), + width: 90, + align: 'center', + render: (_v, b) => { + const label = b.enabled + ? t('dataBudgets.tab.label_enabled') + : t('dataBudgets.tab.state_disabled'); + return ( + + {b.enabled ? ( + + ) : ( + + )} + + ); + }, + }, + { + title: t('dataBudgets.tab.col_actions'), + width: 120, + align: 'right', + render: (_v, b) => ( + <> +
+ ); +} + +function LimitsSummary({ budget }: { budget: DataBudget }) { + const { t } = useTranslation(); + const parts: string[] = []; + if (budget.max_rows != null) { + parts.push(t('dataBudgets.tab.rows_value', { value: budget.max_rows.toLocaleString() })); + } + if (budget.max_bytes != null) { + parts.push(formatBytes(budget.max_bytes) ?? ''); + } + return {parts.join(' · ')}; +} + +function AppliesToSummary({ budget }: { budget: DataBudget }) { + const { t } = useTranslation(); + const parts: string[] = []; + if (budget.applies_to_roles.length > 0) { + parts.push(t('dataBudgets.tab.applies_roles', { count: budget.applies_to_roles.length })); + } + if (budget.applies_to_group_ids.length > 0) { + parts.push(t('dataBudgets.tab.applies_groups', { count: budget.applies_to_group_ids.length })); + } + if (budget.applies_to_user_ids.length > 0) { + parts.push(t('dataBudgets.tab.applies_users', { count: budget.applies_to_user_ids.length })); + } + if (parts.length === 0) { + return ( + + {t('dataBudgets.tab.applies_everyone')} + + ); + } + return {parts.join(' · ')}; +} + +interface DataBudgetFormValues { + name: string; + max_rows?: number | null; + max_bytes?: number | null; + window_amount: number; + window_unit: DataBudgetWindowUnit; + breach_action: DataBudgetBreachAction; + warn_threshold_percent?: number | null; + applies_to_roles?: string[]; + applies_to_group_ids?: string[]; + applies_to_user_ids?: string[]; + enabled: boolean; +} + +interface DataBudgetModalProps { + open: boolean; + dsId: string; + budget: DataBudget | null; + onClose: () => void; +} + +const DEFAULT_VALUES: Partial = { + ...(() => { + const { amount, unit } = splitWindow(DATA_BUDGET_DEFAULT_WINDOW_MINUTES); + return { window_amount: amount, window_unit: unit }; + })(), + breach_action: 'REQUIRE_REVIEW', + warn_threshold_percent: 80, + enabled: true, +}; + +function DataBudgetModal({ open, dsId, budget, onClose }: DataBudgetModalProps) { + const { t } = useTranslation(); + const { message } = App.useApp(); + const queryClient = useQueryClient(); + const [form] = Form.useForm(); + + const usersQuery = useQuery({ + queryKey: userKeys.list({ size: 100 }), + queryFn: () => listUsers({ size: 100 }), + enabled: open, + }); + const groupsQuery = useQuery({ + queryKey: groupKeys.lists(), + queryFn: () => listAllGroups(), + enabled: open, + }); + const rolesQuery = useQuery({ + queryKey: roleKeys.lists(), + queryFn: listRoles, + enabled: open, + }); + const roleOptions = useMemo( + () => roleSelectOptions(rolesQuery.data ?? [], t, 'name'), + [rolesQuery.data, t], + ); + const userOptions = useMemo( + () => + (usersQuery.data?.content ?? []) + .filter((u: User) => u.active) + .map((u: User) => ({ + value: u.id, + label: userDisplay(u.display_name, u.email), + principal_type: u.principal_type ?? 'HUMAN', + })), + [usersQuery.data], + ); + const groupOptions = useMemo( + () => (groupsQuery.data ?? []).map((g) => ({ value: g.id, label: g.name })), + [groupsQuery.data], + ); + const unitOptions = DATA_BUDGET_WINDOW_UNITS.map((unit) => ({ + value: unit, + label: t(`dataBudgets.tab.unit_${unit}` as const), + })); + + useEffect(() => { + if (!open) return; + if (budget) { + const { amount, unit } = splitWindow(budget.window_minutes); + form.setFieldsValue({ + name: budget.name, + max_rows: budget.max_rows ?? null, + max_bytes: budget.max_bytes ?? null, + window_amount: amount, + window_unit: unit, + breach_action: budget.breach_action, + warn_threshold_percent: budget.warn_threshold_percent ?? null, + applies_to_roles: budget.applies_to_roles, + applies_to_group_ids: budget.applies_to_group_ids, + applies_to_user_ids: budget.applies_to_user_ids, + enabled: budget.enabled, + }); + } else { + form.resetFields(); + } + }, [open, budget, form]); + + const saveMutation = useMutation({ + mutationFn: (input: DataBudgetInput) => + budget ? updateDataBudget(dsId, budget.id, input) : createDataBudget(dsId, input), + onSuccess: () => { + void queryClient.invalidateQueries({ queryKey: dataBudgetKeys.all }); + message.success(t('dataBudgets.tab.save_success')); + onClose(); + }, + onError: (err) => { + showApiError(message, err, (e) => apiErrorMessage(e, () => t('dataBudgets.tab.save_error'))); + }, + }); + + const onFinish = (values: DataBudgetFormValues) => { + saveMutation.mutate({ + name: values.name.trim(), + max_rows: values.max_rows ?? null, + max_bytes: values.max_bytes ?? null, + window_minutes: joinWindow(values.window_amount, values.window_unit), + breach_action: values.breach_action, + warn_threshold_percent: values.warn_threshold_percent ?? null, + applies_to_roles: values.applies_to_roles ?? [], + applies_to_group_ids: values.applies_to_group_ids ?? [], + applies_to_user_ids: values.applies_to_user_ids ?? [], + enabled: values.enabled, + }); + }; + + // Backend: @Min(60) @Max(44640) on window_minutes (#942). + const windowRule = { + validator: (_: unknown, amount: number | null | undefined) => { + const unit: DataBudgetWindowUnit = form.getFieldValue('window_unit') ?? 'hours'; + if (amount === null || amount === undefined) { + return Promise.reject(new Error(t('dataBudgets.tab.window_range'))); + } + const minutes = joinWindow(amount, unit); + return minutes >= DATA_BUDGET_WINDOW_MIN_MINUTES && minutes <= DATA_BUDGET_WINDOW_MAX_MINUTES + ? Promise.resolve() + : Promise.reject(new Error(t('dataBudgets.tab.window_range'))); + }, + }; + // Backend: at least one of max_rows / max_bytes (chk_data_budgets_has_limit, 422). + const limitRequiredRule = ({ getFieldValue }: { getFieldValue: (name: string) => unknown }) => ({ + validator: () => + getFieldValue('max_rows') == null && getFieldValue('max_bytes') == null + ? Promise.reject(new Error(t('dataBudgets.tab.limit_required'))) + : Promise.resolve(), + }); + + return ( + form.submit()} + okText={t('common.save')} + cancelText={t('common.cancel')} + confirmLoading={saveMutation.isPending} + destroyOnHidden + width={600} + > + + form={form} + name="data-budget" + layout="vertical" + initialValues={DEFAULT_VALUES} + onFinish={onFinish} + > + + + + +
+ {t('dataBudgets.tab.limits_hint')} +
+
+ + + + + + +
+ + + + + + + + + + + + + + + + + mode="multiple" + allowClear + options={roleOptions} + loading={rolesQuery.isLoading} + /> + + + + + mode="multiple" + allowClear + showSearch={{ optionFilterProp: 'label' }} + options={groupOptions} + loading={groupsQuery.isLoading} + /> + + + + + mode="multiple" + allowClear + showSearch={{ optionFilterProp: 'label' }} + options={userOptions} + optionRender={renderUserOption} + loading={usersQuery.isLoading} + /> + + + + + + + + ); +} diff --git a/frontend/src/components/datasources/SampleDataPreview.test.tsx b/frontend/src/components/datasources/SampleDataPreview.test.tsx index a42e418c8..f9ca74791 100644 --- a/frontend/src/components/datasources/SampleDataPreview.test.tsx +++ b/frontend/src/components/datasources/SampleDataPreview.test.tsx @@ -84,4 +84,19 @@ describe('SampleDataPreview', () => { renderPreview(); expect(screen.getByText(/capped by the result size limit/i)).toBeInTheDocument(); }); + + it('renders the data-budget footer when truncated_reason is DATA_BUDGET', () => { + useTableSampleMock.mockReturnValue({ + data: { + columns: [{ name: 'id', type: 'uuid', restricted: false }], + rows: [['1']], + row_count: 1, + truncated: true, + truncated_reason: 'DATA_BUDGET', + duration_ms: 4, + }, + }); + renderPreview(); + expect(screen.getByText(/capped at your remaining data budget/i)).toBeInTheDocument(); + }); }); diff --git a/frontend/src/components/datasources/SampleDataPreview.tsx b/frontend/src/components/datasources/SampleDataPreview.tsx index b779f4806..a1f57409e 100644 --- a/frontend/src/components/datasources/SampleDataPreview.tsx +++ b/frontend/src/components/datasources/SampleDataPreview.tsx @@ -3,6 +3,7 @@ import type { TableColumnsType } from 'antd'; import { LockOutlined } from '@ant-design/icons'; import { useTranslation } from 'react-i18next'; import { useTableSample } from '@/hooks/useTableSample'; +import type { TruncatedReason } from '@/types/api'; interface SampleDataPreviewProps { datasourceId: string; @@ -81,12 +82,7 @@ export function SampleDataPreview({ datasourceId, schema, table }: SampleDataPre footer={() => data.truncated ? ( - {t( - data.truncated_reason === 'BYTE_LIMIT' - ? 'datasources.settings.sample_truncated_bytes' - : 'datasources.settings.sample_truncated', - { count: data.row_count }, - )} + {t(sampleTruncatedKey(data.truncated_reason), { count: data.row_count })} ) : ( @@ -104,3 +100,9 @@ function formatCell(value: unknown): string { if (typeof value === 'object') return JSON.stringify(value); return String(value); } + +function sampleTruncatedKey(reason: TruncatedReason | null | undefined) { + if (reason === 'BYTE_LIMIT') return 'datasources.settings.sample_truncated_bytes' as const; + if (reason === 'DATA_BUDGET') return 'datasources.settings.sample_truncated_budget' as const; + return 'datasources.settings.sample_truncated' as const; +} diff --git a/frontend/src/components/editor/DataBudgetIndicator.test.tsx b/frontend/src/components/editor/DataBudgetIndicator.test.tsx new file mode 100644 index 000000000..caea9cd15 --- /dev/null +++ b/frontend/src/components/editor/DataBudgetIndicator.test.tsx @@ -0,0 +1,107 @@ +import { describe, expect, it, vi, beforeEach } from 'vitest'; +import { render, screen, waitFor } from '@testing-library/react'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import type { ReactNode } from 'react'; +import type { DataBudgetConsumption, DataBudgetStatus } from '@/types/api'; + +const { getMyDataBudgetStatusMock } = vi.hoisted(() => ({ + getMyDataBudgetStatusMock: vi.fn(), +})); + +vi.mock('@/api/dataBudgets', () => ({ + getMyDataBudgetStatus: getMyDataBudgetStatusMock, + dataBudgetKeys: { mine: (id: string) => ['data-budgets', 'me', id] as const }, +})); + +import { DataBudgetIndicator } from './DataBudgetIndicator'; + +function withClient(ui: ReactNode) { + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return {ui}; +} + +const budget = (overrides: Partial = {}): DataBudgetConsumption => ({ + id: 'b-1', + name: 'Analysts daily', + max_rows: 1000, + max_bytes: 5_000_000_000, + window_minutes: 1440, + breach_action: 'REQUIRE_REVIEW', + warn_threshold_percent: 80, + used_rows: 300, + used_bytes: 1_000_000_000, + remaining_rows: 700, + remaining_bytes: 4_000_000_000, + used_percent: 30, + exhausted: false, + ...overrides, +}); + +const status = (overrides: Partial = {}): DataBudgetStatus => ({ + datasource_id: 'ds-1', + exhausted: false, + budgets: [budget()], + ...overrides, +}); + +describe('DataBudgetIndicator', () => { + beforeEach(() => { + getMyDataBudgetStatusMock.mockReset(); + }); + + it('renders nothing when no budget applies', async () => { + getMyDataBudgetStatusMock.mockResolvedValue(status({ budgets: [] })); + const { container } = render(withClient()); + await waitFor(() => expect(getMyDataBudgetStatusMock).toHaveBeenCalledWith('ds-1')); + expect(container).toBeEmptyDOMElement(); + }); + + it('renders nothing when the standing cannot be read', async () => { + getMyDataBudgetStatusMock.mockRejectedValue(new Error('404')); + const { container } = render(withClient()); + await waitFor(() => expect(getMyDataBudgetStatusMock).toHaveBeenCalled()); + expect(container).toBeEmptyDOMElement(); + }); + + it('shows the remaining rows and bytes for each budget', async () => { + getMyDataBudgetStatusMock.mockResolvedValue(status()); + render(withClient()); + expect(await screen.findByText('Analysts daily')).toBeInTheDocument(); + expect(screen.getByTestId('data-budget-indicator')).toHaveTextContent('700'); + expect(screen.getByTestId('data-budget-indicator')).toHaveTextContent('1,000'); + expect(screen.queryByRole('alert')).not.toBeInTheDocument(); + }); + + it('warns once a budget passes its threshold', async () => { + getMyDataBudgetStatusMock.mockResolvedValue( + status({ budgets: [budget({ used_percent: 85, max_bytes: null, remaining_bytes: null })] }), + ); + render(withClient()); + expect(await screen.findByText('You have used most of your data budget on this datasource.')).toBeInTheDocument(); + }); + + it('explains what happens once a review budget is used up', async () => { + getMyDataBudgetStatusMock.mockResolvedValue( + status({ + exhausted: true, + breach_action: 'REQUIRE_REVIEW', + budgets: [budget({ used_percent: 120, exhausted: true, max_rows: null })], + }), + ); + render(withClient()); + expect(await screen.findByText('Data budget used up')).toBeInTheDocument(); + expect(screen.getByText(/need a reviewer’s approval/)).toBeInTheDocument(); + }); + + it('explains a rejecting budget that is used up', async () => { + getMyDataBudgetStatusMock.mockResolvedValue( + status({ + exhausted: true, + breach_action: 'REJECT', + budgets: [budget({ used_percent: 100, exhausted: true, remaining_rows: null })], + }), + ); + render(withClient()); + expect(await screen.findByText(/are refused until earlier reads/)).toBeInTheDocument(); + }); +}); diff --git a/frontend/src/components/editor/DataBudgetIndicator.tsx b/frontend/src/components/editor/DataBudgetIndicator.tsx new file mode 100644 index 000000000..a4bbe5aa9 --- /dev/null +++ b/frontend/src/components/editor/DataBudgetIndicator.tsx @@ -0,0 +1,103 @@ +import { Alert, Progress } from 'antd'; +import { DashboardOutlined } from '@ant-design/icons'; +import { useQuery } from '@tanstack/react-query'; +import { useTranslation } from 'react-i18next'; +import { dataBudgetKeys, getMyDataBudgetStatus } from '@/api/dataBudgets'; +import type { DataBudgetConsumption } from '@/types/api'; +import { budgetNearlyUsed, formatWindow } from '@/utils/dataBudget'; +import { formatBytes } from '@/utils/queryPlan'; + +const CARD_STYLE = { + background: 'var(--bg-elev)', + border: '1px solid var(--border)', + borderRadius: 'var(--radius-md)', + padding: 14, +} as const; + +/** + * The caller's remaining data-volume allowance on the selected datasource (#942), shown before + * they submit. Renders nothing when no budget applies — or the standing cannot be read — so an + * unbudgeted datasource looks exactly as it did before. + */ +export function DataBudgetIndicator({ dsId }: { dsId: string }) { + const { t } = useTranslation(); + const statusQuery = useQuery({ + queryKey: dataBudgetKeys.mine(dsId), + queryFn: () => getMyDataBudgetStatus(dsId), + retry: false, + }); + const status = statusQuery.data; + if (!status || status.budgets.length === 0) { + return null; + } + + return ( +
+
+ +
{t('dataBudgets.indicator.title')}
+
+ {status.exhausted ? ( + + ) : budgetNearlyUsed(status) ? ( + + ) : null} + {status.budgets.map((budget) => ( + + ))} +
+ ); +} + +function BudgetLine({ budget }: { budget: DataBudgetConsumption }) { + const { t } = useTranslation(); + const window = formatWindow(t, budget.window_minutes); + const parts: string[] = []; + if (budget.max_rows != null) { + parts.push( + t('dataBudgets.indicator.rows_left', { + remaining: (budget.remaining_rows ?? 0).toLocaleString(), + limit: budget.max_rows.toLocaleString(), + }), + ); + } + if (budget.max_bytes != null) { + parts.push( + t('dataBudgets.indicator.bytes_left', { + remaining: formatBytes(budget.remaining_bytes ?? 0), + limit: formatBytes(budget.max_bytes) ?? '', + }), + ); + } + const percent = Math.min(100, budget.used_percent); + return ( +
+
+ {budget.name} + {t('dataBudgets.indicator.per_window', { window })} +
+ +
{parts.join(' · ')}
+
+ ); +} diff --git a/frontend/src/components/policies/decisionTraceDetails.test.ts b/frontend/src/components/policies/decisionTraceDetails.test.ts index 0e4bbcf94..690e63412 100644 --- a/frontend/src/components/policies/decisionTraceDetails.test.ts +++ b/frontend/src/components/policies/decisionTraceDetails.test.ts @@ -205,4 +205,22 @@ describe('formatStepDetails — masking, omission and enum values (#1066 review) 'Rejected: the estimate exceeds the bytes-scanned cap', ); }); + + it('formats the data-budget step with labels, sizes and percentages (#942)', () => { + const rows = formatStepDetails( + { + data_budget_name: 'Analysts daily', + data_budget_action: 'REQUIRE_REVIEW', + data_budget_used_percent: 105, + data_budget_remaining_rows: 0, + data_budget_remaining_bytes: 2_000_000_000, + }, + t, + ); + const byKey = Object.fromEntries(rows.map((r) => [r.key, r.value])); + expect(byKey.data_budget_action).toBe('Send to human review'); + expect(byKey.data_budget_used_percent).toBe('105%'); + expect(byKey.data_budget_remaining_bytes).toBe('2 GB'); + expect(rows.find((r) => r.key === 'data_budget_name')?.label).toBe('Data budget name'); + }); }); diff --git a/frontend/src/components/policies/decisionTraceDetails.ts b/frontend/src/components/policies/decisionTraceDetails.ts index 55cb70050..24f5760b2 100644 --- a/frontend/src/components/policies/decisionTraceDetails.ts +++ b/frontend/src/components/policies/decisionTraceDetails.ts @@ -2,6 +2,7 @@ import type { TFunction } from 'i18next'; import type { BytesScannedCapOutcome, BytesScannedCapSource, + DataBudgetBreachAction, MaskingStrategy, QueryShape, QueryStatus, @@ -15,6 +16,7 @@ import { formatBytes } from '@/utils/queryPlan'; import { BYTES_SCANNED_CAP_OUTCOMES, BYTES_SCANNED_CAP_SOURCES, + DATA_BUDGET_BREACH_ACTIONS, MASKING_STRATEGIES, QUERY_SHAPES, QUERY_TYPES, @@ -22,6 +24,7 @@ import { ROUTING_ACTIONS, bytesScannedCapOutcomeLabel, bytesScannedCapSourceLabel, + dataBudgetBreachActionLabel, maskingStrategyLabel, queryShapeLabel, queryStatusLabel, @@ -68,6 +71,13 @@ export const KNOWN_DETAIL_KEYS = [ 'considered_grant_ids', 'contributing_grants', 'current', + 'data_budget_action', + 'data_budget_id', + 'data_budget_name', + 'data_budget_remaining_bytes', + 'data_budget_remaining_rows', + 'data_budget_suppressed', + 'data_budget_used_percent', 'db_type', 'denied_shapes', 'denied_tables', @@ -172,9 +182,17 @@ function enumValue(key: string, value: unknown, t: TFunction): string | null { && includes(BYTES_SCANNED_CAP_OUTCOMES, value)) { return bytesScannedCapOutcomeLabel(t, value); } - if ((key === 'bytes_scanned_cap' || key === 'estimated_bytes_scanned') && typeof value === 'number') { + if ((key === 'bytes_scanned_cap' || key === 'estimated_bytes_scanned' + || key === 'data_budget_remaining_bytes') && typeof value === 'number') { return formatBytes(value); } + if (key === 'data_budget_action' + && includes(DATA_BUDGET_BREACH_ACTIONS, value)) { + return dataBudgetBreachActionLabel(t, value); + } + if (key === 'data_budget_used_percent' && typeof value === 'number') { + return `${value}%`; + } return null; } diff --git a/frontend/src/components/queries/QueryResultsTable.test.tsx b/frontend/src/components/queries/QueryResultsTable.test.tsx index b5a003900..0b4eb424a 100644 --- a/frontend/src/components/queries/QueryResultsTable.test.tsx +++ b/frontend/src/components/queries/QueryResultsTable.test.tsx @@ -103,6 +103,16 @@ describe('QueryResultsTable', () => { ).toBeInTheDocument(); }); + it('renders the data-budget footer when truncated_reason is DATA_BUDGET', async () => { + getQueryResultsMock.mockResolvedValue( + page({ truncated: true, truncated_reason: 'DATA_BUDGET' }), + ); + renderTable(); + expect( + await screen.findByText(/capped at your remaining data budget/i), + ).toBeInTheDocument(); + }); + it('hides the export button while the decision is unavailable', async () => { getQueryResultsMock.mockResolvedValue(page({})); renderTable(); diff --git a/frontend/src/components/queries/QueryResultsTable.tsx b/frontend/src/components/queries/QueryResultsTable.tsx index c163999f6..a15cc5620 100644 --- a/frontend/src/components/queries/QueryResultsTable.tsx +++ b/frontend/src/components/queries/QueryResultsTable.tsx @@ -234,9 +234,9 @@ export function QueryResultsTable({ queryId, defaultView = 'table' }: Props) { } function truncatedMessageKey(reason: TruncatedReason | null | undefined) { - return reason === 'BYTE_LIMIT' - ? ('queries.detail.results_truncated_bytes' as const) - : ('queries.detail.results_truncated' as const); + if (reason === 'BYTE_LIMIT') return 'queries.detail.results_truncated_bytes' as const; + if (reason === 'DATA_BUDGET') return 'queries.detail.results_truncated_budget' as const; + return 'queries.detail.results_truncated' as const; } function formatCell(value: unknown): string { diff --git a/frontend/src/locales/de.json b/frontend/src/locales/de.json index e7278c1ab..a659e6e9b 100644 --- a/frontend/src/locales/de.json +++ b/frontend/src/locales/de.json @@ -868,7 +868,8 @@ "diff_effective_label": "Effektives SQL" }, "bytes_cap_body": "Geschätzter Scan: {{estimate}} · Limit: {{limit}} ({{source}})", - "bytes_cap_no_estimate": "keine Schätzung" + "bytes_cap_no_estimate": "keine Schätzung", + "results_truncated_budget": "{{count}} Zeilen zurückgegeben (auf Ihr verbleibendes Datenbudget begrenzt)" } }, "reviews": { @@ -1590,7 +1591,9 @@ "grant_bytes_cap_label": "Limit für gescannte Bytes", "grant_bytes_cap_help": "Ein niedrigeres Limit für diesen Grant. Es gilt das strengste aus Datenquellen-Limit und allen Grants.", "grant_bytes_cap_min": "Das Limit für gescannte Bytes muss mindestens 1 Byte betragen.", - "grant_bytes_cap_placeholder": "Standard der Datenquelle" + "grant_bytes_cap_placeholder": "Standard der Datenquelle", + "tab_data_budgets": "Datenbudgets · {{count}}", + "sample_truncated_budget": "{{count}} Zeilen (auf Ihr verbleibendes Datenbudget begrenzt)" } }, "admin": { @@ -2471,7 +2474,8 @@ "minutes_suffix": "Min.", "cicd_present_label": "Ist CI/CD-Herkunft", "scan_type_placeholder": "z. B. Seq Scan, COLLSCAN, Index*", - "bytes_value_label": "Gescannte Bytes" + "bytes_value_label": "Gescannte Bytes", + "budget_percent_label": "Verbrauchter Anteil des Datenbudgets" }, "langfuse": { "title": "Langfuse", @@ -2912,7 +2916,9 @@ "SCHEMA_CHANGE_PROMOTION_FAILED": "Ihre Schemaänderung {{changeSet}} ist auf {{environment}} fehlgeschlagen", "SCHEMA_CHANGE_PROMOTION_FAILED_partial": "Ihre Schemaänderung {{changeSet}} wurde auf {{environment}} nur teilweise angewendet", "SCHEMA_DRIFT_DETECTED_one": "{{count}} neuer Schema-Drift-Befund in {{pipeline}} / {{environment}}", - "SCHEMA_DRIFT_DETECTED_other": "{{count}} neue Schema-Drift-Befunde in {{pipeline}} / {{environment}}" + "SCHEMA_DRIFT_DETECTED_other": "{{count}} neue Schema-Drift-Befunde in {{pipeline}} / {{environment}}", + "DATA_BUDGET_THRESHOLD_REACHED": "Sie haben {{percent}} % Ihres Datenbudgets {{budget}} auf {{datasource}} verbraucht", + "DATA_BUDGET_EXHAUSTED": "Datenbudget {{budget}} für {{user}} auf {{datasource}} ist aufgebraucht" } }, "profile": { @@ -3287,7 +3293,8 @@ "cicd_origin": "CI/CD-Herkunft", "estimated_rows": "Geschätzte Zeilen", "scan_type": "Scan-Typ", - "estimated_bytes_scanned": "Geschätzte gescannte Bytes" + "estimated_bytes_scanned": "Geschätzte gescannte Bytes", + "data_budget_used_percent": "Datenbudget verbraucht (%)" }, "query_shape": { "JOIN": "Join", @@ -3562,7 +3569,8 @@ "EXPORT_POLICY_MANAGE": "Exportrichtlinien verwalten", "ROW_LIMIT_POLICY_MANAGE": "Zeilenlimit-Richtlinien verwalten", "DEPLOYMENT_PIPELINE_MANAGE": "Deployment-Pipelines verwalten", - "DEPLOYMENT_REVIEW": "Deployments prüfen" + "DEPLOYMENT_REVIEW": "Deployments prüfen", + "DATA_BUDGET_MANAGE": "Datenbudgets verwalten" }, "api_variable_kind": { "CONSTANT": "Konstante", @@ -3733,7 +3741,8 @@ "ROW_SECURITY": "Zeilensicherheit", "MASKING": "Maskierung", "BREAK_GLASS": "Break-Glass", - "BYTES_SCANNED_CAP": "Limit für gescannte Bytes" + "BYTES_SCANNED_CAP": "Limit für gescannte Bytes", + "DATA_BUDGET": "Datenbudget" }, "api_decision_step": { "CONNECTOR_GATES": "Konnektor-Prüfungen", @@ -3802,6 +3811,10 @@ "EXCEEDED": "Abgelehnt: die Schätzung überschreitet das Limit für gescannte Bytes", "NO_ESTIMATE_REVIEW": "Zur Prüfung zurückgehalten: keine Byte-Schätzung unter dem Limit", "NO_ESTIMATE_REJECTED": "Abgelehnt: keine Byte-Schätzung unter dem Limit" + }, + "data_budget_breach_action": { + "REQUIRE_REVIEW": "Zur Prüfung senden", + "REJECT": "Ablehnen" } }, "access": { @@ -5812,7 +5825,14 @@ "bytes_scanned_cap": "Limit für gescannte Bytes", "bytes_scanned_cap_outcome": "Ergebnis des Limits", "bytes_scanned_cap_source": "Quelle des Limits", - "estimated_bytes_scanned": "Geschätzte gescannte Bytes" + "estimated_bytes_scanned": "Geschätzte gescannte Bytes", + "data_budget_action": "Aktion bei Budgetüberschreitung", + "data_budget_id": "Datenbudget", + "data_budget_name": "Name des Datenbudgets", + "data_budget_remaining_bytes": "Verbleibende Budget-Bytes", + "data_budget_remaining_rows": "Verbleibende Budget-Zeilen", + "data_budget_suppressed": "Durch das Datenbudget unterdrückt", + "data_budget_used_percent": "Datenbudget verbraucht" }, "use_id": "ID {{id}} verwenden", "user_id_placeholder": "Benutzer-ID einfügen", @@ -5832,5 +5852,93 @@ "key_control_chars": "Der Anwendungsname darf keine Steuerzeichen enthalten.", "rotate_help": "Leer lassen, um den Anwendungsnamen des aktuellen Schlüssels beizubehalten.", "column": "Anwendung" + }, + "dataBudgets": { + "window_hours_one": "{{count}} Stunde", + "window_hours_other": "{{count}} Stunden", + "window_days_one": "{{count}} Tag", + "window_days_other": "{{count}} Tage", + "indicator": { + "title": "Ihr Datenbudget", + "exhausted_title": "Datenbudget aufgebraucht", + "exhausted_reject": "Neue Lesezugriffe auf diese Datenquelle werden abgelehnt, bis frühere Lesezugriffe aus dem Zeitfenster fallen.", + "exhausted_review": "Neue Lesezugriffe auf diese Datenquelle benötigen eine Genehmigung, bis frühere Lesezugriffe aus dem Zeitfenster fallen.", + "nearly_used": "Sie haben den Großteil Ihres Datenbudgets für diese Datenquelle verbraucht.", + "rows_left": "{{remaining}} von {{limit}} Zeilen übrig", + "bytes_left": "{{remaining}} von {{limit}} übrig", + "per_window": "pro {{window}}", + "used_aria": "{{name}}: {{percent}} % verbraucht" + }, + "tab": { + "title": "Datenbudgets", + "description": "Begrenzen Sie, wie viele Zeilen und Ergebnis-Bytes jeder Benutzer in einem gleitenden Zeitfenster aus dieser Datenquelle lesen darf. Eine Abfrage wird auf das verbleibende Kontingent begrenzt; ist ein Budget aufgebraucht, werden neue Lesezugriffe abgelehnt oder zur Prüfung geschickt.", + "add": "Budget hinzufügen", + "empty_title": "Keine Datenbudgets", + "empty_description": "Lesezugriffe auf diese Datenquelle sind nur pro Abfrage begrenzt.", + "col_name": "Name", + "col_limits": "Limits", + "col_window": "Zeitfenster", + "col_action": "Bei Erschöpfung", + "col_applies_to": "Gilt für", + "col_enabled": "Aktiviert", + "col_actions": "Aktionen", + "state_disabled": "Deaktiviert", + "edit": "Budget bearbeiten", + "delete": "Budget löschen", + "delete_confirm_title": "Dieses Datenbudget löschen?", + "delete_confirm_body": "Betroffene Benutzer sind nicht mehr daran gebunden. Erfasste Nutzung bleibt erhalten, bis sie aus dem Zeitfenster fällt.", + "delete_success": "Datenbudget gelöscht", + "delete_error": "Das Datenbudget konnte nicht gelöscht werden", + "save_success": "Datenbudget gespeichert", + "save_error": "Das Datenbudget konnte nicht gespeichert werden", + "create_title": "Neues Datenbudget", + "edit_title": "Datenbudget bearbeiten", + "label_name": "Name", + "name_required": "Geben Sie einen Namen ein", + "name_max": "Höchstens 120 Zeichen", + "label_max_rows": "Zeilenlimit", + "label_max_bytes": "Limit für Ergebnisgröße", + "limits_hint": "Legen Sie ein Zeilenlimit, ein Größenlimit oder beides fest. Jeder Benutzer erhält ein eigenes Kontingent.", + "limit_required": "Legen Sie ein Zeilen- oder Größenlimit fest", + "rows_min": "Das Zeilenlimit muss mindestens 1 betragen", + "bytes_min": "Das Größenlimit muss mindestens 1 Byte betragen", + "label_window": "Gleitendes Zeitfenster", + "window_hint": "Die Nutzung zählt über das zurückliegende Zeitfenster, von jetzt aus gemessen.", + "window_range": "Das Zeitfenster muss zwischen 1 Stunde und 31 Tagen liegen", + "window_unit": "Einheit des Zeitfensters", + "unit_hours": "Stunden", + "unit_days": "Tage", + "label_breach_action": "Wenn das Budget aufgebraucht ist", + "breach_action_hint": "Eine Prüfung lässt legitime Analysten einen intensiven Tag abschließen; Ablehnen stoppt Lesezugriffe sofort. Break-Glass-Zugriff wird nie blockiert, zählt aber mit.", + "label_warn_threshold": "Warnen bei (% verbraucht)", + "warn_threshold_hint": "Der Benutzer wird einmal benachrichtigt, wenn er diesen Anteil überschreitet. Leer lassen für keine Warnung.", + "threshold_range": "Die Schwelle muss zwischen 1 und 99 liegen", + "label_applies_roles": "Gilt für Rollen", + "applies_hint": "Lassen Sie alle drei leer, damit das Budget für jeden Benutzer dieser Datenquelle gilt.", + "label_applies_groups": "Gilt für Gruppen", + "label_applies_users": "Gilt für Benutzer", + "label_enabled": "Aktiviert", + "applies_everyone": "Alle", + "applies_roles_one": "{{count}} Rolle", + "applies_roles_other": "{{count}} Rollen", + "applies_groups_one": "{{count}} Gruppe", + "applies_groups_other": "{{count}} Gruppen", + "applies_users_one": "{{count}} Benutzer", + "applies_users_other": "{{count}} Benutzer", + "rows_value": "{{value}} Zeilen", + "unit_minutes": "Minuten" + }, + "usage": { + "action": "Datennutzung", + "title": "Datennutzung — {{name}}", + "description": "Zeilen und Ergebnis-Bytes, die dieser Benutzer im Zeitfenster jedes für ihn geltenden Datenbudgets gelesen hat.", + "empty": "Für diesen Benutzer gilt kein Datenbudget.", + "load_error": "Die Datennutzung dieses Benutzers konnte nicht geladen werden", + "exhausted": "Aufgebraucht", + "rows_used": "{{used}} von {{limit}} Zeilen", + "bytes_used": "{{used}} von {{limit}}" + }, + "window_minutes_one": "{{count}} Minute", + "window_minutes_other": "{{count}} Minuten" } } diff --git a/frontend/src/locales/en.json b/frontend/src/locales/en.json index 569e80987..5b9439ff3 100644 --- a/frontend/src/locales/en.json +++ b/frontend/src/locales/en.json @@ -868,7 +868,8 @@ "diff_effective_label": "Effective SQL" }, "bytes_cap_body": "Estimated scan: {{estimate}} · Cap: {{limit}} ({{source}})", - "bytes_cap_no_estimate": "no estimate" + "bytes_cap_no_estimate": "no estimate", + "results_truncated_budget": "{{count}} rows returned (capped at your remaining data budget)" } }, "reviews": { @@ -1590,7 +1591,9 @@ "grant_bytes_cap_label": "Bytes-scanned cap", "grant_bytes_cap_help": "A lower cap for this grant. The strictest of the datasource cap and every grant applies.", "grant_bytes_cap_min": "Bytes-scanned cap must be at least 1 byte.", - "grant_bytes_cap_placeholder": "Datasource default" + "grant_bytes_cap_placeholder": "Datasource default", + "tab_data_budgets": "Data budgets · {{count}}", + "sample_truncated_budget": "{{count}} rows (capped at your remaining data budget)" } }, "admin": { @@ -2471,7 +2474,8 @@ "minutes_suffix": "min", "cicd_present_label": "Is CI/CD origin", "scan_type_placeholder": "e.g. Seq Scan, COLLSCAN, Index*", - "bytes_value_label": "Bytes scanned" + "bytes_value_label": "Bytes scanned", + "budget_percent_label": "Share of data budget used" }, "langfuse": { "title": "Langfuse", @@ -2912,7 +2916,9 @@ "SCHEMA_CHANGE_PROMOTION_FAILED": "Your schema change {{changeSet}} failed on {{environment}}", "SCHEMA_CHANGE_PROMOTION_FAILED_partial": "Your schema change {{changeSet}} was only partly applied to {{environment}}", "SCHEMA_DRIFT_DETECTED_one": "{{count}} new schema drift finding on {{pipeline}} / {{environment}}", - "SCHEMA_DRIFT_DETECTED_other": "{{count}} new schema drift findings on {{pipeline}} / {{environment}}" + "SCHEMA_DRIFT_DETECTED_other": "{{count}} new schema drift findings on {{pipeline}} / {{environment}}", + "DATA_BUDGET_THRESHOLD_REACHED": "You have used {{percent}}% of your data budget {{budget}} on {{datasource}}", + "DATA_BUDGET_EXHAUSTED": "Data budget {{budget}} for {{user}} on {{datasource}} is used up" } }, "profile": { @@ -3317,7 +3323,8 @@ "cicd_origin": "CI/CD origin", "estimated_rows": "Estimated rows", "scan_type": "Scan type", - "estimated_bytes_scanned": "Estimated bytes scanned" + "estimated_bytes_scanned": "Estimated bytes scanned", + "data_budget_used_percent": "Data budget used (%)" }, "query_shape": { "JOIN": "Join", @@ -3587,7 +3594,8 @@ "EXPORT_POLICY_MANAGE": "Manage export policies", "ROW_LIMIT_POLICY_MANAGE": "Manage row-limit policies", "DEPLOYMENT_PIPELINE_MANAGE": "Manage deployment pipelines", - "DEPLOYMENT_REVIEW": "Review deployments" + "DEPLOYMENT_REVIEW": "Review deployments", + "DATA_BUDGET_MANAGE": "Manage data budgets" }, "api_variable_kind": { "CONSTANT": "Constant", @@ -3758,7 +3766,8 @@ "ROW_SECURITY": "Row security", "MASKING": "Masking", "BREAK_GLASS": "Break-glass", - "BYTES_SCANNED_CAP": "Bytes-scanned cap" + "BYTES_SCANNED_CAP": "Bytes-scanned cap", + "DATA_BUDGET": "Data budget" }, "api_decision_step": { "CONNECTOR_GATES": "Connector gates", @@ -3827,6 +3836,10 @@ "EXCEEDED": "Rejected: the estimate exceeds the bytes-scanned cap", "NO_ESTIMATE_REVIEW": "Held for review: no bytes estimate under the cap", "NO_ESTIMATE_REJECTED": "Rejected: no bytes estimate under the cap" + }, + "data_budget_breach_action": { + "REQUIRE_REVIEW": "Send to human review", + "REJECT": "Reject" } }, "access": { @@ -5812,7 +5825,14 @@ "bytes_scanned_cap": "Bytes-scanned cap", "bytes_scanned_cap_outcome": "Cap outcome", "bytes_scanned_cap_source": "Cap source", - "estimated_bytes_scanned": "Estimated bytes scanned" + "estimated_bytes_scanned": "Estimated bytes scanned", + "data_budget_action": "Budget breach action", + "data_budget_id": "Data budget", + "data_budget_name": "Data budget name", + "data_budget_remaining_bytes": "Budget bytes left", + "data_budget_remaining_rows": "Budget rows left", + "data_budget_suppressed": "Suppressed by the data budget", + "data_budget_used_percent": "Data budget used" }, "use_id": "Use ID {{id}}", "user_id_placeholder": "Paste a user ID", @@ -5832,5 +5852,93 @@ "key_control_chars": "Application name must not contain control characters.", "rotate_help": "Leave empty to keep the current key's application name.", "column": "Application" + }, + "dataBudgets": { + "window_hours_one": "{{count}} hour", + "window_hours_other": "{{count}} hours", + "window_days_one": "{{count}} day", + "window_days_other": "{{count}} days", + "indicator": { + "title": "Your data budget", + "exhausted_title": "Data budget used up", + "exhausted_reject": "New reads on this datasource are refused until earlier reads age out of the window.", + "exhausted_review": "New reads on this datasource need a reviewer’s approval until earlier reads age out of the window.", + "nearly_used": "You have used most of your data budget on this datasource.", + "rows_left": "{{remaining}} of {{limit}} rows left", + "bytes_left": "{{remaining}} of {{limit}} left", + "per_window": "per {{window}}", + "used_aria": "{{name}}: {{percent}}% used" + }, + "tab": { + "title": "Data budgets", + "description": "Cap how many rows and result bytes each user may read from this datasource over a rolling window. A query is capped to the allowance left; once a budget is used up, new reads are rejected or sent to human review.", + "add": "Add budget", + "empty_title": "No data budgets", + "empty_description": "Reads on this datasource are bounded per query only.", + "col_name": "Name", + "col_limits": "Limits", + "col_window": "Window", + "col_action": "When used up", + "col_applies_to": "Applies to", + "col_enabled": "Enabled", + "col_actions": "Actions", + "state_disabled": "Disabled", + "edit": "Edit budget", + "delete": "Delete budget", + "delete_confirm_title": "Delete this data budget?", + "delete_confirm_body": "Users it applied to are no longer bounded by it. Recorded usage is kept until it ages out.", + "delete_success": "Data budget deleted", + "delete_error": "Could not delete the data budget", + "save_success": "Data budget saved", + "save_error": "Could not save the data budget", + "create_title": "New data budget", + "edit_title": "Edit data budget", + "label_name": "Name", + "name_required": "Enter a name", + "name_max": "Use at most 120 characters", + "label_max_rows": "Row limit", + "label_max_bytes": "Result-size limit", + "limits_hint": "Set a row limit, a result-size limit, or both. Each user gets their own allowance.", + "limit_required": "Set a row limit or a result-size limit", + "rows_min": "The row limit must be at least 1", + "bytes_min": "The result-size limit must be at least 1 byte", + "label_window": "Rolling window", + "window_hint": "Usage counts over the trailing window, measured back from now.", + "window_range": "The window must be between 1 hour and 31 days", + "window_unit": "Window unit", + "unit_hours": "hours", + "unit_days": "days", + "label_breach_action": "When the budget is used up", + "breach_action_hint": "Human review lets a legitimate analyst finish a heavy day; reject stops reads outright. Break-glass access is never blocked but still counts.", + "label_warn_threshold": "Warn at (% used)", + "warn_threshold_hint": "The user is notified once when they cross this share. Leave empty for no warning.", + "threshold_range": "The threshold must be between 1 and 99", + "label_applies_roles": "Applies to roles", + "applies_hint": "Leave all three empty to apply the budget to every user of this datasource.", + "label_applies_groups": "Applies to groups", + "label_applies_users": "Applies to users", + "label_enabled": "Enabled", + "applies_everyone": "Everyone", + "applies_roles_one": "{{count}} role", + "applies_roles_other": "{{count}} roles", + "applies_groups_one": "{{count}} group", + "applies_groups_other": "{{count}} groups", + "applies_users_one": "{{count}} user", + "applies_users_other": "{{count}} users", + "rows_value": "{{value}} rows", + "unit_minutes": "minutes" + }, + "usage": { + "action": "Data usage", + "title": "Data usage — {{name}}", + "description": "Rows and result bytes this user read against each data budget that applies to them, over each budget’s window.", + "empty": "No data budget applies to this user.", + "load_error": "Could not load this user’s data usage", + "exhausted": "Used up", + "rows_used": "{{used}} of {{limit}} rows", + "bytes_used": "{{used}} of {{limit}}" + }, + "window_minutes_one": "{{count}} minute", + "window_minutes_other": "{{count}} minutes" } } diff --git a/frontend/src/locales/es.json b/frontend/src/locales/es.json index db432ca89..9f413f074 100644 --- a/frontend/src/locales/es.json +++ b/frontend/src/locales/es.json @@ -868,7 +868,8 @@ "diff_effective_label": "SQL efectivo" }, "bytes_cap_body": "Escaneo estimado: {{estimate}} · Límite: {{limit}} ({{source}})", - "bytes_cap_no_estimate": "sin estimación" + "bytes_cap_no_estimate": "sin estimación", + "results_truncated_budget": "{{count}} filas devueltas (limitadas a tu presupuesto de datos restante)" } }, "reviews": { @@ -1590,7 +1591,9 @@ "grant_bytes_cap_label": "Límite de bytes escaneados", "grant_bytes_cap_help": "Un límite menor para este permiso. Se aplica el más estricto entre el límite de la fuente de datos y todos los permisos.", "grant_bytes_cap_min": "El límite de bytes escaneados debe ser al menos 1 byte.", - "grant_bytes_cap_placeholder": "Predeterminado de la fuente de datos" + "grant_bytes_cap_placeholder": "Predeterminado de la fuente de datos", + "tab_data_budgets": "Presupuestos de datos · {{count}}", + "sample_truncated_budget": "{{count}} filas (limitadas a tu presupuesto de datos restante)" } }, "admin": { @@ -2471,7 +2474,8 @@ "minutes_suffix": "min", "cicd_present_label": "Es origen CI/CD", "scan_type_placeholder": "p. ej. Seq Scan, COLLSCAN, Index*", - "bytes_value_label": "Bytes escaneados" + "bytes_value_label": "Bytes escaneados", + "budget_percent_label": "Porcentaje del presupuesto de datos usado" }, "langfuse": { "title": "Langfuse", @@ -2912,7 +2916,9 @@ "SCHEMA_CHANGE_PROMOTION_FAILED": "Su cambio de esquema {{changeSet}} falló en {{environment}}", "SCHEMA_CHANGE_PROMOTION_FAILED_partial": "Su cambio de esquema {{changeSet}} solo se aplicó parcialmente en {{environment}}", "SCHEMA_DRIFT_DETECTED_one": "{{count}} hallazgo nuevo de deriva de esquema en {{pipeline}} / {{environment}}", - "SCHEMA_DRIFT_DETECTED_other": "{{count}} hallazgos nuevos de deriva de esquema en {{pipeline}} / {{environment}}" + "SCHEMA_DRIFT_DETECTED_other": "{{count}} hallazgos nuevos de deriva de esquema en {{pipeline}} / {{environment}}", + "DATA_BUDGET_THRESHOLD_REACHED": "Ha usado el {{percent}} % de su presupuesto de datos {{budget}} en {{datasource}}", + "DATA_BUDGET_EXHAUSTED": "El presupuesto de datos {{budget}} de {{user}} en {{datasource}} está agotado" } }, "profile": { @@ -3287,7 +3293,8 @@ "cicd_origin": "Origen CI/CD", "estimated_rows": "Filas estimadas", "scan_type": "Tipo de escaneo", - "estimated_bytes_scanned": "Bytes escaneados estimados" + "estimated_bytes_scanned": "Bytes escaneados estimados", + "data_budget_used_percent": "Presupuesto de datos usado (%)" }, "query_shape": { "JOIN": "Join", @@ -3562,7 +3569,8 @@ "EXPORT_POLICY_MANAGE": "Gestionar políticas de exportación", "ROW_LIMIT_POLICY_MANAGE": "Gestionar políticas de límite de filas", "DEPLOYMENT_PIPELINE_MANAGE": "Gestionar pipelines de despliegue", - "DEPLOYMENT_REVIEW": "Revisar despliegues" + "DEPLOYMENT_REVIEW": "Revisar despliegues", + "DATA_BUDGET_MANAGE": "Gestionar presupuestos de datos" }, "api_variable_kind": { "CONSTANT": "Constante", @@ -3733,7 +3741,8 @@ "ROW_SECURITY": "Seguridad por filas", "MASKING": "Enmascaramiento", "BREAK_GLASS": "Acceso de emergencia", - "BYTES_SCANNED_CAP": "Límite de bytes escaneados" + "BYTES_SCANNED_CAP": "Límite de bytes escaneados", + "DATA_BUDGET": "Presupuesto de datos" }, "api_decision_step": { "CONNECTOR_GATES": "Controles del conector", @@ -3802,6 +3811,10 @@ "EXCEEDED": "Rechazada: la estimación supera el límite de bytes escaneados", "NO_ESTIMATE_REVIEW": "Retenida para revisión: sin estimación de bytes bajo el límite", "NO_ESTIMATE_REJECTED": "Rechazada: sin estimación de bytes bajo el límite" + }, + "data_budget_breach_action": { + "REQUIRE_REVIEW": "Enviar a revisión humana", + "REJECT": "Rechazar" } }, "access": { @@ -5812,7 +5825,14 @@ "bytes_scanned_cap": "Límite de bytes escaneados", "bytes_scanned_cap_outcome": "Resultado del límite", "bytes_scanned_cap_source": "Origen del límite", - "estimated_bytes_scanned": "Bytes escaneados estimados" + "estimated_bytes_scanned": "Bytes escaneados estimados", + "data_budget_action": "Acción al agotar el presupuesto", + "data_budget_id": "Presupuesto de datos", + "data_budget_name": "Nombre del presupuesto de datos", + "data_budget_remaining_bytes": "Bytes restantes del presupuesto", + "data_budget_remaining_rows": "Filas restantes del presupuesto", + "data_budget_suppressed": "Suprimido por el presupuesto de datos", + "data_budget_used_percent": "Presupuesto de datos usado" }, "use_id": "Usar el ID {{id}}", "user_id_placeholder": "Pegue un ID de usuario", @@ -5832,5 +5852,93 @@ "key_control_chars": "El nombre de la aplicación no debe contener caracteres de control.", "rotate_help": "Déjalo vacío para conservar el nombre de aplicación de la clave actual.", "column": "Aplicación" + }, + "dataBudgets": { + "window_hours_one": "{{count}} hora", + "window_hours_other": "{{count}} horas", + "window_days_one": "{{count}} día", + "window_days_other": "{{count}} días", + "indicator": { + "title": "Tu presupuesto de datos", + "exhausted_title": "Presupuesto de datos agotado", + "exhausted_reject": "Las nuevas lecturas en esta fuente de datos se rechazan hasta que las lecturas anteriores salgan de la ventana.", + "exhausted_review": "Las nuevas lecturas en esta fuente de datos necesitan la aprobación de un revisor hasta que las lecturas anteriores salgan de la ventana.", + "nearly_used": "Has usado la mayor parte de tu presupuesto de datos en esta fuente de datos.", + "rows_left": "Quedan {{remaining}} de {{limit}} filas", + "bytes_left": "Quedan {{remaining}} de {{limit}}", + "per_window": "por {{window}}", + "used_aria": "{{name}}: {{percent}} % usado" + }, + "tab": { + "title": "Presupuestos de datos", + "description": "Limita cuántas filas y bytes de resultado puede leer cada usuario de esta fuente de datos en una ventana móvil. Una consulta se limita a la cuota restante; cuando un presupuesto se agota, las nuevas lecturas se rechazan o pasan a revisión humana.", + "add": "Añadir presupuesto", + "empty_title": "Sin presupuestos de datos", + "empty_description": "Las lecturas en esta fuente de datos solo se limitan por consulta.", + "col_name": "Nombre", + "col_limits": "Límites", + "col_window": "Ventana", + "col_action": "Al agotarse", + "col_applies_to": "Se aplica a", + "col_enabled": "Activado", + "col_actions": "Acciones", + "state_disabled": "Desactivado", + "edit": "Editar presupuesto", + "delete": "Eliminar presupuesto", + "delete_confirm_title": "¿Eliminar este presupuesto de datos?", + "delete_confirm_body": "Los usuarios afectados dejan de estar limitados por él. El uso registrado se conserva hasta que caduca.", + "delete_success": "Presupuesto de datos eliminado", + "delete_error": "No se pudo eliminar el presupuesto de datos", + "save_success": "Presupuesto de datos guardado", + "save_error": "No se pudo guardar el presupuesto de datos", + "create_title": "Nuevo presupuesto de datos", + "edit_title": "Editar presupuesto de datos", + "label_name": "Nombre", + "name_required": "Introduce un nombre", + "name_max": "Usa como máximo 120 caracteres", + "label_max_rows": "Límite de filas", + "label_max_bytes": "Límite de tamaño de resultado", + "limits_hint": "Define un límite de filas, de tamaño o ambos. Cada usuario tiene su propia cuota.", + "limit_required": "Define un límite de filas o de tamaño", + "rows_min": "El límite de filas debe ser al menos 1", + "bytes_min": "El límite de tamaño debe ser al menos 1 byte", + "label_window": "Ventana móvil", + "window_hint": "El uso se cuenta en la ventana anterior, medida desde ahora.", + "window_range": "La ventana debe estar entre 1 hora y 31 días", + "window_unit": "Unidad de la ventana", + "unit_hours": "horas", + "unit_days": "días", + "label_breach_action": "Cuando se agota el presupuesto", + "breach_action_hint": "La revisión humana permite a un analista legítimo terminar un día intenso; rechazar detiene las lecturas. El acceso de emergencia nunca se bloquea, pero cuenta.", + "label_warn_threshold": "Avisar al (% usado)", + "warn_threshold_hint": "El usuario recibe un aviso al superar este porcentaje. Déjalo vacío para no avisar.", + "threshold_range": "El umbral debe estar entre 1 y 99", + "label_applies_roles": "Se aplica a roles", + "applies_hint": "Deja los tres vacíos para aplicar el presupuesto a todos los usuarios de esta fuente de datos.", + "label_applies_groups": "Se aplica a grupos", + "label_applies_users": "Se aplica a usuarios", + "label_enabled": "Activado", + "applies_everyone": "Todos", + "applies_roles_one": "{{count}} rol", + "applies_roles_other": "{{count}} roles", + "applies_groups_one": "{{count}} grupo", + "applies_groups_other": "{{count}} grupos", + "applies_users_one": "{{count}} usuario", + "applies_users_other": "{{count}} usuarios", + "rows_value": "{{value}} filas", + "unit_minutes": "minutos" + }, + "usage": { + "action": "Uso de datos", + "title": "Uso de datos — {{name}}", + "description": "Filas y bytes de resultado que este usuario leyó frente a cada presupuesto que le aplica, en la ventana de cada presupuesto.", + "empty": "No se aplica ningún presupuesto de datos a este usuario.", + "load_error": "No se pudo cargar el uso de datos de este usuario", + "exhausted": "Agotado", + "rows_used": "{{used}} de {{limit}} filas", + "bytes_used": "{{used}} de {{limit}}" + }, + "window_minutes_one": "{{count}} minuto", + "window_minutes_other": "{{count}} minutos" } } diff --git a/frontend/src/locales/fr.json b/frontend/src/locales/fr.json index c36d0004c..5c967d1c0 100644 --- a/frontend/src/locales/fr.json +++ b/frontend/src/locales/fr.json @@ -868,7 +868,8 @@ "diff_effective_label": "SQL effectif" }, "bytes_cap_body": "Analyse estimée : {{estimate}} · Limite : {{limit}} ({{source}})", - "bytes_cap_no_estimate": "aucune estimation" + "bytes_cap_no_estimate": "aucune estimation", + "results_truncated_budget": "{{count}} lignes renvoyées (plafonnées à votre budget de données restant)" } }, "reviews": { @@ -1590,7 +1591,9 @@ "grant_bytes_cap_label": "Limite d’octets analysés", "grant_bytes_cap_help": "Une limite plus basse pour cette autorisation. La plus stricte entre la limite de la source de données et toutes les autorisations s’applique.", "grant_bytes_cap_min": "La limite d’octets analysés doit être d’au moins 1 octet.", - "grant_bytes_cap_placeholder": "Valeur par défaut de la source de données" + "grant_bytes_cap_placeholder": "Valeur par défaut de la source de données", + "tab_data_budgets": "Budgets de données · {{count}}", + "sample_truncated_budget": "{{count}} lignes (plafonnées à votre budget de données restant)" } }, "admin": { @@ -2471,7 +2474,8 @@ "minutes_suffix": "min", "cicd_present_label": "Provenance CI/CD", "scan_type_placeholder": "p. ex. Seq Scan, COLLSCAN, Index*", - "bytes_value_label": "Octets analysés" + "bytes_value_label": "Octets analysés", + "budget_percent_label": "Part du budget de données consommée" }, "langfuse": { "title": "Langfuse", @@ -2912,7 +2916,9 @@ "SCHEMA_CHANGE_PROMOTION_FAILED": "Votre modification de schéma {{changeSet}} a échoué sur {{environment}}", "SCHEMA_CHANGE_PROMOTION_FAILED_partial": "Votre modification de schéma {{changeSet}} n'a été que partiellement appliquée sur {{environment}}", "SCHEMA_DRIFT_DETECTED_one": "{{count}} nouvelle constatation de dérive de schéma sur {{pipeline}} / {{environment}}", - "SCHEMA_DRIFT_DETECTED_other": "{{count}} nouvelles constatations de dérive de schéma sur {{pipeline}} / {{environment}}" + "SCHEMA_DRIFT_DETECTED_other": "{{count}} nouvelles constatations de dérive de schéma sur {{pipeline}} / {{environment}}", + "DATA_BUDGET_THRESHOLD_REACHED": "Vous avez utilisé {{percent}} % de votre budget de données {{budget}} sur {{datasource}}", + "DATA_BUDGET_EXHAUSTED": "Le budget de données {{budget}} de {{user}} sur {{datasource}} est épuisé" } }, "profile": { @@ -3287,7 +3293,8 @@ "cicd_origin": "Provenance CI/CD", "estimated_rows": "Lignes estimées", "scan_type": "Type de scan", - "estimated_bytes_scanned": "Octets analysés estimés" + "estimated_bytes_scanned": "Octets analysés estimés", + "data_budget_used_percent": "Budget de données consommé (%)" }, "query_shape": { "JOIN": "Jointure", @@ -3562,7 +3569,8 @@ "EXPORT_POLICY_MANAGE": "Gérer les politiques d'export", "ROW_LIMIT_POLICY_MANAGE": "Gérer les politiques de limite de lignes", "DEPLOYMENT_PIPELINE_MANAGE": "Gérer les pipelines de déploiement", - "DEPLOYMENT_REVIEW": "Réviser les déploiements" + "DEPLOYMENT_REVIEW": "Réviser les déploiements", + "DATA_BUDGET_MANAGE": "Gérer les budgets de données" }, "api_variable_kind": { "CONSTANT": "Constante", @@ -3733,7 +3741,8 @@ "ROW_SECURITY": "Sécurité par ligne", "MASKING": "Masquage", "BREAK_GLASS": "Accès d'urgence", - "BYTES_SCANNED_CAP": "Limite d’octets analysés" + "BYTES_SCANNED_CAP": "Limite d’octets analysés", + "DATA_BUDGET": "Budget de données" }, "api_decision_step": { "CONNECTOR_GATES": "Contrôles du connecteur", @@ -3802,6 +3811,10 @@ "EXCEEDED": "Rejetée : l’estimation dépasse la limite d’octets analysés", "NO_ESTIMATE_REVIEW": "Mise en revue : aucune estimation en octets sous la limite", "NO_ESTIMATE_REJECTED": "Rejetée : aucune estimation en octets sous la limite" + }, + "data_budget_breach_action": { + "REQUIRE_REVIEW": "Envoyer en revue humaine", + "REJECT": "Refuser" } }, "access": { @@ -5812,7 +5825,14 @@ "bytes_scanned_cap": "Limite d’octets analysés", "bytes_scanned_cap_outcome": "Résultat de la limite", "bytes_scanned_cap_source": "Origine de la limite", - "estimated_bytes_scanned": "Octets analysés estimés" + "estimated_bytes_scanned": "Octets analysés estimés", + "data_budget_action": "Action à l’épuisement du budget", + "data_budget_id": "Budget de données", + "data_budget_name": "Nom du budget de données", + "data_budget_remaining_bytes": "Octets restants du budget", + "data_budget_remaining_rows": "Lignes restantes du budget", + "data_budget_suppressed": "Suspendu par le budget de données", + "data_budget_used_percent": "Budget de données consommé" }, "use_id": "Utiliser l'ID {{id}}", "user_id_placeholder": "Collez un ID d'utilisateur", @@ -5832,5 +5852,93 @@ "key_control_chars": "Le nom de l'application ne doit pas contenir de caractères de contrôle.", "rotate_help": "Laissez vide pour conserver le nom d'application de la clé actuelle.", "column": "Application" + }, + "dataBudgets": { + "window_hours_one": "{{count}} heure", + "window_hours_other": "{{count}} heures", + "window_days_one": "{{count}} jour", + "window_days_other": "{{count}} jours", + "indicator": { + "title": "Votre budget de données", + "exhausted_title": "Budget de données épuisé", + "exhausted_reject": "Les nouvelles lectures sur cette source de données sont refusées jusqu’à ce que les lectures précédentes sortent de la fenêtre.", + "exhausted_review": "Les nouvelles lectures sur cette source de données nécessitent l’approbation d’un réviseur jusqu’à ce que les lectures précédentes sortent de la fenêtre.", + "nearly_used": "Vous avez consommé la majeure partie de votre budget de données sur cette source.", + "rows_left": "{{remaining}} lignes restantes sur {{limit}}", + "bytes_left": "{{remaining}} restants sur {{limit}}", + "per_window": "par {{window}}", + "used_aria": "{{name}} : {{percent}} % consommé" + }, + "tab": { + "title": "Budgets de données", + "description": "Limitez le nombre de lignes et d’octets de résultat que chaque utilisateur peut lire depuis cette source sur une fenêtre glissante. Une requête est plafonnée au quota restant ; une fois un budget épuisé, les nouvelles lectures sont refusées ou envoyées en revue humaine.", + "add": "Ajouter un budget", + "empty_title": "Aucun budget de données", + "empty_description": "Les lectures sur cette source ne sont limitées que par requête.", + "col_name": "Nom", + "col_limits": "Limites", + "col_window": "Fenêtre", + "col_action": "Une fois épuisé", + "col_applies_to": "S’applique à", + "col_enabled": "Activé", + "col_actions": "Actions", + "state_disabled": "Désactivé", + "edit": "Modifier le budget", + "delete": "Supprimer le budget", + "delete_confirm_title": "Supprimer ce budget de données ?", + "delete_confirm_body": "Les utilisateurs concernés n’y sont plus soumis. L’usage enregistré est conservé jusqu’à son expiration.", + "delete_success": "Budget de données supprimé", + "delete_error": "Impossible de supprimer le budget de données", + "save_success": "Budget de données enregistré", + "save_error": "Impossible d’enregistrer le budget de données", + "create_title": "Nouveau budget de données", + "edit_title": "Modifier le budget de données", + "label_name": "Nom", + "name_required": "Saisissez un nom", + "name_max": "120 caractères maximum", + "label_max_rows": "Limite de lignes", + "label_max_bytes": "Limite de taille de résultat", + "limits_hint": "Définissez une limite de lignes, de taille, ou les deux. Chaque utilisateur dispose de son propre quota.", + "limit_required": "Définissez une limite de lignes ou de taille", + "rows_min": "La limite de lignes doit être d’au moins 1", + "bytes_min": "La limite de taille doit être d’au moins 1 octet", + "label_window": "Fenêtre glissante", + "window_hint": "L’usage est compté sur la fenêtre écoulée, mesurée à partir de maintenant.", + "window_range": "La fenêtre doit être comprise entre 1 heure et 31 jours", + "window_unit": "Unité de la fenêtre", + "unit_hours": "heures", + "unit_days": "jours", + "label_breach_action": "Quand le budget est épuisé", + "breach_action_hint": "La revue humaine permet à un analyste légitime de finir une journée chargée ; le refus arrête les lectures. L’accès d’urgence n’est jamais bloqué mais reste comptabilisé.", + "label_warn_threshold": "Avertir à (% consommé)", + "warn_threshold_hint": "L’utilisateur est prévenu une fois ce seuil franchi. Laissez vide pour ne pas avertir.", + "threshold_range": "Le seuil doit être compris entre 1 et 99", + "label_applies_roles": "S’applique aux rôles", + "applies_hint": "Laissez les trois vides pour appliquer le budget à tous les utilisateurs de cette source.", + "label_applies_groups": "S’applique aux groupes", + "label_applies_users": "S’applique aux utilisateurs", + "label_enabled": "Activé", + "applies_everyone": "Tout le monde", + "applies_roles_one": "{{count}} rôle", + "applies_roles_other": "{{count}} rôles", + "applies_groups_one": "{{count}} groupe", + "applies_groups_other": "{{count}} groupes", + "applies_users_one": "{{count}} utilisateur", + "applies_users_other": "{{count}} utilisateurs", + "rows_value": "{{value}} lignes", + "unit_minutes": "minutes" + }, + "usage": { + "action": "Usage des données", + "title": "Usage des données — {{name}}", + "description": "Lignes et octets de résultat lus par cet utilisateur pour chaque budget qui le concerne, sur la fenêtre de chaque budget.", + "empty": "Aucun budget de données ne s’applique à cet utilisateur.", + "load_error": "Impossible de charger l’usage des données de cet utilisateur", + "exhausted": "Épuisé", + "rows_used": "{{used}} lignes sur {{limit}}", + "bytes_used": "{{used}} sur {{limit}}" + }, + "window_minutes_one": "{{count}} minute", + "window_minutes_other": "{{count}} minutes" } } diff --git a/frontend/src/locales/hy.json b/frontend/src/locales/hy.json index c20dfd2ef..7b617f4dd 100644 --- a/frontend/src/locales/hy.json +++ b/frontend/src/locales/hy.json @@ -868,7 +868,8 @@ "diff_effective_label": "Փաստացի SQL" }, "bytes_cap_body": "Գնահատված սկանավորում՝ {{estimate}} · Սահմանաչափ՝ {{limit}} ({{source}})", - "bytes_cap_no_estimate": "գնահատական չկա" + "bytes_cap_no_estimate": "գնահատական չկա", + "results_truncated_budget": "Վերադարձվել է {{count}} տող (սահմանափակված ձեր մնացած տվյալների բյուջեով)" } }, "reviews": { @@ -1590,7 +1591,9 @@ "grant_bytes_cap_label": "Սկանավորված բայթերի սահմանաչափ", "grant_bytes_cap_help": "Ավելի ցածր սահմանաչափ այս թույլտվության համար։ Կիրառվում է տվյալների աղբյուրի և բոլոր թույլտվությունների սահմանաչափերից ամենախիստը։", "grant_bytes_cap_min": "Սկանավորված բայթերի սահմանաչափը պետք է լինի առնվազն 1 բայթ։", - "grant_bytes_cap_placeholder": "Տվյալների աղբյուրի լռելյայնը" + "grant_bytes_cap_placeholder": "Տվյալների աղբյուրի լռելյայնը", + "tab_data_budgets": "Տվյալների բյուջեներ · {{count}}", + "sample_truncated_budget": "{{count}} տող (սահմանափակված ձեր մնացած տվյալների բյուջեով)" } }, "admin": { @@ -2471,7 +2474,8 @@ "minutes_suffix": "ր", "cicd_present_label": "CI/CD ծագում", "scan_type_placeholder": "օր.՝ Seq Scan, COLLSCAN, Index*", - "bytes_value_label": "Սկանավորված բայթեր" + "bytes_value_label": "Սկանավորված բայթեր", + "budget_percent_label": "Տվյալների բյուջեի օգտագործված մասը" }, "langfuse": { "title": "Langfuse", @@ -2912,7 +2916,9 @@ "SCHEMA_CHANGE_PROMOTION_FAILED": "Ձեր սխեմայի փոփոխությունը {{changeSet}} ձախողվեց {{environment}}-ում", "SCHEMA_CHANGE_PROMOTION_FAILED_partial": "Ձեր սխեմայի փոփոխությունը {{changeSet}} միայն մասնակիորեն կիրառվեց {{environment}}-ում", "SCHEMA_DRIFT_DETECTED_one": "{{count}} նոր սխեմայի շեղում {{pipeline}} / {{environment}}-ում", - "SCHEMA_DRIFT_DETECTED_other": "{{count}} նոր սխեմայի շեղում {{pipeline}} / {{environment}}-ում" + "SCHEMA_DRIFT_DETECTED_other": "{{count}} նոր սխեմայի շեղում {{pipeline}} / {{environment}}-ում", + "DATA_BUDGET_THRESHOLD_REACHED": "Դուք օգտագործել եք {{datasource}}-ում ձեր {{budget}} տվյալների բյուջեի {{percent}}%-ը", + "DATA_BUDGET_EXHAUSTED": "{{user}}-ի {{budget}} տվյալների բյուջեն {{datasource}}-ում սպառված է" } }, "profile": { @@ -3287,7 +3293,8 @@ "cicd_origin": "CI/CD ծագում", "estimated_rows": "Գնահատված տողեր", "scan_type": "Սկանավորման տեսակ", - "estimated_bytes_scanned": "Սկանավորվող բայթերի գնահատական" + "estimated_bytes_scanned": "Սկանավորվող բայթերի գնահատական", + "data_budget_used_percent": "Տվյալների բյուջեի օգտագործում (%)" }, "query_shape": { "JOIN": "Միացում (JOIN)", @@ -3562,7 +3569,8 @@ "EXPORT_POLICY_MANAGE": "Կառավարել արտահանման քաղաքականությունները", "ROW_LIMIT_POLICY_MANAGE": "Կառավարել տողերի սահմանաչափի քաղաքականությունները", "DEPLOYMENT_PIPELINE_MANAGE": "Կառավարել տեղակայման փիփլայնները", - "DEPLOYMENT_REVIEW": "Վերանայել տեղակայումները" + "DEPLOYMENT_REVIEW": "Վերանայել տեղակայումները", + "DATA_BUDGET_MANAGE": "Կառավարել տվյալների բյուջեները" }, "api_variable_kind": { "CONSTANT": "Հաստատուն", @@ -3733,7 +3741,8 @@ "ROW_SECURITY": "Տողային անվտանգություն", "MASKING": "Քողարկում", "BREAK_GLASS": "Արտակարգ մուտք", - "BYTES_SCANNED_CAP": "Սկանավորված բայթերի սահմանաչափ" + "BYTES_SCANNED_CAP": "Սկանավորված բայթերի սահմանաչափ", + "DATA_BUDGET": "Տվյալների բյուջե" }, "api_decision_step": { "CONNECTOR_GATES": "Միակցիչի ստուգումներ", @@ -3802,6 +3811,10 @@ "EXCEEDED": "Մերժված է. գնահատականը գերազանցում է սկանավորված բայթերի սահմանաչափը", "NO_ESTIMATE_REVIEW": "Պահված է ստուգման համար. սահմանաչափի ներքո բայթերի գնահատական չկա", "NO_ESTIMATE_REJECTED": "Մերժված է. սահմանաչափի ներքո բայթերի գնահատական չկա" + }, + "data_budget_breach_action": { + "REQUIRE_REVIEW": "Ուղարկել մարդկային վերանայման", + "REJECT": "Մերժել" } }, "access": { @@ -5812,7 +5825,14 @@ "bytes_scanned_cap": "Սկանավորված բայթերի սահմանաչափ", "bytes_scanned_cap_outcome": "Սահմանաչափի արդյունք", "bytes_scanned_cap_source": "Սահմանաչափի աղբյուր", - "estimated_bytes_scanned": "Սկանավորվող բայթերի գնահատական" + "estimated_bytes_scanned": "Սկանավորվող բայթերի գնահատական", + "data_budget_action": "Բյուջեի սպառման գործողություն", + "data_budget_id": "Տվյալների բյուջե", + "data_budget_name": "Տվյալների բյուջեի անուն", + "data_budget_remaining_bytes": "Բյուջեի մնացած բայթեր", + "data_budget_remaining_rows": "Բյուջեի մնացած տողեր", + "data_budget_suppressed": "Կասեցված է տվյալների բյուջեով", + "data_budget_used_percent": "Տվյալների բյուջեի օգտագործում" }, "use_id": "Օգտագործել ID {{id}}", "user_id_placeholder": "Տեղադրեք օգտատիրոջ ID", @@ -5832,5 +5852,93 @@ "key_control_chars": "Հավելվածի անունը չպետք է պարունակի կառավարման նիշեր։", "rotate_help": "Թողեք դատարկ՝ ընթացիկ բանալու հավելվածի անունը պահպանելու համար։", "column": "Հավելված" + }, + "dataBudgets": { + "window_hours_one": "{{count}} ժամ", + "window_hours_other": "{{count}} ժամ", + "window_days_one": "{{count}} օր", + "window_days_other": "{{count}} օր", + "indicator": { + "title": "Ձեր տվյալների բյուջեն", + "exhausted_title": "Տվյալների բյուջեն սպառված է", + "exhausted_reject": "Այս տվյալների աղբյուրի նոր ընթերցումները մերժվում են, մինչև նախորդ ընթերցումները դուրս գան պատուհանից։", + "exhausted_review": "Այս տվյալների աղբյուրի նոր ընթերցումները պահանջում են վերանայողի հաստատում, մինչև նախորդ ընթերցումները դուրս գան պատուհանից։", + "nearly_used": "Դուք օգտագործել եք այս տվյալների աղբյուրի ձեր բյուջեի մեծ մասը։", + "rows_left": "Մնացել է {{remaining}} տող {{limit}}-ից", + "bytes_left": "Մնացել է {{remaining}} {{limit}}-ից", + "per_window": "յուրաքանչյուր {{window}}", + "used_aria": "{{name}}՝ օգտագործված է {{percent}}%" + }, + "tab": { + "title": "Տվյալների բյուջեներ", + "description": "Սահմանափակեք, թե քանի տող և արդյունքի բայթ կարող է յուրաքանչյուր օգտատեր կարդալ այս աղբյուրից սահող պատուհանում։ Հարցումը սահմանափակվում է մնացած ծավալով. բյուջեի սպառումից հետո նոր ընթերցումները մերժվում են կամ ուղարկվում մարդկային վերանայման։", + "add": "Ավելացնել բյուջե", + "empty_title": "Տվյալների բյուջեներ չկան", + "empty_description": "Այս աղբյուրի ընթերցումները սահմանափակված են միայն ըստ հարցման։", + "col_name": "Անուն", + "col_limits": "Սահմանաչափեր", + "col_window": "Պատուհան", + "col_action": "Սպառվելիս", + "col_applies_to": "Կիրառվում է", + "col_enabled": "Միացված", + "col_actions": "Գործողություններ", + "state_disabled": "Անջատված", + "edit": "Խմբագրել բյուջեն", + "delete": "Ջնջել բյուջեն", + "delete_confirm_title": "Ջնջե՞լ այս տվյալների բյուջեն", + "delete_confirm_body": "Այն օգտատերերը, որոնց վրա կիրառվում էր, այլևս չեն սահմանափակվի։ Գրանցված օգտագործումը պահպանվում է մինչև ժամկետի ավարտը։", + "delete_success": "Տվյալների բյուջեն ջնջվեց", + "delete_error": "Չհաջողվեց ջնջել տվյալների բյուջեն", + "save_success": "Տվյալների բյուջեն պահպանվեց", + "save_error": "Չհաջողվեց պահպանել տվյալների բյուջեն", + "create_title": "Նոր տվյալների բյուջե", + "edit_title": "Խմբագրել տվյալների բյուջեն", + "label_name": "Անուն", + "name_required": "Մուտքագրեք անուն", + "name_max": "Առավելագույնը 120 նիշ", + "label_max_rows": "Տողերի սահմանաչափ", + "label_max_bytes": "Արդյունքի չափի սահմանաչափ", + "limits_hint": "Սահմանեք տողերի, չափի սահմանաչափ կամ երկուսը։ Յուրաքանչյուր օգտատեր ունի իր սեփական ծավալը։", + "limit_required": "Սահմանեք տողերի կամ չափի սահմանաչափ", + "rows_min": "Տողերի սահմանաչափը պետք է լինի առնվազն 1", + "bytes_min": "Չափի սահմանաչափը պետք է լինի առնվազն 1 բայթ", + "label_window": "Սահող պատուհան", + "window_hint": "Օգտագործումը հաշվվում է անցած պատուհանում՝ հիմա պահից։", + "window_range": "Պատուհանը պետք է լինի 1 ժամից մինչև 31 օր", + "window_unit": "Պատուհանի միավոր", + "unit_hours": "ժամ", + "unit_days": "օր", + "label_breach_action": "Երբ բյուջեն սպառվում է", + "breach_action_hint": "Մարդկային վերանայումը թույլ է տալիս օրինական վերլուծաբանին ավարտել ծանր օրը, մերժումը անմիջապես կանգնեցնում է ընթերցումները։ Արտակարգ մուտքը երբեք չի արգելափակվում, բայց հաշվվում է։", + "label_warn_threshold": "Զգուշացնել (% օգտագործված)", + "warn_threshold_hint": "Օգտատերը ծանուցվում է մեկ անգամ, երբ անցնում է այս մասը։ Թողեք դատարկ՝ առանց զգուշացման։", + "threshold_range": "Շեմը պետք է լինի 1-ից 99", + "label_applies_roles": "Կիրառվում է դերերի վրա", + "applies_hint": "Թողեք երեքն էլ դատարկ՝ բյուջեն այս աղբյուրի բոլոր օգտատերերի վրա կիրառելու համար։", + "label_applies_groups": "Կիրառվում է խմբերի վրա", + "label_applies_users": "Կիրառվում է օգտատերերի վրա", + "label_enabled": "Միացված", + "applies_everyone": "Բոլորը", + "applies_roles_one": "{{count}} դեր", + "applies_roles_other": "{{count}} դեր", + "applies_groups_one": "{{count}} խումբ", + "applies_groups_other": "{{count}} խումբ", + "applies_users_one": "{{count}} օգտատեր", + "applies_users_other": "{{count}} օգտատեր", + "rows_value": "{{value}} տող", + "unit_minutes": "րոպե" + }, + "usage": { + "action": "Տվյալների օգտագործում", + "title": "Տվյալների օգտագործում — {{name}}", + "description": "Տողեր և արդյունքի բայթեր, որոնք այս օգտատերը կարդացել է իր վրա կիրառվող յուրաքանչյուր բյուջեի պատուհանում։", + "empty": "Այս օգտատիրոջ վրա տվյալների բյուջե չի կիրառվում։", + "load_error": "Չհաջողվեց բեռնել այս օգտատիրոջ տվյալների օգտագործումը", + "exhausted": "Սպառված", + "rows_used": "{{used}} տող {{limit}}-ից", + "bytes_used": "{{used}} {{limit}}-ից" + }, + "window_minutes_one": "{{count}} րոպե", + "window_minutes_other": "{{count}} րոպե" } } diff --git a/frontend/src/locales/ru.json b/frontend/src/locales/ru.json index 5367102af..5adf6aaf9 100644 --- a/frontend/src/locales/ru.json +++ b/frontend/src/locales/ru.json @@ -868,7 +868,8 @@ "diff_effective_label": "Фактический SQL" }, "bytes_cap_body": "Оценка сканирования: {{estimate}} · Лимит: {{limit}} ({{source}})", - "bytes_cap_no_estimate": "нет оценки" + "bytes_cap_no_estimate": "нет оценки", + "results_truncated_budget": "Возвращено строк: {{count}} (ограничено оставшимся бюджетом данных)" } }, "reviews": { @@ -1590,7 +1591,9 @@ "grant_bytes_cap_label": "Лимит сканируемых байтов", "grant_bytes_cap_help": "Более низкий лимит для этой выдачи доступа. Применяется самый строгий из лимита источника данных и всех выдач.", "grant_bytes_cap_min": "Лимит сканируемых байтов должен быть не менее 1 байта.", - "grant_bytes_cap_placeholder": "По умолчанию источника данных" + "grant_bytes_cap_placeholder": "По умолчанию источника данных", + "tab_data_budgets": "Бюджеты данных · {{count}}", + "sample_truncated_budget": "Строк: {{count}} (ограничено оставшимся бюджетом данных)" } }, "admin": { @@ -2471,7 +2474,8 @@ "minutes_suffix": "мин", "cicd_present_label": "Источник CI/CD", "scan_type_placeholder": "напр. Seq Scan, COLLSCAN, Index*", - "bytes_value_label": "Сканируемые байты" + "bytes_value_label": "Сканируемые байты", + "budget_percent_label": "Доля использованного бюджета данных" }, "langfuse": { "title": "Langfuse", @@ -2912,7 +2916,9 @@ "SCHEMA_CHANGE_PROMOTION_FAILED": "Ваше изменение схемы {{changeSet}} не выполнено в {{environment}}", "SCHEMA_CHANGE_PROMOTION_FAILED_partial": "Ваше изменение схемы {{changeSet}} применено в {{environment}} лишь частично", "SCHEMA_DRIFT_DETECTED_one": "{{count}} новая находка дрейфа схемы в {{pipeline}} / {{environment}}", - "SCHEMA_DRIFT_DETECTED_other": "Новых находок дрейфа схемы в {{pipeline}} / {{environment}}: {{count}}" + "SCHEMA_DRIFT_DETECTED_other": "Новых находок дрейфа схемы в {{pipeline}} / {{environment}}: {{count}}", + "DATA_BUDGET_THRESHOLD_REACHED": "Вы использовали {{percent}}% бюджета данных {{budget}} в {{datasource}}", + "DATA_BUDGET_EXHAUSTED": "Бюджет данных {{budget}} пользователя {{user}} в {{datasource}} исчерпан" } }, "profile": { @@ -3287,7 +3293,8 @@ "cicd_origin": "Источник CI/CD", "estimated_rows": "Оценка строк", "scan_type": "Тип сканирования", - "estimated_bytes_scanned": "Оценка сканируемых байтов" + "estimated_bytes_scanned": "Оценка сканируемых байтов", + "data_budget_used_percent": "Использовано бюджета данных (%)" }, "query_shape": { "JOIN": "Соединение (JOIN)", @@ -3562,7 +3569,8 @@ "EXPORT_POLICY_MANAGE": "Управлять политиками экспорта", "ROW_LIMIT_POLICY_MANAGE": "Управление политиками лимита строк", "DEPLOYMENT_PIPELINE_MANAGE": "Управлять пайплайнами развертываний", - "DEPLOYMENT_REVIEW": "Проверять развертывания" + "DEPLOYMENT_REVIEW": "Проверять развертывания", + "DATA_BUDGET_MANAGE": "Управление бюджетами данных" }, "api_variable_kind": { "CONSTANT": "Константа", @@ -3733,7 +3741,8 @@ "ROW_SECURITY": "Построчная безопасность", "MASKING": "Маскирование", "BREAK_GLASS": "Экстренный доступ", - "BYTES_SCANNED_CAP": "Лимит сканируемых байтов" + "BYTES_SCANNED_CAP": "Лимит сканируемых байтов", + "DATA_BUDGET": "Бюджет данных" }, "api_decision_step": { "CONNECTOR_GATES": "Проверки коннектора", @@ -3802,6 +3811,10 @@ "EXCEEDED": "Отклонено: оценка превышает лимит сканируемых байтов", "NO_ESTIMATE_REVIEW": "Отправлено на проверку: нет оценки в байтах при лимите", "NO_ESTIMATE_REJECTED": "Отклонено: нет оценки в байтах при лимите" + }, + "data_budget_breach_action": { + "REQUIRE_REVIEW": "Отправить на проверку", + "REJECT": "Отклонить" } }, "access": { @@ -5812,7 +5825,14 @@ "bytes_scanned_cap": "Лимит сканируемых байтов", "bytes_scanned_cap_outcome": "Результат лимита", "bytes_scanned_cap_source": "Источник лимита", - "estimated_bytes_scanned": "Оценка сканируемых байтов" + "estimated_bytes_scanned": "Оценка сканируемых байтов", + "data_budget_action": "Действие при исчерпании бюджета", + "data_budget_id": "Бюджет данных", + "data_budget_name": "Название бюджета данных", + "data_budget_remaining_bytes": "Осталось байтов бюджета", + "data_budget_remaining_rows": "Осталось строк бюджета", + "data_budget_suppressed": "Подавлено бюджетом данных", + "data_budget_used_percent": "Использовано бюджета данных" }, "use_id": "Использовать ID {{id}}", "user_id_placeholder": "Вставьте ID пользователя", @@ -5832,5 +5852,93 @@ "key_control_chars": "Имя приложения не должно содержать управляющих символов.", "rotate_help": "Оставьте пустым, чтобы сохранить имя приложения текущего ключа.", "column": "Приложение" + }, + "dataBudgets": { + "window_hours_one": "{{count}} час", + "window_hours_other": "{{count}} ч", + "window_days_one": "{{count}} день", + "window_days_other": "{{count}} дн.", + "indicator": { + "title": "Ваш бюджет данных", + "exhausted_title": "Бюджет данных исчерпан", + "exhausted_reject": "Новые чтения из этого источника данных отклоняются, пока предыдущие чтения не выйдут за пределы окна.", + "exhausted_review": "Новые чтения из этого источника данных требуют одобрения проверяющего, пока предыдущие чтения не выйдут за пределы окна.", + "nearly_used": "Вы израсходовали большую часть бюджета данных для этого источника.", + "rows_left": "Осталось {{remaining}} из {{limit}} строк", + "bytes_left": "Осталось {{remaining}} из {{limit}}", + "per_window": "за {{window}}", + "used_aria": "{{name}}: использовано {{percent}}%" + }, + "tab": { + "title": "Бюджеты данных", + "description": "Ограничьте, сколько строк и байтов результата каждый пользователь может прочитать из этого источника в скользящем окне. Запрос ограничивается оставшимся объёмом; когда бюджет исчерпан, новые чтения отклоняются или отправляются на проверку.", + "add": "Добавить бюджет", + "empty_title": "Нет бюджетов данных", + "empty_description": "Чтения из этого источника ограничены только на уровне запроса.", + "col_name": "Название", + "col_limits": "Лимиты", + "col_window": "Окно", + "col_action": "При исчерпании", + "col_applies_to": "Применяется к", + "col_enabled": "Включён", + "col_actions": "Действия", + "state_disabled": "Выключен", + "edit": "Изменить бюджет", + "delete": "Удалить бюджет", + "delete_confirm_title": "Удалить этот бюджет данных?", + "delete_confirm_body": "Пользователи больше не будут им ограничены. Учтённое использование хранится, пока не устареет.", + "delete_success": "Бюджет данных удалён", + "delete_error": "Не удалось удалить бюджет данных", + "save_success": "Бюджет данных сохранён", + "save_error": "Не удалось сохранить бюджет данных", + "create_title": "Новый бюджет данных", + "edit_title": "Изменить бюджет данных", + "label_name": "Название", + "name_required": "Введите название", + "name_max": "Не более 120 символов", + "label_max_rows": "Лимит строк", + "label_max_bytes": "Лимит объёма результата", + "limits_hint": "Задайте лимит строк, объёма или оба. У каждого пользователя свой объём.", + "limit_required": "Задайте лимит строк или объёма", + "rows_min": "Лимит строк должен быть не меньше 1", + "bytes_min": "Лимит объёма должен быть не меньше 1 байта", + "label_window": "Скользящее окно", + "window_hint": "Использование считается за прошедшее окно, отсчитанное от текущего момента.", + "window_range": "Окно должно быть от 1 часа до 31 дня", + "window_unit": "Единица окна", + "unit_hours": "часов", + "unit_days": "дней", + "label_breach_action": "Когда бюджет исчерпан", + "breach_action_hint": "Проверка человеком позволяет добросовестному аналитику завершить загруженный день; отклонение сразу останавливает чтения. Экстренный доступ не блокируется, но учитывается.", + "label_warn_threshold": "Предупреждать при (% использовано)", + "warn_threshold_hint": "Пользователь получит уведомление один раз при превышении этой доли. Оставьте пустым, чтобы не предупреждать.", + "threshold_range": "Порог должен быть от 1 до 99", + "label_applies_roles": "Применяется к ролям", + "applies_hint": "Оставьте все три поля пустыми, чтобы бюджет применялся ко всем пользователям источника.", + "label_applies_groups": "Применяется к группам", + "label_applies_users": "Применяется к пользователям", + "label_enabled": "Включён", + "applies_everyone": "Все", + "applies_roles_one": "{{count}} роль", + "applies_roles_other": "{{count}} ролей", + "applies_groups_one": "{{count}} группа", + "applies_groups_other": "{{count}} групп", + "applies_users_one": "{{count}} пользователь", + "applies_users_other": "{{count}} пользователей", + "rows_value": "{{value}} строк", + "unit_minutes": "минут" + }, + "usage": { + "action": "Использование данных", + "title": "Использование данных — {{name}}", + "description": "Строки и байты результата, прочитанные пользователем в окне каждого применимого к нему бюджета.", + "empty": "К этому пользователю не применяется ни один бюджет данных.", + "load_error": "Не удалось загрузить использование данных пользователя", + "exhausted": "Исчерпан", + "rows_used": "{{used}} из {{limit}} строк", + "bytes_used": "{{used}} из {{limit}}" + }, + "window_minutes_one": "{{count}} минута", + "window_minutes_other": "{{count}} мин" } } diff --git a/frontend/src/locales/zh-CN.json b/frontend/src/locales/zh-CN.json index ea8a8cb5f..a7aac24fa 100644 --- a/frontend/src/locales/zh-CN.json +++ b/frontend/src/locales/zh-CN.json @@ -868,7 +868,8 @@ "diff_effective_label": "实际执行的 SQL" }, "bytes_cap_body": "预计扫描：{{estimate}} · 上限：{{limit}}（{{source}}）", - "bytes_cap_no_estimate": "无估算" + "bytes_cap_no_estimate": "无估算", + "results_truncated_budget": "返回 {{count}} 行（受剩余数据预算限制）" } }, "reviews": { @@ -1590,7 +1591,9 @@ "grant_bytes_cap_label": "扫描字节上限", "grant_bytes_cap_help": "为此授权设置更低的上限。取数据源上限与所有授权中最严格的一个。", "grant_bytes_cap_min": "扫描字节上限至少为 1 字节。", - "grant_bytes_cap_placeholder": "数据源默认值" + "grant_bytes_cap_placeholder": "数据源默认值", + "tab_data_budgets": "数据预算 · {{count}}", + "sample_truncated_budget": "{{count}} 行（受剩余数据预算限制）" } }, "admin": { @@ -2471,7 +2474,8 @@ "minutes_suffix": "分钟", "cicd_present_label": "来自 CI/CD", "scan_type_placeholder": "例如 Seq Scan、COLLSCAN、Index*", - "bytes_value_label": "扫描字节数" + "bytes_value_label": "扫描字节数", + "budget_percent_label": "数据预算已用比例" }, "langfuse": { "title": "Langfuse", @@ -2912,7 +2916,9 @@ "SCHEMA_CHANGE_PROMOTION_FAILED": "您的架构变更 {{changeSet}} 在 {{environment}} 上失败", "SCHEMA_CHANGE_PROMOTION_FAILED_partial": "您的架构变更 {{changeSet}} 仅部分应用到 {{environment}}", "SCHEMA_DRIFT_DETECTED_one": "{{pipeline}} / {{environment}} 上有 {{count}} 项新的架构漂移发现", - "SCHEMA_DRIFT_DETECTED_other": "{{pipeline}} / {{environment}} 上有 {{count}} 项新的架构漂移发现" + "SCHEMA_DRIFT_DETECTED_other": "{{pipeline}} / {{environment}} 上有 {{count}} 项新的架构漂移发现", + "DATA_BUDGET_THRESHOLD_REACHED": "您已使用 {{datasource}} 上数据预算 {{budget}} 的 {{percent}}%", + "DATA_BUDGET_EXHAUSTED": "{{user}} 在 {{datasource}} 上的数据预算 {{budget}} 已用尽" } }, "profile": { @@ -3287,7 +3293,8 @@ "cicd_origin": "CI/CD 来源", "estimated_rows": "预估行数", "scan_type": "扫描类型", - "estimated_bytes_scanned": "预计扫描字节数" + "estimated_bytes_scanned": "预计扫描字节数", + "data_budget_used_percent": "数据预算已用（%）" }, "query_shape": { "JOIN": "连接（JOIN）", @@ -3562,7 +3569,8 @@ "EXPORT_POLICY_MANAGE": "管理导出策略", "ROW_LIMIT_POLICY_MANAGE": "管理行数限制策略", "DEPLOYMENT_PIPELINE_MANAGE": "管理部署流水线", - "DEPLOYMENT_REVIEW": "审核部署" + "DEPLOYMENT_REVIEW": "审核部署", + "DATA_BUDGET_MANAGE": "管理数据预算" }, "api_variable_kind": { "CONSTANT": "常量", @@ -3733,7 +3741,8 @@ "ROW_SECURITY": "行级安全", "MASKING": "脱敏", "BREAK_GLASS": "紧急访问", - "BYTES_SCANNED_CAP": "扫描字节上限" + "BYTES_SCANNED_CAP": "扫描字节上限", + "DATA_BUDGET": "数据预算" }, "api_decision_step": { "CONNECTOR_GATES": "连接器检查", @@ -3802,6 +3811,10 @@ "EXCEEDED": "已拒绝：估算超过扫描字节上限", "NO_ESTIMATE_REVIEW": "已转人工审查：上限下没有字节估算", "NO_ESTIMATE_REJECTED": "已拒绝：上限下没有字节估算" + }, + "data_budget_breach_action": { + "REQUIRE_REVIEW": "转交人工审核", + "REJECT": "拒绝" } }, "access": { @@ -5812,7 +5825,14 @@ "bytes_scanned_cap": "扫描字节上限", "bytes_scanned_cap_outcome": "上限结果", "bytes_scanned_cap_source": "上限来源", - "estimated_bytes_scanned": "预计扫描字节数" + "estimated_bytes_scanned": "预计扫描字节数", + "data_budget_action": "预算用尽时的操作", + "data_budget_id": "数据预算", + "data_budget_name": "数据预算名称", + "data_budget_remaining_bytes": "预算剩余字节", + "data_budget_remaining_rows": "预算剩余行数", + "data_budget_suppressed": "被数据预算抑制", + "data_budget_used_percent": "数据预算已用" }, "use_id": "使用 ID {{id}}", "user_id_placeholder": "粘贴用户 ID", @@ -5832,5 +5852,93 @@ "key_control_chars": "应用名称不能包含控制字符。", "rotate_help": "留空则沿用当前密钥的应用名称。", "column": "应用" + }, + "dataBudgets": { + "window_hours_one": "{{count}} 小时", + "window_hours_other": "{{count}} 小时", + "window_days_one": "{{count}} 天", + "window_days_other": "{{count}} 天", + "indicator": { + "title": "你的数据预算", + "exhausted_title": "数据预算已用尽", + "exhausted_reject": "在之前的读取移出窗口之前，此数据源上的新读取将被拒绝。", + "exhausted_review": "在之前的读取移出窗口之前，此数据源上的新读取需要审核人批准。", + "nearly_used": "你已用掉此数据源上的大部分数据预算。", + "rows_left": "剩余 {{remaining}} / {{limit}} 行", + "bytes_left": "剩余 {{remaining}} / {{limit}}", + "per_window": "每 {{window}}", + "used_aria": "{{name}}：已使用 {{percent}}%" + }, + "tab": { + "title": "数据预算", + "description": "限制每个用户在滚动窗口内可从此数据源读取的行数和结果字节数。查询会被限制在剩余额度内；预算用尽后，新的读取将被拒绝或转交人工审核。", + "add": "添加预算", + "empty_title": "没有数据预算", + "empty_description": "此数据源上的读取仅按单个查询限制。", + "col_name": "名称", + "col_limits": "上限", + "col_window": "窗口", + "col_action": "用尽时", + "col_applies_to": "适用于", + "col_enabled": "已启用", + "col_actions": "操作", + "state_disabled": "已停用", + "edit": "编辑预算", + "delete": "删除预算", + "delete_confirm_title": "删除此数据预算？", + "delete_confirm_body": "受其约束的用户将不再受限。已记录的用量会保留至过期。", + "delete_success": "数据预算已删除", + "delete_error": "无法删除数据预算", + "save_success": "数据预算已保存", + "save_error": "无法保存数据预算", + "create_title": "新建数据预算", + "edit_title": "编辑数据预算", + "label_name": "名称", + "name_required": "请输入名称", + "name_max": "最多 120 个字符", + "label_max_rows": "行数上限", + "label_max_bytes": "结果大小上限", + "limits_hint": "设置行数上限、结果大小上限或两者。每个用户拥有各自的额度。", + "limit_required": "请设置行数或结果大小上限", + "rows_min": "行数上限至少为 1", + "bytes_min": "结果大小上限至少为 1 字节", + "label_window": "滚动窗口", + "window_hint": "用量按从现在往回推算的窗口统计。", + "window_range": "窗口必须介于 1 小时到 31 天之间", + "window_unit": "窗口单位", + "unit_hours": "小时", + "unit_days": "天", + "label_breach_action": "预算用尽时", + "breach_action_hint": "人工审核可让正常工作的分析师完成繁忙的一天；拒绝则直接停止读取。紧急访问永不被阻止，但仍计入用量。", + "label_warn_threshold": "预警阈值（已用 %）", + "warn_threshold_hint": "用户超过此比例时会收到一次通知。留空表示不预警。", + "threshold_range": "阈值必须介于 1 到 99 之间", + "label_applies_roles": "适用角色", + "applies_hint": "三项都留空则预算适用于此数据源的所有用户。", + "label_applies_groups": "适用组", + "label_applies_users": "适用用户", + "label_enabled": "已启用", + "applies_everyone": "所有人", + "applies_roles_one": "{{count}} 个角色", + "applies_roles_other": "{{count}} 个角色", + "applies_groups_one": "{{count}} 个组", + "applies_groups_other": "{{count}} 个组", + "applies_users_one": "{{count}} 个用户", + "applies_users_other": "{{count}} 个用户", + "rows_value": "{{value}} 行", + "unit_minutes": "分钟" + }, + "usage": { + "action": "数据用量", + "title": "数据用量 — {{name}}", + "description": "此用户在每个适用预算窗口内读取的行数和结果字节数。", + "empty": "没有适用于此用户的数据预算。", + "load_error": "无法加载此用户的数据用量", + "exhausted": "已用尽", + "rows_used": "{{used}} / {{limit}} 行", + "bytes_used": "{{used}} / {{limit}}" + }, + "window_minutes_one": "{{count}} 分钟", + "window_minutes_other": "{{count}} 分钟" } } diff --git a/frontend/src/pages/admin/AuditLogPage.tsx b/frontend/src/pages/admin/AuditLogPage.tsx index e29a6aac6..801492012 100644 --- a/frontend/src/pages/admin/AuditLogPage.tsx +++ b/frontend/src/pages/admin/AuditLogPage.tsx @@ -86,6 +86,10 @@ const ACTIONS = [ 'ROW_LIMIT_POLICY_CREATED', 'ROW_LIMIT_POLICY_UPDATED', 'ROW_LIMIT_POLICY_DELETED', + 'DATA_BUDGET_CREATED', + 'DATA_BUDGET_UPDATED', + 'DATA_BUDGET_DELETED', + 'QUERY_DATA_BUDGET_ENFORCED', 'AUDIT_SINK_CREATED', 'AUDIT_SINK_UPDATED', 'AUDIT_SINK_DELETED', diff --git a/frontend/src/pages/admin/RoutingPoliciesPage.tsx b/frontend/src/pages/admin/RoutingPoliciesPage.tsx index 6bdbc4b9c..edd99cd13 100644 --- a/frontend/src/pages/admin/RoutingPoliciesPage.tsx +++ b/frontend/src/pages/admin/RoutingPoliciesPage.tsx @@ -958,6 +958,27 @@ function ConditionValueEditor({ name, operand, groups, roleOptions }: ConditionV ); + case 'data_budget_used_percent': + return ( +
+ +
+
Manage datasources———✓—
Manage users———✓—
Manage user groups———✓—
Manage data budgets———✓—
Manage review plans———✓—
Manage deployment pipelines———✓—
View audit log———✓—