feat(budget): enforce durable governed-inference reservations by pallakatos · Pull Request #553 · Azure/kars

Pal Lakatos-Toth (pallakatos) · 2026-09-09T11:15:11Z

Merged integration slice

Merged into kars-bridge at 2026-09-10T20:22:53Z as
ac9f1b96f79c1e0f78bd042a3faec58a6049b9f5.

Head: a5a1aea4f86e634fbb4cf36b6671ac1805ee9ad9.
Base: kars-bridge, incorporating actual integration ancestor
12d1f2a3b82ba82b903b1c6c0907492c9a721437.
This existing PR contains the reviewed recovery and unsupported-route repairs
plus the coherent native fixtures.

The PR targets the protected integration branch directly and now contains
the actually landed MCP foundation. Its combined production source at
1af895bc98dad70f09d2467bef34a01b7ae6fb45 passed focused review and2,453
Rust tests, isolated strict all-target/all-feature Clippy for both crates,
93CLI cases, types, six Python fixtures and Helm/source gates.
Final current-base CI 34520633905 passed before landing. All29required
app-bound checks and review conversations were satisfied; every original
branch protection was restored afterward, with no advisory/test waiver.
No main promotion or customer deployment is part of this PR.

Supported scope

Explicit GovernedInference provides durable token and operator-configured
maximum-price caps. It does not account for compute, GPU/VM, tools/MCP,
storage, networking, tax/FX or all-in invoices. Unbounded standalone defaults
remain unchanged; legacy planning budgets are not silently reinterpreted.

Immutable Task/Team lifetime identity and current full authority.
Durable Kubernetes accounting with atomic ancestor reservations and CAS.
Reserve/begin/settle per actual provider attempt, idempotency/replay fences
and conservative funding for ambiguous acceptance or missing final usage.
Router-only Pod/audience-bound authentication, real TokenReview/live identity,
qualified privacy prerequisites, pinned images and supported model contracts.
Non-destructive admission waits preserve Task UIDs and funded work.
Genuine revocation is distinct from temporary admission exhaustion.
Typed CLI/Helm/controller/router integration and real API/native scenarios.

Focused repair and review

Fresh independent review found a stale-authority recovery race: an observation
of invalid old authority A could close a newly enrolled B session in the latest
ledger. The repair compares the captured authority inside every store CAS
attempt, including conflict retries. Changed authority is deferred for a fresh
check; genuinely invalid unchanged A is still revoked.

Deterministic regressions cover replacement before the transaction, replacement
during a 409 retry, and unchanged-authority revocation. They retain B's new Pod
session/InFlight work and old maximum liabilities. Independent re-review of
exact repair e89040df reported no significant issues.

The audit records the maintainer's explicit delegated-review authorization and
labels Copilot as AI, not a second human:
docs/security-audits/2026-09-08-governed-inference-budgets.md.
No technical gate or audit rule was changed.

Qualification and remaining gates

At exact production source e89040df, guarded local execution passed 82
controller-binary budget cases (including five recovery cases; not additive),
40 router-library cases, strict paired all-target Clippy and formatting.
The later services merge changes only three CLI test files; its 39 forwarded
cases and typecheck pass. The current audit commit changes documentation only.

Native fixtures now verify same-image digest aliases, named-provider routing,
proper server-leaf TLS with negative checks, and bounded reconnection to the
actual UID/generation/image/budget-qualified Pod after a legitimate rollout.
They do not fake ledger state, suppress real 503/authority failures, remove
spending/cancellation assertions or extend the native readiness deadline.

The final combined MCP/budget/SRE qualification has passed. Older image,
TLS, port-forward and cancellation failures remain failed historical
evidence rather than being relabeled. Full credential integration and the
later complete Bridge product qualification remain separately scoped work.

Previous source2a95 CI 34493554446 passed Rust, CLI, benchmarks and the other
gates but failed Kind job 102935839436 (166 passed, one failed).
Four real owned routers now passed both health and governed readiness.
The later /v1/embeddings assertion returned502 instead of503 because
default-provider authentication ran before finite-operation support
classification. The reviewed repair moves only side-effect-free classification
earlier; request authentication/governance and final-send funding are unchanged.
The native503 expectation is retained and strengthened with code/type checks.

Exact repaired source058b7973 passed four new unsupported-route regressions,
61 router budget cases including those four, strict router all-target Clippy
and formatting; 71 JS/CLI cases and typecheck also passed. Its initial test
initializer type error was repaired before those tests ran, not counted as a
pass. Independent review found no significant issues in the production repair.
The actual-base merge changes no tracked source from qualified058, and the
final commit adds audit evidence only.

The prior native failure remains failed; the final native proof below is a
separate successful execution, not a readiness-only substitute.

The later4c2 native run 34509062774 confirmed the503/code/type repair, then
failed at cancellation because kubectl --patch-file - reads a literal file
named -, not stdin. A real-kubectl controlled reproduction made zero API
requests. Fixture-only repair1ee92de3 now sends a genuine merge PATCH via the
shared loopback Kind API client. Every retry retains Task/workspace UID,
generation, full spec/budget binding and fresh RV; only a matching API
Status/Conflict409 permits up to three attempts within30seconds.

Parent validation passed80 budget-related cases including23 cancellation
cases, actual kubectl/proxy tests, types/lint/syntax. Only external-tool unit
integration setup uses bounded30-second test deadlines; native deadlines,
settled8/uncertain30/reserved0 assertions and all production code are unchanged.
Both historical failed native runs are retained. Final job 103026695464
completed 172 passed, zero failed: unsupported503/code/type matched,
namespace GET, Task GET and cancellation PATCH returned200 with the retained
fences, and the actual broker/Pod authentication, sibling token and
maximum-price CAS, plus accepted-work cancellation/accounting assertions passed.

No H100 mutation, public image release, private Bridge visibility change or
main merge has occurred.

Local integration checkpoint only. Rust and real Kind qualification remain pending; no publication or enforcement approval is implied. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Forward exact 7dc7281. Preserve SRE projection and privacy lifecycle logic; use existing workspace TLS package versions and add only controller dependency edges. Budget Rust/Kind qualification remains pending; no Cargo invocation or publication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

…umers Capture stable same-workspace UID/RV ancestry, canonical readiness/attenuation/full authorization, and live Team ownership without coupling identity resolution to budget state. Budget callers verify their persisted pins separately. Add focused API/race/owner tests; Rust execution remains pending the coordinated Cargo lease. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Merge exact 068ae16 without changing Pending authorization semantics. Move the pending budget audit to the existing required security-audit path; no human signature or gate waiver is fabricated. Budget Rust and live Kind qualification remain pending the Cargo allocation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Record actual post-068ae160 static and cached-test results without claiming budget Rust, Kind or human signoff completion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Repair the five bounded source-review findings: module scopes, supported Team launches, non-destructive admission waits, final Anthropic usage evidence, and legacy planning selection. Preserve Task UIDs and existing funded executions while denying new admissions; retain explicit pause/policy/UID revocation. Add full Team/Task API interleaving and actual stream-settlement regressions. Rust tests are authored but unrun pending the parent's Cargo grant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Local-only qualification checkpoint on aa34c3f. Retain all previously uncommitted parent repairs: existing workspace hex and schemars dependency edges, kube 3 typed replace_status, ledger borrow/move, concurrent test root lifetime, required legacy inferenceRef fixture, and real text/event-stream MIME for all three wiremock SSE fixtures. The original HEAD was not compile/test qualified; these repairs are part of this checkpoint. Resolve strict Clippy without new allow/expect attributes, feature flags, or new dependencies. Rename Amounts.add to checked_add with identical arithmetic. Keep OutputField operator wire names via explicit serde renames and add exact serde/JsonSchema compatibility coverage. Move unchanged request normalization, shape validation and final route/egress classification into a shared router-only dispatch module, included by the controller only in tests. Keep the public VerifiedTaskNode.generation field for credential composition and explicitly verify that generation in the second live inventory, with regression coverage. Actual Helm/Rust drift tests revealed three pre-existing candidate failures: Task, Team and Profile budget scope nullability/descriptions, Task's untyped status budget binding, Team account reference schema, and CEL rule ordering. Align these templates to the existing generated Rust contract; retain the same CEL rule strings, lifetime opt-in/retention and legacy launch prohibition. No fallback prices, raw-token authority, all-compute scope, or feature expansion. All Cargo commands below used the existing guard, cwd this worktree, and only /Users/pallakatos/Private/Repos/kars/target. The guard sets CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=2, CARGO_NET_OFFLINE=true and the existing CARGO_TARGET_DIR; it enforces an 8.5 GiB free-space floor. Both packages kept default features. Exact invocation prefix: python3 /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/files/run-cargo-guard.py --cwd /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/kars-pr11-budgets -- Passing guarded commands: 1. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- budget task_identity 75 controller + 57 router = 132 passed, zero failures/ignored. Includes all 16 incomplete Anthropic cases, positive final SSE settlement, complete funded-Team interleavings, new generation and wire-schema tests. 2. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- helm_drift::tests::helm_ kars_task::tests:: kars_task::authorization_tests:: kars_team_reconciler::tests:: kars_team_reconciler::persistence_tests:: routes::model_routing routes::chat_completions routes::anthropic_messages routes::inference routes::spawn_policy forward_proxy:: failover:: blocklist:: 68 controller + 83 router = 151 passed, including all 15 Helm drift tests. Counts describe executions; some selectors overlap command 1. 3. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --test proxy_fake_upstream --test foundry_route_guard --test failover_walk --test multi_provider_guardrails --test anthropic_buffered_guardrail --test chat_output_guardrail_nonjson 24 real loopback HTTP tests passed: proxy 3, Foundry 9, failover 3, multi-provider 5, Anthropic 2, chat non-JSON 2; no external services. 4. cargo clippy --quiet --offline --locked -p kars-controller -p kars-inference-router --all-targets -- -D warnings Passed final all-target check, zero warnings and no new waivers. 5. cargo fmt --package kars-controller --package kars-inference-router 6. cargo fmt --package kars-controller --package kars-inference-router --check Passed. Only affected files required formatting. The compatibility/HTTP test invocations set TMPDIR=$PWD/.qualification-fixtures inside this worktree; that empty scratch directory has been removed. Minimum observed free space across qualification: 10.03 GiB; floor never hit. Passing existing Node/static commands, cwd this worktree unless noted: node cli/node_modules/vitest/vitest.mjs run --root cli --configLoader runner --no-cache src/commands/budget.test.ts 7 passed. Existing symlinked cache only; no dependency install. node cli/node_modules/vitest/vitest.mjs run --root deploy/helm/kars/tests --globals --no-cache src/inference-budget.test.ts src/local-inference.test.ts 18 passed (12 budget + 6 local-inference), rerun after template repair. cd cli && npm run typecheck Passed, no emit. git diff --check git diff --cached --check bash ci/check-copyright-headers.sh 736 source files passed with the new shared module staged. BASE_REF=068ae160 bash ci/check-loc.sh Passed against the reviewed parent. Additional source comparison confirmed normalize, shape validation, Quote accounting, mediated egress and operation classification unchanged modulo whitespace/relocation; Cargo.lock adds only hex and schemars 1.2.1 package edges. Limits: this is source/compile/strict-lint/test qualification, not production acceptance. Real budget/SRE Kind, human review of these repairs, combined credential/budget authority composition, and an independently fully-qualified router image digest remain pending. Privacy epoch remains an additional fence, not Pod UID/custom-audience TokenReview authentication. No public push, merge, main/customer change, Azure, Docker, image build/push or private Bridge actions. cli/node_modules remains untracked and is not part of this checkpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

github-actions · 2026-09-09T11:15:43Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

…andidate Carry the shared js-yaml lock repair and current SRE retirement/fixture work without replacing budget accounting or caller authority. Real combined Kubernetes qualification and human audit gates remain required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Hosted nextest runs each case independently; the pending-leaf fixture constructed a Kubernetes client before selecting the existing AWS-LC provider. Initialize it in the shared test setup so the real authority assertions execute without depending on another test process. No production authority or TLS policy changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Preserve the existing budget ledger, accounting and caller authentication; forward only the prerequisite resource-specific policy correction and its evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Bounded local source checkpoint on 2b2e0b8. Addresses the actual #553 hosted CNCF failures c3_conditions_array_present and c12_state_printer_column without changing conformance checks or scope. Add observational status.phase, observedGeneration, and standard Kubernetes Ready/LedgerValid conditions to both Rust and Helm. The Phase printer points to .status.phase; the existing .status.ledger.phase and ledger wire remain unchanged. Conditions use the existing project helper and standard Condition and jiff-backed Time types, list-map keys by type, and stable transition times. Populate status after UID-anchor creation, during ledger initialization, after sealing, with every changed ledger transaction, and in periodic recovery/backfill. Derive bootstrap, active, temporarily reserved, exhausted, revoked, capacity-blocked, frozen, closing/retired, invalid-ledger and API- unavailable observations from actual account state. No blanket Ready signal, router-health assertion, fallback pricing, private errors, or spend authority. Reporting reloads a fresh account and uses bounded UID/RV whole-status PUT. It preserves the exact ledger, including corrupt or missing data; it cannot initialize/reset money. Initialization also uses complete status PUT. Lost acknowledgements still fail; unchanged ledger mutations retain their original no-write/idempotent return. A failed reporting read/write is not successful publication; an unreachable API necessarily retains its last observation. No funded Task deletion, authority reset, or changed account limit/schema. Add 11 Rust regressions covering real store transitions and fault injection: - reserved headroom versus durable exhaustion, preserving funded dispatch; - expiry/headroom recovery and retained replay fences; - revoked/retired authority and accepted-work liability; - provider breach plus forged observational Ready not authorizing spending; - live recovery API failure, stable Unknown, and subsequent root retirement; - corrupt/missing ledger reporting without repair or reinitialization; - failed reads/writes, UID replacement, conflicts and lost reporting acks; - legacy observation backfill and stable timestamp across generations; - Helm/generated-Rust reporting schema and printer-column equality; - report CAS conflict with concurrent reservation, preserving new money; - lost bootstrap ledger-status acknowledgement without reset before sealing. Extend the existing Helm tests for new reporting fields in enabled/default shapes. Update the directly related operator documentation. FAST GATES ACTUALLY RUN AND PASSED, cwd this worktree: rustfmt --check --edition 2024 controller/src/inference_budget/store.rs Includes nested store/status test modules; syntax/format only, not compilation. rustfmt --check --edition 2024 --config skip_children=true controller/src/inference_budget/account.rs controller/src/inference_budget/status.rs controller/src/inference_budget/status_tests.rs controller/src/inference_budget/store.rs controller/src/inference_budget/store_tests.rs controller/src/inference_budget/recovery.rs controller/src/inference_budget/mod.rs controller/src/kars_team_reconciler/budget_interleaving_tests.rs node cli/node_modules/vitest/vitest.mjs run --root deploy/helm/kars/tests --globals --no-cache src/inference-budget.test.ts src/local-inference.test.ts 20 passed: 14 budget and 6 local-inference. node cli/node_modules/vitest/vitest.mjs run --root cli --configLoader runner --no-cache src/commands/budget.test.ts 7 passed. cd cli && npm run typecheck Passed, no emit. git diff --check git diff --cached --check bash ci/check-copyright-headers.sh 739 tracked source files passed with new files staged. BASE_REF=2b2e0b8018b5059b05467ac76d3aa6115575be78 bash ci/check-loc.sh Passed; every touched Rust module is <=800 lines; no new whitelist/waiver. Parsed-YAML comparison proved the prior financial spec, immutable-grant CEL and status.ledger schema exactly unchanged. No Cargo manifest or lock edits. NOT YET RUST/CNCF QUALIFIED: no Cargo grant was held or used for this checkpoint. The root target remains exclusively leased to the credential RPC owner. After explicit parent grant, run the existing guarded offline/locked commands with existing root target, CARGO_INCREMENTAL=0 and CARGO_BUILD_JOBS=2: cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- budget task_identity helm_drift cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router -p kars-cncf-conformance --test criteria The latter is the actual 17-case CNCF criteria/report runner, not a substitute. cargo clippy --quiet --offline --locked -p kars-controller -p kars-inference-router --all-targets -- -D warnings cargo fmt --package kars-controller --package kars-inference-router --check No public push, parent review claim, merge, customer/main/private Bridge edit, Azure/cloud action, Docker/image action, install or new target. Pending: actual Rust compile/tests, generated schema drift, CNCF17, strict Clippy, parent review, and independent existing SRE Kind/cross-credential production qualification. cli/node_modules remains an unstaged existing symlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Actual guarded Rust qualification of abbfc7d found and closed four bounded issues rather than treating source checks as compilation evidence: - Whole-status acknowledgement comparison now compares exact serialized wire values: standard Kubernetes Time serializes at second precision, so comparing unpublished subsecond clock values falsely reported committed status corrupt. Ledger counters and all other fields remain exact in this comparison. - KarsBudgetAccount::crd() hit kube's structural-schema panic on the existing tagged MaximumPrice union. Its generated ledger schema now uses the same bounded structural envelope already deployed by Helm, with existing contract version and capacity constants. The actual typed Ledger and all runtime financial/authority validation are unchanged. The schema regression now checks the ledger envelope too; only documentation, unordered required sets and equivalent JSON-Schema numeric minima are canonicalized for comparison. - Move the concurrent-reservation and bootstrap-lost-ack tests to module scope. Both are now genuinely registered/executed, not nested inactive functions. - Explicitly install the controller's AWS-LC Rustls provider in the legacy Pod fixture before creating its Client. Verify this fixture in isolation too. PASSING ACTUAL CARGO COMMANDS, cwd kars-pr11-budgets: All invoked through: python3 /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/files/run-cargo-guard.py --cwd /Users/pallakatos/.copilot/session-state/a7f32227-c055-4df4-81ee-ce27d7f2d74d/kars-pr11-budgets -- The guard used only /Users/pallakatos/Private/Repos/kars/target, default core package features, CARGO_INCREMENTAL=0, CARGO_BUILD_JOBS=2, CARGO_NET_OFFLINE=true, and the existing 8.5 GiB free-space floor. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- budget task_identity helm_drift 116 controller + 57 router = 173 passed; zero failed/ignored. Includes all 11 new status regressions, bootstrap/active/reserved/exhausted/revoked/frozen/ corrupt/unavailable transitions, report CAS races and lost acks, timestamp stability, exact generated/Helm reporting+ledger schema checks, and existing budget/interleaving/identity/Helm gates. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router -p kars-cncf-conformance --test criteria 17/17 passed, including c3_conditions_array_present, c12_state_printer_column, full_report_is_all_pass and report stability. cargo test --quiet --offline --locked -p kars-controller -p kars-inference-router --lib --bins -- no_task_owner_no_reference_is_a_byte_safe_legacy_noop_without_api_access 1 controller fixture passed in isolation, without another test installing TLS. cargo clippy --quiet --offline --locked -p kars-controller -p kars-inference-router --all-targets -- -D warnings Passed with no new lint waivers. cargo fmt --package kars-controller --package kars-inference-router --check Passed. FAST GATES: node cli/node_modules/vitest/vitest.mjs run --root deploy/helm/kars/tests --globals --no-cache src/inference-budget.test.ts src/local-inference.test.ts Re-run: 20 passed (14 budget + 6 local-inference). CLI budget 7 and CLI typecheck passed at abbfc7d; no CLI source changed since. git diff --check; git diff --cached --check bash ci/check-copyright-headers.sh 739 source files passed. BASE_REF=2b2e0b8018b5059b05467ac76d3aa6115575be78 bash ci/check-loc.sh Passed; all changed Rust modules remain <=800 lines; no whitelist changes. No Cargo manifest or lock changes, no dependency installs or alternate targets. Minimum observed free space during this lease: 9.84 GiB; guard floor not hit. This qualifies the local budget-account status/CNCF repair, not production rollout. Parent review, hosted CI, real SRE/Kind and cross-credential composition remain independent gates; no claim of a qualified production router digest. No public push/merge, main/customer/private Bridge edits, Azure, Docker or image operations. Existing cli/node_modules symlink remains untracked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Minimal production repair for the actual standalone ReplicaSetCreateError: replace only request.subResource with request.?subResource.orValue('') in the single shared kars-inference-budget-workloads validation expression. The shared JSON feeds both Rust include_str verification and Helm; no parallel copy exists. Deployment creation remains core-controller-only. Exact Kubernetes controller usernames, ephemeral-container restriction, namespace selector, resource rules and every other predicate are unchanged. Parent approved this exact production change from CI34418924042 native missing-subResource evidence. Extend the existing real budget API job with a protected, UID-owned namespace: - create a real Deployment as the core controller and observe its actual native Deployment->ReplicaSet->Pod owner-UID chain, without asserting/faking readiness; - exercise ReplicaSet/Pod primary CREATE under each exact allowed Kubernetes controller identity; - reject direct Deployment creation by those controllers, tenant primary Deployment/ReplicaSet/Pod requests, and ephemeral-container updates by all non-core test principals for the exact shared policy/message/reason; - prove the fixture principals have the relevant RBAC, so arbitrary RBAC failure cannot stand in for admission; preserve UID/RV cleanup with bounded conflicts. The tokenless never-scheduled workload fixtures run no model or image. Existing real kubelet TokenRequest/Pod-UID checks remain unchanged in their unfenced namespace. No account status or Pod readiness is manufactured. Five new fast orchestration/denial/cleanup tests pass in the existing CLI runner; 22 combined CLI/credential/privacy tests, 21 Helm tests, CLI typecheck, 91 Python checks, 10 legacy Helm checks, chart lint and JS/Bash syntax pass on the coherent candidate. Unit orchestration is not native evidence: the new actual workload cases remain pending the one authorized hosted API/standalone run. No local Cargo, Kind/Docker, installs, SDK/private app or unrelated production changes. This commit is the isolated budget patch/ref for forward-port to original #553 once hosted compile/native qualification succeeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Forward the exact shared admission repair and native probes from 63c9a06. Preserve the exact controller allowlist, core-only Deployments, namespace fence, and ephemeral-container denial. Helm and Rust keep consuming the same JSON policy. Native composed run 34428691641/job 102719417174 proves all 19 actor and actual controller-chain cases. Original-branch targeted CLI/probe tests (12), TypeScript, Helm and syntax checks pass. Record honest limits: composed standalone remains 23 passed/1 router image-pull failure; budget spending/cancellation and fresh exact-head hosted qualification remain outstanding. No local Cargo, customer deployment, main change or signature waiver. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

…outs Merge exact services base 6b34d5e into existing PR #553. Preserve the signed SRE privacy/Pending/current-epoch fixes and audits, and combine credential/SRE refresh conditions. Copy only five native test files from c7309f6, retaining byte-identical all-node canonical/literal CRI image verification and named-provider/TLS helpers. Replace first-Running-Pod readiness with pinned Task/account/Sandbox/Namespace/Deployment identities, current RS template and Pod UID checks, a shared 120s deadline and at most three reconnects. Require health and private readiness, revalidate before/after probes, and clean up only owned processes and UID-preconditioned fixture Secrets. Provider attempts, settled/reserved/uncertain spending, sibling denials and cancellation assertions are byte-identical to c4291b0. Read-only run 34479624004 artifact inspection found generation-2 owned Deployments, old ReplicaSets at zero and replacement Pods; empty stage facts do not establish an auth/contract failure. No workflow/lane or production budget behavior changes, no push or dispatch. Validation: 181 targeted CLI/fixture tests; 20 existing Helm contract tests; 108 private SRE fixture tests; CLI typecheck; Helm lint; JS and shell syntax; diff checks; exact candidate helper and signed-services source parity. Lint has only existing warnings. Existing cli/node_modules symlink preserved; Node 24.9.0 / Vitest 4.1.10 differs from lock 4.1.8. Hosted exact-lock/full native E2E remains required. No Cargo, Kind, Docker or package installs. Ephemeral certificate directories removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Recheck the full captured TaskAuthority inside Store.transact before stale-source recovery closes a subtree, including each UID/resourceVersion CAS retry. If a newer enrollment is current, make no ledger mutation and defer its explicit live Task check to the next scan. Unchanged invalid authority still revokes, and accepted old work stays fully funded. Add deterministic wiremock interleavings for B enrollment/new Pod/InFlight work before the revocation transaction and during a 409 retry. Assert B survives, the next scan rechecks its live source, subsequent invalid B revokes, unchanged invalid A still revokes, and old/new maximum liabilities, limits and account UID remain intact. Reuse the existing Store test server. Validation completed without Cargo: rustfmt check for recovery source and regression tests, diff checks, and byte-exact preservation of all seven native fixture files from 5ae7b99. Rust test execution is pending the observer-held sole Cargo lease; requested batch: cargo test --locked --offline --package kars-controller inference_budget. No new target, cleanup, installs, push or CI dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Carry the landed SRE ancestry and hosted-qualified real-Helm test harness; budget production semantics and the qualified recovery-race repair stay unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Record the maintainer-authorized, explicitly AI-attributed audit after focused review and deterministic recovery CAS regression qualification. Preserve failed historical native evidence and require fresh complete budget/SRE acceptance; no technical check or audit rule is waived. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

…tion Native existing-PR run 34493554446 failed only E2E Kind at inference-budget-enforcement.mjs:286: POST /v1/embeddings returned 502 instead of 503. Before this assertion, real monetary siblings passed [200,200,429] and settled 10 usdMicros/16 tokens; held-token siblings passed 429, reserved 30 tokens and settled 8 after release. Four owned router UIDs passed healthz and governed readyz. Agents closure, the following provider-count check and cancellation were not reached. Bounded artifacts contain no detailed proxy error; source places legacy-default credential-scope rejection before the existing unsupported budget-operation classifier. Extract the same canonical method/path classifier into a side-effect-free preflight before buffered/streaming provider credential acquisition. Unsupported finite operations retain the typed 503 inference_budget_unavailable contract. Supported requests still obtain credentials and finish wire normalization before the unchanged final-dispatch reserve/begin boundary; legacy errors and preceding policy denials are unchanged. Add actual Axum endpoint regression for named-primary/default-auth handoff, buffered/streaming/method negative cases, supported-auth-before-grant ordering, and broker-loss no-new-dispatch with accepted unknown work still funded. Reuse the existing broker/ledger fixture and make its ephemeral token directory private. Keep native 503 expectation and strengthen it with code/type assertions plus fixed safe status/category facts. Accounting, cancellation, UID/readiness, image, recovery and shared-ledger source parity verified. Validation: 71 targeted CLI/fixture tests, TypeScript typecheck, Rustfmt checks, JS syntax, diff checks and targeted lint (one pre-existing warning). Existing Node cache is Vitest 4.1.10 versus lock 4.1.8. Four new Rust regressions remain unexecuted pending a fresh immutable-SHA Cargo grant; request router LIB inference_budget::client::tests::unsupported_routes_tests followed by relevant budget regressions and strict checks under the existing guard. No Cargo, new target, cleanup, lease SQL updates, push or native dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

The exact-65d0ecaf guarded router-library regression build found E0308 before executing tests: LoadedInferencePolicy.content_safety is ContentSafetyFloor, not Option. Initialize its default floor without changing production code, route checks, budget semantics or assertions. Guard remained above its 8.5 GiB floor (minimum observed 8.95 GiB). Formatting and diff checks pass; Rust qualification remains pending a fresh immutable-SHA grant. No Cargo retry, cleanup, new target, profile change, public push or lease SQL write. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Use the actual protected kars-bridge ancestor with no tracked-tree change from qualified058b7973. Current-base native accounting/failure-contract qualification remains required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Record the actual late native502/503 failure, reviewed side-effect-free classification fix, four new Rust regressions and61budget cases. Preserve all native accounting/failure expectations and require fresh current-base qualification; no result is relabeled or gate waived. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Compose actual kars-bridge tip12d1f2a3 with the qualified budget recovery and unsupported-route fixes. Preserve both feature families and require fresh combined qualification before landing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

…file Continue from joined existing-PR base 1af895b without changing landed MCP or budget production code. Old-base native run 34509062774 passed the coded unsupported-operation 503, agents 403 and no-dispatch count assertion, then observed accepted provider work before cancellation failed at the kubectl PATCH handoff. The original command error discarded stderr; retained artifacts do not establish an API 409 or broker outage. Inspect the exact pinned kubectl v1.30.5 patch implementation: it calls os.ReadFile(PatchFile), so --patch-file - names a literal file rather than stdin. Reproduce this local-file failure with installed kubectl v1.35.3 against a controlled API, proving zero API requests. Replace the bad handoff with the existing loopback kubectl-proxy API pattern, extracted once for the workload proof and cancellation. Cancel only the created Task UID in its pinned workspace, with unchanged generation/spec/budget binding and fresh resourceVersion. Retry at most three times inside 30 seconds only for actual HTTP 409 with matching Kubernetes Status Conflict. Any identity/intent change, 403/422/transport, malformed status, stale RV or failed result verification stays fatal. Emit only fixed API verb/resource/status/reason facts. Provider, spending, no-dispatch, accepted-work and final uncertain/settled/reserved assertions and the 19-case native workload proof remain byte-identical. Validation: 94 targeted JS/CLI tests including 23 cancellation cases and real kubectl proxy conflict/denial cleanup; TypeScript typecheck; JS syntax and diff checks; lint with only two pre-existing warnings. Existing Node cache uses Vitest 4.1.10 versus lock 4.1.8. Private test configs removed and owned proxy children reaped. Native cancellation/accounting success is still unproven until parent-approved hosted E2E. No Cargo grant used, production changes, lease SQL updates, new agents, push or native cycle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

…ion setup Record the real kubectl literal patch-file reproduction, fenced API replacement, unchanged accounting assertions and combined MCP/budget qualification. Bound only external-tool tests using the established integration-test pattern; native cancellation deadlines and pure HTTP tests stay unchanged. Parent validation passed80budget cases, types/lint/syntax. Full current-base native cancellation acceptance remains required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb3654cd-f1e0-445a-8734-430800af1903

Pal Lakatos-Toth (pallakatos) and others added 7 commits September 8, 2026 22:05

Pal Lakatos-Toth (pallakatos) and others added 5 commits September 9, 2026 14:19

Pal Lakatos-Toth (pallakatos) and others added 9 commits September 10, 2026 05:24

Pal Lakatos-Toth (pallakatos) changed the base branch from public/pr7-governed-services to kars-bridge September 10, 2026 17:34

Pal Lakatos-Toth (pallakatos) and others added 3 commits September 10, 2026 20:15

Pal Lakatos-Toth (pallakatos) marked this pull request as ready for review September 10, 2026 20:22

Pal Lakatos-Toth (pallakatos) requested review from Johnson Shi (johnsonshi) and Lachlan Evenson (lachie83) as code owners September 10, 2026 20:22

Pal Lakatos-Toth (pallakatos) merged commit ac9f1b9 into kars-bridge Sep 10, 2026
36 checks passed

Navigation Menu

Uh oh!

feat(budget): enforce durable governed-inference reservations - #553

feat(budget): enforce durable governed-inference reservations#553
Pal Lakatos-Toth (pallakatos) merged 24 commits into
kars-bridgefrom
public/pr11-inference-budgets

Pal Lakatos-Toth (pallakatos) commented Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Pal Lakatos-Toth (pallakatos) commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merged integration slice

Supported scope

Focused repair and review

Qualification and remaining gates

Uh oh!

github-actions Bot commented Sep 9, 2026

Dependency Review

Scanned Files

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Pal Lakatos-Toth (pallakatos) commented Sep 9, 2026 •

edited

Loading