feat(audit): record HTTP exchanges, tool calls, approvals, capability changes and capsule loads - #2004
Merged
joshuajbouw merged 17 commits intoSep 29, 2026
Conversation
…oads The audit log has no entry kinds for kernel-mediated HTTP, principal grant changes or capsule install/load, and host-call entries cannot say which capsule acted. Approval and tool-call entries have no fields to link a decision to its request or to commit to a tool result. Add, without changing the v1 entry format: - HttpRequest (pre-commit: sequence, method, host, port, path, header and body hashes, redirect hop, injected secret names) and HttpResponse (completion: sequence, request entry id, status, body hash and length, completeness, provider request ids); - CapabilityChanged for principal capability, capsule and group grants; - CapsuleInstalled and CapsuleLoaded binding the wasm hash, manifest hash and engine profile; - a CapsuleActor (capsule id + wasm hash) field on file, network, process, tool-call and approval entries; - call_id and result_hash on CapsuleToolCall, and request_id, request_entry_id and via on the approval entries. New fields on existing variants are optional and omitted when unset, and new variants are appended after the existing ones. An entry written before this change therefore re-serializes to the same signed bytes and still verifies; the new fields are covered by the signature because the action JSON is part of the signing data. The action descriptions and entry tests move into entry/ to keep entry.rs under the file-size cap. Verified with the astrid-audit tests, including new tests that pin the legacy JSON encoding of variants with unset fields, re-verify a legacy entry after a parse/serialize round trip, and show that changing an attributed capsule invalidates the signature; clippy -D warnings on astrid-audit; cargo check of the workspace. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
… writes Host-call audit entries (file, network, process) name the principal but not the capsule that made the call, and every FileWrite entry records the zero hash because the host never reported the written bytes. An auditor cannot tell which code acted or what it wrote. HostAuditSink gains `attributed(actor)`, which returns a sink that stamps a HostAuditActor (capsule id + BLAKE3 of the verified wasm component) on every record. The wasm engine binds the kernel sink this way at load, using the hash it just verified against the install metadata, and the lifecycle (install/upgrade hook) path binds it with the hook component's hash. The actor is therefore host-owned; a guest cannot choose it. The kernel sink maps it onto the new `actor` field of the audit action. `write-file` now reports the BLAKE3 of the bytes it wrote and the kernel records it as the FileWrite content hash. Directory creation and writes denied before any content was accepted keep the zero hash. The bounded writer folds allowed calls of one class per principal; the fold key now includes the actor, so one capsule's calls are never folded into a row attributed to another capsule. The writer is otherwise unchanged; the actor helpers live in new audit_sink/coverage.rs modules in both crates. Verified with new kernel tests (attributed handles stamp the capsule identity and the kernel's own handle does not; two capsules' reads in one window stay separate rows; write hashes are recorded), new capsule tests (the write-file host call reports the content hash; a sink without attribution support is returned unchanged), the existing audit_sink tests in both crates, and clippy -D warnings on astrid-capsule and astrid-kernel. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Every model call an agent makes goes through the capsule HTTP host, but the host only logged it with tracing::debug!. The signed audit log had no record of a request or its response, so an agent could re-ask a model without leaving any trace. Each wire request (each redirect hop included) now produces: - an HttpRequest pre-commit: method, host, port, BLAKE3 hashes of the path+query, the capsule-supplied headers and the body, the body length, the redirect hop, and a kernel-assigned per-principal sequence number. The host awaits a durable append (new HostAuditSink::commit) after the scheme, egress and security-gate checks and before the request is sent, so the entry precedes the request on the chain; - an HttpResponse completion carrying the same sequence and the pre-commit's entry id, the status, provider request ids (x-request-id, request-id and similar), and a BLAKE3 hash of the response body computed over the bytes as the host reads them — incrementally for streams, which complete at end of body, on a read error, or when the guest closes the stream. An exchange dropped before completion (cancelled host call, store teardown) still records a failed completion; - a denied HttpRequest when the principal egress check, the security gate or the SSRF airlock refuses the request. Every HttpRequest entry takes the next number of its principal's sequence, including denials and pre-commits whose append failed, so a missing entry shows as a gap. A failed pre-commit append does not fail the request; it is logged as a security event, matching the existing continue-and-alert behaviour. No content is stored. Commitments are computed after redaction: values of credential headers (authorization, cookie, x-api-key, ...) and credential query parameters (key, api_key, access_token, ...) become "[REDACTED]", as does every occurrence of a secret value get_config handed to the capsule instance. The host remembers those values (zeroized on drop) only for this purpose. The canonical header form and the redaction rules are documented in the http audit module so a verifier holding a request can recompute its commitments. The redirect decision moves into a redirect_step helper so each followed or failed hop can complete its exchange; behaviour is unchanged. The new record types and their kernel mapping live in the audit_sink/coverage.rs modules; the kernel's bounded writer is unchanged. Verified with new capsule tests against loopback servers (the pre-commit resolves before the server sees the request; buffered and streamed responses are hashed and linked to their pre-commit; a stream closed early completes incomplete; a transport failure completes without status; a gate denial is recorded and never sent), unit tests of the redaction and commitment forms (a secret never reaches the hashed bytes), new kernel tests (pre-commits are readable when commit returns and are numbered per principal; denials take a number; completions link to their request and bound provider ids), the existing HTTP host tests, and clippy -D warnings on astrid-capsule and astrid-kernel. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Provider capsules read their API key with get_config and set the
Authorization header themselves (aos-ce capsule-openai-compat does this),
so the key lives in guest memory and passes through the request the audit
log commits to.
A capsule can now name a secret in a request header value with the
placeholder {{secret:NAME}}, for example
`Authorization: Bearer {{secret:api_key}}`. The HTTP host substitutes the
value when it builds the wire request:
- NAME must be a secret-typed [env] key declared in the capsule manifest;
any other name refuses the request with capability-denied and records a
denied HttpRequest entry. The value is resolved like get_config
(invoking principal first, then host-wide) and trimmed.
- An unset or blank secret omits the header, so keyless endpoints still
receive no credential header.
- Only header values are substituted; placeholders in the URL or body are
sent as written. Injected values are marked sensitive.
- On a cross-origin redirect every header carrying a placeholder is
dropped along with Authorization and Cookie, so a secret is only sent to
the origin the capsule addressed.
- The audit commitment is computed over the placeholder form, and the
HttpRequest entry lists the injected secret names, never values.
Capsules that keep reading secrets themselves are unaffected: their values
are still redacted from commitments. docs/models.md describes the provider
request entries and how a capsule adopts injection.
Verified with new capsule tests: substitution rules (multiple and repeated
placeholders, trimming, blank secret omits the header, malformed and
undeclared placeholders and CR/LF values refused, cross-origin strip); an
end-to-end request through the HTTP host where the loopback server
receives the secret, the guest never read it, and the pre-commit's header
hash equals the hash of the placeholder form and names the secret; and an
undeclared placeholder that is refused and recorded without anything being
sent. Existing HTTP host tests and clippy -D warnings on astrid-capsule
pass.
Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Tool invocations and approval decisions determine what an agent did, but neither reached the audit log: CapsuleToolCall and the Approval* actions had no producer, so a tool run or a user's "approve always" left no signed record. Tool calls: when the engine delivers a tool.v1.execute.<tool> request to a tool capsule as an interceptor invocation, it arms a ToolCallAudit. The IPC publish host fn captures the ToolExecuteResult the capsule publishes for that call (on the request topic + ".result" or on tool.v1.execute.result, matched by call id). When the invocation ends the engine records one CapsuleToolCall entry with the tool name taken from the topic, the caller's call id, BLAKE3 hashes of the JSON arguments and of the result content, and the capsule identity. The outcome is a failure when the tool reported an error, published no result, the guest call failed, or the invocation was cancelled (the guard records on drop). The capture is per invocation and cleared when the pooled instance is returned. Approvals: request_approval and the local-egress consent flow now record each check through an ApprovalAudit: - before a prompt is published, an ApprovalRequested entry (host-minted request id, action, resource) is committed and awaited; - the decision is an ApprovalGranted (scope once/session/always) or ApprovalDenied entry carrying the request id and the request entry's id, and how it was reached: user, session_allowance, session_grant, remembered_consent, no_request_owner, timeout, cancelled, or error; - a prompt that ends without a decision (an error path) records a denial when the guard is dropped, so every prompt on the log has a decision. The kernel maps these onto CapsuleToolCall, ApprovalRequested, ApprovalGranted and ApprovalDenied with bounded string fields (in audit_sink/coverage.rs), and stamps "kernel-dispatched tool invocation" or "approval gate" as the system authorization reason instead of the manifest-gate reason. Verified with new capsule tests (a result published through the IPC host fn is recorded with argument and result hashes; tool errors, missing or mismatched results, guest failures and cancellation are failures; non-tool invocations are not armed; each user answer is linked to its committed prompt; the prompt is committed before it is observable on the bus; the missing-owner denial and remembered consent are recorded; egress consent prompts and cached-grant decisions are recorded), new kernel mapping tests, the full astrid-capsule lib suite (771 passed), and clippy -D warnings on astrid-capsule and astrid-kernel. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Capability tokens, principal grants and grant-on-use consent change what an agent may do, but none of them left an audit entry describing the change: admin requests were recorded as AdminRequest rows at authorization time (before the change is applied, and without its result), and grant-on-use was not recorded at all. Applied authority changes are now recorded on the chain of the principal whose authority changed, after the change is saved: - admin.caps.token.mint → CapabilityCreated (token id, resource, permissions, scope); - admin.caps.token.revoke → CapabilityRevoked, chained under the token's subject when the token is still known; - admin.caps.grant / admin.caps.revoke → CapabilityChanged (kind "capability") listing only the patterns actually added, so a repeated grant records nothing; - admin.agent.modify → CapabilityChanged for group and capsule membership differences; - admin.distro.self_grant → CapabilityChanged for the capsules granted; - grant-on-use → an ApprovalRequested entry for the dispatcher's prompt, ApprovalGranted (scope always) or ApprovalDenied (user, timeout) for the decision, linked by request id, and CapabilityChanged (via grant_on_use) for the applied capsule grant. The grant-on-use CapabilityChanged entry is appended after the GrantResult is published so the durable append does not delay the caller. A failed append is logged and does not undo the change, as for admin audit rows. The helpers live in the new grant_audit module. Verified with new kernel tests: caps grant/revoke and agent.modify group changes produce the expected CapabilityChanged entries (and none for a no-op re-grant); token mint and revoke produce CapabilityCreated and CapabilityRevoked under the subject; grant-on-use approve and deny produce linked request/decision entries and the grant entry only on approve. The existing grant_on_use tests pass unchanged, and clippy -D warnings on astrid-kernel passes. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Nothing on the audit log said which code a capsule runtime was running: install and load outcomes were not recorded, and an install or upgrade hook's host calls were not audited at all because the install path passed no audit sink to the lifecycle engine. - The daemon install path records CapsuleInstalled once the capsule is installed and activated: capsule id, version, target principal, the BLAKE3 of the installed wasm component and the BLAKE3 of the exact installed Capsule.toml bytes. - Publishing a runtime generation records CapsuleLoaded with trigger "load"; a live restart/upgrade that replaces a running generation records it with trigger "replace". The entry binds the wasm hash from the install metadata (the engine refuses to load bytes that do not match it), the manifest hash of the runtime directory, and the engine profile: "wasm:<compiled engine ABI>" (COMPILED_ENGINE_ABI, now public), plus "mcp-host" for host-process MCP servers, or "static". - InstallOptions gains audit_sink, threaded through the lifecycle entry points to LifecycleConfig. The daemon passes its signed sink, so hook host calls (file writes, network, HTTP) are recorded and attributed to the hook's code identity. The CLI (workspace installs) passes none. The audit appends are boxed inside the helpers so the request-handler future stays under clippy's large-future bound, and grant_audit is gated to native builds like its callers so the wasm32 portability check gains no warnings. Verified with a kernel test that installs a signed one-component capsule through the daemon install handler and finds CapsuleInstalled and CapsuleLoaded with the component's and manifest's BLAKE3 and the wasm engine profile; an assertion in the durable-upgrade test that the old and new generations are recorded as load and replace with their wasm hashes; a capsule test that the lifecycle host state binds the sink to the hook's capsule id and component hash; astrid-capsule-install tests; clippy -D warnings on astrid-capsule, astrid-capsule-install, astrid-kernel and the CLI; and cargo check for wasm32-unknown-unknown of astrid-audit, astrid-capsule and astrid-kernel. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Summarizes the audit coverage added for astrid-runtime#1998: HTTP pre-commit and completion entries, tool calls, linked approval requests and decisions, applied capability and grant changes, capsule install and load identity, capsule attribution and file-write hashes, audited install hooks, and host-side credential injection. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
The HTTP audit only remembered secret values of at least four bytes and at most 32 distinct values per host state. A shorter value, or any value past the cap, was left out of redaction, so if the capsule put it in a request the signed commitment hashed it in the clear, and a low-entropy secret could be recovered by hashing candidates. Every distinct non-empty value get_config hands out is now remembered and redacted. The set is not guest-growable: values come only from the operator's secret store for the manifest's declared secret keys. Redaction is now one left-to-right pass that replaces the longest secret starting at each position and never rescans replaced text, so the committed form is a deterministic function of the request and the secret set (the previous per-secret passes could rescan the marker), and positions whose byte starts no secret are skipped cheaply. Verified with new tests: a three-byte secret and 40 distinct secrets are all redacted, and overlapping secrets resolve to the longest match without rescanning; the existing HTTP audit and credential tests pass. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
HttpRequest sequence numbers are kept in memory and restart at 1 when the daemon restarts, while the daemon's audit chains are keyed by a session id that stays the same across restarts (the nil session by default). After a restart the same principal chain therefore held two runs of numbers, and a gap could not be told apart from a restart. KernelAuditSink now draws a random run id when it is created and stamps it on every HttpRequest and HttpResponse entry (new run_id field on both variants, which were added in this series). The sequence is defined per principal and run, so a gap within a run means an entry is missing. docs/models.md says so. Verified with the kernel HTTP audit tests (pre-commits carry a non-empty run id and all entries of one kernel share it) and the astrid-audit entry tests; clippy -D warnings on astrid-audit, astrid-capsule and astrid-kernel. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
send_one_hop returned the scheme check's error before any audit record was
built, so a request refused because the caller asked for https-only (or
used a non-HTTP scheme) left no HttpRequest entry and took no sequence
number, unlike requests refused by the egress or security gate.
The scheme refusal is now recorded as a denied HttpRequest ("scheme
denied") whenever the URL parses; the returned error is unchanged.
Verified with a new test that an http:// request with https-only set is
refused with scheme-denied, recorded as denied, and never sent, plus the
existing HTTP host tests and clippy -D warnings on astrid-capsule.
Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
HTTP completion records and approval decisions went through the host-audit queue, which refuses a new unique record once its pending map is full. Under an HTTP burst a durably pre-committed HttpRequest could therefore lose its HttpResponse (status, response hash, provider ids), and an approval prompt committed before it was shown could lose its decision. HostAuditSink::commit now takes the record's outcome, so any record can be appended durably. The HTTP host writes every completion through it: in the async request, redirect, body and stream-read paths it waits for the append; where the exchange ends in a synchronous context (the guest closes or drops a stream, or an unfinished exchange is dropped) it spawns the append on the current runtime instead of queueing it. The decision for an approval whose prompt was committed is appended durably before the host call returns. Decisions made without a prompt (an existing grant), and the denial recorded when an unanswered check is dropped, still use the queue. Verified with the HTTP audit tests (buffered, streamed, early-dropped and failed exchanges complete through the durable path), a new kernel test that a committed completion is readable when commit returns and keeps its failure outcome and run id, the approval and egress-consent audit tests (answered prompts are durable, prompt-less decisions are queued), the full astrid-capsule lib suite, and clippy -D warnings on astrid-capsule and astrid-kernel. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
The grant-on-use prompt (GrantRequired) was recorded by the kernel's approval observer after the dispatcher had already published it, through the droppable host-audit queue. The prompt could be shown and answered before its ApprovalRequested entry existed, and queue pressure could drop it. The dispatcher now commits the ApprovalRequested entry (action "capsule-grant", resource = capsule, the prompt's request id) through HostAuditSink::commit and waits for it before publishing GrantRequired. The sink is carried by CapsuleAccessResolver, which the kernel builds with its audit sink, so the dispatcher's signature is unchanged; the grant signals of one dispatch pass are published after the registry lock is released rather than while it is held. The kernel observer no longer records the prompt and keeps recording the decision and applied grant, linked by the same request id. Verified with a new capsule test that the prompt's entry is committed before the prompt is visible on the bus, the dispatcher and access tests, the grant-on-use kernel tests (decision and grant entries unchanged), and clippy -D warnings on astrid-capsule and astrid-kernel. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
A header value's placeholders were substituted left to right, and the
first one whose secret was unset or blank ended the value: the header was
omitted and the rest of the value was never parsed. An undeclared or
malformed placeholder after it therefore went unchecked, and
`{{secret:unset}} {{secret:undeclared}}` sent the request without the
header instead of refusing it with capability-denied as documented. The
names substituted before the blank one also stayed in the entry's list of
injected secrets although the header was not sent.
Every placeholder in a value is now parsed and resolved before the value
is kept or omitted, so a malformed or undeclared name refuses the request
wherever it appears, and names are recorded only for headers that are
sent.
A test puts an undeclared, a malformed and an unterminated placeholder
after a blank secret, each refused, and checks that an omitted header
names no secret.
Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
The provider-request section said the host waits for the http_request entry before the request leaves, but not what happens when that append fails. As with the rest of the audit log, recording is best-effort: the request is still sent, the failure is logged as a security event, and the missing entry leaves a gap in the run's sequence numbers. Approval prompts behave the same way. The section now says so. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
This was referenced Sep 28, 2026
Merged
An approved grant-on-use prompt recorded its ApprovalGranted decision through the host-audit queue, and appended the applied CapabilityChanged entry directly once the grant result was published. The queue is written later, so the chain could show the capsule grant before the approval that caused it. The decision is now committed durably before the grant result is published, as answered approval prompts already are, and the grant change is appended after it as before. A shutdown right after the result therefore cannot lose the decision either. Denials and timeouts add no grant entry and still use the queue. The approve test now checks that the decision precedes the grant on the chain; on the previous code it fails. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
joshuajbouw
previously approved these changes
Sep 29, 2026
joshuajbouw
left a comment
joshuajbouw
left a comment
Member
There was a problem hiding this comment.
Reviewed 7aa1f73. No blocking findings on this head.
The boundary looks right: Astrid records generic runtime activity and host-stamped capsule identity; it doesn't take on AOS or Codewall policy.
I checked HTTP pre-commit/completion and redaction, credential injection and redirect handling, tool-result capture after IPC validation, approval ordering, applied capability changes, and capsule install/load identity. Independently ran the kernel/capsule audit tests, credential tests, audit library tests and redirect regressions: 194 passes, two ignored tests. Clippy passed for the audit, capsule and kernel libraries with warnings denied.
For integration after #2003, please preserve capsule identity in the replacement writer's run key and rerun the attribution/folding and commit-ordering regressions on the combined code, as the PR description already calls out. This approval is for the reviewed head, not that future integration.
HTTP audit remains explicitly best-effort on persistence failure; this isn't a claim of fail-closed or complete telemetry. GitHub reports no CI checks on this head, and I haven't run the full workspace/platform matrix.
MastaP
marked this pull request as ready for review
September 29, 2026 12:22
5 tasks done
joshuajbouw
added a commit
that referenced
this pull request
Sep 29, 2026
## Linked Issue Closes #1999 Part of #1997. #1996, #2002, and the mount cleanup fix #2008 are merged. Current head `c92562e65087fe47e313e4c029781e89a69927d4` is based on main `26a500a6`. Remaining merge order: this PR, #2004 (recording coverage), #2005 (entry format v2). The integration preserves Pavel's ordered host-audit implementation. The overlapping configuration is resolved in the existing `astrid-config::audit` module, retaining both retention validation and fail-closed host-call settings. Kernel startup loads both settings from the same audit configuration. This PR replaces the host-audit writer. #2004 changes the old writer's fold key to include the capsule. Whichever of the two lands second is rebased onto the other, and the capsule then becomes part of the run key. ## Summary The host-audit sink could lose calls and reorder them without a trace: - It coalesced allowed calls per class over `host_coalesce_ms`, and appended `repeats=N` to the outcome details. The entry signature covers the outcome only as a success bit, so that count was not signed. - It drained pending work per key, so a chain's entries did not follow call order. - A call that met the full queue (`host_queue_capacity`, default 4,096 keys) was dropped, and only a counter moved. - A failed batch was dropped, and calls still queued when the daemon died vanished. As a result, even an externally anchored log could not show that it was complete. This PR records host calls in call order, and makes every loss visible as a signed entry on the chain. It also lets an operator make chosen host-call classes fail closed. ## Changes - **`astrid-audit`: signed host-call records.** These actions are covered by the entry signature: - `HostCallRun`: consecutive calls of one principal. It holds the call count, the first and last call times, a tally per class and outcome, and a fold over every call. - `HostCallLoss`: calls that were accepted but not recorded individually. - `HostCallGap`: an earlier run of the lane stopped without draining. - `HostCallAdmitted`: the write-ahead entry of a fail-closed call. `astrid_audit::host_call` specifies the per-call digest and the fold, so a verifier can recompute them without the kernel. Known-answer tests pin them. - **`astrid-kernel`: ordered lane** (`audit_sink/lane.rs`, `writer.rs`). - Each principal chain has a FIFO of pending slots, so FIFO order is call order. - The writer takes slots from the front and appends each batch atomically with `append_batch_with_principal`. - A failed batch is retried as is, with backoff up to 5 s, before anything behind it is taken. - Signing happens at the durable commit, because only the commit fixes a chain position. The kernel router appends to the same chains directly. - **`astrid-kernel`: lossless coalescing.** - Consecutive allowed and failed calls share one run, and consecutive identical denials share one run. - A one-call run is written as the plain action, as before. - **`astrid-kernel`: loss and gap entries.** - Queue capacity bounds slots, not calls. Overflow folds into one loss slot per chain, which is written as `HostCallLoss` at its place in the chain. - A lane marker in `system:control:audit-lane` records the run and the chains it wrote. After an unclean stop, the next start writes `HostCallGap` into each of those chains and into the session's system chain. - `astrid-storage`: the state-owner resolver admits `system:control:audit-lane` as a system control projection, like `audit`, `invites` and `pair-tokens`. - An unreadable marker, or a marker store that fails, also produces a gap entry. - If the writer thread dies, the lane closes and health reports it. - **`audit.host_fail_closed`** (`astrid-config`, `astrid-capsule`, `astrid-kernel`). - It lists host-call classes whose effect runs only after a write-ahead entry is durable: `file_read`, `file_write`, `file_delete`, `net_connect`, `net_bind`, `process_spawn`. The default is empty. - `HostAuditSink::admit()` is called by the fs, net and process host functions after the security gate and before the effect. - A refusal fails the call with `unknown("audit unavailable")` and is recorded as a denial. - A workspace layer can add classes but not remove them. - **Health:** `accepted` and `persisted` now count calls, and `lost`, `gaps_recorded` and `dropped_after_shutdown` are new. The admin `AuditHealth` wire type is unchanged. - Changelog fragment. **Consequences.** - A denial between allowed calls closes the run. - Order is guaranteed among a chain's host-call entries. Admin rows are still appended when they run. - Calls reported after shutdown are counted in health, not written. **Overhead** (release build, in-memory log, per producer thread; `audit_sink::tests::bench`, ignored by default): | Workload | Before | After | |---|---|---| | 1 thread, 100k mixed allowed calls | 1.61–1.65 µs/call | 0.84–0.92 µs/call | | 8 threads × 8 principals, 200k calls | 15.1–17.9 µs/call | 8.9–12.8 µs/call | | 4 threads, reads with 2% denials | 6.7–7.8 µs/call | 4.2–4.8 µs/call | | 20k distinct denials, 1 thread | 1.35–1.58 µs/call; 15,904 calls dropped | 1.82–1.91 µs/call; none dropped, overflow in 5–7 loss entries | ## Verification Current integrated tree: full kernel library **628 passed, 1 ignored**; capsule audit tests **45 passed**. Earlier validation of the same recorder/retention integration: audit 106 passed / 2 ignored, config 140 passed, kernel audit filter 51 passed / 1 ignored; audit/config/kernel library Clippy passed with warnings denied. The published tree is byte-identical to the tested combined tree; CI must pass on this new head before merge. Prior author verification (before this integration): - **Test suites:** `astrid-audit` 74, `astrid-capsule` 748, `astrid-config` 139 and `astrid-storage` 864 all pass. The kernel lib passes 610 of 611 with `--test-threads=6` and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on `main`. - **Unit and behaviour tests:** - host-call digest and fold known-answer values; every field changes the digest; - a dropped, reordered or altered call fails `matches_calls`; - changing a signed count breaks the signature; - call order across 6 producers, 3 principals, 8-entry batches and a 10 ms window (the same probe fails on `main`); - a 40-call run whose fold, count and time range recompute from the calls; - queue overflow written as one loss entry that commits to the lost calls; - gap entries after an abandoned lane run, none after a clean restart, and gap duties carried across a second unclean stop; - a corrupt marker and a failing marker store, including the unavailable-marker gap surviving a later marker write; - a system-only gap recorded at start with no host call, and retried while idle after a refused append; - the marker namespace written and read back through a runtime principal store; - a batch the log refuses, retried and written in order once the cap is raised; - a fail-closed write durable before `admit()` returns; - refusal within the first failed batch, and after shutdown; - config validation and workspace restriction; - `connect-tcp` and `bind-tcp` refusing without a connection attempt. - **Lint:** `cargo fmt` and `cargo clippy --all-features --all-targets -- -D warnings` on the touched crates. ## AI / Tool Assistance Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, and this description. Assisted-by: Codex CLI. Review passes over the diff. Current integration and verification assisted by Codex. Fresh CI pending; no claim of merge readiness until those checks complete. ## Checklist - [x] Linked to an issue - [x] Changelog fragment added under `changes/{issue}.{kind}.md` (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. Signed-off-by: Joshua J. Bouw <jjb@unicity-labs.com> Co-authored-by: Joshua J. Bouw <jjb@unicity-labs.com>
Signed-off-by: Joshua J. Bouw <jjb@unicity-labs.com>
Member
|
Integrated this onto the now-landed #2003 at c0f4052 without replacing the ordered recorder. The merge needed more than conflict-marker cleanup: capsule identity now participates in its run key, coverage events remain individual, and overflow digests include their payloads. Legacy unattributed digests keep their existing encoding. The combined tree passes 1,531 audit/capsule/kernel tests (3 existing ignored), workspace check, focused Clippy and formatting. The PR description records the commands and the ordering boundary between queued records and synchronous durable commits. Fresh CI is next. I made these integration edits, so the earlier approval should not be read as an independent review of this new delta. |
joshuajbouw
added a commit
that referenced
this pull request
Sep 29, 2026
…d-key verification (#2005) ## Linked Issue Closes #2000 Part of #1997. The current integration includes the landed #1996, #2002, #2003 and #2004. Proposed merge order: #1996, #2002, #2003 (ordered recording), #2004 (recording coverage), this PR. v2 derives an entry's sections from the serde form of the action, authorization and outcome. The new action variants in #2003 and #2004 therefore encode without any change to this format or its specification. ## Summary Format v1 cannot be verified outside Astrid with confidence: - its signed bytes mix binary fields with `serde_json` of the action and authorization, and drop either one if serialization fails; - it signs whole-second time, and only a success bit of the outcome; - it has no sequence number; - verification trusts the public key embedded in each entry, so anyone with write access to the store can re-sign a rewritten chain under any key and still pass; - one runtime key signs audit entries, capability tokens and builds. This PR adds format v2, which is off by default and enabled with `[audit] entry_format = "v2"`: - a deterministic CBOR body that signs every stored field; - a per-chain sequence number; - a dedicated audit key; - verification against a cross-signed key registry instead of the embedded key. v1 history is kept as it is, and the v2 chain links to it. ## Changes - **`astrid-crypto`:** `verify_strict` (RFC 8032 strict Ed25519) and a `random_bytes` helper. - **`astrid-audit`: `entry_v2`.** The module documentation is the byte-level specification, with a known-answer test that an implementation written from the specification alone reproduces. - The body (RFC 8949 §4.2.1 deterministic CBOR) carries: - the format tag; - a chain id derived from the registry, session, principal UID and alias; - a sequence number and the previous hash; - nanosecond time, entry and session ids, and the acting capsule; - the action, authorization and outcome as `[tag, {field => commitment}]` sections derived from their serde form; - the signer key and its key epoch. - Field values are salted commitments. Salts are HMAC-SHA256 of a per-entry salt key, so one field can be disclosed without the others. - The entry hash is SHA-256 of the body, and the audit key signs a domain-separated wrapper of it. - **`astrid-audit`: key registry.** - A hash chain of records binding keys to roles: audit, capability, build and audit-v1. - The genesis is signed by every key it binds, and a rotation by both the old and the new key. A key can never be registered twice. - **`astrid-audit`: verification.** `ChainVerifier` accepts a v2 entry only if all of these hold: - its signer holds the audit role at the entry's epoch; - its chain id derives from the registry; - sequence numbers run 1, 2, 3; - links hold; - epochs never decrease. With a registry present, v1 entries and archive receipts must be signed by registered keys. v2 receipts carry the signer's epoch. - **`astrid-audit`: writing.** - `AuditLog::enable_entry_v2` writes the genesis on first use. - The first v2 entry of each chain has sequence 1 and links to the chain's last v1 entry. - After that, v1 appends are refused (`V1Closed`). - Storage enforces the v1 closure and the current key epoch at commit time, under the durable append lock, so another opener's switch or rotation cannot be bypassed. - The same check applies to a prune before anything is deleted: a new deletion plan is accepted only if its receipt's signer is the registry's current audit key (a receipt without an epoch only while no registry exists). A plan accepted earlier still finishes after a rotation. - `rotate_audit_key` appends a cross-signed rotation. - **`astrid-config`, `astrid-kernel`: `[audit] entry_format`.** It is operator-only, and the default is `"v1"`. - With `"v2"`, the kernel loads or creates `keys/audit.key` (owner-only), enables v2, and on first use writes a genesis that registers the runtime key for the capability, build and v1-audit roles. - A store with a registry stays on v2. - Boot stops if the registry does not verify, if the audit key is missing or different, or if the configuration cannot be read on a store that is not yet on v2. - `keys/audit.key` is protected from admin filesystem edits. - It applies on every native host. - **Docs and changelog.** - `docs/config.md` documents the switch and its one-way migration. - Rolling back to a release without v2 is not supported. - Changelog fragment. With v2 off, v1 behaviour, verification results and archive keys are unchanged. ## Verification ### Current stack integration Head `0dd52fdd48fd25eab5c778353ecadd9701033ff8` reconciles landed #2004 (`3b3360fd`) with integration `c225dfc7`. Its tree is byte-identical to `c225dfc7` (tree `c81645720ec1d81d2c1e447aecdf328f54b4e0a0`). All Actions jobs, including both MUSL smokes, passed on that tree. Three CodeQL hard-coded-value alerts were individually investigated and dismissed as false positives: an HMAC output buffer fully overwritten before return, a test-only known-answer key, and an OS-randomness buffer that cannot return on RNG failure. No scanner rule or test was weakened. Required checks on the ancestry-only commit must still complete before merge. - Retains both anchor-watermark and signing-key-epoch checks before prune-plan acceptance, as well as ordered host recording and capsule attribution. - Keeps `entry_format`, retention settings and fail-closed settings in the shared `AuditConfig`, with both operator-only restrictions intact. - Adds regressions for v2 pruning at the anchor boundary and commitment to host-stamped capsule identity. - The forged-receipt test still requires signature rejection. It now restores the authentic receipt before its subsequent legitimate-prune scenario, because immutable receipt history correctly rejects replacing that generation. - Audit: 171 passed, 2 ignored. Config: 143 passed. Crypto: 28 passed. Kernel: 648 passed, 1 ignored with `--test-threads=4`. The first parallel kernel run hit two unchanged 2-second response timeouts; all three tests in that group passed unchanged in an isolated rerun (0.71s), then the complete four-thread kernel run passed (51.72s). - Audit doctests: 2 passed. Workspace `cargo check --locked --workspace`, formatting and focused audit/config/crypto/kernel Clippy (`--lib --tests -- -D warnings`) passed. - `cargo check --locked -p astrid-kernel --target wasm32-unknown-unknown` passed, with unused/dead-code warnings; this is build evidence, not a browser runtime test. - The prior independent review applies to the earlier source head. These integration checks do not claim a new independent review; completed CI on the identical earlier tree is distinguished from checks on the current commit. ### Original author validation (before integration) - **Test suites:** `astrid-audit` 123 (+2 doc), `astrid-config` 140 and `astrid-crypto` 31 all pass. The kernel lib passes 592 of 593 with `--test-threads=6` and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on `main`. - **Unit and integration tests:** - the known-answer test, and CBOR against RFC 8949 vectors, with non-canonical input rejected; - every stored field covered by the signature: each JSON leaf of a stored entry, and each field of every action, authorization and outcome variant; - sequence gaps, including an entry deleted from the store; - unregistered, foreign, retired and backdated keys rejected, v1 after v2 rejected, and epoch regression across a chain change detected; - registry genesis, rotation and tamper rules; - v1 to v2 migration with mixed chains, persistence and rotation across restart, and principal UID binding; - concurrent single and batch appends producing gap-free sequences; - commit-time refusal of v1 after another opener enables v2, and of a stale epoch after another opener rotates; - archive receipts bound to a registry epoch, and forged receipts rejected; - a stale handle's prune refused, with entries, receipts and verification unchanged: one without v2 after another enabled it, and one at epoch 0 after another rotated to epoch 1; a plan accepted before a rotation finishes after it; - kernel boot: v1 default, key creation, v2 persistence with the setting back at v1, and a missing or replaced key or unreadable config stopping boot. - **Scratch daemon:** boot on v1, switch to v2, then restart with the setting back at v1. The chain reads 8 v1 entries followed by v2 entries 1 to 10, bound to the principal UID, under one registry. It verifies with registered v1 keys required. - **Lint:** `cargo fmt` and `cargo clippy --all-features --all-targets -- -D warnings` on the touched crates. `cargo check --target wasm32-unknown-unknown` of `astrid-kernel`. ## AI / Tool Assistance Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, the format specification, and this description. Assisted-by: Codex CLI. Review passes over the diff. Merge after #2004 and successful checks on the current integration. ## Checklist - [x] Linked to an issue - [x] Changelog fragment added under `changes/{issue}.{kind}.md` (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. --------- Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com> Signed-off-by: Joshua J. Bouw <jjb@unicity-labs.com> Co-authored-by: Joshua J. Bouw <jjb@unicity-labs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked Issue
Closes #1998
Part of #1997. #1996, #2002, #2003, and #2008 are merged. Current head
c0f4052305da424d9625c9f96332448667a9259bintegrates this PR with maine63f65cc. Remaining order: this PR, then #2005 (entry format v2).The integration retains #2003's ordered, loss-accounted writer and fail-closed admission. Capsule identity is part of the run key, so alternating capsules cannot fold together. HTTP/tool/approval coverage records remain individual queue entries; overflow still records a payload-bound loss summary. Existing unattributed call digests keep their exact v1 encoding; attributed and new coverage calls use the documented
astrid.audit.host-call.v2digest domain. This is a call-fold encoding distinction, not #2005's entry-format change.Ordering boundary: queued records retain per-principal FIFO order. Synchronous HTTP/approval commits still use the existing durable append path, like direct administrative audit writes; no new global ordering guarantee between that path and queued records is claimed.
Summary
The audit log did not record the events that determine what an agent did:
tracing::debug!, so an agent turn, including its model request, added no entry;AdminRequestrows at authorization time, before the change was applied and without its result, and grant-on-use was not recorded at all;FileWritestored a zero hash.This PR records all of these on the signed log, without changing the v1 entry format. Content never enters the log, only commitments to it. Provider credentials can now be injected by the host, so they never enter guest memory.
Changes
astrid-audit: entry kinds.HttpRequest(pre-commit) andHttpResponse(completion).CapabilityChanged,CapsuleInstalledandCapsuleLoaded.CapsuleActor(capsule id and wasm hash) on file, network, process, tool-call and approval entries.CapsuleToolCalland the approval entries.New fields are optional and omitted when unset, and new variants are appended after the existing ones. Entries written before this change re-serialize to the same signed bytes.
HTTP exchanges (
astrid-capsuleHTTP host,astrid-kernel).HttpRequestpre-commit before it is sent. The pre-commit holds the method, host, port, BLAKE3 commitments to the path, headers and body, the redirect hop, and a per-principal sequence number scoped to the kernel run.HttpResponsecompletion links to its pre-commit. It carries the status, provider request ids, and a hash of the response body as it was read (incrementally for streams).httpclass ofaudit.host_fail_closedfrom fix(audit): make host-audit recording ordered and loss-accounted #2003 once both have landed.Host-side credentials.
{{secret:NAME}}, and the host substitutes it on the wire.docs/models.mddescribes how a provider capsule adopts this.Tool calls and approvals.
CapsuleToolCallentry pertool.v1.execute.<tool>invocation, with the call id, argument and result hashes, and the capsule.GrantRequiredis published. An approval is committed beforeGrantResultis published, so the chain shows it before the grant it caused.Capability changes. Applied changes are recorded on the affected principal's chain after they are saved: token mint and revoke, caps grant and revoke (only the patterns actually added), group and capsule membership changes, distro self-grants, and grant-on-use.
Code identity.
CapsuleInstalledbinds the wasm and exactCapsule.tomlhashes.CapsuleLoaded(on load or replace) binds the verified wasm hash, the manifest hash and the engine profile.Attribution.
write-filereports the hash of the bytes written.Changelog fragment.
Verification
Current signed integration head: 111 audit tests passed / 2 ignored; 777 capsule tests passed; 643 kernel tests passed / 1 ignored (1,531 passed in total). Commands:
cargo test --locked -p astrid-audit -p astrid-capsule -p astrid-kernel --lib -- --quiet;cargo check --locked --workspace;cargo clippy --locked -p astrid-audit -p astrid-capsule -p astrid-kernel --lib --tests -- -D warnings; formatting and diff checks. All passed locally on macOS. Added integration regressions bind capsule/wasm identity in folded digests and prove coverage events remain individual while overflow binds the original payload. The attribution test now asserts FIFO order across alternating capsules instead of expecting the old unordered folding behavior. Fresh platform CI is still required.Prior author verification before integration:
astrid-audit71,astrid-capsule777,astrid-capsule-install106 and CLI 818 all pass. The kernel lib passes 601 of 602 with--test-threads=6and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails onmain.CapsuleInstalledandCapsuleLoadedwith the expected hashes, and an upgrade recorded as load then replace.cargo fmtandcargo clippy --all-features --all-targets -- -D warningson the touched crates.cargo check --target wasm32-unknown-unknownofastrid-audit,astrid-capsuleandastrid-kernel.AI / Tool Assistance
Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, and this description.
Assisted-by: Codex CLI. Review passes over the diff.
Integration onto the landed recorder and associated regression tests assisted by Codex. The earlier source review does not constitute independent review of these integration edits.
Checklist
changes/{issue}.{kind}.md(docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one)Signed-off-bytrailer.