feat(audit): signed audit.heads snapshot and paged audit.export - #1996
Conversation
e0f4d82 to
49aaefa
Compare
Chain metadata records only the retained count, and a chain keeps only its latest prune receipt, whose omitted_count covers that one generation. After a chain's second prune the total number of entries removed from its front cannot be recovered, so a retained entry has no known absolute position in its chain. ChainMetadata gains omitted_total, the cumulative number of pruned entries, and omitted_generation, the latest receipt generation counted in it. Prune finalization raises omitted_total in the same compare-and-swap that lowers count, under the durable append lock, so the two always come from one committed state and a known total never decreases. The generation marker lets a resumed finalization count its receipt once. Appends carry both fields forward unchanged. Metadata written before this change has neither field. Its first counted prune takes the earlier total from the installed receipt: zero before generation 0, and the generation-0 receipt's omitted_count before generation 1. Any other earlier total is lost; it is stored as u64::MAX, which saturating addition keeps. Absent fields are not serialized, so re-encoding such metadata reproduces its stored bytes, which append-intent recovery compares exactly. Tests cover the counter through three prunes with a close and reopen of an on-disk store in between, the prune-oldest path, a finalization replayed after its receipt was installed, metadata without the fields pruned once and twice before, and the byte-exact re-encoding of such metadata. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Anchoring the audit log externally needs two read-only admin methods that did not exist: a signed statement of every chain head, and a raw export of signed entries with their hashes (the gateway view strips hashes and signatures). audit.heads (capability audit:heads) enumerates every chain-metadata record. For each chain it reads the metadata and prune state together under the durable append lock (one chain at a time, never across the scan), re-reads the head entry and requires it to hash to the recorded head. It returns session, principal, retained count, omitted total, head hash, head id and last timestamp per chain. The runtime Ed25519 key signs a fixed, length-prefixed encoding (domain "astrid.audit.heads.v1", AuditHeadsSnapshot::signed_bytes_v1) directly, with no pre-hash; the response carries the signed bytes, the signature and the public key, so a verifier needs no serde code. The omitted total is the per-chain counter in ChainMetadata, read from the same committed metadata as the retained count. Metadata that has not counted a prune yet takes it from the latest receipt: zero when there is none, and the receipt's omitted_count for generation 0. The total is unknown for a later generation, and while a prune plan that finished deleting is still pending, because an older binary may have lowered the count before it was interrupted. An unknown total is signed as u64::MAX (AUDIT_OMITTED_TOTAL_UNKNOWN), so the claim is covered by the signature instead of an unsigned side flag. audit.export (capability audit:export) pages one chain's stored entries in chain order with id, timestamp, previous hash, content hash (BLAKE3(signing_data)), signature, embedded key, the exact signing bytes and the stored entry, plus the chain's latest prune receipt and its signing bytes. Pages default to 500 entries and are capped at 1000 entries and about 1 MiB of entries. The returned cursor resumes after later appends and is rejected unless it names an entry of the requested chain. The retained index a cursor carries is bounded by the chain's count but not authenticated, and a later prune makes it stale; verifiers take positions from entry links and the signed heads instead. Both methods skip the generic success AdminRequest row, so an anchoring service polling them does not grow the log it anchors. Denied requests are still recorded before dispatch, and an authorized request whose handler fails (unknown chain, rejected cursor, storage error) is recorded after it. Admin request authorization and its audit rows move from admin/mod.rs into admin/request_authorization.rs to keep the module under the 1000-line limit; the context no longer carries the request kind, so it outlives dispatch. The CLI gains `astrid audit heads` and `astrid audit export --session <id> [--agent <alias>] [--from N] [--cursor C] [--limit N]`. Tests cover the encoding (known-answer vector, ordering rules and the unknown sentinel), chain enumeration across sessions and storage pages, the omitted total through several prunes, its derivation for metadata without the counter (never pruned, pruned once, pruned twice) and while a finished prune is pending, cursor paging, the kernel handlers (signature, head hashes, the accumulated omitted total inside the signed bytes, paging, chain links, prune-receipt link, foreign and out-of-range cursors) and, through the admin IPC path, the success-row exemption with its recorded failures and denials. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
An export page read the chain metadata, the cursor entry or the skipped prefix, the page itself and the prune receipt as separate reads. A prune running in between could pair entries from before it with the retained count or receipt from after it, so the first exported entry need not link to the receipt's omitted terminal hash. An entry above the 1 MiB page budget was also returned whatever its size, so a large enough entry produced a response the native uplink cannot deliver in one 2 MiB frame. AuditLog::prune_in_progress reports whether a chain has a prune plan. Entries are deleted only while a plan exists, and a finished prune installs a new receipt. The export checks for a prune in progress and then reads the receipt, before and after its other reads, and fails with a retryable error unless no prune was in progress and the receipt is unchanged. A chain whose prune was interrupted cannot be exported until its next prune resumes and finishes the plan. An entry above the page budget is still returned alone, up to 1.5 MiB serialized, which leaves room for the page envelope in one frame; a larger entry fails the page with an error that names it. The admin IPC test helper now waits up to 30 s for a response instead of 5 s, since the kernel suite runs it alongside heavier tests. Tests cover prune_in_progress with and without a pending plan, and an export over entries of about 1.1 MiB and 1.6 MiB serialized that returns the first alone and rejects the second. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
…runtime#1995 Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
An export page asked storage for limit + 1 entries (up to 1001) and applied the 1 MiB page budget only afterwards, and skipping to a `from` index read 256 entries per call. An entry can be nearly as large as the page budget, so one request over a chain of large entries could hold hundreds of megabytes in memory to return a single entry. The page is now filled 32 entries per storage call, each call asking for one more entry than the page still needs when that fits, and reading stops as soon as the page is full or its byte budget is reached; the extra entry still tells whether the page is complete. Skipping to a `from` index reads the same 32-entry batches. At most one batch of entries is held besides the page itself. A new test exports a 70-entry chain as one page, with limits of 64 and 70, and as a 64-entry page followed by a cursor page, crossing several batch boundaries; the existing paging, cursor and large-entry tests pass unchanged. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
An export cursor's storage part is a session-index key "<session>:<sequence>:<entry id>", and paging resumes after it by lexical comparison. The kernel checked only that the entry id loaded and belonged to the requested chain, so a caller could keep a real entry id and raise the sequence to skip any part of the chain, and the page came back looking like a valid, even complete, continuation. AuditLog::chain_cursor_entry returns the entry a cursor names only when the cursor is a session-index key that storage holds, and the export uses it to resume. An altered key, or one whose entry a prune removed, is rejected with an error that asks the caller to restart from an index. Tests cover a real cursor, an altered sequence, a malformed key and a pruned entry at the audit-log level, and a forged-sequence cursor through the export handler alongside the existing foreign and out-of-range cases. Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
49aaefa to
d9d4d16
Compare
joshuajbouw
left a comment
There was a problem hiding this comment.
Reviewed d9d4d16. This is a good first step for the series: signed heads and raw export are generic Astrid mechanisms, with external anchoring policy kept outside the runtime.
I checked pruning counters and replay, legacy metadata compatibility, cursor validation, paging, and admin authorization. No blocking findings. Independently ran 540 tests across audit/core/uplink and the focused kernel audit handlers (2 audit tests ignored), plus focused Clippy with warnings denied; all passed. The kernel tests cover signatures, paging after pruning, rejected cursors, and recording failed/denied requests without logging every successful poll.
GitHub currently reports no CI checks for this branch. This approval is the code review, not confirmation of CI or a live external-anchoring run. #2002 should follow this merge, as planned.
Wasmtime and wasmtime-wasi 48.0.1 are affected by three advisories published on 2026-09-24: - RUSTSEC-2026-0315 (GHSA-m63x-6p34-q65x): code compiled for call_ref, and for exception catch blocks, skips fuel accounting, so a guest can execute far more instructions than the fuel it was given. - RUSTSEC-2026-0316 (GHSA-jqpg-j7w6-42pr): lifting a record or tuple through the dynamic component Val API can allocate past the hostcall fuel limit. - RUSTSEC-2026-0314 (GHSA-j2g9-4prp-pf6h): a guest can panic the host through a datetime overflow in the wasmtime-wasi filesystem implementation. The capsule engine enables fuel metering: each pooled interceptor call is seeded with a fixed fuel budget that caps a runaway call, and the fuel it consumes is recorded in the invoking principal's fuel ledger. Function references are part of the default Wasm 3.0 feature set and stay enabled, so under 48.0.1 a capsule using call_ref could exceed that budget and under-report its instruction count. Exceptions are disabled, so only the call_ref path applies. Host calls use the typed bindgen bindings rather than the dynamic Val API, and no wasi:* interfaces are linked for capsules or hooks, so the other two advisories are not reachable by guests today; both crates are updated so the runtime no longer ships the affected code. Both workspace requirements move from 48.0.1 to 48.0.3, the first patched 48.x release. The lockfile change is limited to the wasmtime family: wasmtime, wasmtime-wasi, wasmtime-wasi-io, wasmtime-environ, the wasmtime-internal-* crates, pulley-interpreter, pulley-macros, wiggle, wiggle-generate and wiggle-macro move to 48.0.3, and the cranelift-* crates from 0.135.1 to 0.135.3. wasmtime-wasi 48.0.3 no longer depends on cap-fs-ext, which leaves the lockfile, and now depends on rustix-linux-procfs and winx, which were already locked. 49.0.1 is also patched but requires Rust 1.96, above the workspace MSRV of 1.95. The compiled-artifact cache is in memory only and keyed by the Wasmtime 48 ABI domain, so no persisted compiled code needs invalidating. Verified with cargo check --workspace --all-targets, the astrid-capsule and astrid-hooks test suites, the astrid-kernel library tests, clippy with -D warnings on astrid-capsule and astrid-hooks, and cargo audit, which reports no vulnerabilities (the three advisories above against the previous lockfile). Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
joshuajbouw
left a comment
There was a problem hiding this comment.
Rechecked 5984977 after the Wasmtime security update. No blocking findings in the new dependency delta. Wasmtime/WASI and the associated compiler crates are aligned on the patched 48.0.3 release; no audit/export implementation or test changes were added since my earlier review.
Independently ran cargo audit against this exact Cargo.lock: passed with 1,277 advisories loaded and 836 dependencies scanned. This addresses the three advisories without suppressing them. The earlier audit/export review still applies to the unchanged code; the refreshed CI is now running to validate compatibility with the updated runtime dependencies. This approval is not a claim that those pending jobs have passed.
## Linked Issue Closes #2001 #1996 is merged. The retention-only commits are rebased onto main `ed2df49b`, preserving the Wasmtime 48.0.3 security update. Current head: `896c6e4c3d9f5fb305278a3a408250a7908d7204`. The integration adds one Windows Clippy fix: directory syncing and its stored path are compiled only on Unix; Windows retains the existing write-through rename behavior. No global lint suppression. ## Summary Today, pruning deletes entries permanently. That holds whether an operator runs it or the global cap triggers it (1,000,000 entries / 1 GiB), and only a chain's latest prune receipt is kept. So history no external party has seen can be erased, and a verifier cannot connect retained entries to history it anchored several prunes earlier. This PR changes that: - An anchoring service records, per chain, how far it has certified the log: an anchored watermark. - Retention never deletes past a watermark. When the cap is reached and nothing can be pruned, the log keeps the entries and reports degraded instead of deleting them. - Every prune receipt is kept and can be exported. - Operators can require a watermark before any prune, and can archive pruned segments. Installs that do not anchor keep today's bounded retention. ## Changes - **`astrid-audit`: anchored watermarks.** - `AuditLog::mark_anchored` records that a chain was certified through a position. The position counts entries from the chain's genesis, pruned ones included (`omitted_total + count`). The mark carries the head hash at that position and up to 4,096 bytes of opaque evidence. - The claim is checked against the chain: the entry at `position - 1`, or, at the pruned boundary, the latest receipt's terminal hash, must hash to the given head. - Verification resumes from the previous watermark's stored cursor, so a mark costs only the entries appended since. - Watermarks never decrease. Re-marking the same position and hash is a no-op. - Marks live in their own namespace (`audit:anchor_marks`) and are replaced by compare-and-swap. Chain metadata, which append-intent recovery compares byte for byte, is untouched. - **`astrid-audit`: prunes never pass a watermark.** - Each prune is checked against the chain's pruned total after its retention scan. The check runs again, under the durable append lock, when the deletion plan is created. - Marks are installed under that lock only while no plan exists and the receipt is unchanged since verification. This closes the window in which a chain's first mark could land between a prune's check and its plan. - A refused prune deletes nothing and writes no receipt. - `prune_oldest` (used by `audit prune` and the cap) takes the oldest sealed segment, in seal order, whose removal is allowed. It searches the whole seal-order index before refusing. - **`astrid-audit`: the cap holds instead of deleting.** When an append reaches the cap and every candidate holds unanchored history: - a durable retention hold is set in the global metadata; - appends are admitted over the cap; - the global state reports degraded with the reason, and an error is logged once. The hold clears when a watermark advances, a prune finishes, a segment is sealed or the caps change. - **`astrid-audit`: every receipt is kept.** Receipts are recorded under `audit:receipt_history/<chain>/<generation>` before installation. A chain pruned before this change keeps its latest receipt. `AuditLog::prune_receipts` pages them. - **`astrid-audit`: archive, then drop.** With an `AuditArchiver` set, a prune writes the entries it removes to an archive and commits it before the deletion plan is created. The archive must match the receipt's count, digest and terminal hash, or the prune is abandoned. Prunes are serialized from segment selection to plan installation, so two prunes of one chain cannot archive the same generation. - **`astrid-core`, `astrid-kernel`: admin methods.** - `audit.anchor_mark` (capability `audit:anchor`) accepts one certification's evidence and up to 4,096 chains. Each chain is reported as advanced, unchanged or rejected with the reason. Malformed evidence, an empty or oversized list, or a duplicate chain rejects the whole request. - The kernel checks the evidence's shape, not the certification itself. - Successful marks skip the generic success row, as `audit.heads` and `audit.export` do. Rejections and denials are recorded. - `audit.anchor_status` (capability `audit:heads`) reports each chain's count, pruned total, watermark, its time and evidence, whether anchoring is required before prune, and the retention hold. - `audit.export` gains `receipts_from`, which pages prune receipts with their hashes and signing bytes. - `audit.stats` reports the hold. - The file archiver does its filesystem work on the blocking thread pool. - **`astrid-config`: `[audit.retention]`, operator-only.** - `require_anchor` (default `false`) makes every prune require a watermark. - `archive_dir` must be an absolute path. It writes each pruned segment as `<session>/<system|principal.alias>/<generation>.jsonl`: the signed receipt, then the removed entries. The file is written under a temporary name, synced and renamed, in directories private to the daemon's user. - A workspace layer cannot override either key. - `AuditConfig` moves to its own module. - **`astrid-cli`:** - `astrid audit anchor-status` shows per-chain watermarks and lag, and exits 2 while the hold is set. - `audit prune` explains a refusal. - `audit stats` shows the hold. - `audit export --receipts-from` prints prune receipts. - Scenario entries. - Changelog fragment. New storage fields are optional and absent by default, so existing metadata re-encodes to its stored bytes. ## Verification Current integration validation: audit library 99 passed / 2 ignored; config library 138 passed; kernel retention 2 passed; kernel-library Clippy with warnings denied, formatting and diff checks passed on macOS. All 33 checks on 896c6e4 now pass, including Windows and both MUSL targets. #2008 is merged. The exact combined tree with main bdc1fe8 (tree 6c15a1d) also passes the full kernel library suite: 605 passed, zero failed or ignored. Prior author validation of the retention implementation follows: - **Test suites:** `astrid-audit` 99 (+2 doc), `astrid-core` 390, `astrid-config` 138, `astrid-uplink` 58 and CLI 818 all pass. The kernel lib passes 602 of 603 with `--test-threads=6` and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on the base commit. With full parallelism on a loaded host, some kernel-router tests exceed their 2-second admin-response timeout; #1996's branch shows the same under the same load. - **Unit and handler tests:** - watermark verification and monotonicity; - positions across prunes and at the pruned boundary; - persistence across reopen; - a first mark recorded while a prune is held between its check and its plan: the prune is refused with nothing deleted; - a mark refused while a prune is pending; - prune refusal past a watermark; - the require-anchor switch; - oldest-eligible segment selection across refused chains; - the cap hold, which keeps every append and clears on anchoring; - cap pruning unchanged for chains without watermarks; - receipt history order and backfill; - archive contents and permissions, and abort on archive failure; - two concurrent prunes of one chain archiving one at a time; - per-chain `anchor_mark` outcomes and whole-request rejections; - receipt pages in export; - no row written for a successful mark, while rejections and denials write one; - config restriction and validation. - **Lint:** `cargo fmt` and `cargo clippy --all-features --all-targets -- -D warnings` on the touched crates. ## AI / Tool Assistance Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, and this description. Assisted-by: Codex CLI. Review passes over the diff. #1996 and #2008 are merged; all current-head CI checks pass. ## Checklist - [x] Linked to an issue - [x] Changelog fragment added under `changes/{issue}.{kind}.md` (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. --------- Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com> Signed-off-by: Joshua J. Bouw <jjb@unicity-labs.com> Co-authored-by: Joshua J. Bouw <jjb@unicity-labs.com>
## Linked Issue Closes #1999 Part of #1997. #1996, #2002, and the mount cleanup fix #2008 are merged. Current head `c92562e65087fe47e313e4c029781e89a69927d4` is based on main `26a500a6`. Remaining merge order: this PR, #2004 (recording coverage), #2005 (entry format v2). The integration preserves Pavel's ordered host-audit implementation. The overlapping configuration is resolved in the existing `astrid-config::audit` module, retaining both retention validation and fail-closed host-call settings. Kernel startup loads both settings from the same audit configuration. This PR replaces the host-audit writer. #2004 changes the old writer's fold key to include the capsule. Whichever of the two lands second is rebased onto the other, and the capsule then becomes part of the run key. ## Summary The host-audit sink could lose calls and reorder them without a trace: - It coalesced allowed calls per class over `host_coalesce_ms`, and appended `repeats=N` to the outcome details. The entry signature covers the outcome only as a success bit, so that count was not signed. - It drained pending work per key, so a chain's entries did not follow call order. - A call that met the full queue (`host_queue_capacity`, default 4,096 keys) was dropped, and only a counter moved. - A failed batch was dropped, and calls still queued when the daemon died vanished. As a result, even an externally anchored log could not show that it was complete. This PR records host calls in call order, and makes every loss visible as a signed entry on the chain. It also lets an operator make chosen host-call classes fail closed. ## Changes - **`astrid-audit`: signed host-call records.** These actions are covered by the entry signature: - `HostCallRun`: consecutive calls of one principal. It holds the call count, the first and last call times, a tally per class and outcome, and a fold over every call. - `HostCallLoss`: calls that were accepted but not recorded individually. - `HostCallGap`: an earlier run of the lane stopped without draining. - `HostCallAdmitted`: the write-ahead entry of a fail-closed call. `astrid_audit::host_call` specifies the per-call digest and the fold, so a verifier can recompute them without the kernel. Known-answer tests pin them. - **`astrid-kernel`: ordered lane** (`audit_sink/lane.rs`, `writer.rs`). - Each principal chain has a FIFO of pending slots, so FIFO order is call order. - The writer takes slots from the front and appends each batch atomically with `append_batch_with_principal`. - A failed batch is retried as is, with backoff up to 5 s, before anything behind it is taken. - Signing happens at the durable commit, because only the commit fixes a chain position. The kernel router appends to the same chains directly. - **`astrid-kernel`: lossless coalescing.** - Consecutive allowed and failed calls share one run, and consecutive identical denials share one run. - A one-call run is written as the plain action, as before. - **`astrid-kernel`: loss and gap entries.** - Queue capacity bounds slots, not calls. Overflow folds into one loss slot per chain, which is written as `HostCallLoss` at its place in the chain. - A lane marker in `system:control:audit-lane` records the run and the chains it wrote. After an unclean stop, the next start writes `HostCallGap` into each of those chains and into the session's system chain. - `astrid-storage`: the state-owner resolver admits `system:control:audit-lane` as a system control projection, like `audit`, `invites` and `pair-tokens`. - An unreadable marker, or a marker store that fails, also produces a gap entry. - If the writer thread dies, the lane closes and health reports it. - **`audit.host_fail_closed`** (`astrid-config`, `astrid-capsule`, `astrid-kernel`). - It lists host-call classes whose effect runs only after a write-ahead entry is durable: `file_read`, `file_write`, `file_delete`, `net_connect`, `net_bind`, `process_spawn`. The default is empty. - `HostAuditSink::admit()` is called by the fs, net and process host functions after the security gate and before the effect. - A refusal fails the call with `unknown("audit unavailable")` and is recorded as a denial. - A workspace layer can add classes but not remove them. - **Health:** `accepted` and `persisted` now count calls, and `lost`, `gaps_recorded` and `dropped_after_shutdown` are new. The admin `AuditHealth` wire type is unchanged. - Changelog fragment. **Consequences.** - A denial between allowed calls closes the run. - Order is guaranteed among a chain's host-call entries. Admin rows are still appended when they run. - Calls reported after shutdown are counted in health, not written. **Overhead** (release build, in-memory log, per producer thread; `audit_sink::tests::bench`, ignored by default): | Workload | Before | After | |---|---|---| | 1 thread, 100k mixed allowed calls | 1.61–1.65 µs/call | 0.84–0.92 µs/call | | 8 threads × 8 principals, 200k calls | 15.1–17.9 µs/call | 8.9–12.8 µs/call | | 4 threads, reads with 2% denials | 6.7–7.8 µs/call | 4.2–4.8 µs/call | | 20k distinct denials, 1 thread | 1.35–1.58 µs/call; 15,904 calls dropped | 1.82–1.91 µs/call; none dropped, overflow in 5–7 loss entries | ## Verification Current integrated tree: full kernel library **628 passed, 1 ignored**; capsule audit tests **45 passed**. Earlier validation of the same recorder/retention integration: audit 106 passed / 2 ignored, config 140 passed, kernel audit filter 51 passed / 1 ignored; audit/config/kernel library Clippy passed with warnings denied. The published tree is byte-identical to the tested combined tree; CI must pass on this new head before merge. Prior author verification (before this integration): - **Test suites:** `astrid-audit` 74, `astrid-capsule` 748, `astrid-config` 139 and `astrid-storage` 864 all pass. The kernel lib passes 610 of 611 with `--test-threads=6` and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on `main`. - **Unit and behaviour tests:** - host-call digest and fold known-answer values; every field changes the digest; - a dropped, reordered or altered call fails `matches_calls`; - changing a signed count breaks the signature; - call order across 6 producers, 3 principals, 8-entry batches and a 10 ms window (the same probe fails on `main`); - a 40-call run whose fold, count and time range recompute from the calls; - queue overflow written as one loss entry that commits to the lost calls; - gap entries after an abandoned lane run, none after a clean restart, and gap duties carried across a second unclean stop; - a corrupt marker and a failing marker store, including the unavailable-marker gap surviving a later marker write; - a system-only gap recorded at start with no host call, and retried while idle after a refused append; - the marker namespace written and read back through a runtime principal store; - a batch the log refuses, retried and written in order once the cap is raised; - a fail-closed write durable before `admit()` returns; - refusal within the first failed batch, and after shutdown; - config validation and workspace restriction; - `connect-tcp` and `bind-tcp` refusing without a connection attempt. - **Lint:** `cargo fmt` and `cargo clippy --all-features --all-targets -- -D warnings` on the touched crates. ## AI / Tool Assistance Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, and this description. Assisted-by: Codex CLI. Review passes over the diff. Current integration and verification assisted by Codex. Fresh CI pending; no claim of merge readiness until those checks complete. ## Checklist - [x] Linked to an issue - [x] Changelog fragment added under `changes/{issue}.{kind}.md` (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. Signed-off-by: Joshua J. Bouw <jjb@unicity-labs.com> Co-authored-by: Joshua J. Bouw <jjb@unicity-labs.com>
… changes and capsule loads (#2004) ## Linked Issue Closes #1998 Part of #1997. #1996, #2002, #2003, and #2008 are merged. Current head `c0f4052305da424d9625c9f96332448667a9259b` integrates this PR with main `e63f65cc`. Remaining order: this PR, then #2005 (entry format v2). The integration retains #2003's ordered, loss-accounted writer and fail-closed admission. Capsule identity is part of the run key, so alternating capsules cannot fold together. HTTP/tool/approval coverage records remain individual queue entries; overflow still records a payload-bound loss summary. Existing unattributed call digests keep their exact v1 encoding; attributed and new coverage calls use the documented `astrid.audit.host-call.v2` digest domain. This is a call-fold encoding distinction, not #2005's entry-format change. Ordering boundary: queued records retain per-principal FIFO order. Synchronous HTTP/approval commits still use the existing durable append path, like direct administrative audit writes; no new global ordering guarantee between that path and queued records is claimed. ## Summary The audit log did not record the events that determine what an agent did: - LLM and other HTTP exchanges appeared only in `tracing::debug!`, so an agent turn, including its model request, added no entry; - capsule tool calls and approval decisions had no producer; - capability grants and revocations were recorded as `AdminRequest` rows at authorization time, before the change was applied and without its result, and grant-on-use was not recorded at all; - capsule install and load were not recorded, and install hooks ran without an audit sink; - host-call entries did not name the capsule, and `FileWrite` stored a zero hash. This PR records all of these on the signed log, without changing the v1 entry format. Content never enters the log, only commitments to it. Provider credentials can now be injected by the host, so they never enter guest memory. ## Changes - **`astrid-audit`: entry kinds.** - `HttpRequest` (pre-commit) and `HttpResponse` (completion). - `CapabilityChanged`, `CapsuleInstalled` and `CapsuleLoaded`. - A `CapsuleActor` (capsule id and wasm hash) on file, network, process, tool-call and approval entries. - Linking fields on `CapsuleToolCall` and the approval entries. New fields are optional and omitted when unset, and new variants are appended after the existing ones. Entries written before this change re-serialize to the same signed bytes. - **HTTP exchanges** (`astrid-capsule` HTTP host, `astrid-kernel`). - Each wire request, each redirect hop included, gets a durable `HttpRequest` pre-commit before it is sent. The pre-commit holds the method, host, port, BLAKE3 commitments to the path, headers and body, the redirect hop, and a per-principal sequence number scoped to the kernel run. - An `HttpResponse` completion links to its pre-commit. It carries the status, provider request ids, and a hash of the response body as it was read (incrementally for streams). - Completions are appended durably. - Refusals by the scheme check, egress, the security gate or the SSRF airlock are recorded as denied requests and take a sequence number. - Credential headers, credential query parameters and every secret value the capsule received are redacted before hashing. The redaction rules are documented so a verifier can recompute the commitments. - Recording is best-effort, like the rest of the audit log since 0.9.0. If the pre-commit cannot be written, the request is still sent, a security event is logged, and the missing entry shows as a gap in the run's sequence numbers. Approval prompts behave the same way. Making these fail closed is an operator policy; it would fit as an `http` class of `audit.host_fail_closed` from #2003 once both have landed. - **Host-side credentials.** - A header value can name a manifest-declared secret as `{{secret:NAME}}`, and the host substitutes it on the wire. - Undeclared names are refused and recorded. Placeholders are stripped on a cross-origin redirect. - Commitments cover the placeholder form, and the entry lists injected secret names, never values. - `docs/models.md` describes how a provider capsule adopts this. - **Tool calls and approvals.** - One `CapsuleToolCall` entry per `tool.v1.execute.<tool>` invocation, with the call id, argument and result hashes, and the capsule. - Approval prompts are committed before they are published. Every decision is linked to its prompt, with how it was reached: user, session allowance, remembered consent, timeout and so on. - Grant-on-use prompts are committed by the dispatcher before `GrantRequired` is published. An approval is committed before `GrantResult` is published, so the chain shows it before the grant it caused. - **Capability changes.** Applied changes are recorded on the affected principal's chain after they are saved: token mint and revoke, caps grant and revoke (only the patterns actually added), group and capsule membership changes, distro self-grants, and grant-on-use. - **Code identity.** - `CapsuleInstalled` binds the wasm and exact `Capsule.toml` hashes. - `CapsuleLoaded` (on load or replace) binds the verified wasm hash, the manifest hash and the engine profile. - Install and upgrade hooks run by the daemon get the signed sink, attributed to the hook's component. - **Attribution.** - Host-call entries name the capsule and wasm hash, stamped by the host from the verified component; a guest cannot choose them. - `write-file` reports the hash of the bytes written. - Changelog fragment. ## Verification Current signed integration head: **111 audit tests passed / 2 ignored; 777 capsule tests passed; 643 kernel tests passed / 1 ignored** (1,531 passed in total). Commands: `cargo test --locked -p astrid-audit -p astrid-capsule -p astrid-kernel --lib -- --quiet`; `cargo check --locked --workspace`; `cargo clippy --locked -p astrid-audit -p astrid-capsule -p astrid-kernel --lib --tests -- -D warnings`; formatting and diff checks. All passed locally on macOS. Added integration regressions bind capsule/wasm identity in folded digests and prove coverage events remain individual while overflow binds the original payload. The attribution test now asserts FIFO order across alternating capsules instead of expecting the old unordered folding behavior. Fresh platform CI is still required. Prior author verification before integration: - **Test suites:** `astrid-audit` 71, `astrid-capsule` 777, `astrid-capsule-install` 106 and CLI 818 all pass. The kernel lib passes 601 of 602 with `--test-threads=6` and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on `main`. - **Unit and integration tests:** - legacy JSON encodings pinned, and a legacy entry re-verified after a round trip; - pre-commits resolve before a loopback server sees the request; - buffered and streamed responses hashed and linked, including streams closed early, transport failures and gate denials; - redaction: short secrets, many secrets, overlapping secrets; - credential injection end to end: the server receives the secret, the guest never reads it, and the commitment covers the placeholder; - placeholder rules: undeclared, malformed and unterminated placeholders refused, also after a blank secret, and an omitted header names no secret; - tool results captured through the IPC host function, with error, missing, mismatched and cancelled cases; - approval prompts committed before they are visible on the bus; - capability grant, revoke and membership entries, with none for a no-op re-grant, and a grant-on-use approval ahead of its grant; - a daemon install producing `CapsuleInstalled` and `CapsuleLoaded` with the expected hashes, and an upgrade recorded as load then replace. - **Lint:** `cargo fmt` and `cargo clippy --all-features --all-targets -- -D warnings` on the touched crates. `cargo check --target wasm32-unknown-unknown` of `astrid-audit`, `astrid-capsule` and `astrid-kernel`. ## AI / Tool Assistance Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, and this description. Assisted-by: Codex CLI. Review passes over the diff. Integration onto the landed recorder and associated regression tests assisted by Codex. The earlier source review does not constitute independent review of these integration edits. ## Checklist - [x] Linked to an issue - [x] Changelog fragment added under `changes/{issue}.{kind}.md` (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. --------- Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com> Signed-off-by: Joshua J. Bouw <jjb@unicity-labs.com> Co-authored-by: Joshua J. Bouw <jjb@unicity-labs.com>
…d-key verification (#2005) ## Linked Issue Closes #2000 Part of #1997. The current integration includes the landed #1996, #2002, #2003 and #2004. Proposed merge order: #1996, #2002, #2003 (ordered recording), #2004 (recording coverage), this PR. v2 derives an entry's sections from the serde form of the action, authorization and outcome. The new action variants in #2003 and #2004 therefore encode without any change to this format or its specification. ## Summary Format v1 cannot be verified outside Astrid with confidence: - its signed bytes mix binary fields with `serde_json` of the action and authorization, and drop either one if serialization fails; - it signs whole-second time, and only a success bit of the outcome; - it has no sequence number; - verification trusts the public key embedded in each entry, so anyone with write access to the store can re-sign a rewritten chain under any key and still pass; - one runtime key signs audit entries, capability tokens and builds. This PR adds format v2, which is off by default and enabled with `[audit] entry_format = "v2"`: - a deterministic CBOR body that signs every stored field; - a per-chain sequence number; - a dedicated audit key; - verification against a cross-signed key registry instead of the embedded key. v1 history is kept as it is, and the v2 chain links to it. ## Changes - **`astrid-crypto`:** `verify_strict` (RFC 8032 strict Ed25519) and a `random_bytes` helper. - **`astrid-audit`: `entry_v2`.** The module documentation is the byte-level specification, with a known-answer test that an implementation written from the specification alone reproduces. - The body (RFC 8949 §4.2.1 deterministic CBOR) carries: - the format tag; - a chain id derived from the registry, session, principal UID and alias; - a sequence number and the previous hash; - nanosecond time, entry and session ids, and the acting capsule; - the action, authorization and outcome as `[tag, {field => commitment}]` sections derived from their serde form; - the signer key and its key epoch. - Field values are salted commitments. Salts are HMAC-SHA256 of a per-entry salt key, so one field can be disclosed without the others. - The entry hash is SHA-256 of the body, and the audit key signs a domain-separated wrapper of it. - **`astrid-audit`: key registry.** - A hash chain of records binding keys to roles: audit, capability, build and audit-v1. - The genesis is signed by every key it binds, and a rotation by both the old and the new key. A key can never be registered twice. - **`astrid-audit`: verification.** `ChainVerifier` accepts a v2 entry only if all of these hold: - its signer holds the audit role at the entry's epoch; - its chain id derives from the registry; - sequence numbers run 1, 2, 3; - links hold; - epochs never decrease. With a registry present, v1 entries and archive receipts must be signed by registered keys. v2 receipts carry the signer's epoch. - **`astrid-audit`: writing.** - `AuditLog::enable_entry_v2` writes the genesis on first use. - The first v2 entry of each chain has sequence 1 and links to the chain's last v1 entry. - After that, v1 appends are refused (`V1Closed`). - Storage enforces the v1 closure and the current key epoch at commit time, under the durable append lock, so another opener's switch or rotation cannot be bypassed. - The same check applies to a prune before anything is deleted: a new deletion plan is accepted only if its receipt's signer is the registry's current audit key (a receipt without an epoch only while no registry exists). A plan accepted earlier still finishes after a rotation. - `rotate_audit_key` appends a cross-signed rotation. - **`astrid-config`, `astrid-kernel`: `[audit] entry_format`.** It is operator-only, and the default is `"v1"`. - With `"v2"`, the kernel loads or creates `keys/audit.key` (owner-only), enables v2, and on first use writes a genesis that registers the runtime key for the capability, build and v1-audit roles. - A store with a registry stays on v2. - Boot stops if the registry does not verify, if the audit key is missing or different, or if the configuration cannot be read on a store that is not yet on v2. - `keys/audit.key` is protected from admin filesystem edits. - It applies on every native host. - **Docs and changelog.** - `docs/config.md` documents the switch and its one-way migration. - Rolling back to a release without v2 is not supported. - Changelog fragment. With v2 off, v1 behaviour, verification results and archive keys are unchanged. ## Verification ### Current stack integration Head `0dd52fdd48fd25eab5c778353ecadd9701033ff8` reconciles landed #2004 (`3b3360fd`) with integration `c225dfc7`. Its tree is byte-identical to `c225dfc7` (tree `c81645720ec1d81d2c1e447aecdf328f54b4e0a0`). All Actions jobs, including both MUSL smokes, passed on that tree. Three CodeQL hard-coded-value alerts were individually investigated and dismissed as false positives: an HMAC output buffer fully overwritten before return, a test-only known-answer key, and an OS-randomness buffer that cannot return on RNG failure. No scanner rule or test was weakened. Required checks on the ancestry-only commit must still complete before merge. - Retains both anchor-watermark and signing-key-epoch checks before prune-plan acceptance, as well as ordered host recording and capsule attribution. - Keeps `entry_format`, retention settings and fail-closed settings in the shared `AuditConfig`, with both operator-only restrictions intact. - Adds regressions for v2 pruning at the anchor boundary and commitment to host-stamped capsule identity. - The forged-receipt test still requires signature rejection. It now restores the authentic receipt before its subsequent legitimate-prune scenario, because immutable receipt history correctly rejects replacing that generation. - Audit: 171 passed, 2 ignored. Config: 143 passed. Crypto: 28 passed. Kernel: 648 passed, 1 ignored with `--test-threads=4`. The first parallel kernel run hit two unchanged 2-second response timeouts; all three tests in that group passed unchanged in an isolated rerun (0.71s), then the complete four-thread kernel run passed (51.72s). - Audit doctests: 2 passed. Workspace `cargo check --locked --workspace`, formatting and focused audit/config/crypto/kernel Clippy (`--lib --tests -- -D warnings`) passed. - `cargo check --locked -p astrid-kernel --target wasm32-unknown-unknown` passed, with unused/dead-code warnings; this is build evidence, not a browser runtime test. - The prior independent review applies to the earlier source head. These integration checks do not claim a new independent review; completed CI on the identical earlier tree is distinguished from checks on the current commit. ### Original author validation (before integration) - **Test suites:** `astrid-audit` 123 (+2 doc), `astrid-config` 140 and `astrid-crypto` 31 all pass. The kernel lib passes 592 of 593 with `--test-threads=6` and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on `main`. - **Unit and integration tests:** - the known-answer test, and CBOR against RFC 8949 vectors, with non-canonical input rejected; - every stored field covered by the signature: each JSON leaf of a stored entry, and each field of every action, authorization and outcome variant; - sequence gaps, including an entry deleted from the store; - unregistered, foreign, retired and backdated keys rejected, v1 after v2 rejected, and epoch regression across a chain change detected; - registry genesis, rotation and tamper rules; - v1 to v2 migration with mixed chains, persistence and rotation across restart, and principal UID binding; - concurrent single and batch appends producing gap-free sequences; - commit-time refusal of v1 after another opener enables v2, and of a stale epoch after another opener rotates; - archive receipts bound to a registry epoch, and forged receipts rejected; - a stale handle's prune refused, with entries, receipts and verification unchanged: one without v2 after another enabled it, and one at epoch 0 after another rotated to epoch 1; a plan accepted before a rotation finishes after it; - kernel boot: v1 default, key creation, v2 persistence with the setting back at v1, and a missing or replaced key or unreadable config stopping boot. - **Scratch daemon:** boot on v1, switch to v2, then restart with the setting back at v1. The chain reads 8 v1 entries followed by v2 entries 1 to 10, bound to the principal UID, under one registry. It verifies with registered v1 keys required. - **Lint:** `cargo fmt` and `cargo clippy --all-features --all-targets -- -D warnings` on the touched crates. `cargo check --target wasm32-unknown-unknown` of `astrid-kernel`. ## AI / Tool Assistance Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, the format specification, and this description. Assisted-by: Codex CLI. Review passes over the diff. Merge after #2004 and successful checks on the current integration. ## Checklist - [x] Linked to an issue - [x] Changelog fragment added under `changes/{issue}.{kind}.md` (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. --------- Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com> Signed-off-by: Joshua J. Bouw <jjb@unicity-labs.com> Co-authored-by: Joshua J. Bouw <jjb@unicity-labs.com>
## Linked Issue Closes #2011 Follow-up to #1996 and #2005. ## Summary `audit.export` reported every entry's `content_hash_hex` as `BLAKE3(signing_data)`. For a format-v1 entry that is its content hash. For a format-v2 entry the signing data is the signature wrapper of the entry hash, while chain links, chain heads and prune receipts use the SHA-256 entry hash of the canonical body. On a chain switched to v2, an exported entry's hash therefore matched neither the next entry's `previous_hash_hex` nor the head in `audit.heads`, and a verifier following the export could not link the chain. The export now reports `AuditEntry::content_hash()`, which is unchanged for v1 entries. ## Changes - **`astrid-kernel`: `audit.export`.** `content_hash_hex` is the entry's content hash in either format: `BLAKE3(signing_data)` for v1, the SHA-256 entry hash for v2. - **`astrid-core`: export wire types.** Their documentation states which hash each format exports, that v2 signing data is the signature wrapper of that hash, and that only a v1 signature covers whole seconds. - Changelog fragment. ## Verification - **New kernel test** `export_reports_the_hash_that_links_a_chain_switched_to_v2`. It exports a chain with two v1 entries followed by three v2 entries and checks: - each exported hash against the stored entry; - the signature over the exported signing data; - that each v2 entry's signing data wraps its entry hash; - the links across the switch; - the head in the export page and in `audit.heads`. It fails on the previous hash computation (the v2 hashes differ) and passes with this change. - **Test suites:** `astrid-core` 391 pass. The kernel lib passes 647 of 648 with `--test-threads=6` and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on `main`. - **Lint and changelog:** `cargo fmt`, `cargo clippy --all-features --all-targets -- -D warnings` on `astrid-core` and `astrid-kernel`, and `scripts/changelog.py check`. ## AI / Tool Assistance Assisted-by: Claude Code:claude-opus-5-5. Covers the fix, the test, the documentation and this description. Assisted-by: Codex CLI. Review pass over the diff. ## Checklist - [x] Linked to an issue - [x] Changelog fragment added under `changes/{issue}.{kind}.md` (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one) - [x] I understand every change in this PR and can explain its design, risks, and validation. - [x] I reviewed and tested any meaningful tool-generated output included in this PR. - [x] Every non-bot, non-merge commit has a matching `Signed-off-by` trailer. --------- Signed-off-by: Pavel Grigorenko <pavel@unicity-labs.com>
Linked Issue
Closes #1995
Summary
Adds two read-only, admin-scoped audit operations so the audit log can be anchored and verified outside the runtime:
audit.headsreturns a snapshot of every chain head, signed by the runtime key over a fixed byte layout.audit.exportpages one chain's raw signed entries, with their hashes and exact signing bytes.To give entries an absolute position after pruning, chain metadata now counts the entries pruned from each chain. This is the chain-head publication follow-up that #678 leaves out of scope. It also provides an admin read path that a remote-CLI verifier can use.
Changes
astrid-audit: per-chain pruned-entry counter.ChainMetadatagainsomitted_totalandomitted_generation.omitted_totalin the same compare-and-swap that lowerscount, under the durable append lock. A resumed finalization counts its receipt once.u64::MAX(unknown).astrid-audit: read functions.heads_snapshot()reads each chain's metadata and prune state together, one chain at a time, and re-reads the head entry to confirm it hashes to the recorded head.chain_entries_page()reads entries in bounded batches of 32. A page never mixes pre- and post-prune state: it fails with a retryable error while a prune is in progress or if one completes mid-page.AuditLog::prune_in_progressandAuditLog::chain_cursor_entry.astrid-core: request and response types.AuditHeads/AuditExportrequests and their responses, plusAuditHeadsSnapshot::signed_bytes_v1.astrid.audit.heads.v1, snapshot time in ns, then per chain: session, principal or none, count,omitted_total, head hash.u64::MAXsignals an unknownomitted_totalinside the signed bytes.astrid-kernel: handlers. Capabilitiesaudit:headsandaudit:export. Both methods skip the generic successAdminRequestrow, so a poller does not grow the log it reads. Denied requests are recorded before dispatch; authorized requests whose handler fails are recorded as failure rows. Admin request authorization moves toadmin/request_authorization.rsto keepadmin/mod.rsunder the 1,000-line limit.astrid-uplink,astrid-cli, e2e. The admin client,astrid audit headsandastrid audit export --session <id> [--agent <alias>] [--from N] [--cursor C] [--limit N], and CLI scenario entries.wasmtimeandwasmtime-wasimove from 48.0.1 to 48.0.3 for RUSTSEC-2026-0314, -0315 and -0316, published after this branch was cut, which fail the security audit job onmain's lockfile as well. RUSTSEC-2026-0315 (fuel accounting skipped aroundcall_ref) reaches capsules, since fuel metering is on and function references are enabled; the other two are not reachable by guests today. The lockfile change is limited to the wasmtime, cranelift, pulley and wiggle crates. 49.0.1 needs Rust 1.96, above the 1.95 MSRV. Changelog fragmentchanges/1995.security.md.Verification
astrid-audit82 (+2 doc),astrid-core387 (+1),astrid-uplink58, CLI 818, and kernel router all pass. The kernel lib passes 581; its 15 remaining failures are host-environment ones that also fail on the base commit: 14 depend on umask and pass under umask 0022, and 1 needs a copy-on-write workspace backend.cargo auditreports no vulnerabilities on the new lockfile, and exactly the three advisories on the previous one.astrid-capsuleandastrid-hooks(820) pass, the kernel lib passes 595 of 596 (the copy-on-write backend test), andcargo check --workspace --all-targetsis clean.cargo fmtandcargo clippy --all-features --all-targets -- -D warningson the touched crates.mainand its 22 capsules on this runtime. Chain heads were anchored on the Unicity testnet and re-verified against exported entries, including detection of a store rolled back to an earlier backup.AI / Tool Assistance
Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, and this description.
Assisted-by: Codex CLI. Review passes over the diff.
Checklist
changes/{issue}.{kind}.md(docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one)Signed-off-bytrailer.