fix(audit): make host-audit recording ordered and loss-accounted

[MastaP]

Linked Issue

Closes #1999

Part of #1997. #1996, #2002, and the mount cleanup fix #2008 are merged. Current head c92562e65087fe47e313e4c029781e89a69927d4 is based on main 26a500a6. Remaining merge order: this PR, #2004 (recording coverage), #2005 (entry format v2).

The integration preserves Pavel's ordered host-audit implementation. The overlapping configuration is resolved in the existing astrid-config::audit module, retaining both retention validation and fail-closed host-call settings. Kernel startup loads both settings from the same audit configuration.

This PR replaces the host-audit writer. #2004 changes the old writer's fold key to include the capsule. Whichever of the two lands second is rebased onto the other, and the capsule then becomes part of the run key.

Summary

The host-audit sink could lose calls and reorder them without a trace:

• It coalesced allowed calls per class over host_coalesce_ms, and appended repeats=N to the outcome details. The entry signature covers the outcome only as a success bit, so that count was not signed.
• It drained pending work per key, so a chain's entries did not follow call order.
• A call that met the full queue (host_queue_capacity, default 4,096 keys) was dropped, and only a counter moved.
• A failed batch was dropped, and calls still queued when the daemon died vanished.

As a result, even an externally anchored log could not show that it was complete.

This PR records host calls in call order, and makes every loss visible as a signed entry on the chain. It also lets an operator make chosen host-call classes fail closed.

Changes

• astrid-audit: signed host-call records. These actions are covered by the entry signature:

  • HostCallRun: consecutive calls of one principal. It holds the call count, the first and last call times, a tally per class and outcome, and a fold over every call.
  • HostCallLoss: calls that were accepted but not recorded individually.
  • HostCallGap: an earlier run of the lane stopped without draining.
  • HostCallAdmitted: the write-ahead entry of a fail-closed call.

  astrid_audit::host_call specifies the per-call digest and the fold, so a verifier can recompute them without the kernel. Known-answer tests pin them.

• astrid-kernel: ordered lane (audit_sink/lane.rs, writer.rs).

  • Each principal chain has a FIFO of pending slots, so FIFO order is call order.
  • The writer takes slots from the front and appends each batch atomically with append_batch_with_principal.
  • A failed batch is retried as is, with backoff up to 5 s, before anything behind it is taken.
  • Signing happens at the durable commit, because only the commit fixes a chain position. The kernel router appends to the same chains directly.

• astrid-kernel: lossless coalescing.

  • Consecutive allowed and failed calls share one run, and consecutive identical denials share one run.
  • A one-call run is written as the plain action, as before.

• astrid-kernel: loss and gap entries.

  • Queue capacity bounds slots, not calls. Overflow folds into one loss slot per chain, which is written as HostCallLoss at its place in the chain.
  • A lane marker in system:control:audit-lane records the run and the chains it wrote. After an unclean stop, the next start writes HostCallGap into each of those chains and into the session's system chain.
  • astrid-storage: the state-owner resolver admits system:control:audit-lane as a system control projection, like audit, invites and pair-tokens.
  • An unreadable marker, or a marker store that fails, also produces a gap entry.
  • If the writer thread dies, the lane closes and health reports it.

• audit.host_fail_closed (astrid-config, astrid-capsule, astrid-kernel).

  • It lists host-call classes whose effect runs only after a write-ahead entry is durable: file_read, file_write, file_delete, net_connect, net_bind, process_spawn. The default is empty.
  • HostAuditSink::admit() is called by the fs, net and process host functions after the security gate and before the effect.
  • A refusal fails the call with unknown("audit unavailable") and is recorded as a denial.
  • A workspace layer can add classes but not remove them.

• Health: accepted and persisted now count calls, and lost, gaps_recorded and dropped_after_shutdown are new. The admin AuditHealth wire type is unchanged.

• Changelog fragment.

Consequences.

• A denial between allowed calls closes the run.
• Order is guaranteed among a chain's host-call entries. Admin rows are still appended when they run.
• Calls reported after shutdown are counted in health, not written.

Overhead (release build, in-memory log, per producer thread; audit_sink::tests::bench, ignored by default):

Workload	Before	After
1 thread, 100k mixed allowed calls	1.61–1.65 µs/call	0.84–0.92 µs/call
8 threads × 8 principals, 200k calls	15.1–17.9 µs/call	8.9–12.8 µs/call
4 threads, reads with 2% denials	6.7–7.8 µs/call	4.2–4.8 µs/call
20k distinct denials, 1 thread	1.35–1.58 µs/call; 15,904 calls dropped	1.82–1.91 µs/call; none dropped, overflow in 5–7 loss entries

Verification

Current integrated tree: full kernel library 628 passed, 1 ignored; capsule audit tests 45 passed. Earlier validation of the same recorder/retention integration: audit 106 passed / 2 ignored, config 140 passed, kernel audit filter 51 passed / 1 ignored; audit/config/kernel library Clippy passed with warnings denied. The published tree is byte-identical to the tested combined tree; CI must pass on this new head before merge.

Prior author verification (before this integration):

• Test suites: astrid-audit 74, astrid-capsule 748, astrid-config 139 and astrid-storage 864 all pass. The kernel lib passes 610 of 611 with --test-threads=6 and umask 0022. The remaining failure needs a copy-on-write workspace backend and also fails on main.
• Unit and behaviour tests:
  • host-call digest and fold known-answer values; every field changes the digest;
  • a dropped, reordered or altered call fails matches_calls;
  • changing a signed count breaks the signature;
  • call order across 6 producers, 3 principals, 8-entry batches and a 10 ms window (the same probe fails on main);
  • a 40-call run whose fold, count and time range recompute from the calls;
  • queue overflow written as one loss entry that commits to the lost calls;
  • gap entries after an abandoned lane run, none after a clean restart, and gap duties carried across a second unclean stop;
  • a corrupt marker and a failing marker store, including the unavailable-marker gap surviving a later marker write;
  • a system-only gap recorded at start with no host call, and retried while idle after a refused append;
  • the marker namespace written and read back through a runtime principal store;
  • a batch the log refuses, retried and written in order once the cap is raised;
  • a fail-closed write durable before admit() returns;
  • refusal within the first failed batch, and after shutdown;
  • config validation and workspace restriction;
  • connect-tcp and bind-tcp refusing without a connection attempt.
• Lint: cargo fmt and cargo clippy --all-features --all-targets -- -D warnings on the touched crates.

AI / Tool Assistance

Assisted-by: Claude Code:claude-opus-5-5. Covers design, implementation and tests in all touched crates, and this description.
Assisted-by: Codex CLI. Review passes over the diff.

Current integration and verification assisted by Codex. Fresh CI pending; no claim of merge readiness until those checks complete.

Checklist

• Linked to an issue
• Changelog fragment added under changes/{issue}.{kind}.md (docs/CI-only may skip; release PRs roll fragments into the version section instead of adding one)
• I understand every change in this PR and can explain its design, risks, and validation.
• I reviewed and tested any meaningful tool-generated output included in this PR.
• Every non-bot, non-merge commit has a matching Signed-off-by trailer.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[joshuajbouw]

Reviewed eff6be4. No blocking findings in this head. The per-principal FIFO, retry-before-later-work behavior, signed overflow records, and durable restart marker address the ordering/loss problem without putting Codewall policy into Astrid. Fail-closed admission runs after authorization and before the protected effect.

Independently ran 254 tests across audit/config and the focused kernel and capsule audit paths (3 ignored), plus focused audit/config Clippy with warnings denied. All passed. This includes refusal before a TCP effect, failed-batch recovery, overflow accounting, unreadable/unavailable markers, and restart gaps.

Please preserve capsule identity in the run key when integrating #2004, as noted in the description; that combined change needs its attribution regression checked. This approval covers this draft head, not that future integration. GitHub currently reports no CI checks for this branch.