diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index caeb94f..56caa09 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -16,6 +16,8 @@ on: - "overrides/**" - "index.md" - "CODE_OF_CONDUCT.md" + - "LIMITATIONS.md" + - "SPONSORS.md" workflow_dispatch: permissions: @@ -59,7 +61,7 @@ jobs: # measurement/REPORT.md is in the nav as Self-verification. if [ -d measurement ]; then cp -r measurement $BUILD/measurement; fi - for fname in index.md CHANGELOG.md CONTRIBUTING.md CODE_OF_CONDUCT.md PRIVACY.md CNAME; do + for fname in index.md CHANGELOG.md CONTRIBUTING.md CODE_OF_CONDUCT.md LIMITATIONS.md SPONSORS.md PRIVACY.md CNAME; do if [ -f "$fname" ]; then cp "$fname" "$BUILD/$fname"; fi done diff --git a/LIMITATIONS.md b/LIMITATIONS.md new file mode 100644 index 0000000..5a4cdcc --- /dev/null +++ b/LIMITATIONS.md @@ -0,0 +1,67 @@ +# Known Limitations + +What this suite does **not** establish. A conformance report is only useful if the reader knows +what it was never checking, so this is the companion to the report rather than a footnote to it. + +## What a pass means + +**A pass describes the record, not the agent.** +Conformance means the record is well formed, internally consistent, and carries what its level +requires. It says nothing about whether the agent behaved well, whether the policy it ran under +was a sensible policy, or whether the run should have been allowed. A record of a bad run passes +exactly as cleanly as a record of a good one. + +**The suite is one implementation, not the definition.** +[trace-spec](https://github.com/agentrust-io/trace-spec) is normative. Where this suite and the +specification disagree, the specification is what other implementations were written against and +the disagreement is a bug worth reporting here. + +## Where the checks stop + +**`TR-RTE` checks the shape of the attestation fields, not the attestation.** +It validates that `runtime.platform` is a recognised value, that `runtime.measurement` is a well +formed digest, and that the RIM URI parses. It does not obtain a quote, and it does not verify one +against AMD, Intel or a TPM manufacturer root. A record can satisfy `TR-RTE` at Level 1 carrying a +syntactically perfect measurement that no hardware ever produced. Verifying the quote against the +silicon vendor is the relying party's job, and `cmcp_verify` is where that happens. + +**`TR-ANC-002` proves inclusion relative to the receipt you hand it.** +It replays the audit path against the Merkle root carried in that receipt. It does not fetch the +`transparency` URI, and it does not establish that the root is one a public log actually +published. A self-consistent receipt over a tree the submitter built themselves will pass. What +the check rules out is a record that has been modified since anchoring, or that was never in the +tree the receipt commits to. Establishing that the tree is real is a separate step and is not in +scope here. + +`TR-ANC-001` is explicit that it checks the pointer rather than the anchor. Supplying no receipt +fails `TR-ANC-002`, so Level 2 cannot be reached on a well formed URI alone. + +## Statuses that are easy to misread + +**`UNVERIFIED` is about reachability, not correctness.** +It means the check could not be run against the evidence the record cites. The evidence may be +perfectly good and simply out of reach. It is deliberately held apart from a skip so that it can +never be read as a benign omission. + +Whether an unverified finding fails a run is decided per code, in +`src/trace_tests/modules/unverified.py`, not by one blanket rule. `TR-POL-003` is tolerated until +Level 2; anything the table does not name fails from Level 1, so a newly added code fails closed +rather than turning a run quietly permissive. + +**A `TR-ENV` profile failure is a version mismatch before it is a defect.** +Suite 0.4.0 and later require v0.2 records. Run against a v0.1 record, the profile sentinel +produces a confident failure on a record that is fine. Check the suite and record versions before +believing that result. + +**Results are perishable.** +`TR-ENV` validates `iat`, so a record that passes today can fail later with nothing about the +record having changed. A report is a statement about a moment, and it needs its timestamp to be +read correctly. + +## Who is asserting what + +**Nothing here is independently assessed.** +A report is produced by whoever ran the suite, on evidence they supplied. There is no third-party +assessor and no certification programme behind it. This is why the generated report tells a reader +who does not trust the sender to go and check the record themselves rather than trusting the +summary. diff --git a/README.md b/README.md index bd7b6b1..960ea2c 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,7 @@ and CI. | 🗂 Test schemas | [schemas/](schemas/) | | 💬 Discussions | [GitHub Discussions](https://github.com/orgs/agentrust-io/discussions) | | 📋 Changelog | [CHANGELOG.md](CHANGELOG.md) | +| ⚠️ Known limitations | [LIMITATIONS.md](LIMITATIONS.md) | ## Contributing diff --git a/mkdocs.yml b/mkdocs.yml index fbba382..101c3f6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -175,6 +175,7 @@ nav: - Contributing: CONTRIBUTING.md - Code of Conduct: CODE_OF_CONDUCT.md - Privacy: PRIVACY.md + - Known limitations: LIMITATIONS.md extra_javascript: - https://agentrust-io.com/supernav.js