From 992cbbb14372494298888eebe1c00d1f7851902d Mon Sep 17 00:00:00 2001 From: Imran Siddique Date: Wed, 2 Sep 2026 12:45:36 -0700 Subject: [PATCH] docs: add LIMITATIONS.md, and fix the SPONSORS page 404 while registering it Every other public AgenTrust project ships a LIMITATIONS.md and this one did not, which matters more here than elsewhere: this suite issues pass and fail verdicts about other people's records, so what it does not check is part of reading the verdict. Written against the code rather than from memory. The three that matter: - TR-RTE validates the shape of the attestation fields and does not verify a quote against a silicon vendor root, so Level 1 can be reached with a well formed measurement no hardware produced. - TR-ANC-002 replays the audit path against the Merkle root carried in the receipt supplied to it. It does not fetch the transparency URI and does not establish that the root is one a public log published. It rules out a record modified since anchoring, not a self-made tree. - UNVERIFIED is about reachability rather than correctness, and whether it fails a run is per code from the table in modules/unverified.py, not one blanket rule. Also fixes a live 404. SPONSORS.md is in the mkdocs nav and exists in the repo, but was never in the allowlist that assembles .docs_build, so tests.agentrust-io.com/SPONSORS/ has been returning 404 from a nav link. Both files are now in the allowlist and in the deploy paths filter, since a file that is in the build but not the filter merges green and never deploys. Verified by replicating the CI assemble-and-build steps locally: both LIMITATIONS and SPONSORS pages are produced. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uc16CsknaQ9PTLxGj8TXXj --- .github/workflows/docs.yml | 4 ++- LIMITATIONS.md | 67 ++++++++++++++++++++++++++++++++++++++ README.md | 1 + mkdocs.yml | 1 + 4 files changed, 72 insertions(+), 1 deletion(-) create mode 100644 LIMITATIONS.md diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index caeb94f..56caa09 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -16,6 +16,8 @@ on: - "overrides/**" - "index.md" - "CODE_OF_CONDUCT.md" + - "LIMITATIONS.md" + - "SPONSORS.md" workflow_dispatch: permissions: @@ -59,7 +61,7 @@ jobs: # measurement/REPORT.md is in the nav as Self-verification. if [ -d measurement ]; then cp -r measurement $BUILD/measurement; fi - for fname in index.md CHANGELOG.md CONTRIBUTING.md CODE_OF_CONDUCT.md PRIVACY.md CNAME; do + for fname in index.md CHANGELOG.md CONTRIBUTING.md CODE_OF_CONDUCT.md LIMITATIONS.md SPONSORS.md PRIVACY.md CNAME; do if [ -f "$fname" ]; then cp "$fname" "$BUILD/$fname"; fi done diff --git a/LIMITATIONS.md b/LIMITATIONS.md new file mode 100644 index 0000000..5a4cdcc --- /dev/null +++ b/LIMITATIONS.md @@ -0,0 +1,67 @@ +# Known Limitations + +What this suite does **not** establish. A conformance report is only useful if the reader knows +what it was never checking, so this is the companion to the report rather than a footnote to it. + +## What a pass means + +**A pass describes the record, not the agent.** +Conformance means the record is well formed, internally consistent, and carries what its level +requires. It says nothing about whether the agent behaved well, whether the policy it ran under +was a sensible policy, or whether the run should have been allowed. A record of a bad run passes +exactly as cleanly as a record of a good one. + +**The suite is one implementation, not the definition.** +[trace-spec](https://github.com/agentrust-io/trace-spec) is normative. Where this suite and the +specification disagree, the specification is what other implementations were written against and +the disagreement is a bug worth reporting here. + +## Where the checks stop + +**`TR-RTE` checks the shape of the attestation fields, not the attestation.** +It validates that `runtime.platform` is a recognised value, that `runtime.measurement` is a well +formed digest, and that the RIM URI parses. It does not obtain a quote, and it does not verify one +against AMD, Intel or a TPM manufacturer root. A record can satisfy `TR-RTE` at Level 1 carrying a +syntactically perfect measurement that no hardware ever produced. Verifying the quote against the +silicon vendor is the relying party's job, and `cmcp_verify` is where that happens. + +**`TR-ANC-002` proves inclusion relative to the receipt you hand it.** +It replays the audit path against the Merkle root carried in that receipt. It does not fetch the +`transparency` URI, and it does not establish that the root is one a public log actually +published. A self-consistent receipt over a tree the submitter built themselves will pass. What +the check rules out is a record that has been modified since anchoring, or that was never in the +tree the receipt commits to. Establishing that the tree is real is a separate step and is not in +scope here. + +`TR-ANC-001` is explicit that it checks the pointer rather than the anchor. Supplying no receipt +fails `TR-ANC-002`, so Level 2 cannot be reached on a well formed URI alone. + +## Statuses that are easy to misread + +**`UNVERIFIED` is about reachability, not correctness.** +It means the check could not be run against the evidence the record cites. The evidence may be +perfectly good and simply out of reach. It is deliberately held apart from a skip so that it can +never be read as a benign omission. + +Whether an unverified finding fails a run is decided per code, in +`src/trace_tests/modules/unverified.py`, not by one blanket rule. `TR-POL-003` is tolerated until +Level 2; anything the table does not name fails from Level 1, so a newly added code fails closed +rather than turning a run quietly permissive. + +**A `TR-ENV` profile failure is a version mismatch before it is a defect.** +Suite 0.4.0 and later require v0.2 records. Run against a v0.1 record, the profile sentinel +produces a confident failure on a record that is fine. Check the suite and record versions before +believing that result. + +**Results are perishable.** +`TR-ENV` validates `iat`, so a record that passes today can fail later with nothing about the +record having changed. A report is a statement about a moment, and it needs its timestamp to be +read correctly. + +## Who is asserting what + +**Nothing here is independently assessed.** +A report is produced by whoever ran the suite, on evidence they supplied. There is no third-party +assessor and no certification programme behind it. This is why the generated report tells a reader +who does not trust the sender to go and check the record themselves rather than trusting the +summary. diff --git a/README.md b/README.md index bd7b6b1..960ea2c 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,7 @@ and CI. | 🗂 Test schemas | [schemas/](schemas/) | | 💬 Discussions | [GitHub Discussions](https://github.com/orgs/agentrust-io/discussions) | | 📋 Changelog | [CHANGELOG.md](CHANGELOG.md) | +| ⚠️ Known limitations | [LIMITATIONS.md](LIMITATIONS.md) | ## Contributing diff --git a/mkdocs.yml b/mkdocs.yml index fbba382..101c3f6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -175,6 +175,7 @@ nav: - Contributing: CONTRIBUTING.md - Code of Conduct: CODE_OF_CONDUCT.md - Privacy: PRIVACY.md + - Known limitations: LIMITATIONS.md extra_javascript: - https://agentrust-io.com/supernav.js