Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions PRIVACY.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Privacy

agentrust-trace-tests (the TRACE conformance suite) collects and transmits no personal data.
The TRACE test suite reads the record and optional evidence you supply. CLI options can also load policy bundles, receipts, and related local files. Findings and exported reports may reproduce identifiers, artifact locations, and error details from those inputs; review reports before sharing them.

It runs locally as a command-line tool. It processes only the inputs you give it, entirely on your machine, and sends no telemetry, analytics, or usage data to the TRACE Tests project or any third party. There is no account, login, or tracking, and no cookies or background network calls.
The CLI does not send project telemetry or analytics and does not fetch arbitrary record URLs. Its policy-directory resolver uses local files. Library callers can supply their own resolver callbacks, whose network and data handling behavior belongs to the calling application.

It reads only the record file you pass on the command line and makes no outbound calls.
Uninstalling the package does not delete input records, evidence files, exported reports, badges, logs, or backups. Manage those artifacts through your application's retention and deletion procedures.

Uninstalling removes it completely. Questions or corrections: https://github.com/agentrust-io/trace-tests/issues
[Report a correction](https://github.com/agentrust-io/trace-tests/issues).
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,15 @@

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![TRACE Spec](https://img.shields.io/badge/TRACE-Spec_v0.2-0ea5e9)](https://github.com/agentrust-io/trace-spec)
[![Tests](https://img.shields.io/badge/Conformance_Tests-7_modules-green)]()
[![Tests](https://img.shields.io/badge/Conformance_Tests-8_modules-green)](docs/modules.md)
[![CI](https://github.com/agentrust-io/trace-tests/actions/workflows/ci.yml/badge.svg)](https://github.com/agentrust-io/trace-tests/actions/workflows/ci.yml)
[![Discord](https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white&style=flat)](https://discord.gg/grgzFEHgkj)

> **Test suite v0.2.** Tracks [TRACE Spec v0.2](https://github.com/agentrust-io/trace-spec).

Conformance tests for TRACE (Trust Runtime Attestation and Compliance Evidence). Run this suite against your implementation to verify it meets the spec before claiming TRACE compliance.
Check a TRACE record, inspect the findings, and produce a reproducible conformance report. The suite checks the record and supplied evidence; a passing report does not establish that an entire implementation meets every specification requirement.

Seven test modules covering the full specification: envelope structure, signature algorithms, TEE runtime claims, policy binding, tool-call transcripts, SCITT transparency anchoring, and supply chain provenance.
Eight modules cover envelope, signature, runtime, policy, appraisal, transcript, transparency, and provenance checks. Read the [limitations](LIMITATIONS.md) to interpret their results.

## Quick start

Expand Down
2 changes: 1 addition & 1 deletion docs/modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The TRACE conformance suite is divided into eight modules. Each module maps to a
|--------|-----------|--------------|---------------|
| [Envelope](modules/tr-env.md) | TR-ENV | §3.2 | `eat_profile` URI, `iat` validity, `subject` form, presence of `cnf.jwk.kty` |
| [Signature](modules/tr-sig.md) | TR-SIG | §3.2.1 | Private key leak detection, key type support, and the Ed25519 signature verification outcome |
| [Runtime](modules/tr-rte.md) | TR-RTE | §3.1 | TEE platform enum, measurement format, RIM URI scheme |
| [Runtime](modules/tr-rte.md) | TR-RTE | §3.1 | TEE platform enum, measurement format, RIM URI scheme, and verifier-issued nonce comparison |
| [Policy](modules/tr-pol.md) | TR-POL | §3.1 | Policy bundle hash format, enforcement mode values, and whether the bundle at `policy_uri` has the declared digest |
| [Appraisal](modules/tr-apr.md) | TR-APR | §3.1 | Appraisal status enum, verifier URI form, `policy_ref` URI form, and timestamp plausibility. The appraisal policy is never resolved |
| [Transcript](modules/tr-txn.md) | TR-TXN | §3.1 | Tool-call transcript hash binding |
Expand Down
21 changes: 13 additions & 8 deletions docs/modules/tr-rte.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
# TR-RTE Runtime
# TR-RTE: Runtime

Tests TEE platform attestation in the `runtime` field.
Checks runtime claim fields and, at Level 1 or above, the verifier's nonce. This module does not fetch reference manifests, verify hardware quotes, authenticate platform certificates, or compare measurements against an approved image.

## Required at Level 1+
## Checks

| Test ID | Description | Positive Case | Negative Case |
|---------|-------------|---------------|---------------|
| TR-RTE-001 | `runtime.platform` is in the registered set, and is not `software-only` above Level 0 | `intel-tdx`, `amd-sev-snp`, `nvidia-h100`, `tpm2` | `"software"`, `"cloud"`, `"sev-snp"`, absent |
| TR-RTE-002 | `runtime.measurement` is a valid `sha256:` digest | `sha256:` followed by 64 hex chars | missing, wrong prefix, all zeros |
| TR-RTE-003 | RIM URI (if present) resolves to a valid reference image | valid `https://` URI returning a reference manifest | non-HTTPS URI, 404 response |
| Test ID | Actual check | Boundary |
|---|---|---|
| TR-RTE-001 | Platform belongs to the registered set; `software-only` is rejected above Level 0 | A registered platform string is not hardware evidence |
| TR-RTE-002 | Measurement has `sha256:` plus 64 lowercase hex digits or `sha384:` plus 96 | Format only; all-zero values also match this check |
| TR-RTE-003 | If present, `rim_uri` is a string starting with `https://`; absent means skipped | No network request, HTTP status check, URI resolution, or reference-image appraisal |
| TR-RTE-004 | At Level 1 or above, a nonempty runtime nonce matches the verifier's nonempty expected nonce | Compares claim values; does not establish binding to a hardware quote |

The registered set includes `intel-tdx`, `amd-sev-snp`, `azure-cvm-sev-snp`, `nvidia-h100`, `nvidia-blackwell`, `aws-nitro`, `arm-cca`, `google-confidential-space`, `tpm2`, and `software-only`.

Use independently trusted hardware evidence, expected measurements, key binding, and provider-specific appraisal outside these format checks. See the [suite limitations](../../LIMITATIONS.md) and [runtime checker source](https://github.com/agentrust-io/trace-tests/blob/main/src/trace_tests/modules/tr_rte.py).
2 changes: 1 addition & 1 deletion docs/tutorials/writing-conformance-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,6 @@ When a finding carries `status == Status.UNVERIFIED`, the record has no signatur

## Summary

You ran the existing suite with pytest, called individual module `check()` functions directly, and built a signed test fixture from scratch. The `Finding` dataclass with `code`, `status`, and `message` fields is the single interface across all seven modules.
You ran the existing suite with pytest, called individual module `check()` functions directly, and built a signed test fixture from scratch. The `Finding` dataclass with `code`, `status`, and `message` fields is the single interface across all eight modules.

Next steps: [CI Integration](./ci-integration.md) shows how to run these tests in GitHub Actions with matrix Python versions and artifact upload.
6 changes: 3 additions & 3 deletions index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ description: Run this suite against your implementation to find out which TRACE

# TRACE Test Suite

Conformance tests for [TRACE](https://trace.agentrust-io.com) (Trust, Runtime Attestation, and Compliance Evidence). Run this suite against your implementation to verify it meets the specification before claiming TRACE compliance.
Check a TRACE record, inspect the findings, and produce a reproducible conformance report. The suite checks the record and supplied evidence; a passing report does not establish that an entire implementation meets every requirement of the [TRACE specification](https://trace.agentrust-io.com).

**Seven modules cover the full specification: envelope structure, signature algorithms, TEE runtime claims, policy binding, tool-call transcripts, SCITT transparency anchoring, and supply-chain provenance.**
The [eight modules](docs/modules.md) cover envelope, signature, runtime, policy, appraisal, transcript, transparency, and provenance checks. Read the [limitations](LIMITATIONS.md) to interpret what each result establishes.

!!! tip "TL;DR"
- `verify` answers a question for the person running it. `report` produces an artifact for somebody who was not there.
Expand Down Expand Up @@ -54,7 +54,7 @@ A conformance report that looks authoritative and cannot be checked is the same

---

The seven test modules, the `TR-*` error codes they emit, and what each one checks.
The eight test modules, the `TR-*` error codes they emit, and what each one checks.

[Test Modules](docs/modules.md)

Expand Down
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ plugins:
- docs/modules/tr-sig.md
- docs/modules/tr-rte.md
- docs/modules/tr-pol.md
- docs/modules/tr-apr.md
- docs/modules/tr-txn.md
- docs/modules/tr-anc.md
- docs/modules/tr-sca.md
Expand Down Expand Up @@ -163,6 +164,7 @@ nav:
- Signature (TR-SIG): docs/modules/tr-sig.md
- Runtime (TR-RTE): docs/modules/tr-rte.md
- Policy (TR-POL): docs/modules/tr-pol.md
- Appraisal (TR-APR): docs/modules/tr-apr.md
- Transcript (TR-TXN): docs/modules/tr-txn.md
- Transparency (TR-ANC): docs/modules/tr-anc.md
- Provenance (TR-SCA): docs/modules/tr-sca.md
Expand Down
21 changes: 21 additions & 0 deletions tests/test_docs_match_the_modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import jsonschema

from trace_tests.modules.unverified import UNVERIFIED_FAILS_FROM_LEVEL
from trace_tests.runner import _LEVEL_MODULES

REPO = pathlib.Path(__file__).resolve().parents[1]
MODULES = REPO / "src" / "trace_tests" / "modules"
Expand All @@ -35,6 +36,26 @@
_UNVERIFIED_ROW = re.compile(r"^\| (TR-[A-Z]{3}-\d{3}) \| (\d+) \|", re.M)


def test_published_module_counts_match_the_runner() -> None:
"""Keep total counts on reader entry pages aligned with the active modules."""
expected = len(set().union(*_LEVEL_MODULES.values()))
words = ["zero", "one", "two", "three", "four", "five", "six",
"seven", "eight", "nine", "ten", "eleven", "twelve"]
pattern = re.compile(r"\b(" + "|".join(words) + r"|\d+) (?:test )?modules\b", re.I)
for relative in ("README.md", "index.md", "docs/modules.md",
"docs/tutorials/writing-conformance-tests.md"):
text = (REPO / relative).read_text(encoding="utf-8")
counts = pattern.findall(text)
assert counts, f"No published total found in {relative}"
for count in counts:
actual = int(count) if count.isdigit() else words.index(count.lower())
assert actual == expected, f"{relative}: says {count} modules; runner has {expected}"
readme = (REPO / "README.md").read_text(encoding="utf-8")
badge = re.search(r"Conformance_Tests-(\d+)_modules-", readme)
assert badge is not None
assert int(badge.group(1)) == expected


def _codes(text: str) -> set[str]:
return set(_CODE.findall(text))

Expand Down