test(report): add machine-readable obligation execution accounting

[solloek369-arch]

Current gap

A report can show attempted levels and PASS/FAIL findings without showing whether
every obligation in a bounded expected set had an applicable checker result.

trace-tests#63 is a
public example. appraisal has schema coverage, but no TR-APR module-level
findings. To keep the first pilot atomic, use one child rather than treating all
“appraisal semantics” as one row: account for module/report coverage of the
pinned appraisal.status enum while explicitly preserving the existing schema
coverage.

Mechanically, Finding carries code, status, and message; the JSON report
adds level/module context and an aggregate verdict. Central result policy also
already exists: UNVERIFIED_FAILS_FROM_LEVEL determines from which attempted
level selected finding codes block a run. The proposal exposes that relationship;
it does not claim central verdict logic is absent.

The missing report relationship is:

immutable obligation source
→ bounded suite coverage
→ applicability at attempted level
→ evaluation state
→ observed {finding code, status}
→ centrally computed level contribution

Proposed bounded pilot

Pair two additive artifacts:

1. a small current-only obligation registry;
2. per-attempted-level obligation accounting in the existing report.

The registry should be consumed by the runner/report or generated mechanically
from the same executable metadata. A detached JSON inventory would create a
second truth. Each report should therefore bind registry_id and
registry_sha256.

A static row could carry:

suite_obligation_key
obligation_family
target_surface
normative_source = repository + commit + path + locator
atomic_condition
applicability_condition
required_effect
coverage_disposition = IMPLEMENTED | UNIMPLEMENTED_TRACKED | OUT_OF_SUITE_SCOPE
checker binding
expected findings = [{condition, code, status}]
positive/negative vectors
wrong-finding control
tracking URL

All rows are current obligations at their pinned source. Open decisions and
prospective work belong in separate design-backlog metadata rather than the
obligation universe. The suite-local key does not claim an upstream-issued clause
identifier.

For every attempted_level × suite_obligation_key, the report would emit one
row with candidate concepts such as:

attempted_level
profile_or_floor
applicability = APPLICABLE | NOT_APPLICABLE | UNDETERMINED
evaluation_state =
  COMPLETED
  | ATTEMPTED_UNRESOLVED
  | BLOCKED_BY_PREREQUISITE
  | NOT_ATTEMPTED
state_reason
blocking_finding_codes
observed_findings = [{code, status}]
counts_as_level_failure = true | false | null

status would use the report's lowercase wire values (pass, fail, skip,
unverified). For overloaded finding codes, the row's atomic precondition must
also hold; {code, status} alone is not assumed globally branch-unique.

Contribution must come from the same central logic that computes the displayed
result. The existing TR-POL-003 / UNVERIFIED threshold is a useful control: the
same finding is non-blocking below Level 2 and blocking at Level 2.

Small first set

Start with exactly three atomic rows:

1. pinned build_provenance.digest format → implemented TR-SCA-002;
2. resolved policy-bundle digest correspondence → implemented TR-POL-003;
3. module/report coverage of the pinned appraisal.status enum →
   UNIMPLEMENTED_TRACKED under
   trace-tests#63.

The #63 row does not absorb verifier URI, policy_ref, timestamp, or Level-1
affirming semantics; those remain separate obligations.
Its applicability must not require prior JSON Schema success, because that would
pre-satisfy the enum and make the negative vector unreachable. Missing appraisal
shape is accounted as a separate prerequisite block.

agentrust-io/trace-spec#66
is motivation/tracking, not the normative locator. Provenance-depth requirements
already live in pinned normative §3.3.1 after
trace-spec PR #173.
They are a current gap outside this first pilot for scope only. The specific
prospective action-receipt profile axis tracked by #66 remains separate; this
does not classify all current action-receipt semantics as future.

agentrust-io/trace-spec#190
is a boundary/open decision, not a registry row or invented runtime outcome.

Wrong-finding and prerequisite properties

A red aggregate counts as target coverage only when the target row completed and
contains the expected {code, status} pair. If a sibling rejects while the
target pair is absent, target coverage is absent.

Likewise, when build_provenance is missing and TR-SCA-001 returns early, the
TR-SCA-002 row should remain visible as BLOCKED_BY_PREREQUISITE; it should not
silently become NOT_APPLICABLE.

Limits

This remains an unsigned suite self-report, not independent proof that code ran
or that a field was accessed. Current checker interfaces receive broad record
structures; input supplied to a checker must not be represented as independently
observed field consumption.

accounting_complete should mean that every row in this explicitly bounded
three-row registry has one accounting row for each attempted level.
evaluation_complete is separate. Neither means complete TRACE coverage.

Smallest implementation

• one runner-consumed or mechanically generated three-row registry;
• one additive/versioned JSON report extension bound to registry identity/hash;
• one explicit tracked-unimplemented path;
• one wrong-finding differential;
• one prerequisite-block differential;
• one attempted-level contribution control;
• no initial verdict-policy change.

Would this bounded report-layer pilot fit the direction of trace-tests?

[imran-siddique]

Yes to the direction, with one constraint that decides when it can ship rather than whether.

What I am accepting

The relationship you set out is the right one to make explicit, and the two design calls in it are the ones that matter:

Generated from executable metadata, not a detached inventory. You are right that a hand-maintained JSON file would be a second truth, and a second truth about coverage is worse than no statement about coverage, because it drifts silently toward flattering. Binding registry_id and registry_sha256 into each report is what makes a report re-checkable against the registry it was produced under rather than against whatever the registry says today.

Accounting at the obligation level, not the finding level. A report that lists findings answers "what did the suite say", and a report that lists obligations answers "what did the suite look at". Those are different questions and only the second bounds the claim.

That is the same discipline the TR-APR module applied one level down, where absent optional fields report SKIP rather than PASS, on the reasoning that nothing was checked and a pass would say otherwise. This generalises it, and I would like the two to use consistent vocabulary rather than inventing a second one.

The constraint

No report ships an unreconciled registry. Every obligation in the bounded set must be either covered by a checker or explicitly marked out of scope with a stated reason, before the accounting appears in any report a user can produce.

The reason is specific to this repository rather than general caution. A conformance suite's output is a claim about conformance, so a machine-readable list of obligations it does not check is a published statement that TRACE conformance is less verified than the suite's verdicts imply. I declined trace-tests#57 on exactly that ground: it named ten unguarded conformance paths without the ten tests, and the instrument and the remediation have to land together.

Your proposal is the general case of that PR, which makes the constraint more load-bearing here, not less. Building the accounting is fine and I want it. Emitting not_covered rows to users before those rows have been worked through is the part to sequence.

Practically: land the registry and the accounting with the reconciliation done for the bounded pilot set, so the first report that carries it shows a complete picture. out_of_scope with a reason is a perfectly good answer for a row; not_covered with no decision behind it is the one that should not reach a user.

One thing you could not have known

#63 closed a few hours ago. The TR-APR module landed in #82, which adds five codes over appraisal.status, verifier, policy_ref and timestamp, with TR-APR-005 skipping at Level 0 because docs/levels.md's own minimum Level 0 record carries "status": "none".

So your motivating example is no longer a gap. That does not weaken the proposal, and I would rather it strengthened it: TR-APR is now a good first row for the pilot, because you can express a real obligation set with a real checker behind it and demonstrate the accounting on something reconciled rather than on something outstanding.

Worth rebasing your pilot scope onto that before you write the registry rows.

[solloek369-arch]

Thank you Imran 🙏!

I’m picking this up against current main and #82 now.
I’ll treat your constraint as the emission boundary: no user-facing accounting report until every obligation in the bounded pilot is either checker-covered or explicitly out of scope with a stated reason. The registry will be derived from checker-owned executable metadata, and the accounting will remain at atomic obligation level.
I had already prototyped and attacked this path locally. Two controls from that work are worth preserving:

• execution must be observed at the checker branch that owns the atomic obligation, rather than inferred later from a finding code, aggregate result, or sibling failure; and
• accounting rows must be validated against the exact findings from the same run, so independently valid outputs cannot be spliced into a plausible report.
  I won’t carry forward the old UNIMPLEMENTED_TRACKED appraisal row now that feat(tr-apr): check appraisal well-formedness at every level #82 / TR-APR has landed. I’ll rebase that part onto the implemented, atomic TR-APR-001 obligation, preserve TR-APR’s SKIP semantics, and keep the remaining TR-APR obligations separate.
  I’ll keep the report wire/version as an explicit open product choice rather than silently deciding it in the implementation. I’ll come back with a compact current diff and hostile tests, not another design round. Thank you for the clear constraint.