Procedure-compliance metric + pure scorer module (#85) - #91
Merged
Conversation
scoring.py turns a recorded trajectory into a ScoreCard with no Sim, no network, no compiled binding: the vector (coverage, order, end_state, safety, extraneous) is always reported and the derived scalar (safety x tidiness x (0.4 end_state + 0.4 coverage + 0.2 order), order weight redistributed when null) is for rankings only. Safety is a gate, not a trade-off: one dangerous forbidden action floors the scalar to 0. Only command actions score; reads, failed commands and off-surface attempts are reported but never scored. Matching is a deterministic canonical assignment (injective, multiplicity counts) that classifies every effective command as matched_required / optional / forbidden / extraneous, kept in the ScoreCard detail. docs/fase5-metrica.md is the full spec (matching, every component, the edge-case table, the contamination section) and D-026 records the shape decision. Refactor: parse_ground_truth/parse_success extracted from load_scenario so the scorer parses the embedded ground truth through the same path that validated it. Lazy exports keep the scorer binding-free. 21 synthetic tests, one per edge case, plus the import-without-binding guard and the acceptance-case strict ordering (full 1.00 > omitted 0.90 > ext-pwr-only 0.55 > nothing 0.00 -- the real smoke finding). 77 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…ror (#85) score_trajectory takes raw dicts from a trajectory file, not the jsonschema-validated objects load_scenario yields, so a structurally corrupt record must not crash deep in a comprehension. Three spots that raised a bare KeyError/ValueError now raise a named ScoringError that says what is malformed: - a successful set_control (is_error=false) whose args lack a 'control' or carry a non-numeric 'value'; - a run whose meta.scenario has no ground_truth (names the run_id). _end_state also tolerates a null final_state/ecam_clear_of list, and a null meta.scenario no longer trips the id lookup. ScoringError is exported for parity with ScenarioError so slice G can skip corrupt files. Corruption stays as loud as read_trajectory: never a silent wrong number. New tests cover each malformed shape plus the previously untested defensible edges (empty procedure, forbidden/optional overlap, ecam-only success_eval, asdict round-trip of the nested CommandClass). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…just <=1 executed) Review follow-up on #85: two required actions in one unordered block also yield zero applicable constraints; the parenthetical now says so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #85 (slice F of epic #20). The measurement half of the benchmark begins here.
What
bench/a320_bench/scoring.py— a recorded trajectory becomes aScoreCard, pure and offline: no Sim, no network, no compiled binding (a paper reviewer runs it withpip install -e bench/alone). The vector (coverage,order,end_state,safety,extraneous+ informative signals) is always reported; the derived scalarsafety × tidiness × (0.4·end_state + 0.4·coverage + 0.2·order)(order weight redistributed to coverage when null) is for rankings only.set_controlas matched_required / optional / forbidden / extraneous, kept indetail.clear_failure) are reported (errored_commands,off_surface_attempts) but never scored.dangerousforbidden action floors the scalar to 0;anti_procedure→ ×0.5. Unsafe runs are flagged, not ranked.invalid_scenariodoesn't score;provider_errorscores but is excluded from means by default (--include-errorson the aggregator); a truncated trajectory isincomplete(scalar null) but still reports its trajectory components.docs/fase5-metrica.mdis the full spec (matching math, every component, edge-case table, contamination section); D-026 records the vector+scalar shape decision.parse_ground_truth/parse_successextracted fromload_scenarioso the scorer parses the trajectory's embedded ground truth through the same path that validated it (no drift). Lazy exports keep the scorer binding-free.Tests
77 passed (
pytest bench/tests mcp/tests -q). 21 synthetic scoring tests (trajectories built by hand, no Sim), one per edge case: multiplicity, OFF→ON reset pair, wrong-block, wrong value, toggle-back, dangerous floor, anti_procedure halving, errored forbidden, optional, extraneous cap, off-surface, observed-before-acting, invalid_scenario, provider_error flag, truncated→incomplete, aggregation. Plus the import-without-binding guard and the acceptance case as a strict-order CI assertion: full 1.00 > omitted-isolation 0.90 > ext-pwr-only 0.55 > nothing 0.00 — the exact real smoke finding.🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm