Scenario schema + APU GEN FAULT scenario + validated loader (#69) - #75
Merged
Conversation
…ader (#69) scenarios/ holds the data: a JSON Schema for declarative failure scenarios (ordered procedure blocks with optionally-ordered action sets, validity-gate ECAM expectations, tolerance predicates, QRH citation metadata) and the first scenario: elec.apu_gen.1 on apu-running, ground truth verified empirically against vendor pin 13bce4b (pb OFF resets the fault circuit, pb ON re-raises the caution, EXT PWR retires it) and cited to the FBW A32NX docs + FCOM PRO-ABN-24 identity. bench/ is the new a320-bench package: the loader validates shape via jsonschema and cross-checks every control, failure id and start state against the live catalogs, so a bad reference fails at load time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
… table form The inverted-bounds check only covered success.final_state; a between predicate with min > max in failures[].at.when would pass load-time validation and hang the injection wait mid-episode. Check every predicate in the file and say which one. The pyproject license moves to the table form: the PEP 639 bare string needs setuptools>=77, contradicting the declared >=61 floor (mcp/pyproject.toml already uses the table). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
This was referenced Jul 23, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #69 (slice B of the Phase 5 epic #19). Independent of #73 (slice A); the episode runner (#70) needs both.
What
scenarios/— data onlyschema/scenario.schema.json: declarative scenario format. Procedure ground truth is an ordered list of blocks, each block an optionally-ordered set of actions (ordered: truedemands sequence) — the QRH's real dependencies without over-penalizing interchangeable steps.expected_ecamis the run's validity gate (not the agent's task): if the injected failure does not manifest, the run aborts as invalid instead of being scored. Success criteria are tolerance predicates (eq/ne/gt/ge/lt/le/between), never snapshot equality — the vendor has real randomness.elec/apu_gen_fault.yaml: first scenario —elec.apu_gen.1onapu-runningwith a GPU pre-plugged (ext_pwr_availis adomain=worldcontrol, fixed by the harness per the Phase 3 closure note). Ground truth verified empirically against vendor pin13bce4b: APU GEN pb OFF retires the caution (fault circuit reset), pb back ON re-raises it (observable failed reset), EXT PWR restores AC 1/2/ESS and retires every caution. Cited to the FBW A32NX docs (URL, accessed date) + FCOM PRO-ABN-24 procedure identity; the fidelity notes record what was verified and that the action list encodes the procedure's structure, not a verbatim FCOM transcript.HYD ENG 2 PUMP FAULTappeared in some probe runs only, so it is deliberately not asserted.bench/— newa320-benchpackagea320_bench/scenario.py: loader with two validation layers — jsonschema shape errors with a JSON-path to the offender, then live cross-checks of every failure id, control name, start state andworld_controlsdomain against the core catalogs (same philosophy as D-017: a name that does not exist fails at load time, loudly, not mid-LLM-run).Tests
python -m pytest bench/tests -q: 11 passed — the shipped scenario loads with full catalog cross-checks, a task-prompt-leak guard runs over every scenario in the suite, and shape/reference errors name the file and field (elec.flux_capacitor.1, cockpit control inworld_controls, invertedbetweenbounds, typo'd top-level key…).🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm