Phase 5 decisions, ENG 1 GEN scenario, citation policy, Python CI job (#72) - #80
Merged
Conversation
…ons, Python CI job (#72) D-022..D-025 in docs/decisiones.md: benchmark determinism as checkpoints-with-tolerance without patching the vendor RNG (reinterpreting #19's deterministic replay; seed field reserved in every trajectory), benchmark tool profile (no inject/clear, report_done), in-process runner over the memory transport with the self-contained JSONL trajectory, and litellm pinned exactly as the provider layer. Second scenario: elec.gen.1 on engines-running — the network never sags (bus tie feeds AC 1 from GEN 2), so the exam is restraint: reset attempt, isolate, and keep both engines running; shutting an engine down over a generator fault is the classic forbidden action. Empirically verified on pin 13bce4b, scripted-procedure test included. scenarios/README.md documents the scenario anatomy and the citation policy (procedure identity + public secondary source + fidelity notes with pin and date; FCOM wording never reproduced). CI gets the Python layer for the first time: maturin build of bindings/ (cargo cache on its workspace) + pytest over mcp/tests and bench/tests, wired into ci-success. No [providers] extra: CI runs no LLM and no network. a320_bench imports go lazy (PEP 562) so the #20 scorer can read trajectories without the compiled binding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #72 (slice E of the Phase 5 epic #19).
What
shared/random.rsnative branch is OS-seeded and unseedable; EPIC: scenario suite + procedure ground truth #19's 'deterministic replay' reinterpreted as same input → same trajectory class, withseedreserved in every trajectory meta for a futureSim::set_seed); the benchmark tool profile; the in-process runner + self-contained JSONL trajectory; litellm pinned exactly.scenarios/elec/eng1_gen_fault.yaml):elec.gen.1onengines-running. The bus tie holds the whole AC network from GEN 2 (verified on pin13bce4b), so the exam is restraint: reset attempt → isolate → leave both engines alone.forbidden_actionscarry the classic wrong-lever error (shutting down a healthy engine over a generator fault); success predicates include both engines still running. Scripted-procedure test included.pythonjob — first CI coverage of the whole Python layer (maturin build ofbindings/with cargo cache on its workspace, editable installs,pytest mcp/tests bench/tests), wired intoci-success. No[providers]extra: no LLM, no network in CI.a320_benchlazy imports (PEP 562):read_trajectoryno longer drags the compiled binding — the EPIC: scoring metric + baselines/ablations #20 scorer can consume trajectories in a plain Python env (review follow-up from In-process episode runner + JSONL trajectory recording (#70) #78).Tests
python -m pytest bench/tests mcp/tests -q: 38 passed locally, including the new ENG 1 GEN scripted procedure. The CI job itself is exercised by this PR's own run.🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm