Repair multiseed and canonical resume orchestration

[charlesmartin14]

Motivation

• Make resume orchestration deterministic and auditable so each canonical/multiseed arm is either verified-and-skipped, resumed from a verified checkpoint, created as a fresh append-only run, or rejected on ambiguity.
• Ensure checkpoint compatibility includes code/provenance metadata (repo commit/dirty state, WeightWatcher/WW-PGD versions, PyTorch, optimizer implementation) and make code-version mismatches explicit and auditable rather than silently ignored.

Description

• Implemented repository identity capture and extended run manifest/compatibility to include git_commit, git_dirty, weightwatcher_version, wwpgd_commit, torch_version, and optimizer_implementation_version. (src/wwgpt/train.py, src/wwgpt/checkpointing.py)
• Reworked resume selection _select_resume_run to classify runs as complete, resume, or new, verify checkpoint integrity and completion markers, skip verified completed runs, resume a single compatible incomplete run, create a new append-only run when none exists, and fail with an explicit ambiguity error if multiple compatible incomplete runs exist. (src/wwgpt/train.py)
• Made assert_checkpoint_compatible return mismatches and accept allow_code_version_mismatch so code-version fields may be permitted while other scientific mismatches remain blocking; when allowed, mismatches are recorded to a persistent audit JSON and mark runs non-publication-eligible unless an explicit audit override is supplied. (src/wwgpt/checkpointing.py, src/wwgpt/train.py, src/wwgpt/integrity.py)
• Added --allow-code-version-mismatch support and a new --audit-override-code-version-mismatch flag in the CLI and propagated the audit override through run-multiseed and run-canonical-trials. (src/wwgpt/cli.py, src/wwgpt/train.py)
• Audit integration: resume-recorded code-version mismatches are persisted as code_version_mismatch_*.json and audit_* checks in integrity detect unaudited mismatches to exclude runs from publication eligibility. (src/wwgpt/integrity.py)

Testing

• Ran the full test suite with pytest -q, which completed successfully (tests passed; some tests skipped for environment specifics).
• Ran focused suites pytest -q tests/test_critical_fixes.py tests/test_wwpgd_training_cadence.py tests/test_canonical_trials.py and pytest -q tests/test_canonical_trials.py tests/test_wwpgd_training_cadence.py tests/test_cli_execution_interface.py tests/test_integrity_publication_grade.py, which passed (targeted tests green).
• Performed static checks: python -m py_compile src/wwgpt/train.py src/wwgpt/checkpointing.py src/wwgpt/cli.py and git diff --check, both succeeded.

---

Codex Task