Summary
Prevent a version-only manifest.diagnostic_version update from allowing a stale regression baseline to pass the CI coupling gate.
Rationale
The current gate can accept this sequence:
Diagnostic.version increases.
- A pull request updates only
manifest.diagnostic_version to the new value.
test_case_version, committed, and native remain unchanged.
- The gate returns
SKIP.
This allows committed and native baseline data produced by the earlier diagnostic version to remain in use.
This behaviour predates #854. It is intentionally out of scope for that pull request.
Required changes
Update the regression gate policy so that an increase in diagnostic_version requires a re-mint-authorised baseline update. A test_case_version increase is one possible authorisation mechanism.
Add regression coverage for a version-only manifest edit. The test must verify that the gate does not return SKIP when diagnostic_version increases without the required baseline update.
Affected areas
packages/climate-ref-core/src/climate_ref_core/regression/gate.py
packages/climate-ref-core/tests/unit/regression/test_gate.py
- Related regression-gate documentation, if the policy changes documented behaviour.
Acceptance criteria
- A
diagnostic_version increase cannot pass with unchanged test_case_version, committed, and native baseline data.
- The gate emits a clear failure or required-action result for this case.
- Unit tests cover the version-only manifest update.
- Documentation describes the final policy.
Backlinks
Requested by @lewisjared.
Summary
Prevent a version-only
manifest.diagnostic_versionupdate from allowing a stale regression baseline to pass the CI coupling gate.Rationale
The current gate can accept this sequence:
Diagnostic.versionincreases.manifest.diagnostic_versionto the new value.test_case_version,committed, andnativeremain unchanged.SKIP.This allows committed and native baseline data produced by the earlier diagnostic version to remain in use.
This behaviour predates #854. It is intentionally out of scope for that pull request.
Required changes
Update the regression gate policy so that an increase in
diagnostic_versionrequires a re-mint-authorised baseline update. Atest_case_versionincrease is one possible authorisation mechanism.Add regression coverage for a version-only manifest edit. The test must verify that the gate does not return
SKIPwhendiagnostic_versionincreases without the required baseline update.Affected areas
packages/climate-ref-core/src/climate_ref_core/regression/gate.pypackages/climate-ref-core/tests/unit/regression/test_gate.pyAcceptance criteria
diagnostic_versionincrease cannot pass with unchangedtest_case_version,committed, andnativebaseline data.Backlinks
Requested by @lewisjared.