Sub-issue of #639 — surface 2 of 3 ("What flows where").
Depends on #674 for the extraction carrier (FieldSpec.deprecated).
Scope
Let a data producer ask "which rows still populate a deprecated field?" and get an answer through the existing validation error channel.
Deprecation is a validation mode, not a new severity. Generated PySpark validation is strictly pass/fail today — every check returns an error string or null, and explain_errors unpivots violations into (field, check, message) rows. A deprecation check is an ordinary check that fires when a deprecated field is present and non-null; it just is not evaluated by default.
Default is ignore, deliberately: during a deprecation window real published data legitimately still carries the deprecated field, so erroring by default would fail every ordinary validation run. Strict mode is what a producer runs to find the rows to clean up before the field is removed.
This is also where the deprecation message gets its runtime home — the check's error string is the field's deprecation message, so "Use `new_field` instead." reaches the producer at the point of the violation.
Work
- Generate one deprecation check per deprecated field. For each
FieldSpec.deprecated, emit a Check that errors when the field is present and non-null, carrying the deprecation message as its error string.
- Keep them off the default path. Emit them into a separate
deprecation_checks: Callable[[], list[Check]] builder on ModelValidation, alongside the existing checks, so the default check list is byte-for-byte unchanged and empty for models with no deprecated fields. (The alternative — tagging each Check with a category and filtering at runtime — leaves the Check dataclass dirtied for one caller; the separate builder does not.)
- Toggle at validation time.
validate_model gains deprecated: Literal["ignore", "error"] = "ignore"; in "error" mode it concatenates validation.deprecation_checks() into the evaluated set. evaluate_checks, explain_errors, and the (field, check, message) contract are unchanged.
- Expose the flag.
overture-validate --deprecated [ignore|error] (default ignore), wired straight through, next to --skip-schema-check / --suppress.
- Conformance coverage. Generated tests cover both modes for a model with a deprecated field: a populated deprecated field is a violation under
error and clean under ignore.
Open question
The trigger is "present and non-null", not mere schema presence — confirm that is what we want before implementing. A schema-presence trigger would fire on every row of a partition regardless of content, which answers a different question.
Acceptance
- With no deprecated fields anywhere, generated modules and their default check lists are unchanged.
overture-validate --deprecated error reports a violation whose message is the authored deprecation message, on rows populating a deprecated field.
overture-validate with no flag, and with --deprecated ignore, reports nothing for those same rows.
Integration points
- Codegen:
packages/overture-schema-codegen/.../codegen/pyspark/renderer.py, render_model_module.
- Carrier:
packages/overture-schema-pyspark/.../pyspark/check.py — ModelValidation (~56-61); leave the Check dataclass (~22-51) untouched.
- Toggle:
packages/overture-schema-pyspark/.../pyspark/validate.py — validate_model (~298), evaluate_checks (~125), explain_errors (~187).
- CLI:
packages/overture-schema-pyspark/.../pyspark/cli.py, validate_cli.
_registry.py picks the extra builder up automatically via MODEL_VALIDATION.
Sub-issue of #639 — surface 2 of 3 ("What flows where").
Depends on #674 for the extraction carrier (
FieldSpec.deprecated).Scope
Let a data producer ask "which rows still populate a deprecated field?" and get an answer through the existing validation error channel.
Deprecation is a validation mode, not a new severity. Generated PySpark validation is strictly pass/fail today — every check returns an error string or null, and
explain_errorsunpivots violations into(field, check, message)rows. A deprecation check is an ordinary check that fires when a deprecated field is present and non-null; it just is not evaluated by default.Default is
ignore, deliberately: during a deprecation window real published data legitimately still carries the deprecated field, so erroring by default would fail every ordinary validation run. Strict mode is what a producer runs to find the rows to clean up before the field is removed.This is also where the deprecation message gets its runtime home — the check's error string is the field's deprecation message, so
"Use `new_field` instead."reaches the producer at the point of the violation.Work
FieldSpec.deprecated, emit aCheckthat errors when the field is present and non-null, carrying the deprecation message as its error string.deprecation_checks: Callable[[], list[Check]]builder onModelValidation, alongside the existingchecks, so the default check list is byte-for-byte unchanged and empty for models with no deprecated fields. (The alternative — tagging eachCheckwith a category and filtering at runtime — leaves theCheckdataclass dirtied for one caller; the separate builder does not.)validate_modelgainsdeprecated: Literal["ignore", "error"] = "ignore"; in"error"mode it concatenatesvalidation.deprecation_checks()into the evaluated set.evaluate_checks,explain_errors, and the(field, check, message)contract are unchanged.overture-validate --deprecated [ignore|error](defaultignore), wired straight through, next to--skip-schema-check/--suppress.errorand clean underignore.Open question
The trigger is "present and non-null", not mere schema presence — confirm that is what we want before implementing. A schema-presence trigger would fire on every row of a partition regardless of content, which answers a different question.
Acceptance
overture-validate --deprecated errorreports a violation whose message is the authored deprecation message, on rows populating a deprecated field.overture-validatewith no flag, and with--deprecated ignore, reports nothing for those same rows.Integration points
packages/overture-schema-codegen/.../codegen/pyspark/renderer.py,render_model_module.packages/overture-schema-pyspark/.../pyspark/check.py—ModelValidation(~56-61); leave theCheckdataclass (~22-51) untouched.packages/overture-schema-pyspark/.../pyspark/validate.py—validate_model(~298),evaluate_checks(~125),explain_errors(~187).packages/overture-schema-pyspark/.../pyspark/cli.py,validate_cli._registry.pypicks the extra builder up automatically viaMODEL_VALIDATION.