Skip to content

[FEATURE] Opt-in strict deprecation mode in PySpark validation #676

Description

Sub-issue of #639 — surface 2 of 3 ("What flows where").

Depends on #674 for the extraction carrier (FieldSpec.deprecated).

Scope

Let a data producer ask "which rows still populate a deprecated field?" and get an answer through the existing validation error channel.

Deprecation is a validation mode, not a new severity. Generated PySpark validation is strictly pass/fail today — every check returns an error string or null, and explain_errors unpivots violations into (field, check, message) rows. A deprecation check is an ordinary check that fires when a deprecated field is present and non-null; it just is not evaluated by default.

Default is ignore, deliberately: during a deprecation window real published data legitimately still carries the deprecated field, so erroring by default would fail every ordinary validation run. Strict mode is what a producer runs to find the rows to clean up before the field is removed.

This is also where the deprecation message gets its runtime home — the check's error string is the field's deprecation message, so "Use `new_field` instead." reaches the producer at the point of the violation.

Work

  • Generate one deprecation check per deprecated field. For each FieldSpec.deprecated, emit a Check that errors when the field is present and non-null, carrying the deprecation message as its error string.
  • Keep them off the default path. Emit them into a separate deprecation_checks: Callable[[], list[Check]] builder on ModelValidation, alongside the existing checks, so the default check list is byte-for-byte unchanged and empty for models with no deprecated fields. (The alternative — tagging each Check with a category and filtering at runtime — leaves the Check dataclass dirtied for one caller; the separate builder does not.)
  • Toggle at validation time. validate_model gains deprecated: Literal["ignore", "error"] = "ignore"; in "error" mode it concatenates validation.deprecation_checks() into the evaluated set. evaluate_checks, explain_errors, and the (field, check, message) contract are unchanged.
  • Expose the flag. overture-validate --deprecated [ignore|error] (default ignore), wired straight through, next to --skip-schema-check / --suppress.
  • Conformance coverage. Generated tests cover both modes for a model with a deprecated field: a populated deprecated field is a violation under error and clean under ignore.

Open question

The trigger is "present and non-null", not mere schema presence — confirm that is what we want before implementing. A schema-presence trigger would fire on every row of a partition regardless of content, which answers a different question.

Acceptance

  • With no deprecated fields anywhere, generated modules and their default check lists are unchanged.
  • overture-validate --deprecated error reports a violation whose message is the authored deprecation message, on rows populating a deprecated field.
  • overture-validate with no flag, and with --deprecated ignore, reports nothing for those same rows.

Integration points

  • Codegen: packages/overture-schema-codegen/.../codegen/pyspark/renderer.py, render_model_module.
  • Carrier: packages/overture-schema-pyspark/.../pyspark/check.pyModelValidation (~56-61); leave the Check dataclass (~22-51) untouched.
  • Toggle: packages/overture-schema-pyspark/.../pyspark/validate.pyvalidate_model (~298), evaluate_checks (~125), explain_errors (~187).
  • CLI: packages/overture-schema-pyspark/.../pyspark/cli.py, validate_cli.
  • _registry.py picks the extra builder up automatically via MODEL_VALIDATION.

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions