Pre-flight metadata validation API for draft datasets (non-throwing, schema-aware, Unicode-safe)

[youseihuayu-wonderful]

Summary

I'd like to propose a pre-flight validation layer in pyDataverse: a non-throwing API that checks dataset metadata before it is sent to a Dataverse instance and returns structured, machine-readable findings instead of failing at the first problem.

While working on #240 / #247 and #241 / #249 I noticed that several open issues share the same shape: metadata the user believed was valid, failing late, one problem at a time, as a traceback rather than as a field-level message. This proposal addresses that class of problem rather than each instance.

Layer 2 (out of scope for this issue) is written up in a separate public repo/branch, not in pyDataverse:

• Repo: https://github.com/youseihuayu-wonderful/dataverse-agentic-deposit
• Branch: main
• Markdown: dataverse_agentic_deposit_proposal.md

That document describes the multi-agent / External Tool design. This issue (#254) is only Layer 1 (deterministic pre-flight validation in pyDataverse).

Motivation

1. Metadata errors surface late, one at a time, as tracebacks.

• "TypeError: issubclass() arg 1 must be a class" when using "identifier_scheme": "ORCID" #241 / Fix nested controlled-vocabulary serialization (ORCID / issue #241) #249 — nested controlled-vocabulary serialization failed with an opaque TypeError: issubclass() arg 1 must be a class deep inside MetadataBlockBase._process_field_value. The failure involved a controlled-vocabulary field (identifier_scheme / authorIdentifierScheme), but the message said nothing useful about fields, blocks, or allowed values. A pre-flight pass could surface field-level problems before the request path collapses into a traceback.
• API fails to handle metadataBlocks: "astrophysics" & "biomedical" #164 — API fails to handle the astrophysics and biomedical metadata blocks.
• json Validator Fails when License is included #161 — JSON validator fails when a license is included.
• error: {'status': 'ERROR', 'message': 'Error parsing Json: incorrect typeClass for field kindOfData, should be controlledVocabulary'} #153 — incorrect typeClass for controlled-vocabulary fields such as kindOfData.

In each case the problem is discovered late. A shared validation pass would surface these as field-level messages before the request is made, and would report all problems in one run rather than one per round trip.

2. Non-Latin metadata appears to be untested.

Dataverse installations hold multilingual metadata. Controlled-vocabulary values, author names, keywords, and descriptions in CJK and other non-Latin scripts pass through the same serialization path touched in #249, but I could not find test coverage for:

• Unicode normalization (NFC vs NFD) round-tripping
• non-ASCII values in controlled-vocabulary fields
• non-Latin author names and affiliations surviving serialize → POST → retrieve unchanged

These are possible sources of silent data corruption in a preservation system. I would like to add this coverage regardless of whether the broader agentic-deposit proposal is accepted.

3. Multi-stage deposits have no consistency check (later / optional).

Researchers depositing across stages of one study often need consistency between related datasets (authors, funding, controlled-vocabulary choices, linkage). That is valuable, but I would treat it as a follow-on after a solid single-dataset validator exists—not part of the first PR.

Proposed scope

A small, dependency-free module — deterministic and rule-based:

from pyDataverse.validation import validate_dataset_metadata

report = validate_dataset_metadata(
    dataset,
    blocks=("citation", "biomedical"),
    server_schema=api.get_metadatablocks(),  # optional; enables schema-aware checks
)

report.ok          # bool
report.findings    # list[Finding]

# Finding(
#     level="error" | "warning",
#     block="citation",
#     field="authorIdentifierScheme",
#     code="cv_value_not_allowed",
#     message="'ORCID ' is not an allowed value for authorIdentifierScheme",
#     value="ORCID ",
#     allowed=["ORCID", "ISNI", "LCNA", "VIAF", "GND", "DAI", "ResearcherID", "ScopusID"],
# )

Properties:

• Collects all findings; never raises. Returns the complete set in one pass.
• Structured output. CI, notebooks, and external tools can consume it without parsing prose.
• Optionally schema-aware. Installations differ in which metadata blocks they enable; validating against the blocks a target server advertises is more useful than a hard-coded schema. Offline mode can still catch structural and Unicode issues.
• No new runtime dependency. pydantic is already in use (Annotated[str, AfterValidator(...)]).

First PR slice (proposed)

1. Public validation API for the citation block (required / empty fields + controlled vocabulary).
2. Non-Latin / Unicode round-trip test coverage described above.
3. Documented example: validate → inspect findings → fix → then create / upload.
4. No LLM / AI dependency.

Design principle: draft vs. published

This constrains anything built on top of the validation layer:

Automated correction acts only on draft (unpublished) versions. For published versions, tooling proposes and a human decides.

Published Dataverse versions are permanent and citable. Drafts are private and mutable, so acting on them carries no preservation risk.

Layer 1 itself only validates; it does not publish and does not auto-correct.

Non-goals

• Not changing the raise / no-raise behaviour of any existing call. This is additive.
• Not bundling any LLM or AI dependency into pyDataverse. Validation is deterministic.
• Not proposing UI. A guided deposit experience belongs in a Dataverse External Tool (Configure type, dataset scope) built on this API — not in the client library.
• Not requiring Dataverse core changes for this issue.

Why this belongs in pyDataverse

These rules already exist in the client; they are just implicit and only observable by triggering them. Exposing them as an inspectable, non-throwing API means one implementation can serve library users, CI checks, and external tools, instead of each reimplementing the same knowledge about metadata blocks and controlled vocabularies.

Acceptance criteria (verifiable)

• Public API accepts a dataset / metadata structure and returns a structured report without raising on the first error.
• At least citation block rules covered for required fields + controlled vocabulary.
• Tests for Unicode / non-Latin round-trip cases (suite fails if corruption is detected).
• Documented example: validate → inspect findings → fix → create / upload.
• No required dependency on external LLM APIs.
• Existing create / serialize paths remain behavior-compatible (additive feature only).

Related

• "TypeError: issubclass() arg 1 must be a class" when using "identifier_scheme": "ORCID" #241 / Fix nested controlled-vocabulary serialization (ORCID / issue #241) #249 — nested controlled-vocabulary serialization
• Errors when using pyDataverse 0.4.0.beta1 with Python 3.14: AsyncLibraryNotFoundError #240 / Fix Python 3.14 async compatibility #247 — Python 3.14 async compatibility
• API fails to handle metadataBlocks: "astrophysics" & "biomedical" #164 — astrophysics & biomedical metadata blocks
• json Validator Fails when License is included #161 — JSON validator fails when license is included
• error: {'status': 'ERROR', 'message': 'Error parsing Json: incorrect typeClass for field kindOfData, should be controlledVocabulary'} #153 — incorrect typeClass for controlled-vocabulary fields
• Layer 2 design notes (separate repo, branch main): https://github.com/youseihuayu-wonderful/dataverse-agentic-deposit/blob/main/dataverse_agentic_deposit_proposal.md

Questions for maintainers

1. Is a validation module in scope for pyDataverse, or would you rather it live in a separate package that depends on pyDataverse?
2. Should offline (schema-less) validation be supported, or should validation always require a live instance's metadata blocks?
3. Preference on the report data shape — dataclasses, pydantic models, or plain dicts?
4. Preferred API shape — top-level function, dedicated module/class, or method on Dataset / Dataverse?
5. Should findings mirror Dataverse server validation messages where possible?
6. Any in-flight model / metadata work I should align with?

I'm happy to implement this. If the direction seems reasonable, I'd start small: the citation block plus the non-Latin / Unicode round-trip tests above, as a self-contained PR — and leave multi-stage consistency / External Tool work until there's agreement on the shape.

[youseihuayu-wonderful]

@pdurbin could you take a look at this new feature request and see if you agree with the proposal?

[JR-1991]

@youseihuayu-wonderful do I understand this proposal correctly as a sort of schema validation for metatdatablocks? In the current version, the metadatablocks of a Dataverse/Collection are fetched at runtime and turned into Pydantic BaseModel classes and CVs to enums. These are schema-aware, according to what api/metadatablocks/{id} returns and should throw upon setting invalid metadata in a Dataset. If it helps, you can get the respective Pydantic models on a Dataverse instance via to_pydantic (see here)

[youseihuayu-wonderful]

@JR-1991 @pdurbin
Yes — schema validation for metadatablocks is the frame. to_pydantic is the schema source I would build on rather than introducing a second one.

(One small note on the current path: CVs are constrained against the allowed values from api/metadatablocks/{id}, typically via validators / JSON Schema enum rather than generated Enum classes — but the schema-aware intent is the same.)

The difference is in when and how results are returned, not in whether validation exists:

• Whole payload, one pass. Throwing at assignment returns one problem per run. Metadata assembled from a spreadsheet, a notebook, or another system often has several invalid fields at once, and enumerating them currently requires a try/except around each assignment. Validating the assembled payload against the model from to_pydantic returns all of them together — Pydantic already collects every error during whole-model validation.
• Non-throwing, with a stable output type. Finding(block, field, code, message, value, allowed) rather than an exception, so CI and batch-ingest tooling can consume results without depending on the internal structure of a ValidationError.
• Usable without a live instance. to_pydantic requires a Dataverse connection to fetch the blocks. Validating against a cached or serialized schema covers CI, and structural and Unicode checks run with no schema at all.

On "should throw upon setting invalid metadata": in #241, "identifier_scheme": "ORCID" produced TypeError: issubclass() arg 1 must be a class from MetadataBlockBase._process_field_value (fixed in #249). The value was not rejected; the validation path failed on an Annotated[...] type. #161 and #164 have the same shape — a metadata problem surfacing as a library error rather than as a field-level result.

On structured output rather than exception messages: the same interface is also useful to a downstream tool. I am sketching a human-in-the-loop deposit assistant as a Dataverse External Tool that proposes draft metadata (including from a depositor writing in a non-English language) and calls a deterministic checker in a loop — propose, validate, repair, re-validate — with a human accepting or rejecting each field. The layers are separable: Layer 1’s other callers are CI, notebooks, and batch ingest, and nothing in it depends on Layer 2. No LLM dependency enters pyDataverse either way. Design notes: https://github.com/youseihuayu-wonderful/dataverse-agentic-deposit

The first PR can be small: build the model via to_pydantic, validate the payload in one pass, map Pydantic’s errors to Finding, and add non-Latin / Unicode round-trip tests. I can shape it to match your preference, including having it live outside pyDataverse as a dependent package.

[pdurbin]

@youseihuayu-wonderful thanks for kicking off this proposal.

@JR-1991 thanks for explaining (or reminding me) about how pyDataverse already fetches the metadata blocks to enforce what it can.

A while back, @poikilotherm created this issue, which seems relevant:

• bklog: Deliverable - As a system integrator, I would appreciate a JSON Schema for validating my dataset JSON before uploading via API IQSS/dataverse-pm#26

This suggestion ultimately resulted in the ability to download the JSON Schema for a dataset, documented here: https://guides.dataverse.org/en/6.11/api/native-api.html#retrieve-a-dataset-json-schema-for-a-collection

Is that JSON Schema useful? If not, should we try to make it useful?

[youseihuayu-wonderful]

@pdurbin @JR-1991 @@poikilotherm
This is from the source, not live runs — my demo token isn't valid and both endpoints 401 for
me.

isDatasetJsonValid validates in two sequential stages inside one try block: Everit against
the schema from getCollectionDatasetSchema, then JSONDataValidation.validate(...), which
checks controlled vocabulary through
datasetFieldService.findControlledVocabularyValueByDatasetFieldTypeAndStrValue — a DB lookup.

Three consequences:

The downloadable schema has no enum. Its constraints are structural — required,
const, contains, minItems. CV values can't be in the artifact because the check hits
the database. Of the three use cases in #26 — pre-creation validation, command-line
auto-completion, client-side form generation — the artifact alone supports the first; the
other two need allowed values.
Stage 2 is unreachable while stage 1 fails. Any structural error, and the response
carries no CV information.
Stage 1 accumulates errors; stage 2 throws a single ValidationException, so one CV
error surfaces per call. A payload with 3 structural errors and 4 bad CV values takes 1 + 4
calls to diagnose, each requiring AddDataset on the collection. The result returns as one
concatenated String, so a caller acting on findings programmatically parses prose.

On "did he get what we originally wanted": the endpoint works; the downloadable artifact is
thinner than the story it was meant to serve.

Two corrections on my side. I didn't know the schema was downloadable when I opened #254. And
#254 is in the wrong repo for its two strongest points — a structured result type, and running
both stages before returning, are server-side. pyDataverse can only parse the string it is
handed.

So I'd rather split it: an issue on IQSS/dataverse for the structured result and un-gating
stage 2, and a small pyDataverse PR for cached-schema offline structural validation plus
non-Latin round-trip test coverage. Happy to close #254 in favour of those, or keep it as the
pyDataverse half.

Will confirm the live responses once I have a working token. And agreed on @poikilotherm — he
would know whether the artifact matches what he had in mind in #26.

[JR-1991]

Whole payload, one pass. Throwing at assignment returns one problem per run.

I think you can use .model_validate to validate an entire dictionary instead of iterative assigments.

Non-throwing, with a stable output type. Finding(block, field, code, message, value, allowed) rather than an exception, so CI and batch-ingest tooling can consume results without depending on the internal structure of a ValidationError.

Wouldnt it make sense to keep the validation on the pydantic side, parse ValidationError and wrap this in validate_dataset_metadata for users to not have to engage in a try/except pattern?

Usable without a live instance. to_pydantic requires a Dataverse connection to fetch the blocks. Validating against a cached or serialized schema covers CI, and structural and Unicode checks run with no schema at all.

Hm this has been a pathology in the "old" pyDataverse and I believe that hard-coding the metadata blocks into this library puts an additional maintenance burden on us. Blocks can vary across installations and I fear the variation could cause problems we could avoid by inferring them ad hoc. Also, if one wants to validate whether their dataset is compliant do that against the instance of interest?

When speaking about CI, do you mean test runs? If so, I would generally advise to use the Dataverse Action as instance within the CI. We also use it in the CI of this repo.

[youseihuayu-wonderful]

@JR-1991 @pdurbin
Thanks — this narrows it usefully.

model_validate — that removes the premise. I described iterative assignment as if it were
the only path; one-pass collection already works, so it's a usage question rather than a gap.

Wrapping ValidationError — that's the shape I had in mind, and smaller than what I wrote:
validate_dataset_metadata calls model_validate, catches ValidationError, and maps
.errors() into a stable result type. No new validation logic — a non-throwing surface, and an
output contract that doesn't expose pydantic's internal error shape to callers.

Offline / cached schema — dropping it. A cached schema can pass a dataset against blocks an
installation has since changed, which produces a confident wrong answer rather than a missing
one; compliance is relative to the target instance either way.

On CI — yes, test runs. I hadn't seen that dataverse-action is already used in this repo's
CI. That covers the case I had in mind, and it's a better answer than a cached schema, since it
validates against a real instance rather than a snapshot of one.

That leaves a narrower scope than #254 as written:

validate_dataset_metadata(...) — thin wrapper over model_validate, non-throwing, stable
result type
non-Latin / Unicode round-trip coverage — NFC vs NFD, non-ASCII controlled-vocabulary
values, non-Latin author names surviving serialize → POST → retrieve; test coverage rather
than validation, and missing either way

I can rewrite #254 down to those two, or close it and open them separately. Do you have a
preference for the result type — dataclass, pydantic model, or plain dicts?