Core spec: allow ai_context on the document root

[eisber]

Summary

The document root is the only node in the core spec with no ai_context. The key appears on
SemanticModel, Dataset, Field, Metric and Relationship; the root is closed
(additionalProperties: false) with only version and semantic_model.

Since semantic_model is a list, guidance that governs every model in a document has
nowhere to live. This PR adds one optional key to the document root, reusing the existing
definition:

Key	Type
ai_context	$ref: #/$defs/AIContext

Semantics — scope only. Document-level ai_context applies to every semantic model in
the document. How it composes with model-level ai_context is deliberately left undefined
by this version; see "Changes since the first review" below.

The case

examples/multi_model_ai_context.yaml (new) is the motivating document: two models, one per
federated data source, each queried in its own dialect, sharing no keys.

version: "0.2.0.dev0"

ai_context:
  instructions: "The fiscal year starts in July: a date in July 2024 belongs to FY2025.
    These two models come from separate systems and share no keys, so never join across
    them; answer questions that span both by querying each and reporting the results
    side by side."

semantic_model:
  - name: finance          # ANSI_SQL
    ai_context:
      instructions: "Amounts are in USD. Rows with a null posted_at are drafts."
    datasets: [...]
  - name: telemetry        # DATABRICKS
    datasets: [...]

The fiscal-year convention and the "never join across these" rule are properties of the
document. Today the only way to express them is to copy them into every model, which means:

• shared instruction is duplicated N times, and drifts;
• a consumer cannot distinguish "this was document-wide" from "this is genuinely specific to
  this model" — note finance's USD rule above, which really is model-specific.

The example is wired into validation-ci.yml so it cannot rot. That workflow previously
validated only examples/tpcds_semantic_model.yaml, by hardcoded path, so a new example
would not otherwise have been checked by anything.

Changes since the first review

Thanks to @khush-bhatia for the review — both points led to changes:

• Dropped root custom_extensions. It resembled re-adding root vendors, and unlike
  ai_context it had no motivating document. The PR is one key now.
• Cut the precedence rule back to scope. It previously said model-level ai_context
  "adds to" document-level context and "takes precedence where the two conflict". AIContext
  is oneOf: [string, object], and neither "adds" nor "conflict" is well-defined for the
  string form. Scope alone delivers the point of Core spec: document root cannot carry ai_context or custom_extensions #322 — say it once instead of N times — so
  composition is left to a later change with its own discussion.
• Split the unrelated spec.yaml sectioning fix out into core-spec: move semantic_model out of the Enumerations section in spec.yaml #367, since it was noise here.

On dialects / vendors

These look like the opposite case rather than a precedent against this change.

They were never in the schema root. main is {version, semantic_model} with
additionalProperties: false, so a document carrying root dialects is rejected by
validate.py today. They are fields on the Python OssieDocument model only, and #148/#306
remove them because the same fact is already carried at the leaf — dialect per expression,
vendor per custom_extension.

Document-wide ai_context has no leaf that already carries it. This PR also keeps the root
closed and adds nothing to required.

Precedent

An ontology document already carries ai_context on its root, and $refs this spec's
definition:

"ai_context": { "$ref": ".../core-spec/ossie-schema.json#/$defs/AIContext" }

The core spec already defines document-wide AI context — the core document root was simply
the only root that never consumed it. A test pins both roots to the same AIContext
definition so they cannot drift.

Not a breaking change

• No new $defs — the key reuses the existing definition.
• required is unchanged; the key is optional.
• additionalProperties: false is unchanged, so the root stays closed.
• examples/tpcds_semantic_model.yaml validates unchanged, and an absent key does not
  serialize.

Asserted directly against the edited schema:

Document	Result
no ai_context	accepted (back-compat)
root ai_context as string	accepted
root ai_context as object	accepted
root ai_context: 42	rejected — not valid under any of the given schemas
unknown root key	rejected — root stays closed
root custom_extensions	rejected — confirms it is fully removed
root dialects	rejected — was never valid at the root

Stacked on #367

This PR is based on #367 (a 5-line, comment-only sectioning fix in spec.yaml) and so shows
two commits until that one merges. Only the second commit,
"Core spec: allow ai_context on the document root", belongs to this PR.

Interaction with other open work

• fix(converters): emit schema-conformant OSI documents (drop non-spec root dialects/vendors) #148 / Keep converter outputs aligned with root schema #306 remove root keys (dialects, vendors) from the Python model that were
  never in the schema. Compatible in substance: this PR goes the other way round — it adds to
  the schema first and leaves the root closed. Under Keep converter outputs aligned with root schema #306's extra="forbid", declared fields
  still validate.
• core-spec/spec.yaml declares ai_context: string but the JSON schema + canonical example use a structured object #141 tracks spec.yaml declaring ai_context: string while the schema and canonical
  example use the structured object. The root entry added here shows the structured form and
  points at AIContext in ossie-schema.json as authoritative for both forms; aligning the
  four remaining model-level sites is core-spec/spec.yaml declares ai_context: string but the JSON schema + canonical example use a structured object #141's scope and is untouched here.
• does ai_context really nessary? Strengthen description and name to reduce reliance on ai_context #234 asks whether ai_context should exist at all. That is a broader question, and a
  fair one — if the direction is to shrink ai_context, this PR should wait on that outcome
  rather than iterate. This PR only makes the existing construct available at the one level
  where it is missing.

Related Issues

Refs #322

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue — see Core spec: document root cannot carry ai_context or custom_extensions #322
• Breaking changes to the spec are clearly called out in the summary — there are none; see "Not a breaking change"

Ontology

• Ontology changes in ontology/ are consistent with spec changes — ontology/ is untouched; it already had root ai_context and this aligns core with it
• New or modified terms are defined and documented

Converters

• Converter logic in converters/ is updated to reflect spec or ontology changes — not applicable, the key is optional and no converter emits it
• New converters include tests under the converter's test directory — not applicable

Validation

• Validation rules in validation/ are updated if the spec changed — validate.py is schema-driven and needed no change; verified positive, negative and back-compat cases against it
• New validation cases are covered by tests — plus examples/multi_model_ai_context.yaml is now validated in CI

Documentation

• docs/ is updated to reflect any user-facing changes
• New features or behaviors are documented with examples where appropriate — new "Document" section in spec.md, and a full example under examples/
• CONTRIBUTING.md is updated if the contribution process changed — not applicable

Examples

• examples/ are added or updated for any new spec constructs — multi_model_ai_context.yaml added; tpcds_semantic_model.yaml intentionally left unchanged to demonstrate back-compat

Tests

• All existing tests pass (pytest / CI green) — python/tests 9 -> 12 passing; validation/test_validate.py 34 passing; validation/tests/ 10 passing
• New functionality is covered by tests — 3 new tests, including one pinning the core and ontology roots to the same AIContext definition

Compliance

• ASF license headers are present on all new source files — present on the new example
• No third-party dependencies are added without PMC/IPMC approval — none added

[Copilot]

Pull request overview

Adds support for document-scoped guidance and vendor metadata by allowing ai_context and custom_extensions on the core spec document root, aligning the core document root with patterns already used elsewhere (and with the ontology root).

Changes:

• Extend core-spec/osi-schema.json document root with optional ai_context and custom_extensions (root remains closed).
• Update the Python reference model and tests to parse/serialize ai_context / custom_extensions at the document level.
• Update spec/docs to document the new document-level fields and their precedence semantics.

Show a summary per file

File	Description
python/tests/test_models.py	Adds regression tests asserting schema/root parity and round-trip serialization for document-level ai_context and custom_extensions.
python/src/ossie/models.py	Adds ai_context and custom_extensions to the OSIDocument Pydantic model.
docs/index.md	Updates docs text to include document-level AI context.
core-spec/spec.yaml	Documents root-level ai_context / custom_extensions in the YAML spec summary.
core-spec/spec.md	Adds a new “Document” section with a worked example and precedence semantics.
core-spec/osi-schema.json	Adds the new optional root properties with descriptions; keeps additionalProperties: false.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 6/6 changed files
• Comments generated: 2
• Review effort level: Lite

[eisber]

Confirmed — the diagnosis is right. Reproduced against this branch:

emitted root keys: ['version', 'dialects', 'vendors', 'semantic_model']
schema errors: ["Additional properties are not allowed ('dialects', 'vendors' were unexpected)"]

Two reasons I'm not making that change here, though:

1. It's pre-existing, not introduced by this PR. dialects/vendors are already on OSIDocument in main; this PR only adds ai_context and custom_extensions alongside them.

2. A removal limited to models.py would break converters. Root vendors is still emitted by the Honeydew converter (converters/honeydew/src/honeydew_osi/converter.py:605-611) and asserted by its tests (test_vendors_roundtrip). Dropping the model fields without the converter side would just relocate the failure.

That work is already in flight in two open PRs — #148 (dbt side + OSIDocument) and #306 (dbt + Honeydew + rejecting root extras). #306 in particular covers the converter half. Issue #322 calls this out explicitly: this PR "goes the other way round — it adds to the schema first and keeps the root closed", deliberately compatible with #148/#306.

So I'd rather let those land the removal than half-do it here and conflict with both. Nothing in this PR makes the situation worse: the root stays additionalProperties: false, and the two optional keys it adds are the only additions.

_{Side note for whoever picks up #148/#306: OSIVendor has no HONEYDEW member, so OSIDocument.model_validate() currently rejects the Honeydew converter's own output. Schema-side $defs/Vendor is free-form, so custom_extensions is unaffected — it's purely the root vendors field, which those PRs remove anyway.}

[eisber]

Good catch on the inconsistency — but I've fixed it in the opposite direction, because the string form is the side that's already been agreed to be wrong.

That contradiction is tracked in #141: spec.yaml says ai_context: string at every level, while osi-schema.json ($defs/AIContext) and the canonical examples/tpcds_semantic_model.yaml both use the structured object. The resolution direction there is to tighten spec.yaml up to the schema, not to push the schema's structured form down to string. Declaring the root as string here would move away from that and remove the only structured example in the file.

While checking this I did find a real problem with the block — one that's only visible with more of the file in view. The new keys had landed inside the # Enumerations section (after the --- on line 27, among dialects/datatypes/vendor_name, which are enum definitions, not root keys). Parsed, that section read:

['dialects', 'datatypes', 'vendor_name', 'ai_context', 'custom_extensions', 'semantic_model']

Fixed by giving the root keys their own --- section with a # Document root header, matching how every other section in this file is delimited. It now parses as:

0 -> ['version']
1 -> ['dialects', 'datatypes', 'vendor_name']
2 -> ['ai_context', 'custom_extensions', 'semantic_model']
3 -> ['datasets'] ...

which also pulls semantic_model out of the enum section, where it had been sitting on main.

I did take your "point readers to the authoritative definition" suggestion: the comment now states that the structured form is the one shown and refers to AIContext in osi-schema.json as authoritative for both forms. Aligning the four remaining model-level ai_context: string sites is #141's scope, so I've left those untouched here.

[khush-bhatia]

Not sure why we would need this ? Also dialects and vendors were removed as top level fields in the document.