fix: resolve $ref and combinator keywords in JSON Schema to Python conversion

[DerMayer1]

What

schema_type_to_python handles enum, const, type, arrays and objects. Everything else falls through to return Any. $ref, anyOf, oneOf and allOf are all unhandled, so a schema using them loses its constraints. For $ref that loss is not silent: it makes the converted model unusable.

Pydantic emits $ref for a nested model, so a round trip drops the nesting. This is not a
quiet widening of the type: the converted model is then unusable, because the regex backend
refuses a property with no type.

to_regex(JsonSchema(json_schema_dict_to_pydantic(Person.model_json_schema(), "Person")))
ValueError: Unsupported JSON Schema structure {"title":"Address"}

So on main the converter cannot round-trip any pydantic model that contains another model.

class Address(BaseModel):
    street: str
    city: str

class Person(BaseModel):
    name: str
    address: Address

json_schema_dict_to_pydantic(Person.model_json_schema(), "Person")
# before: address -> typing.Any
# after:  address -> Address(street: str, city: str)

The three converters also read properties directly off the document. Pydantic emits a top-level $ref for a self-referential model — {"$defs": {...}, "$ref": "#/$defs/Node"} — so properties is empty and the generated model has no fields at all:

class Node(BaseModel):
    value: int
    children: Optional[List["Node"]] = None

json_schema_dict_to_pydantic(Node.model_json_schema(), "Node")
# before: {}   (no fields)
# after:  value: int, children: Optional[List[Any]]

Why it's user-visible

Gemini is the one backend that converts a JSON schema into a Python type and ships it as response_schema — models/gemini.py:181, JsonSchema.convert_to(output_type, ["dataclass", "typeddict", "pydantic"]). So passing a schema with a nested object to Gemini sends a constraint that doesn't contain the nested structure, with no warning.

Other backends are unaffected: OpenAI, Ollama and LMStudio convert to dict, dottxt to str, and the local backends compile the raw schema.

Scope

In. Local $ref (#, #/$defs/…, #/definitions/…), anyOf/oneOf → Union, and a one-element allOf → its subschema (the form Pydantic emits for a $ref with sibling metadata).

Out. Remote refs, dangling pointers and multi-element allOf stay Any rather than raising, so an unsupported schema still degrades instead of breaking. Merging two or more allOf subschemas means reconciling conflicting keywords and competing required sets; that's a separate change.

Reference cycles resolve to Optional[Any]. A recursive schema has no finite Python type
here, and the alternative is recursing until the stack ends. It is Optional[Any] rather
than a bare Any so the back edge round-trips to {"anyOf": [{}, {"type": "null"}]}, which
the regex backend accepts; a bare Any serializes to a property with no type and raises
the same ValueError as above when the recursive property is required. Null is also the
only way an instance of a required self-reference can terminate.

One incidental change: items given as a list (the draft-7 tuple form) previously raised AttributeError from schema.get on a list. It is now List[Any]. Happy to drop that if you'd rather keep the PR to the $ref path.

Why this is a fix rather than a feature

tests/types/test_json_schema_utils.py already covers nested objects and asserts full recursion into them (test_json_schema_dict_to_pydantic_nested_object and the typeddict/dataclass equivalents). $ref is the same intent expressed through $defs rather than inline, so the existing tests describe behaviour that $ref schemas were silently not getting.

Tests

17 added, covering local $ref, the draft-07 definitions keyword, unresolvable refs, root-level $ref, recursive $ref, anyOf/oneOf/allOf, the typeddict and dataclass paths, the end-to-end Pydantic round trip, and to_regex over
converted recursive and nested models rather than their annotations alone.

tests/types/: 302 passed, 2 failed. Both failures are test_dsl.py::test_dsl_cfg_from_file and test_dsl_json_schema_from_file, which fail identically on a clean checkout of main on Windows (NamedTemporaryFile cannot be reopened while held). Unrelated to this change.

ruff 0.9.1 with --config=pyproject.toml and mypy 1.14.1 with --allow-redefinition are both clean on the changed files.

[ErenAta16]

Ran this against a clone of main at be2cd15 and then against d986758, using the Person/Address case from the description plus a set of schemas I picked to probe the edges. It does what it says, and the headline case holds up end to end:

Person.address    main: typing.Any        this PR: Address(street: str, city: str)

Generation follows through, which is the part I cared about most:

{"name": "a", "address": {"street": "s", "city": "c"}}   -> True
{"name": "a", "address": "not an object"}                -> False
{"name": "a", "address": 5}                              -> False

One thing you should put in the description, because it makes the case stronger than what is written there now. On main this is not a silent loss of constraints for nested models, it is a hard failure. Any round-trips to a property with no type, and the regex builder refuses it:

to_regex(JsonSchema(json_schema_dict_to_pydantic(Person.model_json_schema(), "Person")))
ValueError: Unsupported JSON Schema structure {"title":"Address"}

So today the converter cannot round-trip any pydantic model that contains another model, it raises rather than generating something loose. "Silently loses its constraints" undersells it and might be why this has sat unreviewed.

Combinators, all Any on main:

anyOf [str, int]      Union[str, int]
oneOf [str, null]     Optional[str]
anyOf [int, null]     Optional[int]
allOf single          int
allOf multi           Any
enum / array / plain  unchanged

The recursion question was the one I expected to break, and it does not. Both a self-referential schema and a mutually recursive pair terminate:

Node   value -> int,  child -> Optional[Any]
A      b     -> Optional[AnonymousPydanticModel]

No RecursionError, and generation still works on the converted recursive model because the back edge lands as Optional[Any], which round-trips to {"anyOf": [{}, {"type": "null"}]} and the regex builder accepts that shape even though it rejects a bare untyped property. That is a nice accident of the two representations and I would not want it to stay accidental. There is no test covering a recursive schema, so if the cycle guard is ever tightened to return something other than Optional[Any], or loosened to plain Any, this goes straight back to the ValueError above and nothing in the suite catches it. Worth one test that converts a self-referential schema and calls to_regex on the result, not just on the annotations.

It is also worth a line in the docstring saying the back edge is intentionally unconstrained. It is the correct call, a finite regex cannot express an unbounded tree, but a reader hitting child -> Optional[Any] will otherwise read it as the bug this PR set out to fix rather than the designed stopping point.

Cross-PR note, since these will collide. #1949 changes field_definitions[property] = (Optional[typ], None) to (typ, None), and that exact line sits inside one of this PR's hunks as context, so merge order matters textually. Semantically they fit together rather than fight: this PR is what makes an explicitly nullable schema produce Optional, and #1949 stops a merely non-required field from producing it. Landing #1963 first and #1949 second reads cleanest, because after this PR the Optional that #1949 removes is genuinely redundant for the anyOf/oneOf null cases rather than being the only thing providing them. Flagging so they are not reviewed as independent.

One smaller thing, take it or leave it. Malformed subschemas are handled two different ways:

{"anyOf": [{"type": "integer"}, "junk"]}    ->  int     (non-dict member dropped)
{"allOf": [{"type": "integer"}, {"type": "string"}]}  ->  Any   (bails out)

Dropping the junk member is defensible, and so is bailing, but right now which one you get depends on the keyword. If that is deliberate a short comment saying so would help, since a schema with a typo'd member currently narrows to int without complaint.

Otherwise this is a solid change and the test additions cover the parts that matter.

[DerMayer1]

Thanks for actually running it, and for the to_regex point. You were right that the recursion held up by accident, and chasing that turned up a case that is broken today rather than only after some future change.

The back edge is safe only when the recursive property is optional, because then json_schema_dict_to_pydantic wraps it and you get {"anyOf": [{}, {"type": "null"}]}. Make it required and there is no wrapper:

{"$defs": {"Node": {"properties": {"value": {...}, "next": {"$ref": "#/$defs/Node"}},
                    "required": ["value", "next"]}}, "$ref": "#/$defs/Node"}

next -> typing.Any  ->  {"title": "Next"}
ValueError: Unsupported JSON Schema structure {"title":"Next"}

Same for a mutually recursive pair where both references are required. So the exact error this PR removes for nested models was still reachable through recursion.

The cycle now returns Optional[Any] directly instead of relying on the caller to wrap it. Null is also the only way an instance of a required self-reference can terminate, so it is the honest type rather than a patch. Four shapes now round-trip: required back edge, optional back edge, mutual pair, and list-based recursion.

Tests call to_regex on the converted model rather than only checking annotations, which is the part that would have caught this. Reverting the guard to a bare Any fails three of them. Added the Notes section saying the back edge is deliberately unconstrained and why narrowing it breaks the regex backend.

On the description, you are right and I have updated it. Calling it a silent loss of constraints was wrong: to_regex refuses the converted model outright, so today the converter cannot round-trip any pydantic model containing another model.

On the malformed members, I looked into it and the behaviour is already consistent, though my examples in the code did not say so. All three combinators drop a non-dict member and fall back to Any when none are left:

anyOf [{"type": "integer"}, "junk"]   ->  int
oneOf [{"type": "integer"}, "junk"]   ->  int
allOf [{"type": "integer"}, "junk"]   ->  int
anyOf ["junk"]                        ->  Any
allOf ["junk"]                        ->  Any

The Any you saw from allOf was the multi-member case, allOf [{"type": "integer"}, {"type": "string"}], which is a well-formed intersection rather than a malformed member. Merging those means reconciling conflicting keywords and competing required sets, so it stays Any. Added a comment stating both rules, and a parametrized test over the three keywords so the malformed-member behaviour is pinned rather than incidental.

Agreed on #1949 and the ordering. This PR is what makes an explicitly nullable schema produce Optional, so landing it first leaves #1949 removing an Optional that is genuinely redundant. Happy to rebase on whichever lands first.

tests/types: 302 passed, 2 failed, both test_dsl.py file-handling tests that fail the same way on a clean checkout of main on Windows. ruff 0.9.1 and mypy 1.14.1 clean.

[ErenAta16]

Reproduced your required-versus-optional split, and the asymmetry comes from two adjacent lines in json_schema_dict_to_pydantic:

if property not in required:
    field_definitions[property] = (Optional[typ], None)
else:
    field_definitions[property] = (typ, ...)

With typ degraded to Any by the unresolved back edge, those two branches serialize very differently:

required  (typ, ...)              -> {"title": "Next"}
optional  (Optional[typ], None)   -> {"anyOf": [{}, {"type": "null"}], "default": null, "title": "Next"}

and that is the whole story:

required   type=False  anyOf=False   -> no structural keyword to dispatch on
optional   type=False  anyOf=True    -> dispatchable

The optional spelling smuggles in an anyOf that the container path is willing to walk, so the cycle appeared to work while carrying no type information at all. The required spelling has nothing to dispatch on and surfaces as the Unsupported JSON Schema structure you hit. So the recursion was not merely fragile against a future change, it was already broken for the more common way to write a linked structure, and the optional case was masking it.

The consequence for the tests is the part I would make sure lands. A recursive fixture written with an optional next passes on main today and would keep passing under a change that reintroduces this, because the anyOf wrapper does the dispatching rather than the fix. Any regression test for the cycle needs the required spelling, and ideally both, with a comment saying the optional one is a compatibility case rather than a guard. Otherwise the suite looks like it covers recursion and covers only the accident.

The mutually-recursive pair with both references required is the same shape one step out, and worth including for the reason you give: it is the case where neither side has a wrapper to hide behind.

Thanks for chasing it rather than taking the "it happens to work" answer. I would not have found the required case from where I was looking.