[DOCS] Consolidate the Python docs into SCHEMA_GUIDE.md

CONTRIBUTING.md

@@ -6,6 +6,14 @@ Thank you for your interest in contributing.
 > [DevOps tracking issue #490](https://github.com/OvertureMaps/schema/issues/490)
 > for current status and what is planned next.
 
+## Working with the Python packages
+
+The schema is authored as Pydantic models. [SCHEMA_GUIDE.md](SCHEMA_GUIDE.md) covers
+installing and using the packages; [AUTHORING.md](AUTHORING.md) covers authoring new
+schema models and the development workflow (`uv sync`, `make check`, ruff and
+docformatter). [TROUBLESHOOTING.md](TROUBLESHOOTING.md) collects the errors that cost
+people time.
+
 ## Where to send your change
 
 This repository uses a two-branch model. Target the branch that matches your

GLOSSARY.md

@@ -1,18 +1,173 @@
-# Entity
-An entity is a thing in the physical world. It can relate both to physical objects or more abstract concepts that have a spatial presence (e.g. administrative areas are not physical objects as such, but they have physical properties that define their location and extend). An entity can only exist once. 
+# Glossary
 
-# Feature
-A Feature is an abstraction of a specific entity in the map. It exists only in its digital form.  
+Two vocabularies meet in this repository. The first describes the map: what an entity is,
+what a feature is, how feature types are named. The second describes the Python packages
+that define the schema: workspaces, entry points, specs. Both are collected here.
+
+Terms in the second group are cross-referenced to the page and section that covers them
+in full — [SCHEMA_GUIDE.md](SCHEMA_GUIDE.md) for using the schema,
+[AUTHORING.md](AUTHORING.md) for extending it, and [CONCEPTS.md](CONCEPTS.md) for
+background.
+
+---
+
+## Data model
+
+### Entity
+
+An entity is a thing in the physical world. It can relate both to physical objects or more
+abstract concepts that have a spatial presence (e.g. administrative areas are not physical
+objects as such, but they have physical properties that define their location and extent).
+An entity can only exist once.
+
+### Feature
+
+A Feature is an abstraction of a specific entity in the map. It exists only in its digital
+form.
+
+### Feature Class
 
-# Feature Class
 Feature Class is a synonym for "Feature Type". The preferred term is "Feature Type".
 
-# Feature Type
-A Feature Type is a type of entities with common properties as described in the Overture Schema.
+### Feature Type
+
+A Feature Type is a type of entities with common properties as described in the Overture
+Schema.
+
+### Instance
 
-# Instance
 Instance is a synonym for "Feature". The preferred term is "Feature".
 
-# Object
+### Object
+
 Object is a synonym for "Instance" or "Feature". The preferred term is "Feature".
 
+### Theme
+
+The top-level grouping a feature type belongs to, and a mandatory property on every
+feature. The term is deliberately chosen over "layer" to avoid that word's baggage. There
+are six: `addresses`, `base`, `buildings`, `divisions`, `places`, `transportation`. Each
+ships as its own Python package, `overture-schema-theme-*`.
+
+### Type
+
+The feature type within a theme, and a mandatory property on every feature — for example
+`theme=buildings`, `type=building`. Together `theme` and `type` identify the feature type.
+
+### Subtype
+
+An optional third property that further refines the feature type — for example
+`theme=transportation`, `type=segment`, `subtype=road`. Where a type has subtypes, the
+model for that type is usually a [discriminated union](#discriminated-union) with one arm
+per subtype. Note the spelling: the field is `subtype`, not `subType`.
+
+### GERS
+
+The Global Entity Reference System. A feature's `id` may be a GERS ID if — and only if —
+the feature represents an entity that is part of GERS.
+
+---
+
+## Toolchain
+
+### Alias
+
+Three unrelated things in this codebase go by this name. Which one is meant is almost
+always clear from context, but they are worth separating:
+
+1. **Field alias** — a mapping from a Python attribute name to the name the data uses,
+   declared with `Field(alias=...)`. It exists because some data field names are not legal
+   Python identifiers: `Building.class_` carries `alias="class"`, since `class` is a
+   reserved word. Dump with `by_alias=True` to get the data name back.
+2. **Type alias** — a module-level name bound to a type expression rather than to a class.
+   `Segment` is one: it is an `Annotated[Union[...], Discriminator(...)]`, *not* a class.
+   This is why `Segment.model_validate(...)` raises `AttributeError` and you need
+   `TypeAdapter(Segment).validate_python(...)` instead. See
+   [Working with Segment and other unions](SCHEMA_GUIDE.md#35-working-with-segment-and-other-unions).
+3. **Entry-point name** — the short name a model registers under, which need not match the
+   class name. `building = "overture.schema.buildings:Building"` registers the class
+   `Building` under the name `building`.
+
+### Entry point
+
+The mechanism by which a package advertises something to the rest of the installed
+environment, declared in `pyproject.toml`. Nothing imports these directly; they are
+discovered at runtime. The schema uses four groups: `overture.models` (feature types),
+`overture.tag_providers` ([tags](#tag)), `project.scripts` (the CLIs), and `pytest11`
+(test plugins). Registering your own model is a matter of adding an entry point — see
+[Register your own feature types](AUTHORING.md#register-your-own-feature-types).
+
+### Workspace
+
+One repository containing several independently-versioned packages that share a single
+lockfile and a single virtual environment. Declared by `[tool.uv.workspace]` in the root
+`pyproject.toml`. The schema repo is a `uv` workspace of thirteen packages. See
+[What "workspace" means](CONCEPTS.md#what-workspace-means).
+
+### Metapackage
+
+A package that ships no code of its own and exists only to depend on others.
+`overture-schema` is one: installing it pulls in every theme. Its namespace root contains
+nothing but a `py.typed` marker, which is why `from overture.schema import Building`
+fails.
+
+### Flat shape
+
+The form a feature takes as a single record with no nesting of core properties — `id`,
+`geometry`, `theme`, `type` and the rest all sitting at the same level. This is the shape
+the Pydantic models use, and the shape Overture publishes in Parquet.
+`model_validate()` expects it.
+
+### Envelope
+
+The GeoJSON form of a feature, where most properties are tucked under a `properties` key
+alongside a top-level `type: "Feature"`. Distinct from the [flat shape](#flat-shape), and
+the single most common source of confusion when validating: `model_validate()` rejects it,
+`model_validate_json()` accepts it. See
+[Two representations](SCHEMA_GUIDE.md#32-the-one-thing-to-understand-two-representations).
+
+### Discriminated union
+
+A union of models where one field's value decides which arm applies. `Segment` is
+discriminated on `subtype`: `road` selects `RoadSegment`, `rail` selects `RailSegment`,
+`water` selects `WaterSegment`. Pydantic uses the discriminator to pick an arm without
+trying each in turn, which also makes validation errors point at the right model.
+
+### NewType
+
+A distinct type wrapping an existing one, used to give a plain value a name and a set of
+constraints — `Id`, `CountryCodeAlpha2`, `LanguageTag`. At runtime the value is still a
+`str`; the wrapper carries the validation rules and survives into the generated artifacts,
+which is why it is preferred over a bare `str` with a `Field` constraint.
+
+### ModelKey
+
+The key type returned by `discover_models()`. Carries the model's entry-point `name`, its
+`entry_point` string, and its [tags](#tag). Not a plain string, and not a tuple — code
+that assumes either will break.
+
+### Tag
+
+A string attached to a model during discovery, classifying it orthogonally to its name —
+`feature`, or `overture:theme=buildings`. Tags are what the CLI's `--tag` / `--filter` /
+`--exclude` options select on. They are produced by *tag providers* registered on the
+`overture.tag_providers` entry-point group; third parties can register their own. Eight
+tags exist today: `feature`, `overture`, and one `overture:theme=*` per theme. `overture`
+marks a model built on Overture's feature model — it subclasses `OvertureFeature` — which
+a third party's own type can also be; it is not a claim that the type belongs to the
+Overture schema.
+
+### Spec
+
+The target-independent description of a model produced by the codegen's extraction layer,
+before any output format is chosen — `RecordSpec` for a model, `UnionSpec` for a
+discriminated union, `FieldSpec` for a field, `EnumSpec` for an enum. Renderers consume
+specs; they never touch Pydantic models directly. This split is what lets a new output
+format be a new renderer rather than new extraction logic. See
+[Write a new codegen target](AUTHORING.md#write-a-new-codegen-target).
+
+### Extra
+
+An optional dependency group declared under `[project.optional-dependencies]` and
+installed with bracket syntax (`package[extra]`) or `uv sync --all-extras`. Distinct from
+"extra fields", which is what `no_extra_fields` forbids on a model.

Makefile

@@ -1,4 +1,4 @@
-.PHONY: default uv-sync clean-pyspark generate-pyspark check check-namespace test-all test test-only docformat docformat-only doctest doctest-only mypy mypy-only lint-only update-baselines
+.PHONY: default uv-sync clean-pyspark generate-pyspark check check-namespace test-all test test-only docformat docformat-only doctest doctest-only mypy mypy-only lint-only format update-baselines
 
 TESTMON ?= --testmon
 
@@ -47,13 +47,13 @@ check-namespace:
 # the PySpark output first: that tree is no longer tracked in git, so the
 # generated conformance tests only exist once generation has run.
 test-all: uv-sync generate-pyspark
-	@uv run pytest -W error packages/
+	@uv run pytest -W error packages/ tests/
 
 test: uv-sync
-	@uv run pytest -W error $(TESTMON) packages/ -x -q --tb=short
+	@uv run pytest -W error $(TESTMON) packages/ tests/ -x -q --tb=short
 
 test-only:
-	@uv run pytest -W error $(TESTMON) packages/ -x -q --tb=short
+	@uv run pytest -W error $(TESTMON) packages/ tests/ -x -q --tb=short
 
 coverage: uv-sync
 	@uv run pytest packages/ --cov overture.schema --cov-report=term --cov-report=html && open htmlcov/index.html
@@ -90,11 +90,20 @@ mypy-only:
 		| tr - . \
 		| sed 's|^packages/|-p |' \
 		| xargs uv run mypy --no-error-summary
-	@for d in packages/*/tests; do find "$$d" -name "*.py" | sort | xargs uv run mypy --no-error-summary || exit 1; done
+	@for d in packages/*/tests tests; do find "$$d" -name "*.py" | sort | xargs uv run mypy --no-error-summary || exit 1; done
 
 lint-only:
-	@uv run ruff check -q packages/
-	@uv run ruff format --check packages/
+	@uv run ruff check -q packages/ tests/
+	@uv run ruff format --check packages/ tests/
+	@# Python embedded in Markdown -- keeps documented snippets in the
+	@# same style as the code they describe.
+	@git ls-files -z '*.md' | xargs -0 --no-run-if-empty uv run ruff format -q --check
+
+# The fixing counterpart to lint-only: what to run when that gate fires.
+format:
+	@uv run ruff check -q --fix packages/ tests/
+	@uv run ruff format -q packages/ tests/
+	@git ls-files -z '*.md' | xargs -0 --no-run-if-empty uv run ruff format -q
 
 update-baselines:
 	@uv run pytest --update-baselines -m baseline -q packages/

README.md

@@ -1,15 +1,136 @@
-Overture Maps Schema
+Overture Schema
 ===
 
-The Overture Maps schema working group is responsible for designing the Overture Maps Data Schema and the Global Entity Reference System (GERS).
+This code in this repository defines the Overture schema.
 
-## Documentation
-The contents of this repository are presented in a more human-friendly format at [docs.overturemaps.org](https://docs.overturemaps.org/)
+
+_ Note: You'll find reference documentation, tutorials, and examples of working with Overture data
+at [docs.overturemaps.org](https://docs.overturemaps.org/).__
+
+## What's in this repository
+
+| Path | What it is |
+|---|---|
+| `packages/` | The schema, authored as [Pydantic](https://docs.pydantic.dev/latest/) models and published as Python packages. |
+| `reference/examples/` | Feature instances that are **expected to validate** — one file per case, organized by theme. |
+| `reference/counterexamples/` | Feature instances that are **expected to fail**, most carrying the specific error they should raise. |
+| `tests/` | Tests that span packages, including the check that keeps the imports in these docs working. |
+| `docs/` | Source for the schema pages on docs.overturemaps.org, plus the versioning reference. |
+| `schema/` | **Deprecated.** The YAML JSON Schema — see [The YAML schema](#the-yaml-schema-deprecated). |
+| `examples/`, `counterexamples/` | **Deprecated.** Fixtures for the YAML schema, not the Pydantic models. |
+
+Note the two sets of examples. `reference/examples/` and `reference/counterexamples/` are
+the current ones, exercised by the Python test suite. The top-level `examples/` and
+`counterexamples/` belong to the deprecated YAML schema. They overlap heavily but have
+drifted apart; when you add a case, add it under `reference/`.
+
+## Python packages
+
+Fourteen packages under `packages/`, versioned and released independently:
+
+- **`overture-schema`** — the umbrella package. Depends on all themes and support
+  packages for a coherent set. **This is what consumers pin.**
+- **Themes** — `theme-addresses`, `theme-base`, `theme-buildings`, `theme-divisions`,
+  `theme-places`, `theme-transportation`. One package per theme, each holding its feature
+  types and the structures they share.
+- **Foundation** — `system` (the base types every model builds on) and `common`
+  (structures shared across themes, like names and sources).
+- **Tooling** — `cli` (validate data, generate JSON Schema), `codegen` (generate docs and
+  code from the models), `validation` (validate against the union of all discovered
+  models), `pyspark` (validation expressions for Spark).
+- **Extensions** — `extensions-operating-hours`, an optional add-on.
+
+These pages are for people working with the packages in code:
+
+- [SCHEMA_GUIDE.md](SCHEMA_GUIDE.md) — install the packages, explore the models, validate
+  data, generate artifacts. **Start here.**
+- [AUTHORING.md](AUTHORING.md) — register your own feature types, author new schema
+  models, build an SDK or CLI on the models.
+- [CONCEPTS.md](CONCEPTS.md) — why the schema is Pydantic, why it is many packages, and
+  what the GeoJSON envelope is.
+- [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — symptom-indexed fixes and known gotchas.
+- [GLOSSARY.md](GLOSSARY.md) — vocabulary for both the data model (entity, feature type,
+  theme) and the Python toolchain (entry point, workspace, discriminated union).
+
+Run the full test and quality suite with:
+
+```bash
+make check
+```
+
+## The YAML schema (deprecated)
+
+`schema/` holds the previous definition of the Overture schema as JSON Schema written in
+YAML, with `schema/schema.yaml` as its entry point. It is validated by `./test.sh` (which
+needs [`jv`](https://github.com/santhosh-tekuri/jsonschema)) against the top-level
+`examples/` and `counterexamples/`, wired up in
+[test-schema.yaml](.github/workflows/test-schema.yaml).
+
+**It is deprecated and scheduled for removal in December 2026.** It stays until then
+because docs.overturemaps.org still builds from it: the interactive schema blocks come
+from `schema/`, and the sample features come from `examples/`. See
+[docs/README.md](docs/README.md).
+
+**Until it is removed, backward-compatible changes belong in both places.** A change to
+the published data should land in the Pydantic models *and* in the YAML, so the two
+definitions stay in step for as long as both are live.
+
+## Project history
+
+The schema working group opened this repository in January 2023, and the artifacts of
+every phase of development are still here. 
+
+| | |
+|---|---|
+| **Jan 2023** | Repository created as `schema-wg`, chartered to design the data schema and GERS. |
+| **Mar 2023** | The schema takes shape as JSON Schema written in YAML: `schema/`, `examples/`, `counterexamples/`, and `test.sh` all arrive together. |
+| **Aug 2023** | The Schema Task Force writes a *doctrine* on top of the project's tenets, to aim the work at a vision rather than react week to week. |
+| **Jul 2024** | Overture data reaches general availability. |
+| **Jul 2025** | Pydantic rewrite begins.
+| **Nov 2025** | `packages/` are merged to main. The Pydantic schema and the YAML schema are developed in parallel. |
+| **Aug 2026** | Repository docs consolidated into the guides listed above. |
+| **Dec 2026** | `schema/`, `examples/`, and `counterexamples/` scheduled for removal. |
+
+1,788 commits from 71 contributors.
+
+### The tenets and the doctrine
+
+The working group set down six tenets early on, under the heading *"These are our tenets
+unless you know better ones"* — address the core and enable the periphery; invent across
+the gap; backward-compatible is forward-compatible; the world is neither flat nor still;
+empower, don't dictate; always open, never closed. They still govern the design, and are
+recorded with their consequences in [CONCEPTS.md](CONCEPTS.md#the-tenets).
+
+At the Schema Task Force meeting on 2023-08-30, the group decided to work toward a stated
+vision rather than react week to week, and wrote a short doctrine built on those tenets for
+the Working Group to ratify. It made five commitments:
+
+- **The schema optimizes for specific use cases, but allows extension.** Structure targets
+  selected use cases; user extensions unblock everyone else.
+- **The schema gets better over time.** Properties for core use cases keep landing, known
+  issues keep getting fixed, and the schema advances in a backward-compatible way.
+- **The schema is a cohesive whole.** Same problems get same solutions and same things get
+  same names, within and across themes, so that understanding one theme carries to the rest.
+- **The schema is usable for data consumption by humans.**
+- **The schema is usable for data consumption by open source tools.**
+
+The doctrine flagged the last two as an unresolved tension and said the project should lean
+toward one. The 2025 move to Pydantic is what settled it: the models are authored for humans
+to read and edit, and the artifacts tools need — JSON Schema, PySpark expressions,
+documentation — are generated from them. See
+[Why Pydantic rather than JSON Schema](CONCEPTS.md#why-pydantic-rather-than-json-schema).
+
+The specifics the doctrine argued over have all since been resolved. The `admins` theme is
+now `divisions`; the camelCase property names it cited (`isoCountryCodeAlpha2`,
+`road.roadNames`) became snake_case in February 2024; and the `entityId` property it
+proposed as the schema/GERS interface never shipped — features carry a GERS `id` directly.
 
 ## Contributing
-See [CONTRIBUTING.md](CONTRIBUTING.md) for branching strategy, workflow, and contribution guidelines.
 
-## Feedback
-Please provide feedback or ask questions at [Discussions](https://github.com/orgs/OvertureMaps/discussions).
+See [CONTRIBUTING.md](CONTRIBUTING.md) for branching strategy, workflow, and contribution
+guidelines.
 
+## Feedback
 
+Please provide feedback or ask questions at
+[Discussions](https://github.com/orgs/OvertureMaps/discussions).