Skip to content

feat(gis): shareable QGIS and ArcGIS Pro artifacts for the OGC layers - #878

Open
jirhiker wants to merge 4 commits into
fix/edr-waterlevels-thing-release-gatefrom
feat/ogc-desktop-gis-artifacts
Open

feat(gis): shareable QGIS and ArcGIS Pro artifacts for the OGC layers#878
jirhiker wants to merge 4 commits into
fix/edr-waterlevels-thing-release-gatefrom
feat/ogc-desktop-gis-artifacts

Conversation

@jirhiker

Copy link
Copy Markdown
Member

Stacked on #877 (→ #876#874#866). Base is fix/edr-waterlevels-thing-release-gate; GitHub retargets as the chain merges.

Adds /gis: downloadable QGIS and ArcGIS Pro files so a desktop GIS user reaches our OGC API - Features collections without configuring a connection by hand.

Two levels, per client

QGIS ArcGIS Pro
Everything .xml connections file — generated .ogcnot generated, see below
One layer .qlr × 6 — generated, verified in QGIS .lyrx × 6 — generated, not verified
GET /gis                                  landing page; ?f=json for the catalogue
GET /gis/qgis/connections.xml             public mount
GET /gis/qgis/connections-internal.xml    public + internal (viewer role)
GET /gis/qgis/layers/{id}.qlr
GET /gis/arcgis/layers/{id}.lyrx

Curated layers: water-wells, depth-to-water, water-level-trend, actively-monitored-wells, springs, latest-tds. Declared in core/gis-curated-layers.yml; adding one needs no code change.

Generated, not committed

Every artifact embeds an absolute service URL, and there are three environments × two mounts. Static files would mean six copies of each, all going stale as collections are added — production already advertises 30 collections against the 13 defined in core/pygeoapi.py.

The base URL comes from _server_url(), the same value pygeoapi stamps into its own self/next links, so a client that imports a connection and then pages through items never crosses hosts — the failure PYGEOAPI_INTERNAL_SERVER_URL was added to fix.

Derived, so it cannot drift

Field aliases and value maps come from core/ogc-field-descriptions.yml, the file that already feeds /schema and /queryables.

_defaults in that file is a shared pool, not a set of universal columns — it carries well and geothermal fields side by side, and describe_fields only applies the ones a view reflects. collection_fields() does the same intersection, because unfiltered a nine-column collection ships aliases for 42 fields including api and depth_unit. QGIS drops what it cannot match; ArcGIS Pro takes fieldDescriptions at its word.

Value maps are emitted only where the label differs from the stored value. The lexicon columns already store prose, so mapping them to themselves would add kilobytes of noise per layer. trend_category is the real case — increasing means the water table is falling.

No artifact embeds a credential

Both formats allow it — QGIS connections have username/password attributes, CIMInternetServerConnection has a user field. Internal access uses per-user API keys so they can be revoked per user, and a shared file carrying one person's key defeats that. connections-internal.xml ships the internal URL credential-free. Guarded by test_no_artifact_embeds_a_credential.

Two deliberate omissions

The .ogc connection file. Esri documents where Pro writes it but not what is in it, and it is absent from the CIM spec. /gis gives the two-step click path instead of shipping a guess that fails in the one client we cannot test against. Someone with Pro sending back one exported .ogc makes this a small follow-up.

The EDR collections. waterlevels and water_chemistry publish no /items endpoint and neither client has an EDR reader, so such a layer file would not open. The curated water-level layers use the feature collections carrying the same measurements summarised per site. test_no_curated_layer_points_at_an_edr_collection enforces it.

Verification

The .qlr format was established by loading every curated layer into a real QGIS 4.0.1 (QgsLayerDefinition.loadLayerDefinition) against live production. All six load valid on the OAPIF provider, serve live features (2453 for the trend layer) and apply renderer, aliases, value map and scale visibility.

Two findings from that are pinned by tests:

  • A flattened value map segfaults QGIS 4.0.1 rather than erroring. Each entry must be its own <Option type="Map"> wrapper whose single child is named for the display label and carries the stored value.
  • A curated layer must name a collection this branch serves. The first draft of the water-level layer pointed at latest_depth_to_water_wells, which only production has; it would have 404'd on first open. test_every_curated_layer_names_a_collection_this_branch_serves reads both core/pygeoapi.py and the config template.

QGIS is not a CI dependency; docs/ogc-desktop-gis-artifacts.md records the PyQGIS bootstrap for re-running that check by hand.

The .lyrx files follow Esri's published CIM spec (CIMLayerDocumentCIMFeatureLayerCIMOGCAPIServiceConnection) but have not been opened in ArcGIS Pro — none is available. Treat the first open in Pro as the real test.

The other two commits

  • 9c2d79dc — the routes advertised text/plain while sending text/xml/application/json. Media type now comes from the Response subclass in both places, so the schema and the wire cannot disagree.
  • 7b5136e5/gis was HTML-only, forcing a frontend to hardcode the six layer ids. It now negotiates ?f=json, same precedence as api/disclaimer.py. Also declares the 404 both layer routes already raised.

Tests

  • uv run pytest --ignore=tests/transfers1148 passed, 84 skipped, 6 xpassed
  • tests/test_gis_artifacts.py → 49 covering XML/JSON shape, datasource URI, tree/maplayer agreement, renderer fields existing on the collection, alias coverage and trimming, EDR exclusion, credential absence, QGIS/ArcGIS renderer agreement, catalogue href resolution and content-type agreement
  • Four new anonymous routes added to the tests/test_authorization.py allowlist
  • pre-commit (black, flake8) clean

🤖 Generated with Claude Code

jirhiker and others added 4 commits August 23, 2026 03:02
Serves downloadable connection and layer files from /gis so a desktop GIS user
reaches our OGC API - Features collections without configuring a connection by
hand. Two levels per client: a connections file that registers the whole
service in one import, and six curated layer files carrying symbology, field
aliases, value maps and scale visibility.

Generated rather than committed because every artifact embeds an absolute
service URL, and there are three environments times two mounts. Static files
would mean six copies of each, all going stale as collections are added --
production already advertises 30 against the 13 defined in core/pygeoapi.py.
The base URL comes from _server_url(), the same value pygeoapi stamps into its
own self/next links, so a client that imports a connection and then pages
through items never crosses hosts.

Aliases and value maps derive from core/ogc-field-descriptions.yml, the file
that already feeds /schema and /queryables, so a renamed field cannot drift
between the API and the shipped layer files. Value maps are emitted only where
the label differs from the stored value: the lexicon columns store terms that
already read as prose, and mapping them to themselves would add kilobytes of
noise per layer. trend_category is the real case -- "increasing" means the
water table is falling.

_defaults in that file is a shared pool, not a set of universal columns; it
carries well and geothermal fields side by side and describe_fields only
applies the ones a view reflects. collection_fields() does the same
intersection here, because unfiltered a nine-column collection ships aliases
for 42 fields. QGIS drops what it cannot match; ArcGIS Pro would not.

No artifact embeds a credential. Both formats allow it -- QGIS connections have
username/password attributes, CIMInternetServerConnection has a user field --
but internal access uses per-user API keys so they can be revoked per user, and
a shared file carrying one person's key defeats that.

The two EDR collections are excluded: they publish no /items endpoint and
neither client has an EDR reader, so such a layer file would not open. The
curated water-level layers use the feature collections carrying the same
measurements summarised per site.

The ArcGIS .ogc connection file is NOT generated. Esri documents where Pro
writes it but not what is in it, and it is absent from the CIM spec, so /gis
gives the two-step click path instead of shipping a guess that fails in the one
client we cannot test against.

Verification: the .qlr format was established by loading every curated layer
into a real QGIS 4.0.1 against live production -- all six load valid on the
OAPIF provider, serve live features and apply renderer, aliases, value map and
scale. Two findings are pinned by tests: a flattened value map segfaults QGIS
rather than erroring, and a curated layer must name a collection this branch
serves (the first draft of the water-level layer named one only production
has). The .lyrx files follow Esri's published CIM spec but have not been opened
in Pro; none is available here.

uv run pytest --ignore=tests/transfers -> 1140 passed, 84 skipped, 6 xpassed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The four download routes carried response_class=PlainTextResponse, chosen for
convenience rather than accuracy. Each returns a raw Response setting its own
media_type, so OpenAPI advertised text/plain while the endpoint sent text/xml
or application/json. Harmless to a client that ignores the schema, wrong for
anything generating from it.

Rather than restate the media type in the decorator, it now comes from the
Response subclass in both places: XmlAttachment and JsonAttachment each declare
`media_type`, FastAPI reads it off `response_class` to document the operation,
and `_attachment` instantiates that same class. One definition, so the two
cannot drift apart again.

JsonAttachment deliberately subclasses Response rather than JSONResponse. The
.lyrx body is already serialised, and re-encoding it would escape the whole CIM
document into a JSON string.

test_openapi_advertises_the_content_type_actually_returned compares the
documented content type against the served one for all three download shapes;
it fails on the previous code.

uv run pytest --ignore=tests/transfers -> 1143 passed, 84 skipped, 6 xpassed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/gis returned HTML only, so a frontend wanting to offer these downloads had to
hardcode the six curated layer ids and reconstruct their URLs. It now
negotiates: HTML by default for a human following the link, JSON on ?f=json or
an Accept: application/json header. Same precedence as api/disclaimer.py and
pygeoapi, so the surfaces behave alike.

The payload carries the service URL, the connection file, and per layer its id,
title, abstract, backing collection, that collection's URL, geometry, renderer
kind, and one download entry per client with href, media type and filename.
Enough to build a download UI without knowing anything about this router.

Hrefs are absolute and built from _app_base_url() rather than from the request:
a browser app on another origin has to use them unchanged, and a proxy
rewriting Host must not be able to redirect the caller. Tests fetch every
advertised href and assert the media type and filename match what the download
actually sends, so the catalogue cannot drift from the files.

Also declares 404 on the two layer routes. Both already raised it for an
unknown id, but only 422 was documented, so a generated client had no case for
the error it is most likely to hit.

uv run pytest --ignore=tests/transfers -> 1148 passed, 84 skipped, 6 xpassed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant