feat(gis): shareable QGIS and ArcGIS Pro artifacts for the OGC layers - #878
Open
jirhiker wants to merge 4 commits into
Open
feat(gis): shareable QGIS and ArcGIS Pro artifacts for the OGC layers#878jirhiker wants to merge 4 commits into
jirhiker wants to merge 4 commits into
Conversation
Serves downloadable connection and layer files from /gis so a desktop GIS user reaches our OGC API - Features collections without configuring a connection by hand. Two levels per client: a connections file that registers the whole service in one import, and six curated layer files carrying symbology, field aliases, value maps and scale visibility. Generated rather than committed because every artifact embeds an absolute service URL, and there are three environments times two mounts. Static files would mean six copies of each, all going stale as collections are added -- production already advertises 30 against the 13 defined in core/pygeoapi.py. The base URL comes from _server_url(), the same value pygeoapi stamps into its own self/next links, so a client that imports a connection and then pages through items never crosses hosts. Aliases and value maps derive from core/ogc-field-descriptions.yml, the file that already feeds /schema and /queryables, so a renamed field cannot drift between the API and the shipped layer files. Value maps are emitted only where the label differs from the stored value: the lexicon columns store terms that already read as prose, and mapping them to themselves would add kilobytes of noise per layer. trend_category is the real case -- "increasing" means the water table is falling. _defaults in that file is a shared pool, not a set of universal columns; it carries well and geothermal fields side by side and describe_fields only applies the ones a view reflects. collection_fields() does the same intersection here, because unfiltered a nine-column collection ships aliases for 42 fields. QGIS drops what it cannot match; ArcGIS Pro would not. No artifact embeds a credential. Both formats allow it -- QGIS connections have username/password attributes, CIMInternetServerConnection has a user field -- but internal access uses per-user API keys so they can be revoked per user, and a shared file carrying one person's key defeats that. The two EDR collections are excluded: they publish no /items endpoint and neither client has an EDR reader, so such a layer file would not open. The curated water-level layers use the feature collections carrying the same measurements summarised per site. The ArcGIS .ogc connection file is NOT generated. Esri documents where Pro writes it but not what is in it, and it is absent from the CIM spec, so /gis gives the two-step click path instead of shipping a guess that fails in the one client we cannot test against. Verification: the .qlr format was established by loading every curated layer into a real QGIS 4.0.1 against live production -- all six load valid on the OAPIF provider, serve live features and apply renderer, aliases, value map and scale. Two findings are pinned by tests: a flattened value map segfaults QGIS rather than erroring, and a curated layer must name a collection this branch serves (the first draft of the water-level layer named one only production has). The .lyrx files follow Esri's published CIM spec but have not been opened in Pro; none is available here. uv run pytest --ignore=tests/transfers -> 1140 passed, 84 skipped, 6 xpassed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The four download routes carried response_class=PlainTextResponse, chosen for convenience rather than accuracy. Each returns a raw Response setting its own media_type, so OpenAPI advertised text/plain while the endpoint sent text/xml or application/json. Harmless to a client that ignores the schema, wrong for anything generating from it. Rather than restate the media type in the decorator, it now comes from the Response subclass in both places: XmlAttachment and JsonAttachment each declare `media_type`, FastAPI reads it off `response_class` to document the operation, and `_attachment` instantiates that same class. One definition, so the two cannot drift apart again. JsonAttachment deliberately subclasses Response rather than JSONResponse. The .lyrx body is already serialised, and re-encoding it would escape the whole CIM document into a JSON string. test_openapi_advertises_the_content_type_actually_returned compares the documented content type against the served one for all three download shapes; it fails on the previous code. uv run pytest --ignore=tests/transfers -> 1143 passed, 84 skipped, 6 xpassed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/gis returned HTML only, so a frontend wanting to offer these downloads had to hardcode the six curated layer ids and reconstruct their URLs. It now negotiates: HTML by default for a human following the link, JSON on ?f=json or an Accept: application/json header. Same precedence as api/disclaimer.py and pygeoapi, so the surfaces behave alike. The payload carries the service URL, the connection file, and per layer its id, title, abstract, backing collection, that collection's URL, geometry, renderer kind, and one download entry per client with href, media type and filename. Enough to build a download UI without knowing anything about this router. Hrefs are absolute and built from _app_base_url() rather than from the request: a browser app on another origin has to use them unchanged, and a proxy rewriting Host must not be able to redirect the caller. Tests fetch every advertised href and assert the media type and filename match what the download actually sends, so the catalogue cannot drift from the files. Also declares 404 on the two layer routes. Both already raised it for an unknown id, but only 422 was documented, so a generated client had no case for the error it is most likely to hit. uv run pytest --ignore=tests/transfers -> 1148 passed, 84 skipped, 6 xpassed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #877 (→ #876 → #874 → #866). Base is
fix/edr-waterlevels-thing-release-gate; GitHub retargets as the chain merges.Adds
/gis: downloadable QGIS and ArcGIS Pro files so a desktop GIS user reaches our OGC API - Features collections without configuring a connection by hand.Two levels, per client
.xmlconnections file — generated.ogc— not generated, see below.qlr× 6 — generated, verified in QGIS.lyrx× 6 — generated, not verifiedCurated layers:
water-wells,depth-to-water,water-level-trend,actively-monitored-wells,springs,latest-tds. Declared incore/gis-curated-layers.yml; adding one needs no code change.Generated, not committed
Every artifact embeds an absolute service URL, and there are three environments × two mounts. Static files would mean six copies of each, all going stale as collections are added — production already advertises 30 collections against the 13 defined in
core/pygeoapi.py.The base URL comes from
_server_url(), the same value pygeoapi stamps into its ownself/nextlinks, so a client that imports a connection and then pages throughitemsnever crosses hosts — the failurePYGEOAPI_INTERNAL_SERVER_URLwas added to fix.Derived, so it cannot drift
Field aliases and value maps come from
core/ogc-field-descriptions.yml, the file that already feeds/schemaand/queryables._defaultsin that file is a shared pool, not a set of universal columns — it carries well and geothermal fields side by side, anddescribe_fieldsonly applies the ones a view reflects.collection_fields()does the same intersection, because unfiltered a nine-column collection ships aliases for 42 fields includingapianddepth_unit. QGIS drops what it cannot match; ArcGIS Pro takesfieldDescriptionsat its word.Value maps are emitted only where the label differs from the stored value. The lexicon columns already store prose, so mapping them to themselves would add kilobytes of noise per layer.
trend_categoryis the real case —increasingmeans the water table is falling.No artifact embeds a credential
Both formats allow it — QGIS connections have
username/passwordattributes,CIMInternetServerConnectionhas auserfield. Internal access uses per-user API keys so they can be revoked per user, and a shared file carrying one person's key defeats that.connections-internal.xmlships the internal URL credential-free. Guarded bytest_no_artifact_embeds_a_credential.Two deliberate omissions
The
.ogcconnection file. Esri documents where Pro writes it but not what is in it, and it is absent from the CIM spec./gisgives the two-step click path instead of shipping a guess that fails in the one client we cannot test against. Someone with Pro sending back one exported.ogcmakes this a small follow-up.The EDR collections.
waterlevelsandwater_chemistrypublish no/itemsendpoint and neither client has an EDR reader, so such a layer file would not open. The curated water-level layers use the feature collections carrying the same measurements summarised per site.test_no_curated_layer_points_at_an_edr_collectionenforces it.Verification
The
.qlrformat was established by loading every curated layer into a real QGIS 4.0.1 (QgsLayerDefinition.loadLayerDefinition) against live production. All six load valid on theOAPIFprovider, serve live features (2453 for the trend layer) and apply renderer, aliases, value map and scale visibility.Two findings from that are pinned by tests:
<Option type="Map">wrapper whose single child is named for the display label and carries the stored value.latest_depth_to_water_wells, which only production has; it would have 404'd on first open.test_every_curated_layer_names_a_collection_this_branch_servesreads bothcore/pygeoapi.pyand the config template.QGIS is not a CI dependency;
docs/ogc-desktop-gis-artifacts.mdrecords the PyQGIS bootstrap for re-running that check by hand.The
.lyrxfiles follow Esri's published CIM spec (CIMLayerDocument→CIMFeatureLayer→CIMOGCAPIServiceConnection) but have not been opened in ArcGIS Pro — none is available. Treat the first open in Pro as the real test.The other two commits
9c2d79dc— the routes advertisedtext/plainwhile sendingtext/xml/application/json. Media type now comes from theResponsesubclass in both places, so the schema and the wire cannot disagree.7b5136e5—/giswas HTML-only, forcing a frontend to hardcode the six layer ids. It now negotiates?f=json, same precedence asapi/disclaimer.py. Also declares the404both layer routes already raised.Tests
uv run pytest --ignore=tests/transfers→ 1148 passed, 84 skipped, 6 xpassedtests/test_gis_artifacts.py→ 49 covering XML/JSON shape, datasource URI, tree/maplayer agreement, renderer fields existing on the collection, alias coverage and trimming, EDR exclusion, credential absence, QGIS/ArcGIS renderer agreement, catalogue href resolution and content-type agreementtests/test_authorization.pyallowlistpre-commit(black, flake8) clean🤖 Generated with Claude Code