[FEATURE] Rust port: WIP, do not merge

[Francisco Jiménez (jjcfrancisco)]

Tracking PR for the rust branch — a Rust port of overture-stac with a Python face on top. Do not merge yet.

Closes #100

What it is

A fully-functional STAC generator that is ~4x faster than the current Python implementation. The core is Rust; Python bindings expose the same functionality as an async API.

import asyncio
import overture_stac

async def main():
    ids = await overture_stac.list_releases()
    print("current releases:", ids)

    # Also: validate_catalog(dir=...) for a local build,
    #       validate_catalog_uri(catalog_uri="s3://...") for the bucket.
    report = await overture_stac.validate_url("https://stac.overturemaps.org")
    if not report.ok:
        for f in report.failures:
            print(f"[{f.kind}] {f.location}: {f.message}")

asyncio.run(main())

CLI overview

$ overture-stac
Usage: overture-stac <COMMAND>

Commands:
  build          Build a STAC catalog
  list-releases  List release IDs currently in the data bucket, newest first
  reconcile      Compare the live STAC catalog against the data bucket and report drift
  validate       Validate a built STAC catalog (JSON schema, link integrity, Overture rules)

build

Fast catalog build with optional post-write validation. Supports any object_store URI scheme (s3://, gs://, az://, file://), per-release builds, and a --root-href override for staging previews.

reconcile

Read-only diff between the live catalog and the data bucket. A dry-run against a stale catalog completes in ~0.5s:

$ overture-stac reconcile --catalog-uri s3://overture-managed-scratch-dev/franciscojimenez/stac/
Read  s3://overture-managed-scratch-dev/franciscojimenez/stac/catalog.json (2 releases)
Listed s3://overturemaps-us-west-2 (2 releases)

+ 1 release to add (in bucket, not in catalog):
    + 2026-08-19.0
- 1 release to remove (in catalog, not in bucket):
    - 2026-06-17.0

Catalog drift: 2 release(s) differ.
Re-run with --apply to fix.

--apply performs the mutation; --backup-catalog snapshots the old catalog.json first; --validate runs local + bucket-mode validation around the apply.

validate

CI-focused validator (local dir, URL, or bucket URI). Currently flags known drift on prod (~1 min):

$ overture-stac validate --url https://stac.overturemaps.org
Validated 2006 document(s) — 2 file(s) with failures (8 total)

  https://stac.overturemaps.org/2026-07-22.0/catalog.json
    [overture-rule] schema:tag contains 'vNone' — bug #115 marker
    [overture-rule] schema:version is null — bug #115 marker; omit the field instead
    [schema]        Catalog[id=2026-07-22.0]: is not valid under any of the schemas listed in the 'oneOf' keyword
    ...

Distribution ideas

• Publish to crates.io for discoverability (example: https://crates.io/crates/osm-waters).
• Ship a Homebrew formula — single static binary makes this straightforward.
• Python wheels via maturin (already the build path for the bindings).

[John McCall (lowlydba)]

Reviewed the Rust port against main (merge-base b1f0a09). Since this is explicitly WIP/do-not-merge, I only flagged correctness/safety bugs in code that's already here, not gaps or missing pieces. Everything outside apply_diff (stac/, storage/, python.rs, the PyO3 bindings vs. __init__.pyi) looked solid: bounded concurrency, defensive parsing on external parquet/JSON, bindings match the stub exactly.

One real cluster, all in apply_diff (reconcile --apply), the path that mutates the published production catalog:

• issue: src/main.rs#L510 — get_json(...).ok() collapses every failure mode (network timeout, transient 5xx, auth error, corrupt JSON) into the same None as "catalog doesn't exist yet." A transient blip makes this fall back to build_empty_root(), and the diff loop only re-adds diff.to_add links, silently dropping every release link outside this run's diff. read_catalog_children propagates the same get_json error with ? moments earlier, so the inconsistency is self-inflicted. Worth distinguishing object_store's not-found error from everything else before treating a fetch failure as "no catalog."
• issue: src/main.rs#L512-519 — downstream of the above: --backup-catalog prints "No existing catalog.json to back up" and skips the backup whenever the fetch merely failed, removing the one safeguard that would've limited the blast radius. Fixes itself once the not-found distinction lands.
• issue (non-blocking): src/main.rs#L584-585, L597, L602, L635, L640 — .expect() on as_object_mut()/as_array_mut() assumes a shape on JSON fetched live from the bucket, not JSON the process just constructed. A hand-edited or foreign-tool-written catalog.json panics mid-mutation, possibly after some put_json calls already landed.

[Francisco Jiménez (jjcfrancisco)]

Reviewed the Rust port against main (merge-base b1f0a09). Since this is explicitly WIP/do-not-merge, I only flagged correctness/safety bugs in code that's already here, not gaps or missing pieces. Everything outside apply_diff (stac/, storage/, python.rs, the PyO3 bindings vs. __init__.pyi) looked solid: bounded concurrency, defensive parsing on external parquet/JSON, bindings match the stub exactly.

One real cluster, all in apply_diff (reconcile --apply), the path that mutates the published production catalog:

• issue: src/main.rs#L510 — get_json(...).ok() collapses every failure mode (network timeout, transient 5xx, auth error, corrupt JSON) into the same None as "catalog doesn't exist yet." A transient blip makes this fall back to build_empty_root(), and the diff loop only re-adds diff.to_add links, silently dropping every release link outside this run's diff. read_catalog_children propagates the same get_json error with ? moments earlier, so the inconsistency is self-inflicted. Worth distinguishing object_store's not-found error from everything else before treating a fetch failure as "no catalog."
• issue: src/main.rs#L512-519 — downstream of the above: --backup-catalog prints "No existing catalog.json to back up" and skips the backup whenever the fetch merely failed, removing the one safeguard that would've limited the blast radius. Fixes itself once the not-found distinction lands.
• issue (non-blocking): src/main.rs#L584-585, L597, L602, L635, L640 — .expect() on as_object_mut()/as_array_mut() assumes a shape on JSON fetched live from the bucket, not JSON the process just constructed. A hand-edited or foreign-tool-written catalog.json panics mid-mutation, possibly after some put_json calls already landed.

Thanks for highlighting these issues: 1st one was ugly. All comments addressed.