Simplify fiboa publish for catalog-driven publication

CHANGELOG.md

@@ -7,6 +7,20 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+- Fix `use_variant_as_determination`: the determination:datetime column was dropped again because it was not listed in `columns` (affected DK, HR)
+- Declare the beautifulsoup4 dependency that the ES-PV and ES-VC converters import
+- EE: name the cached WFS responses (ee_gsaa_<year>.gml)
+- ES-MD: the archive no longer nests RECINTO.shp in a folder
+- SK: KODKD is the (non-unique, sometimes empty) LPIS block code, keep it as block_id and use the row index as id
+- ES-CAT: the 2024 download is a shapefile package, not a GeoPackage; 34 crop names new in 2024 added to the mapping
+- CZ: find the shapefile in nested archive folders (2026)
+- DE-BB: read the shapefile as cp1252 (its .cpg wrongly says UTF-8)
+- NL: new PDOK download location (rvo/gewaspercelen/atom), add the 2026 concept edition
+- DE-TH: note the INSPIRE download service
+- Drop rows without a crop:code (required by the crop extension) with a warning instead of failing the conversion (BE-VLG 2023, ES-CN had one such row each); more than 1% missing is an error
+- Europe-LAND converters: use crop_name as crop:code when the file's crop_code column is empty (LT 2024)
+- BE-VLG: derive determination:datetime from the variant year instead of a constant date
+- `fiboa publish` no longer uploads to S3 or generates README/LICENSE files. It creates GeoParquet, PMTiles and a STAC Collection with relative links, `file:size`/`file:checksum` and a web-map-links v1.3.0 `pmtiles` link. Publishing is done by catalogs such as the [harmonized field data catalog](https://github.com/fieldsoftheworld/harmonized-field-data-catalog).
 - Add Italy Tuscany (IT-1) basd on EuroCrops v2
 - Suuport multiple years for CZ
 - Multiple years for DE_sh

README.md

@@ -62,7 +62,7 @@ fiboa CLI supports various commands to work with the files:
     - [Improve a fiboa Parquet file](#improve-a-fiboa-parquet-file)
     - [Update an extension template with new names](#update-an-extension-template-with-new-names)
     - [Converter for existing datasets](#converter-for-existing-datasets)
-    - [Publish datasets to source coop or your own s3 repository](#publish-datasets-to-source-coop-or-your-own-s3-repository)
+    - [Publish datasets](#publish-datasets)
   - [Development](#development)
     - [Implement a converter](#implement-a-converter)
   - [Run in Docker](#run-in-docker)
@@ -193,46 +193,43 @@ Use any of the IDs from the list to convert an existing dataset to fiboa:
 
 See [Implement a converter](#implement-a-converter) for details about how to
 
-### Publish datasets to source coop or your own s3 repository
+### Publish datasets
 
 `fiboa publish <dataset> -o <target>`
 
-The publish converts and publishes a fiboa dataset to source coop or your own s3 repository. The target directory 
-will be filled with the following files:
+Converts and validates a fiboa dataset and prepares everything that is needed to publish it
+in a (STAC-based) catalog. The target directory will be filled with the following files:
 
 ```
 <target>/
-  <dataset>.parquet
-  <dataset>.pmtiles      # requires working ogr2ogr and tippecanoe
-  stac/collection.json
-  README.md              # generated if --generate-meta/-gm flag is present
-  LICENSE.txt            # generated if --generate-meta/-gm flag is present
+  <dataset>[-<variant>].parquet
+  <dataset>[-<variant>].pmtiles   # requires working ogr2ogr and tippecanoe
+  collection.json                 # STAC Collection with relative links to the files above
 ```
 
-This directory is synchronized to the s3 repository (default source.coop/fiboa/data).
+The STAC Collection carries `file:size` and `file:checksum` for the files and a
+`pmtiles` link (web-map-links extension). Existing files in the target directory are reused,
+delete them to regenerate. Uploading to a bucket and catalog-specific metadata
+(README, styles, thumbnails, ...) are the job of the catalog that publishes the data, e.g. the
+[harmonized field data catalog](https://github.com/fieldsoftheworld/harmonized-field-data-catalog).
 
-**Requirements**: Requires the [aws CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to be installed, 
-and `AWS_ACCESS_KEY_ID` with `AWS_SECRET_ACCESS_KEY` environment variables. Also, for generating the pmtiles file, 
-it requires [ogr2ogr](https://gdal.org/programs/ogr2ogr.html) and [tippecanoe](https://github.com/mapbox/tippecanoe).
+The command runs:
 
-The command executes the following steps:
-
-- `fiboa convert` to generate a fiboa parquet dataset. All convert parameters are passed to the converter.
-- `fiboa validate` to validate the fiboa dataset
-- creates a <dataset>.pmtiles from the parquet file. Uses ogr2ogr and tippecanoe
-- `fiboa create-stac-collection` to create a STAC collection
-- `fiboa publish` to publish the fiboa dataset to a source coop or your own s3 repository
+- `fiboa convert` to create a `<dataset>[-<variant>].parquet` file
+- `fiboa validate` to validate the GeoParquet file
+- `ogr2ogr | tippecanoe` to create the PMTiles file
+- `fiboa create-stac-collection` to create the STAC Collection
 
 Examples:
 
-- `fiboa publish at_crop -o data/at_crop`
-- `fiboa publish -c /tmp/cache -gm br_conab -o data/br_conab`
+- `fiboa publish at -o data/at`
+- `fiboa publish -c /tmp/cache nl --variant 2025 -o data/nl/2025`
 
 Relevant parameters:
 
-- `--generate-meta/-gm` Generatse the README.md and LICENSE.txt files if absent, based on data-survey and converter properties.
-- `--data-url` The URL to the data repository, used when generating the README
-- `--s3-upload-path` The `aws s3 sync` target. Defaults to `s3://source.coop/fiboa/data` . Uploading requires the `aws` CLI, and `AWS_ACCESS_KEY_ID` with `AWS_SECRET_ACCESS_KEY` environment variables.
+- `--variant` Choose the variant (e.g. year) of a dataset, defaults to the first variant.
+- `--no-pmtiles` Skip PMTiles generation.
+- `--tippecanoe-opts` Options passed to tippecanoe, defaults to `-zg --drop-densest-as-needed --extend-zooms-if-still-dropping`.
 
 Check `fiboa publish --help` for more details.
 

fiboa_cli/conversion/fiboa_converter.py

@@ -4,20 +4,47 @@
 from ..fiboa.version import get_fiboa_uri
 
 AREA_KEY = "metrics:area"
+# Properties that a schema requires to be non-null; rows lacking them cannot
+# validate, so they are dropped (with a warning) rather than failing the run.
+REQUIRED_NON_NULL = ("crop:code",)
 
 
 class FiboaBaseConverter(BaseConverter):
     area_is_in_ha = True
     area_calculate_missing = False
     use_variant_as_determination = False
+    # rows lacking a REQUIRED_NON_NULL value are dropped up to this share, else it's an error
+    max_dropped_share = 0.01
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.extensions.add(get_fiboa_uri())
+        if self.use_variant_as_determination:
+            # The column is added in post_migrate; list it so it survives the
+            # "remove unlisted columns" step of the base converter.
+            self.columns = {**self.columns, "determination:datetime": "determination:datetime"}
 
     def post_migrate(self, gdf):
         gdf = super().post_migrate(gdf)
 
+        # post_migrate runs before columns are renamed, so look up the source column
+        for key in REQUIRED_NON_NULL:
+            for src, dst in self.columns.items():
+                targets = dst if isinstance(dst, (list, tuple)) else [dst]
+                if key in targets and src in gdf.columns:
+                    nulls = gdf[src].isna()
+                    if nulls.any():
+                        share = nulls.mean()
+                        if share > self.max_dropped_share:
+                            raise ValueError(
+                                f"{int(nulls.sum())} of {len(gdf)} rows ({share:.1%}) have no "
+                                f"{key} ({src}); fix the converter instead of dropping them"
+                            )
+                        self.warning(
+                            f"Dropping {int(nulls.sum())} rows without a value for {key} ({src})"
+                        )
+                        gdf = gdf[~nulls]
+
         gdf_area_key = next((k for k, v in self.columns.items() if v == AREA_KEY), None)
         if self.area_calculate_missing:
             # If CRS is not in meters, reproject to an equal-area projection for area calculation

fiboa_cli/datasets/be_vlg.py

@@ -10,6 +10,7 @@ class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
     variants = {
         str(k): {PREFIX + v: [v.replace("_GPKG.zip", ".gpkg")]}
         for k, v in (
+            (2026, "agpa_2026_2026-06-02_public.zip"),
             (2025, "Landbouwgebruikspercelen_2025_-_Voorlopig_(extractie_02-06-2025)_GPKG.zip"),
             (2024, "Landbouwgebruikspercelen_2024_-_Definitief_(extractie_27-03-2025)_GPKG.zip"),
             (2023, "Landbouwgebruikspercelen_2023_-_Definitief_(extractie_28-03-2024)_GPKG.zip"),
@@ -42,9 +43,9 @@ class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
         "GWSCOD_H": "crop:code",
         "GWSNAM_H": "crop:name",
     }
-    column_additions = {
-        "determination:datetime": "2024-03-28T00:00:00Z",
-    }
+    # Each edition is the campaign year of its variant; the old constant
+    # "2024-03-28" was the extraction date of one edition applied to all of them.
+    use_variant_as_determination = True
     ec_mapping_csv = "be_vlg_2021.csv"
 
     missing_schemas = {"properties": {"typology": {"type": "string"}}}

fiboa_cli/datasets/commons/euro_land.py

@@ -47,3 +47,11 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         provider = "Europe-LAND HE Project <https://doi.org/10.5281/zenodo.14230620>"
         self.provider = (f"{self.provider}, {provider}") if self.provider else provider
+
+    def migrate(self, gdf):
+        # Some Europe-LAND files (e.g. LT 2024) ship an empty crop_code column next to
+        # a populated crop_name; the name is then the best available crop code.
+        if "crop_code" in gdf.columns and gdf["crop_code"].isna().all():
+            self.warning("crop_code is empty, using crop_name as crop:code")
+            gdf["crop_code"] = gdf["crop_name"]
+        return super().migrate(gdf)

fiboa_cli/datasets/cz.py

@@ -20,7 +20,8 @@
 
 class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
     # see https://mze.gov.cz/public/app/eagriapp/lpisdata/
-    variants = {str(k): {BASE.format(v): ["*.shp"]} for k, v in ITEMS.items()}
+    # the 2026 archive nests the shapefile in a folder, older ones are flat
+    variants = {str(k): {BASE.format(v): ["**/*.shp"]} for k, v in ITEMS.items()}
     id = "cz"
     short_name = "Czech"
     title = "Field boundaries for Czech"

fiboa_cli/datasets/data-files/es_cat.csv

@@ -297,3 +297,37 @@ original_code,original_name,translated_name
 310,MANDARINER,MANDARIN
 311,'ALGARROBA' HERBÀCIA,ALGARROBA
 312,ESPÍGOL O LAVANDA,LAVENDER
+313,ALOE VERA,ALOE VERA
+314,ANTARA VINYA,VINEYARD INTER-ROW
+315,"ARANYONER, PRUNYONER",BLACKTHORN (SLOE)
+316,"BITXO, VITXO",CHILI PEPPER
+317,CIBULET,CHIVES
+318,CIVADA I BLAT,OATS AND WHEAT
+319,CIVADA I ORDI,OATS AND BARLEY
+320,CIVADA I TRITICALE,OATS AND TRITICALE
+321,COL KALE,KALE
+322,CROTALÀRIA,CROTALARIA (SUNN HEMP)
+323,ERBS I CIVADA,BITTER VETCH AND OATS
+324,ESPELTA PETITA,EINKORN
+325,GINJOLER,JUJUBE
+326,LAVANDA X ESPIGOL,LAVANDIN
+327,LOT CORNICULAT,BIRD'S-FOOT TREFOIL
+328,MALVA,MALLOW
+329,MARIALLUÏSA,LEMON VERBENA
+330,"MARXANT, BLET",AMARANTH
+331,"MENTA VERDA, HERBA DE SANTA MARIA",SPEARMINT
+332,"MILL ITALIÀ, CUA DE GUILLA",FOXTAIL MILLET
+333,MONGETA VERMELLA,RED KIDNEY BEAN
+334,MORER,MULBERRY
+335,"NYÀMERA, PATATA DE CANYA",JERUSALEM ARTICHOKE
+336,OKRA,OKRA
+337,PASSACAMINS,KNOTGRASS
+338,"PISANA, ESPELTA BESSONA",EMMER
+339,"RAVE PICANT, RAVE RUSTICÀ",HORSERADISH
+340,RUIBARBRE,RHUBARB
+341,SULLA O ENCLOVA,SULLA (FRENCH HONEYSUCKLE)
+342,TARONGER AGRE,BITTER ORANGE TREE
+343,VEÇA I RAIGRÀS,VETCH AND RYEGRASS
+344,VIMETERA,OSIER WILLOW
+345,VIVER - PRODUCTOR MVR,NURSERY (MVR PRODUCER)
+346,XIRIMOIER,CHERIMOYA TREE

fiboa_cli/datasets/de_bb.py

@@ -14,6 +14,8 @@ class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
     license = "DL-DE-BY-2.0"
     provider = "Land Brandenburg <https://geobroker.geobasis-bb.de/gbss.php?MODE=GetProductInformation&PRODUCTID=9e95f21f-4ecf-4682-9a44-e5f7609f6fa0>"
     ec_mapping_csv = "de.csv"
+    # The .cpg claims UTF-8 but the DBF is cp1252 (June 2026 download)
+    open_options = dict(encoding="cp1252")
 
     columns = {
         "geometry": "geometry",

fiboa_cli/datasets/de_th.py

@@ -8,6 +8,8 @@
 
 class Converter(AdminConverterMixin, FiboaBaseConverter):
     sources = "https://www.geoproxy.geoportal-th.de/download-service/opendata/agrar/DGK_Thue.zip"
+    # https://www.geoproxy.geoportal-th.de/inspire-dl/
+    # http://www.geoproxy.geoportal-th.de/inspire-dl/atom/DataSet/DataSet_06cd3e2f-ed4a-4507-b5e7-14973d4d6968.xml
 
     id = "de_th"
     admin_subdivision_code = "TH"

fiboa_cli/datasets/ee.py

@@ -14,10 +14,11 @@
 
 
 class Convert(AddHCATMixin, FiboaBaseConverter):
+    # explicit cache names: the WFS URL has no usable file name
     variants = {
-        str(
-            year
-        ): f"https://kls.pria.ee/geoserver/inspire_gsaa/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=inspire_gsaa:LU.GSAA.AGRICULTURAL_PARCELS_{year}&propertyName={ATTRIBUTES}"
+        str(year): {
+            f"https://kls.pria.ee/geoserver/inspire_gsaa/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=inspire_gsaa:LU.GSAA.AGRICULTURAL_PARCELS_{year}&propertyName={ATTRIBUTES}": f"ee_gsaa_{year}.gml"
+        }
         for year in range(2024, 2009, -1)
     }
     ec_mapping_csv = "https://fiboa.org/code/ee/ee.csv"

fiboa_cli/datasets/es_cat.py

@@ -9,7 +9,7 @@ class ESCatConverter(FiboaBaseConverter):
     variants = {
         "2024": {
             "https://analisi.transparenciacatalunya.cat/api/views/yh94-j2n9/files/d90f5fca-ddd8-405d-a0d5-90609985e98e?download=true&filename=Cultius_DUN2024_SHP.zip": [
-                "Cultius_DUN2024_GPKG/CULTIUS_DUN2024.gpkg"
+                "Cultius_DUN2024_SHP/Cultius_DUN2024_SHP.shp"
             ]
         },
         "2023": {
@@ -66,7 +66,7 @@ def layer_filter(self, layer, uri):
 
     def migrate(self, gdf):
         # In 2023 gpkg, names are lowercase. But in 2022 shapefile, case is mixed
-        to_lower = {k: k.lower() for k in gdf.columns if k != k.lower}
+        to_lower = {k: k.lower() for k in gdf.columns if k != k.lower()}
         if to_lower:
             gdf.rename(columns=to_lower, inplace=True)
 

fiboa_cli/datasets/es_md.py

@@ -4,7 +4,7 @@
 class ESCLConverter(ESBaseConverter):
     sources = {
         "https://idem.comunidad.madrid/recursos_cat_geo/Catalogo/recursos/UsoDelSuelo/spacm_sigpac.cm.zip": [
-            "2024_SIGPAC_shape_toda_la_com/RECINTO.shp"
+            "**/RECINTO.shp"
         ]
     }
     id = "es_md"

fiboa_cli/datasets/nl.py

@@ -4,13 +4,15 @@
 from ..conversion.fiboa_converter import FiboaBaseConverter
 from .commons.hcat import AddHCATMixin
 
-# see https://service.pdok.nl/rvo/brpgewaspercelen/atom/v1_0/basisregistratie_gewaspercelen_brp.xml
-base = "https://service.pdok.nl/rvo/brpgewaspercelen/atom/v1_0/downloads"
+# see https://service.pdok.nl/rvo/gewaspercelen/atom/basisregistratie_gewaspercelen_brp.xml
+# (the old feed rvo/brpgewaspercelen/atom/v1_0/ redirects here since 2026)
+base = "https://service.pdok.nl/rvo/gewaspercelen/atom/downloads"
 
 
 class NLCropConverter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
     area_calculate_missing = True
     variants = {
+        "2026": f"{base}/gewaspercelen_concept_2026.gpkg",
         **{str(y): f"{base}/brpgewaspercelen_definitief_{y}.gpkg" for y in range(2025, 2020, -1)},
         **{str(y): f"{base}/brpgewaspercelen_definitief_{y}.zip" for y in range(2020, 2009, -1)},
     }
@@ -27,9 +29,9 @@ class NLCropConverter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
 A dataset is generated for each year with reference date May 15.
 A view service and a download service are available for the most recent BRP crop plots.
 
-<https://service.pdok.nl/rvo/brpgewaspercelen/atom/v1_0/index.xml>
+<https://service.pdok.nl/rvo/gewaspercelen/atom/index.xml>
 
-Data is currently available for the years 2009 to 2024.
+Data is currently available for the years 2009 to 2025 (final) and 2026 (concept).
     """
 
     provider = (

fiboa_cli/datasets/sk.py

@@ -27,16 +27,20 @@ class Converter(AdminConverterMixin, AddHCATMixin, FiboaBaseConverter):
     provider = "Pôdohospodárska platobná agentúra <https://www.apa.sk>"
     license = "CC0-1.0"  # "Open Data"
     ec_mapping_csv = "https://fiboa.org/code/sk/sk.csv"
+    # KODKD is the LPIS block code, shared by several fields and sometimes empty;
+    # the row index is the field id and the code is kept as block_id.
+    index_as_id = True
     columns = {
         "geometry": "geometry",
-        "KODKD": "id",
+        "KODKD": "block_id",
         "PLODINA": "crop:name",
         "KULTURA_NA": "crop_group",
         "LOKALITA_N": "municipality",
         "VYMERA": "metrics:area",
     }
     missing_schemas = {
         "properties": {
+            "block_id": {"type": "string"},
             "crop_group": {"type": "string"},
             "municipality": {"type": "string"},
         }