Reorganize into a multi-domain toolkit (geo, storage, weather)

.github/workflows/ci.yml

@@ -0,0 +1,38 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Create virtual environment
+        run: uv venv --python ${{ matrix.python-version }} .venv
+
+      - name: Install project with dev extras
+        run: uv pip install --python .venv -e ".[dev]"
+
+      - name: Lint (ruff)
+        run: .venv/bin/ruff check .
+
+      - name: Run tests
+        run: .venv/bin/pytest -q

README.md

@@ -1,6 +1,90 @@
 # simlab-tools
-Research toolchain for SimLab.
 
-Main functionalities are listed below, grouped by topic.
+Shared research tooling for SIM lab projects: loading weather data, geodata, and
+transferring files to and from object storage. The package is organised into
+topical subpackages, each keeping its heavy dependencies behind an optional
+extra so consumers install only what they use.
 
-## Online Storage (SwitchDrive)
+## Installation
+
+Install straight from GitHub, selecting the extras you need:
+
+```bash
+# Object storage only
+uv pip install "simlab-tools[storage] @ git+https://github.com/simlab-vs/simlab-tools.git"
+
+# Geodata utilities
+uv pip install "simlab-tools[geo] @ git+https://github.com/simlab-vs/simlab-tools.git"
+
+# Everything
+uv pip install "simlab-tools[all] @ git+https://github.com/simlab-vs/simlab-tools.git"
+```
+
+Available extras: `storage`, `geo`, `smoothing`, `weather`, `all`, and `dev`
+(the full toolkit plus test/lint tooling).
+
+## Subpackages
+
+### `simlab_tools.storage` — object storage (extra: `storage`)
+
+Transfer files to and from S3-compatible buckets (e.g. SwitchCloud) with
+progress bars and multipart transfers.
+
+```python
+from simlab_tools.storage import get_s3_client, download_files_from_bucket
+
+client = get_s3_client("https://zhw-a.s3.cloud.switch.ch")  # reads S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY
+download_files_from_bucket(client, "ofen", "dejection_cones_dem", "data/", file_extensions=[".tif"])
+```
+
+`get_s3_client` accepts explicit `key_id` / `key_secret` arguments, or falls back
+to the `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` environment variables.
+
+For backwards compatibility, the storage functions are also importable from the
+top-level package: `from simlab_tools import get_s3_client`.
+
+### `simlab_tools.geo` — geodata (extra: `geo`)
+
+- `raster`: `load_and_merge_rasters`, `export_multiband_geotiff`,
+  `convert_raster_to_xarray` — load/merge raster tiles, write GeoTIFFs, wrap
+  arrays as rioxarray `DataArray`s.
+- `terrain`: `compute_slope_components`, `create_geometry_mask` — DEM slope
+  gradients and polygon rasterisation.
+- `swisstopo`: `Point`, `BoundingBox`, `query_layer`, `query_layer_from_tiles`,
+  `merge_river_segments` — a client for the swisstopo / geo.admin.ch REST API.
+
+```python
+from simlab_tools.geo import load_and_merge_rasters, compute_slope_components
+
+mosaic, x, y, transform, crs = load_and_merge_rasters(["tile_a.tif", "tile_b.tif"])
+slopes = compute_slope_components(mosaic, pixel_size=0.5)
+```
+
+### `simlab_tools.smoothing` — 1D smoothers (extra: `smoothing`)
+
+Interchangeable 1D curve smoothers behind a common `smooth(x, y)` protocol:
+`MovingAverageSmoother`, `SplineSmoother`, `SavGolSmoother`, `APLRSmoother`, and
+a `get_smoother(method)` factory.
+
+### `simlab_tools.logging` — context-aware logging
+
+`configure_logging`, `set_site_context` and `SiteFormatter` attach a per-context
+key (e.g. a site or job id) to every log record, keeping logs attributable across
+concurrent workers.
+
+### `simlab_tools.weather` — weather data (planned)
+
+Placeholder for provider-agnostic loaders of weather observations and forecasts.
+The public surface is documented in `simlab_tools/weather/__init__.py`;
+implementations are tracked as future work.
+
+## Development
+
+```bash
+uv pip install -e ".[dev]"
+ruff check .
+pytest
+```
+
+Continuous integration (ruff + pytest on Python 3.11 and 3.12) runs on every push
+and pull request via GitHub Actions.

examples/s3_demo.ipynb

@@ -24,9 +24,9 @@
     "from pathlib import Path\n",
     "\n",
     "from simlab_tools import (\n",
+    "    download_files_from_bucket,\n",
     "    get_s3_client,\n",
     "    upload_files_to_bucket,\n",
-    "    download_files_from_bucket,\n",
     ")"
    ]
   },

pyproject.toml

@@ -2,25 +2,67 @@
 name = "simlab-tools"
 version = "0.1.0"
 description = "Shared utilities for the SIM lab"
+readme = "README.md"
 requires-python = ">=3.11"
-dependencies = [
+dependencies = []
+
+[project.optional-dependencies]
+# Transfer files to/from S3-compatible object storage.
+storage = [
     "boto3>=1.35.0",
-    "moto[all]>=5.1.20",
     "mypy-boto3-s3>=1.35.0",
-    "pytest>=9.0.2",
     "tqdm>=4.66.0",
 ]
-
-[project.optional-dependencies]
-dev = ["pytest>=8.0.0"]
-
-[tool.pytest.ini_options]
-testpaths = ["tests"]
-python_files = "test_*.py"
+# Raster handling, terrain analysis and geodata API clients.
+geo = [
+    "numpy>=1.26",
+    "affine>=2.4",
+    "rasterio>=1.3",
+    "rioxarray>=0.15",
+    "xarray>=2024.1",
+    "pandas>=2.0",
+    "geopandas>=0.14",
+    "shapely>=2.0",
+    "requests>=2.32",
+]
+# 1D signal/curve smoothers.
+smoothing = [
+    "numpy>=1.26",
+    "scipy>=1.11",
+    "aplr>=10.0",
+]
+# Weather observations and forecasts (planned; see simlab_tools.weather).
+weather = []
+# Everything above, for consumers that want the full toolkit.
+all = [
+    "simlab-tools[storage,geo,smoothing,weather]",
+]
+# Development / CI: full toolkit plus test and lint tooling.
+dev = [
+    "simlab-tools[all]",
+    "pytest>=8.0.0",
+    "moto[all]>=5.1.20",
+    "ruff>=0.6.0",
+]
 
 [build-system]
 requires = ["hatchling"]
 build-backend = "hatchling.build"
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/simlab_tools"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = "test_*.py"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "D"]
+
+[tool.ruff.lint.per-file-ignores]
+# Allow undocumented tests and empty package docstrings.
+"tests/*" = ["D"]
+"**/__init__.py" = ["D104"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "numpy"

src/simlab_tools/__init__.py

@@ -1,21 +1,45 @@
-"""Shared utilities for the SIM lab."""
-
-from simlab_tools.s3 import (
-    download_file,
-    download_files_from_bucket,
-    download_specific_files_from_bucket,
-    get_s3_client,
-    list_bucket_contents,
-    upload_file,
-    upload_files_to_bucket,
+"""Shared utilities for the SIM lab.
+
+Organised into topical subpackages:
+
+- :mod:`simlab_tools.storage` -- transfer files to/from S3-compatible buckets.
+- :mod:`simlab_tools.geo` -- raster handling, terrain analysis and geodata APIs.
+- :mod:`simlab_tools.weather` -- weather observations and forecasts (planned).
+- :mod:`simlab_tools.logging` -- context-aware logging configuration.
+- :mod:`simlab_tools.smoothing` -- 1D signal/curve smoothers.
+
+Each subpackage keeps its heavy dependencies behind an optional extra
+(``storage``, ``geo``, ``smoothing``), so consumers only install what they use.
+"""
+
+__version__ = "0.1.0"
+
+# Names re-exported from :mod:`simlab_tools.storage` for backwards compatibility
+# with ``from simlab_tools import get_s3_client``. Resolved lazily so that
+# importing the top-level package does not require the ``storage`` extra.
+_STORAGE_EXPORTS = frozenset(
+    {
+        "get_s3_client",
+        "list_bucket_contents",
+        "upload_file",
+        "upload_files_to_bucket",
+        "download_file",
+        "download_files_from_bucket",
+        "download_specific_files_from_bucket",
+    }
 )
 
-__all__ = [
-    "get_s3_client",
-    "list_bucket_contents",
-    "upload_file",
-    "upload_files_to_bucket",
-    "download_file",
-    "download_files_from_bucket",
-    "download_specific_files_from_bucket",
-]
+__all__ = ["__version__", *sorted(_STORAGE_EXPORTS)]
+
+
+def __getattr__(name: str):
+    """Lazily resolve backwards-compatible storage exports (PEP 562)."""
+    if name in _STORAGE_EXPORTS:
+        from simlab_tools import storage
+
+        return getattr(storage, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+def __dir__() -> list[str]:
+    return sorted(__all__)

src/simlab_tools/geo/__init__.py

@@ -0,0 +1,43 @@
+"""Geodata utilities: raster handling, terrain analysis and geodata APIs.
+
+Submodules
+----------
+- ``raster``: load, merge, export and wrap raster tiles (rasterio / rioxarray).
+- ``terrain``: DEM slope components and geometry masks.
+- ``swisstopo``: client for the swisstopo / geo.admin.ch REST API.
+- ``types``: shared file-path type aliases.
+"""
+
+from simlab_tools.geo.raster import (
+    convert_raster_to_xarray,
+    export_multiband_geotiff,
+    load_and_merge_rasters,
+)
+from simlab_tools.geo.swisstopo import (
+    BoundingBox,
+    Point,
+    merge_river_segments,
+    query_layer,
+    query_layer_from_tiles,
+)
+from simlab_tools.geo.terrain import compute_slope_components, create_geometry_mask
+from simlab_tools.geo.types import FileName, FileNames
+
+__all__ = [
+    # raster
+    "load_and_merge_rasters",
+    "export_multiband_geotiff",
+    "convert_raster_to_xarray",
+    # terrain
+    "compute_slope_components",
+    "create_geometry_mask",
+    # swisstopo
+    "Point",
+    "BoundingBox",
+    "query_layer",
+    "query_layer_from_tiles",
+    "merge_river_segments",
+    # types
+    "FileName",
+    "FileNames",
+]