P0: Strict two-map/two-head open-vocabulary segmentation foundation

.github/workflows/ovs-heritage-p0.yml

@@ -0,0 +1,24 @@
+name: ovs-heritage-p0
+
+on:
+  pull_request:
+    paths:
+      - "ovs_heritage/**"
+      - ".github/workflows/ovs-heritage-p0.yml"
+      - "environment.yml"
+  push:
+    branches: ["codex/-p0"]
+
+jobs:
+  cpu-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.9"
+      - run: python -m pip install --upgrade pip
+      - run: python -m pip install --extra-index-url https://download.pytorch.org/whl/cpu torch==1.12.1 numpy==1.26.4 pillow==10.4.0 pyyaml==6.0.2 pytest==8.3.3 ruff==0.6.9
+      - run: python -m compileall -q ovs_heritage
+      - run: pytest -q ovs_heritage/tests
+      - run: ruff check ovs_heritage

environment.yml

@@ -214,6 +214,8 @@ dependencies:
       - python-dotenv==1.0.1
       - python-json-logger==2.0.7
       - pytz==2023.4
+      - pytest==8.3.3
+      - ruff==0.6.9
       - pyyaml==6.0.2
       - pyzmq==26.2.0
       - referencing==0.35.1

ovs_heritage/AUDIT.md

@@ -0,0 +1,71 @@
+# P0 audit of the current working tree
+
+This document records code observations, not assumptions. Legacy results and paths are not changed.
+
+## 1. Actual fine-tuning path
+
+**Confirmed:** `tools/finetune.py:FineTuneWrapper.forward` selects `base_model.clip_backbone` when present, calls that MaskCLIP object with `return_feat=True`, then directly calls `decode_head.cls_seg`. It does not call `LPOSS.forward`. `tools/finetune_tiled.py` imports this wrapper as `common.FineTuneWrapper` and follows the same path. Consequently this is MaskCLIP-branch fine-tuning, not full-LPOSS fine-tuning. DINO graph refinement is absent from training and from `evaluate_stitched`; the latter stitches wrapper logits. The full LPOSS path exists separately in `models/lposs/lposs.py:LPOSS.forward`.
+
+`configure_trainable_layers` first freezes the whole model, unfreezes the requested suffix (`depth=-1` means all) of CLIP visual transformer blocks, and unfreezes the entire decode head. This includes `decode_head.proj`; the text prototypes are buffers, not parameters. Mixers are separate trainable modules added after this configuration.
+
+## 2. Imbalance versus catastrophic forgetting
+
+Class imbalance changes the relative number/gradient contribution of supervised examples. Catastrophic forgetting changes foundation feature geometry: enabled visual blocks and the image projection (`decode_head.proj`) are optimized for the closed heritage labels, while text embeddings remain fixed. That can damage text/image alignment for unseen concepts. Class weighting or oversampling can rebalance heritage gradients, but does not constrain preservation of the original CLIP geometry and therefore cannot guarantee open-vocabulary retention.
+
+## 3. Softmax before cross entropy
+
+**Confirmed mathematical defect in the legacy training path.** `models/maskclip/maskclip.py:MaskClipHead.cls_seg` computes cosine convolution, multiplies by 100, then returns `F.softmax(...)`. `FineTuneWrapper` averages these probabilities and its direct second `cls_seg` result; `tools/finetune.py:compute_loss` passes that tensor to `F.cross_entropy`, which expects raw logits. Full `models/lposs/lposs.py:LPOSS.forward` does its own graph label-propagation scorer from normalized CLIP/DINO features rather than merely consuming the MaskCLIP probability map. P0 provides a new isolated raw scorer/loss; this does not retroactively repair checkpoints or historical measurements.
+
+## 4. Vocabulary-specific state
+
+`MaskClipHead.__init__` uses `register_buffer("class_embeddings", ...)`, so prototypes are persistent in `state_dict` and checkpoint shape/order depends on aliases and vocabulary. `class_mapping` is a plain tensor attribute, not a registered buffer, and is not saved. `class_names` is a plain Python value. `proj.weight` is persistent vocabulary-independent image projection state. P0 `PrototypeSet` is returned at runtime and `RawCosineScorer.state_dict()` is empty.
+
+## 5. Aliases and update_vocab
+
+Semicolon-separated labels are expanded in `_get_class_embeddings`: every alias produces another prototype and hence another `cls_seg` output channel. `class_mapping` is created but never used by `cls_seg`; `reduce_to_true_classes` in the LPOSS inferencer only collapses extra leading background expansion and is not a general alias reduction. `update_vocab` replaces embeddings but does not update `self.class_names`, does not move the newly created text model to CUDA, and does not explicitly preserve the caller device. `_embed_label` hard-codes prompts to `cuda`; constructor hard-codes `model.cuda()`. These choices permit CPU failure and device mismatch. List iteration preserves input order before alias expansion. P0 aggregates all prompt variants into exactly one prototype in exact runtime order.
+
+## 6. ADVERTISEMENTS loss risk
+
+**Confirmed.** `tools/finetune.py:_sanitize_targets` replaces every target outside `[0, num_classes)` with 255. With the legacy eleven-channel datasets, ID 11 is therefore silently ignored. P0 `validate_mask_ids`, dataset validation, and `supervised_cross_entropy` raise and enumerate invalid IDs; they never rewrite labels.
+
+## 7. Consumers of the old ontology
+
+| file | symbol/location | current assumption | evidence | required action | status |
+|---|---|---|---|---|---|
+| `tools/convert_brush_coco_to_masks.py` | `LABELS` | IDs 0..10; no advertisements | eleven literal entries | retain as historical converter; use a reviewed v2 conversion path later | legacy intentionally preserved |
+| `tools/convert_brush_coco_to_masks.py` | annotation loop | overlap is input-order “last annotation wins” | unconditional `mask[ann_mask > 0] = label_id` | document/resolve overlap policy before any migration | deferred to P1/P2 |
+| `mmseg/datasets/facades_train.py` and sibling facade datasets | `classes`, `palette` | eleven classes/channels | eleven literals | keep v1 meaning; new P0 adapter reads v2 source | legacy intentionally preserved |
+| `segmentation/configs/_base_/datasets/facades_test.py` | `classes`, `palette` | eleven classes | eleven literals | do not use with masks containing ID 11 | legacy intentionally preserved |
+| `tools/finetune.py` | metric groups | HUMAN_ACTIVITY omits advertisements | two-name set | consume v2 groups in future trainer | deferred to P1/P2 |
+| `tools/compare_models_facades.py` | groups | eleven-class evaluation | local sets | re-evaluate both models on common v2 test set | deferred to P1/P2 |
+| `tools/render_temporal_qualitative_grids.py` | defaults | eleven names/colors | literal lists | legacy figures remain reproducible | legacy intentionally preserved |
+| `models/maskclip/maskclip.py` | head outputs | channel count follows expanded strings | embedding convolution | legacy file is unchanged; separate P0 scorer supports runtime C | legacy intentionally preserved |
+| `ovs_heritage/configs/datasets/heritage_facades_v2.py` | adapter exports | twelve concepts projected to 11 main channels plus one ornament channel | values loaded from canonical source and projection | use only with explicit two-map v2 manifests | changed |
+| README temporal semantics | ontology prose | text/signage combined | explicitly says combined class | update only when downstream temporal contract migrates | legacy intentionally preserved |
+
+The legacy/base code audited before P0 had no tracked `ADVERTISEMENTS` class in its converter or eleven-class dataset configurations. This PR now contains the canonical P0 `ADVERTISEMENTS` concept at semantic ID 11 with visualization RGB `(216, 27, 96)`, while the legacy consumers listed above remain unchanged. No annotation pixels were created, moved, or converted.
+
+## 8. LPOSS inference
+
+**Confirmed syntax/semantic defect:** `segmentation/evaluation/lposs_eval.py:LPOSS_Infrencer.forward` has a duplicated conditional expression immediately after `else i`. Python parses this as attempting to call `i` (often a Tensor) with the following parenthesized result. P0.1 removed the earlier AST test because it passed only while the bug remained and therefore encoded the defect as expected behavior. The safe legacy fix and a regression test that exercises successful inference belong with the P1 LPOSS wrapper/integration work. A later wrapper must distinguish DINO graph refinement (feature graph propagation in LPOSS), LPOSS+ pixel refinement (`pixel_refine`, CuPy Laplacian), and fallback: without CuPy pixel refinement is explicitly skipped; FAISS/CUDA availability affects graph implementations and is not equivalent to LPOSS+.
+
+`LPOSS_Infrencer.encode_decode` currently calls `self.model(img)` and does not reference an undefined `x`; the earlier audit statement claiming otherwise was incorrect and has been removed. The duplicated conditional-expression defect above remains separately verified. Any LPOSS fix and positive integration regression test remain outside P0.
+
+## 9. Historical metrics
+
+Values such as mIoU 0.0551→0.1676 or DAMAGE_MACRO_MIOU 0.0209→0.0802, wherever retained as experiment references, are not P0 results. Legacy single-mask metrics and v2 two-map metrics are not directly comparable. Stock and adapted models must be evaluated again on the identical 12-concept two-map test set. Future reports must distinguish `stock_repo_exact` from `stock_shared_scorer` (stock dense features with the P0 scorer).
+
+## 10. P0 two-map correction
+
+The current P0 representation supersedes the earlier single-raster v2 assumption.
+Canonical concept 8 is now `ornament_region`: visible decorative geometry in an
+independent binary target, not a mutually exclusive “intact ornament” class.
+Legacy converters and v1 masks flatten overlaps and cannot be losslessly
+reinterpreted without original overlapping annotations. The new projection,
+losses, and validator do not modify legacy files or implement a two-head model.
+
+The repository has a temporal semantic backend registry, but no shared neutral
+experiment-metadata record suitable for segmentation validation/loss settings.
+P0 therefore exposes a small deterministic JSON-serializable record only; it
+does not duplicate the backend registry or implement the future experiment
+ledger. LPOSS model integration remains outside P0.

ovs_heritage/README.md

@@ -0,0 +1,123 @@
+# Heritage open-vocabulary foundations (P0)
+
+P0 defines target representation, ontology projection, runtime prototypes, raw
+scoring, strict losses, validation, deterministic metadata, and CPU tests. It
+does **not** implement the complete two-head model, LPOSS integration, adapter
+training, retention evaluation, a general registry, or an experiment ledger;
+those integrations belong to P1 or later.
+
+## V2: 12 concepts, two target maps
+
+`heritage_facades_v2_12concepts_two_heads` has stable semantic IDs 0–11, but
+these are not 12 mutually exclusive model channels. Stored targets are:
+
+* `Y_main`: `{0,1,2,3,4,5,6,7,9,10,11,255}`. Values remain semantic IDs on
+  disk. Concept 8 is invalid here.
+* `Y_ornament`: `{0,1,255}`. Positive `1` means visible decorative geometry
+  (`ornament_region`, semantic concept 8); the binary mask does not store 8.
+
+`ornament_region` is a pixel-level visible-geometry label independent of damage
+or surface condition. Thus corrosion (`Y_main=7`) or water stain (`Y_main=5`)
+may overlap `Y_ornament=1`. A completely missing ornament has `Y_ornament=0`;
+its absence can be `missing_element` in `Y_main`. This mask never represents an
+expected/reconstructed historical footprint.
+
+The legacy name `ornament_intact` was misleading. It is only a deprecated alias
+requiring explicit resolution. It does not trigger dataset migration. A legacy
+flattened raster cannot reveal the hidden main label under an ornament region;
+without original Label Studio/COCO annotations that overlap cannot be recovered
+losslessly and must not be invented.
+
+Legacy `heritage_facades_v1_11classes` remains an explicit, separate single-mask
+schema. V1 masks are never automatically interpreted as v2.
+
+## Canonical output projection versus runtime vocabulary
+
+Canonical future supervision expects raw `main_logits [N,11,H,W]` and raw
+`ornament_logits [N,1,H,W]`. Main semantic IDs map as follows:
+
+```text
+semantic ID:  0 1 2 3 4 5 6 7 9 10 11
+main channel: 0 1 2 3 4 5 6 7 8  9 10
+semantic ID 8 -> ornament head, channel 0
+```
+
+`OntologyProjection` performs semantic/channel round trips and preserves 255.
+Conversion to contiguous channels occurs only at the main loss boundary;
+export converts argmax channels back to semantic IDs. Ornament inference uses
+sigmoid and a configurable threshold and remains a separate binary mask. The
+two predictions are never flattened together.
+
+The stateless open-vocabulary scorer is different: it returns raw
+`[N,C_main,H,W]` logits for an arbitrary reordered/subset/extended/mixed runtime
+vocabulary. Runtime entries may have `semantic_id=None`. Such dynamic channels
+must not be passed to the canonical supervised loss without an explicit mapping.
+Prototype metadata records channel order, nullable semantic IDs, ontology hash,
+prompt settings, and a deterministic specification hash.
+
+## Raw-logit losses
+
+`main_segmentation_loss` validates exactly 11 canonical channels, maps stored
+semantic IDs to channel indices, and sends raw logits directly to cross entropy.
+Main 255 is handled through `ignore_index`; all-ignore input returns a
+differentiable zero rather than NaN.
+
+`ornament_region_loss` sends raw logits directly to element-wise
+`binary_cross_entropy_with_logits`. It replaces ignored targets with a safe zero,
+then averages only pixels where `Y_ornament != 255`; ignored pixels contribute
+to neither numerator nor denominator. All-ignore input is a differentiable
+zero. `combined_two_head_loss` records finite non-negative `lambda_ornament` and
+optional positive `pos_weight`; P0 does not tune either value or a threshold.
+
+## Validation and reproducibility
+
+V2 manifests explicitly contain `main_mask_path`, `ornament_mask_path`, and
+`facade_id` (optionally `image_path` and `source_id`). The validator checks both
+files, shape equality, strict dtypes/IDs, empty splits, missing facade IDs,
+facade/source-ID leakage, and reused image or mask paths. Optional `image_path`
+values are opened and verified; when present, their pixel dimensions must match
+both target grids. Reports count those files as `verified_image_count` rather
+than treating every manifest row as a verified image. Missing advertisements is a warning.
+Reports distinguish manifest rows, valid and failed samples, main and ornament
+mask counts, and source counts; they do not call unchecked rows “images”.
+Unknown IDs are excluded from valid statistics.
+
+Validation also requires explicit `schema_version` and `ontology_version`
+declarations, either at the dataset-config level or through the corresponding
+CLI options. V2 uses `heritage_two_map_v2`; legacy v1 uses
+`heritage_single_mask_v1`. Missing, unknown, conflicting, or ontology-mismatched
+declarations are errors and never trigger schema inference. Per-split reports
+separate manifest rows, valid/failed sample rows, and inventory-level split
+errors; error-message count is not used as a sample count.
+
+Legacy-v1 directory inventories remain supported. Their deterministic
+fingerprint uses root-relative canonical paths plus each mask's content hash, so
+content changes alter the fingerprint while moving an unchanged inventory does
+not. Relative directory paths resolve files exactly once.
+
+Reports include the component/schema versions, ontology version/hash, complete
+projection, split fingerprints, overlaps, duplicated paths, warnings/errors,
+and a deterministic neutral metadata record. Hashed payloads contain no current
+time. `MetadataRecord` is the intended adapter point for a future shared ledger;
+no competing registry or provenance JSONL is introduced.
+
+```bash
+python -m ovs_heritage.validate_dataset \
+  --ontology ovs_heritage/configs/heritage_vocab.yaml \
+  --schema-version heritage_two_map_v2 \
+  --ontology-version heritage_facades_v2_12concepts_two_heads \
+  --train train.csv --val val.csv --test test.csv \
+  --output validation-report.json --strict
+```
+
+## Checks
+
+```bash
+python -m compileall -q ovs_heritage
+pytest -q ovs_heritage/tests
+ruff check ovs_heritage
+```
+
+The CPU GitHub Actions workflow runs these commands on Python 3.9 with versions
+compatible with the project environment. No model or dataset download occurs in
+the unit tests.

ovs_heritage/__init__.py

@@ -0,0 +1,5 @@
+"""P0 foundations for open-vocabulary heritage-facade segmentation."""
+
+from .ontology import IGNORE_INDEX, Ontology, load_ontology
+
+__all__ = ["IGNORE_INDEX", "Ontology", "load_ontology"]

ovs_heritage/configs/datasets/README.md

@@ -0,0 +1,8 @@
+# Dataset schemas
+
+`heritage_facades_v2.py` exports metadata for the v2 two-map contract: stored
+`Y_main` uses stable semantic IDs while stored `Y_ornament` uses binary
+`0/1/255`. It defines 11 canonical main channels and one independent ornament
+channel, but does not implement a model. Existing MMSeg facade configs remain
+explicit legacy-v1 single-mask configurations and must not be used to infer or
+migrate v2 overlaps.

ovs_heritage/configs/datasets/heritage_facades_v2.py

@@ -0,0 +1,20 @@
+"""Metadata adapter for the v2 two-target representation; not a model config."""
+
+from ovs_heritage.ontology import load_ontology
+from ovs_heritage.projection import OntologyProjection
+
+_ONTOLOGY = load_ontology()
+_PROJECTION = OntologyProjection.from_ontology(_ONTOLOGY)
+ONTOLOGY_VERSION = _ONTOLOGY.version
+ONTOLOGY_HASH = _ONTOLOGY.hash
+DATASET_SCHEMA_VERSION = "heritage_two_map_v2"
+SEMANTIC_CONCEPTS = _ONTOLOGY.display_names
+PALETTE = _ONTOLOGY.palette
+MAIN_SEMANTIC_IDS = tuple(entry.semantic_id for entry in _PROJECTION.main_entries)
+MAIN_CLASSES = tuple(entry.canonical_name for entry in _PROJECTION.main_entries)
+MAIN_NUM_CHANNELS = _PROJECTION.main_channel_count
+ORNAMENT_SEMANTIC_ID = 8
+ORNAMENT_NUM_CHANNELS = 1
+OUTPUT_MAPPING = _PROJECTION.as_dict()
+EVALUATION_GROUPS = _ONTOLOGY.groups
+IGNORE_INDEX = _ONTOLOGY.ignore_index