FinaliseableDatasetAdapterMixin.finalise_datasets writes parsed values straight into the catalogue frame:
datasets.at[idx, key] = value # mixins.py:97
If the column was inferred as float64 and the file stores that attribute as a string, pandas raises rather than coercing:
TypeError: Invalid value '60225' for dtype 'float64'
Trigger
CMIP6 data from KIOST stores branch_time_in_child and branch_time_in_parent as strings. In a DRS-ingested catalogue both columns come back float64, so finalising any KIOST dataset raises.
next institution KIOST rows 32
MISMATCH column= branch_time_in_child dtype= float64 parsed value= '60225'
MISMATCH column= branch_time_in_parent dtype= float64 parsed value= '60225'
Why it matters
The exception propagates out of DataCatalog.finalise and kills ref solve. On a DRS-ingested catalogue, finalisation happens inside the first solve (see #837), so 32 files from one model abort a run that had already spent hours reading. Nothing is committed, because finalisation only writes at the end of the batch, so the work is lost too.
A single non-conforming model should not be able to end a solve over 11,000 datasets.
Suggested fix
Coerce on assignment rather than trusting the inferred dtype. Either build the finalised frame column-wise from the parsed records and let pandas infer once, or upcast the target column to object when the incoming value does not fit. Failing that, catch per-dataset and mark it invalid rather than propagating, so one bad model is skipped rather than fatal.
Seen on climate-ref v0.16.2.
FinaliseableDatasetAdapterMixin.finalise_datasetswrites parsed values straight into the catalogue frame:If the column was inferred as
float64and the file stores that attribute as a string, pandas raises rather than coercing:Trigger
CMIP6 data from KIOST stores
branch_time_in_childandbranch_time_in_parentas strings. In a DRS-ingested catalogue both columns come backfloat64, so finalising any KIOST dataset raises.Why it matters
The exception propagates out of
DataCatalog.finaliseand killsref solve. On a DRS-ingested catalogue, finalisation happens inside the first solve (see #837), so 32 files from one model abort a run that had already spent hours reading. Nothing is committed, because finalisation only writes at the end of the batch, so the work is lost too.A single non-conforming model should not be able to end a solve over 11,000 datasets.
Suggested fix
Coerce on assignment rather than trusting the inferred dtype. Either build the finalised frame column-wise from the parsed records and let pandas infer once, or upcast the target column to
objectwhen the incoming value does not fit. Failing that, catch per-dataset and mark it invalid rather than propagating, so one bad model is skipped rather than fatal.Seen on climate-ref v0.16.2.