Proposal: evalport-sdk adapter for COMET's Prediction output

[adhabnr-ux]

Hi, I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open interchange spec + SDK (evalport-sdk on PyPI) for portable LLM/MT evaluation results — a common JSON schema (checked by openeval.validate.validate_result_set()) so results produced by different eval tools can be compared or archived outside each tool's own format.

COMET's model.predict(...) already returns exactly the kind of structured object EvalPort adapters are built around, not a bare list of floats. From comet/models/base.py / comet/models/utils.py, CometModel.predict() returns a Prediction (a ModelOutput, i.e. an ordered-dict-like object), and per the README this carries:

• Prediction.scores — segment-level scores (list, one per input sample)
• Prediction.system_score — the aggregated system-level score
• Prediction.metadata — itself a Prediction, e.g. metadata.error_spans for XCOMET (list of per-segment error-span dicts with start/end/severity/confidence/text), or metadata.mcd_scores/metadata.mcd_std when MC-Dropout is enabled

That's a real per-segment + system-level + rich-metadata result, which maps cleanly onto an EvalPort result set: each input sample → one item result (segment score + any error spans/uncertainty metadata attached), and system_score → the run-level aggregate.

Concretely, an adapter would look like:

# comet_openeval_adapter/__init__.py
from comet.models.utils import Prediction

def results_to_openeval(output: Prediction, samples: list[dict]) -> dict:
    """Convert a COMET Prediction into an EvalPort result-set dict."""
    error_spans = getattr(output.metadata, "error_spans", None) if "metadata" in output else None
    return {
        "results": [
            {
                "item_id": str(i),
                "input": samples[i],
                "score": score,
                "metadata": {"error_spans": error_spans[i]} if error_spans else {},
            }
            for i, score in enumerate(output.scores)
        ],
        "aggregates": [{"metric": "comet_system_score", "mean": output.system_score}],
    }

from openeval.validate import validate_result_set
validate_result_set(results_to_openeval(model_output, data))

Before/after: today model_output is a COMET-specific Prediction object usable only within Python code that imports comet; with an adapter it becomes a spec-validated, portable JSON result set consumable by any EvalPort-aware tool (dashboards, cross-metric comparisons against BLEU/chrF/other MT metrics, archival), without those tools needing to import COMET or pytorch-lightning.

I'd like to build this as comet-openeval-adapter/ following EvalPort's standard adapter layout (standalone installable package depending on unbabel-comet + evalport-sdk, to_openeval()/results_to_openeval(), tests, README) and submit it as a PR — either against adapters/ in this repo or against EvalPort's own adapters/ directory, whichever you'd prefer. Zero footprint on COMET itself. Happy to also handle the non-XCOMET case (no error_spans) and the MC-Dropout metadata shape if that's useful to include.

Let me know if this is of interest.

— independent contributor, not affiliated with Unbabel