Annotation + calibration interface v0.1 (draft for partner review)

[zhiganov]

Draft spec for the shared vocabulary that lets participating platforms exchange the eval commons' objects, so that labels collected on one platform's data can calibrate a judge anywhere, and a published number is auditable from its artifacts.

Open as a draft deliberately — this is for comment, not merge. It should not land until Crown Shy has reviewed the scope seam and we know whether CIP already runs annotation infrastructure (see below).

What's here

annotation-interface/ — six object schemas, a six-endpoint service API, and two coordination notes.

Object	What it is
Rubric	A facilitator-authored criterion, before any machine encoding. The intake object.
SessionOutput	A real session in canonical form: weval-compatible messages + an explicit target (final_turn / transcript) + scrub attestation + consent posture.
AnnotationPack	A labelling assignment: records + criteria, with the date range and seed needed to reproduce the record set.
Labels	One labeller's verdicts, carrying labeller identity and expertise class.
JudgeVerdicts	A judge's scores over the same records, keyed by a deterministic judge fingerprint.
CalibrationReport	TPR/TNR per axis per split, bias-corrected rates, inter-labeller agreement — computed from stored artifacts, never self-reported.

Two things worth arguing about

1. Facilitators never author machine-readable rubrics. Per the scoping call of 2026-08-04, facilitators write criteria in whatever form suits them and a platform maintainer transcribes them, with the author confirming the transcription. Rubric.provenance.confirmed is the gate: an unconfirmed rubric can be used for internal iteration, but a calibration claim resting on one is a claim about the transcriber's judgment, not the facilitator's. Please check whether that gate is in the right place.

2. This does not compete with weval. Weval owns eval specs and machine judging — blueprint grammar, multi-judge consensus, judge-vs-judge Krippendorff's α. It has no human-annotation layer, no judge-vs-human validity reporting, and no notion of ingesting a real production session. This adds exactly those three, and reuses weval at three seams: criteria travel in weval point form so the human and the judge rate the same words, the session form extends weval messages, and α is reused human-vs-human. Full boundary analysis in docs/cip-weval-alignment.md.

Relationship to calibration/

The manual flow already running in calibration/ (markdown packs, labels by PR) becomes the L0 conformance profile — same loop, file-shaped. It stays valid; a platform can participate without any service at all. L1 adds the service API.

Open questions for reviewers

1. Crown Shy — does the messages + target shape fit RAG-based knowledge-encountering sessions, and is the proposed per-record retrieved_context attachment the right seam? Five specific questions in docs/crown-shy-scope.md.
2. CIP — does Weval already have infrastructure for collecting and managing human expert labels? The design assumes OFL hosts a service; if that already exists we would rather plug into it than duplicate it. This blocks any implementation work. Question posed in docs/cip-weval-alignment.md.
3. Verdict scale — binary pass/fail with an optional graded score, or weval's 5-point scale as the primary?
4. The cross_session unit. Criteria about whole-process outcomes across all participants can be authored (Rubric.applies_to.unit) but have no SessionOutput representation yet. That gap needs the outcome workstream's scoping first.

Not in this PR

Hosting and operation of any service (a governance question), the reference implementation, and the labelling UI. Schemas first, deliberately.