Draft: lighteval converter (for #189)

[Mandark-droid]

Draft for #189, which is assigned to @gbemike and stays theirs. They said they'd
been delayed by a PC problem and were happy to build on a draft, so this is that
draft — a starting point to take, cannibalise or bin. Not asking for
reassignment, and not asking for this to be merged over their work.

Adds every_eval_ever/converters/lighteval/, following the lm_eval layout
(__init__ / __main__ / adapter / utils), plus convert lighteval in the
top-level CLI and an offline fixture test. Pure JSON parsing, so lighteval isn't
a dependency and nothing needs guarding in the core matrix.

uv run every_eval_ever convert lighteval --log_path tests/data/lighteval

The question I'd like an opinion on

lighteval's results (MetricsLogger.metric_aggregated) mixes rows it measured
with rows it averaged. aggregate() appends a per-parent mean under
<parent>:_average|<fewshot> and a mean over everything under the literal key
all, into the same dict as the real tasks. Emit every key and you publish an
aggregate and its own parts as siblings.

I skip the averaged rows and record which keys were skipped in
source_metadata.additional_details.lighteval_derived_rows_not_converted, so the
omission is visible rather than silent. Reasons:

• They're unweighted means, and all averages across every task-metric pair
  regardless of whether those are comparable.
• aggregate() builds them from metric_aggregated[subtask].keys(), which
  includes the _stderr entries. So the averaged rows carry a mean of standard
  errors, which isn't the standard error of the mean. Emitting that as
  uncertainty would publish a number that is simply wrong.
• They have no entry in config_tasks, so there's no source_data for them, and
  publish_evaluation_logs needs one.
• A consumer can always recompute a mean. They can't un-mix a double-counted one.

The counter-argument is that all is what people quote from a lighteval run, and
dropping it loses the headline number. If you'd rather have them with a marker on
the record, that's a small change and I'd rather you decide it than me.

Other calls worth a look

stderr. MetricsLogger writes {metric}_stderr into the same dict as the
metric, so a naive loop over results[task] yields acc_stderr as its own
metric. It's attached as score_details.uncertainty.standard_error instead. It's
set to float("nan") when the estimate overflows, and lighteval dumps that as a
bare NaN token; that reads as "no uncertainty reported", so the field is
omitted rather than written as 0.0. There's a fixture and a test for exactly
this.

stderr method. Not always bootstrap — get_stderr_function returns the
analytic mean_stderr when the aggregation's name contains mean. The converter
applies the same rule against the corpus_level_fn recorded in config_tasks,
and omits method when the file doesn't say. Labelling everything bootstrap
would have been wrong about half the time.

Timestamps. config_general.start_time and end_time are
time.perf_counter() values despite the docstring calling them Unix timestamps,
so they can't be converted to an instant. The only wall clock in a run is the
date_id in results_{date_id}.json (datetime.now().isoformat() with :
swapped for -). I parse that back to ISO-8601 and use it. Note this differs
from the lm_eval and inspect converters, which emit a Unix epoch — going to
epoch here would mean inventing a timezone for a naive local time, so I left it
as the string the source actually recorded. Happy to switch if you'd rather have
consistency across converters.

total_evaluation_time_secondes really is spelled that way upstream, so the
converter reads that key; it's written out under the corrected spelling.

lighteval_sha is "?" for any pip install (it shells out to git against
its own source tree). It's recorded in eval_library.additional_details when
real and omitted when "?", and it never becomes eval_library.version — a git
SHA isn't a version.

Non-finite scores. aggregate() sets a metric to NaN on OverflowError. The
metric is skipped and counted in
source_metadata.additional_details.metrics_dropped_non_finite, because the
validator rejects a non-finite score and coercing it to 0 would invent a result.
If that takes out every metric of a task, the task produces no record at all, so
its key is listed under tasks_without_finite_scores on the logs the same file
did produce — otherwise it would disappear silently.

ModelInfo. name/id from config_general.model_name, developer from
the org prefix, and everything else stringified into additional_details. On
inference_engine: lighteval dumps model_config via pydantic model_dump(),
which carries no discriminator for the config subclass, so the backend isn't
recoverable from the file — it's a CLI flag rather than a guess.
inference_platform is read from model_config.provider when present (LiteLLM
and inference-provider runs state it outright), CLI flag otherwise.

One thing I'd flag upstream too: LiteLLMModelConfig.api_key is a plain
str, and model_dump() puts it in the results file. Anyone who passed a key in
--model-args has it sitting in results_*.json in cleartext. The converter
drops credential-bearing keys before they reach additional_details and records
which key names it dropped. Worth knowing about regardless of what happens to
this PR.

Fixture

Written from the upstream writers at v0.13.0
(logging/evaluation_tracker.py, logging/info_loggers.py), not captured from a
run — lighteval pulls torch, which I didn't want to make a prerequisite for
reading this diff. Everything in it traces to a line in those files, but it has
not been checked against a real run, and that's the thing most worth a second
pair of eyes. It contains a bare NaN token on purpose: that's what
json.dumps emits with allow_nan left at its default, which is how lighteval
calls it.

Verification

• uv run pytest tests — 31 new tests pass, no existing test changes behaviour
• uv run ruff check . — clean
• Round-tripped through the real CLI: convert lighteval then
  every_eval_ever validate, 3/3 records pass with semantic checks on

Run locally on Windows, where four tests unrelated to this change already fail on
main (path separators and CRLF). CI here is sitting at action_required — it
needs a maintainer to approve the workflow run for a first-time contributor.

Shared-file change

One line outside the new directory: SupportedLibrary gains a LIGHTEVAL
member. The alternative was returning CUSTOM, which would be false.

[mrshu]

⚒️ review-anvil report

Review decision: COMMENT — This run is comment-only; two high-priority and six medium suggestions remain.
Result: The draft covers several tricky lighteval shapes well. Stable identity and complete source accounting need attention before publication.
Scope: Add a dependency-free lighteval aggregate converter with CLI, docs, fixture, and offline tests.
Checks: 3 single-reviewer concerns checked; 1 confirmed and 2 narrowed.
Second check: Targeted, 2 reviewers; all findings kept, with deployment, count, credential, and test guidance clarified.

Earlier review comments

No earlier review comments were found.

What I noticed

ID	Priority	Topic	Code location	What I noticed
RAV-RUN1-R1-F001	high	evaluation identity	every_eval_ever/converters/lighteval/adapter.py:375	evaluation_id includes retrieval time, so identical input gets a new identity on every conversion.
RAV-RUN1-R1-F002	high	source accounting	every_eval_ever/converters/lighteval/adapter.py:524	A directory whose measured tasks all lack finite scores can exit successfully with zero records and no failure ledger.
RAV-RUN1-R1-F004	medium	score coverage	every_eval_ever/converters/lighteval/adapter.py:185	Capped runs expose original dataset size, but do not clearly preserve the smaller population that produced the score.
RAV-RUN1-R1-F005	medium	metric semantics	every_eval_ever/converters/lighteval/adapter.py:287	An undeclared result key is published with an assumed higher-is-better direction and no verified scale.
RAV-RUN1-R1-F006	medium	metric identity	every_eval_ever/converters/lighteval/adapter.py:297	Every metric omits metric_id, including global metrics such as accuracy.
RAV-RUN1-R1-F007	medium	coverage totals	every_eval_ever/converters/lighteval/adapter.py:538	total_records combines task-level output logs with file-level failures.
RAV-RUN1-R1-F008	medium	failure reporting	every_eval_ever/cli.py:234	Single-file parse or conversion failures exit before a structured failure report is saved.
RAV-RUN1-R1-F009	medium	credential filtering	every_eval_ever/converters/lighteval/utils.py:185	Supported nested model configuration such as env_vars is serialized without recursive secret filtering.

ID legend: RUN is the observed PR review run, R is the origin round, and F is a finding.

Non-blocking low-priority suggestions (2 items)

• RAV-RUN1-R1-F003 [low] deployment metadata — Supported logs normally publish deployment_type and model_availability as unknown, with no CLI override for known facts.
• RAV-RUN1-R1-F010 [low] test coverage — The thorough synthetic fixture is useful. An end-to-end publication assertion would also prove strict output parsing through the real CLI.

Things to try (10 items)

• [high] evaluation identity — Stable raw model identity, task, source timestamp, and a non-secret run-config digest can form an idempotent ID. (RAV-RUN1-R1-P001; covers RAV-RUN1-R1-F001)
• [high] source accounting — A file with measured but unconvertible tasks can become a failure; a file containing only intentional derived rows can remain an exclusion. (RAV-RUN1-R1-P002; covers RAV-RUN1-R1-F002)
• [low] deployment metadata — Validated CLI overrides could follow the existing platform and engine override pattern. (RAV-RUN1-R1-P003; covers RAV-RUN1-R1-F003)
• [medium] score coverage — Both original and effective document counts can stay visible, with the evaluated count identified as the score population. (RAV-RUN1-R1-P004; covers RAV-RUN1-R1-F004)
• [medium] metric semantics — Source metric specs or an operator definition can supply direction and scale. Unresolved keys can enter the failure ledger after valid metrics are preserved. (RAV-RUN1-R1-P005; covers RAV-RUN1-R1-F005)
• [medium] metric identity — One metric-definition resolver can provide ID, direction, type, and bounds without duplicate maps. (RAV-RUN1-R1-P006; covers RAV-RUN1-R1-F006)
• [medium] coverage totals — Results files can remain the source-record grain, with output-log count reported separately. (RAV-RUN1-R1-P007; covers RAV-RUN1-R1-F007)
• [medium] failure reporting — A small lighteval-local single-file result path can reuse directory reporting behavior before raising. (RAV-RUN1-R1-P008; covers RAV-RUN1-R1-F008)
• [medium] credential filtering — One recursive sanitizer can protect nested mappings and lists before model configuration is serialized. (RAV-RUN1-R1-P009; covers RAV-RUN1-R1-F009)
• [low] test coverage — The existing fixture can drive one CLI publication test that strictly parses and validates each emitted file. (RAV-RUN1-R1-P010; covers RAV-RUN1-R1-F010)

Run details

• Target: PR #235 at 75a03c269eb60dbaf5247ec4a36dc79af18dcbdf (11 files, +1663/-4)
• Run ordinal: 1
• Rounds: 1/1 completed; adaptive off
• Mix: 3 codex-exec
• Focus: correctness, maintainability, simplicity, production blast radius, converter content checks, and constructive suggestions
• Earlier review comments: none
• Finding counts: 0 critical, 2 high, 6 medium, 2 low, 0 nit
• Checks: concerns=3; confirmed=1; narrowed=2; set-aside=0; removed=0
• Second check: targeted; reviewers=2; kept=10; clarified=4; set-aside=0; removed=0
• Fixes applied: 0 (review-only)

---

Reviewed with review-anvil.

[mrshu]

RAV-RUN1-R1-F001 [high] evaluation-identity — evaluation_id includes the current retrieval time. The same source file therefore gets a different logical identity on every conversion, which allows duplicate evaluations on re-ingest.

A stable ID can use the raw model identity, task key, source timestamp, and a deterministic digest of non-secret run configuration. Retrieval time can remain only in retrieved_timestamp.

[mrshu]

RAV-RUN1-R1-F002 [high] source-accounting — An input file whose measured tasks all lack finite scores returns an empty list without a failure. A directory made from those files then reports zero converted logs and exits successfully, so automation cannot distinguish total conversion loss.

A file with measured but unconvertible tasks can enter the failure ledger. Files containing only intentionally skipped derived rows can stay explicit exclusions.

[mrshu]

RAV-RUN1-R1-F004 [medium] score-coverage — Capped or deduplicated runs put original_num_docs in samples_number, while uncertainty uses effective_num_docs. The dataset size is valid source context, but the record does not clearly preserve the smaller population that produced the score.

Both counts can remain visible, with the effective count identified as score coverage and the original count retained as dataset provenance.

[mrshu]

RAV-RUN1-R1-F005 [medium] metric-semantics — When a result key has no matching metric spec, the converter assumes higher-is-better and publishes no verified scale or bounds. A finite custom key does not prove that direction.

Source metric specs or an operator-supplied definition can establish the semantics. Unresolved keys can be reported after the valid metrics are preserved.

[mrshu]

RAV-RUN1-R1-F006 [medium] metric-identity — Both MetricConfig branches omit metric_id, including common metrics such as accuracy. metric_name keeps the source label but does not provide the cross-source join key.

The same metric-definition mapping used for direction and bounds can supply canonical global IDs and stable namespaced IDs for defined lighteval-specific metrics.

[mrshu]

RAV-RUN1-R1-F007 [medium] coverage-totals — total_records adds task-level output logs to file-level failures. One successful file can contribute several counts, while one failed file contributes one, so the coverage denominator has no consistent source unit.

Results files can remain the source-record grain, with converted output-log count reported separately.

[mrshu]

RAV-RUN1-R1-F008 [medium] failure-reporting — The single-file branch calls transform_from_file before it creates a SourceConversionResult. Parse or conversion errors exit non-zero, but the structured failure report is never saved; directory input reports the same failure class correctly.

A small lighteval-local file-result path can give both entry modes the same report-before-raise behavior.

[mrshu]

RAV-RUN1-R1-F009 [medium] credential-filtering — Secret filtering checks only top-level model-config keys before nested values are serialized. Supported v0.13 configurations can carry unrestricted env_vars, so a nested token or API key can reach published additional_details.

One recursive sanitizer for mappings and lists can protect these values before serialization. Tests with provider-prefixed key names would cover the reachable path without recording secret values.

[Mandark-droid]

All eight findings are addressed in deb0b8d, so this is ready for another pass whenever you want to run one.

The two high ones:

F009 was the worst of them and slightly worse than the report describes. The filter tested only top-level keys, and the else branch json.dumps() the whole value, so a nested env_vars mapping carrying OPENAI_API_KEY was serialised intact into additional_details — a live credential reaching a published record, not just an unfiltered field. Sanitising is recursive now over mappings and lists, and redacted keys are reported as dotted paths (env_vars.OPENAI_API_KEY). Matching is on exact names plus suffixes rather than substrings, so tokenizer and max_tokens aren't swept up; there's a test for that alongside the provider-prefixed one you suggested.

F001 — the defect is real, but the cause is a documentation conflict rather than an oversight. CONTRIBUTING.md line 102 prescribes {benchmark_name/model_id/retrieved_timestamp}, which is what I implemented; .claude/skills/eee-dataset-conversion/reference/fields.md says to key on a stable value and never on now. I've followed fields.md — task key, the model name as the file states it, the run's own wall-clock stamp, plus a digest of non-secret config so runs differing only in settings stay distinct — and conversion time stays in retrieved_timestamp alone. The two documents still disagree, and the next converter author will hit it.

The rest: F002 files with measured but unconvertible tasks now enter the failure ledger, while derived-only files are recorded as explicit exclusions; F007 the source-record grain is the results file, not a mix of task logs and file failures; F008 both CLI entry modes go through a shared transform_from_file_result, so a single-file parse error writes its report before raising; F004 samples_number stays dataset provenance with the smaller scored population recorded as scored_num_docs; F005 an undeclared direction is still assumed, since the schema requires one, but operators can now declare it and unresolved names are reported instead of the guess passing silently.

F006 I did the conservative version. Names with an unambiguous cross-source identity map to it, everything else gets a stable lighteval/<name> rather than an invented canonical id, and metric_id_source records which route was taken. That's deliberate — a wrong join key silently merges two different measurements, which is worse than leaving one unjoined — and it means a later pass can resolve the namespaced ones against the eval-card-registry without re-deriving which were canonical. Given evaleval/eval-card-registry#57, that resolution is probably where this should end up anyway.

39 lighteval tests now (was 31), full suite 428 passed / 20 skipped, ruff clean, and convert → validate still passes 3/3.

One question: the numbering skips F003. Dropped deliberately, or did a finding not post?

[mrshu]

⚒️ review-anvil report

Review decision: COMMENT — The latest commit addresses many earlier concerns
and the focused suite is green; a few important trust-boundary and conversion
edge cases remain as focused suggestions.
Result: Stable ordinary IDs, recursive model-info redaction, file-level
failure reporting, metric IDs, and score-population provenance are meaningful
improvements. Three high-priority, five medium-priority, and two low-priority
follow-ups remain.
Scope: Adds a LightEval aggregate converter, two CLI entry points, shared
publication and accounting integration, documentation, a realistic fixture,
and offline tests.
Checks: 5 concerns checked; 3 kept (1 narrowed), 1 lowered in priority, and
1 ruled out.
Second check: targeted, 2 Codex reviewers; 10 findings kept, 7 fix paths
clarified, 0 set aside, 0 removed.

Earlier review comments

Earlier review comments (10 unique items)

• Fixed: RAV-RUN1-R1-F004 now records original and effective document counts;
  RAV-RUN1-R1-F007 now uses source-file totals and separate output counts;
  RAV-RUN1-R1-F008 now writes structured single-file failure reports.
• Narrower but still present: RAV-RUN1-R1-F001 is stable for normal timestamped
  files, but accepted timestamp-less names can collide; RAV-RUN1-R1-F009
  recursively sanitizes model-info fields, but another raw projection can still
  publish credentials.
• Still present: RAV-RUN1-R1-F002, RAV-RUN1-R1-F003,
  RAV-RUN1-R1-F005, RAV-RUN1-R1-F006, and RAV-RUN1-R1-F010 remain in
  narrower forms described below.

What I noticed

ID	Priority	Topic	Code location	What I noticed
RAV-RUN1-R1-F009	high	credential filtering	every_eval_ever/converters/lighteval/adapter.py:232	The recursive sanitizer is a strong improvement for model_info, but generation configuration rereads raw generation_parameters and serializes unfamiliar keys. A credential there, or an Authorization value in nested headers, can still reach the published EvaluationLog.
RAV-RUN2-R1-F001	high	model routing	every_eval_ever/converters/lighteval/adapter.py:106	A valid bare model name leaves developer=None. The adapter can build the log, but the shared publisher rejects its one-component model ID because it has no safe developer route. (inline)
RAV-RUN1-R1-F002	high	partial accounting	every_eval_ever/converters/lighteval/adapter.py:587	A measured task with no finite score becomes a metadata note when a sibling task succeeds. The remaining logs publish and the CLI exits zero, so automation cannot detect the missing task.
RAV-RUN1-R1-F001	medium	stable identity	every_eval_ever/converters/lighteval/adapter.py:426	Ordinary timestamped files now reconvert idempotently. Accepted names such as results_latest.json use unknown and omit task-specific run/content identity, so distinct runs can still collide.
RAV-RUN1-R1-F005	medium	metric semantics	every_eval_ever/converters/lighteval/adapter.py:320	When required metric direction is unavailable, the converter still publishes an assumed higher-is-better value. The internal override is not exposed by either CLI; unknown optional bounds can remain explicitly unknown.
RAV-RUN1-R1-F006	medium	metric identity	every_eval_ever/converters/lighteval/utils.py:185	The private metric table labels values canonical without a registry lookup, pinned snapshot, revision, or review status. This makes global join-key claims difficult to verify and lets ID, direction, and bounds drift independently.
RAV-RUN2-R1-F002	medium	task isolation	every_eval_ever/converters/lighteval/adapter.py:598	If a later task raises, the file-level exception boundary reports the failure but loses valid sibling logs accumulated earlier in the loop. (inline)
RAV-RUN2-R1-F003	medium	CLI provenance	every_eval_ever/converters/lighteval/__main__.py:25	The module entry point defaults to an empty organization and first_party; the top-level command defaults to unknown and third_party. Identical input therefore gets different provenance based only on entry point. (inline)

Non-blocking low-priority findings (2 items)

• RAV-RUN1-R1-F003 [low] deployment metadata — Both deployment axes still
  become unknown, and neither supported CLI accepts known operator values.
• RAV-RUN1-R1-F010 [low] integration tests — The thorough adapter tests do
  not yet exercise a real CLI parse, atomic publication, and semantic validation
  of the final datastore path.

Things to try

Things to try (8 suggestions)

• [high] credential filtering — Sanitize model configuration once before
  every projection, explicitly handle credential-bearing header/cookie fields,
  keep redaction bookkeeping out of stable identity, and scan the complete
  serialized log in tests. (RAV-RUN2-R1-P001; covers RAV-RUN1-R1-F009)
• [high] model routing — Resolve known bare names with shared helpers and
  registry resolution, add a bounded/offline path plus explicit identity
  override, and turn unresolved unsafe routes into structured source failures.
  The untouched raw name can remain the record-identity input.
  (RAV-RUN2-R1-P002; covers RAV-RUN2-R1-F001)
• [high] task accounting and isolation — Parse each file once and convert
  measured tasks independently. Keep valid logs, aggregate failed-task details
  into one file-level failure, publish successes, then exit nonzero so the
  existing file-grain denominator stays coherent. (RAV-RUN2-R1-P003; covers
  RAV-RUN1-R1-F002 and RAV-RUN2-R1-F002)
• [medium] stable identity — For an unparseable timestamp, hash a canonical
  task-local, secret-safe payload with raw model, sanitized generation settings,
  relevant task configuration, and that task's result. Exclude paths, secrets,
  redaction lists, canonical registry IDs, and CLI provenance.
  (RAV-RUN2-R1-P004; covers RAV-RUN1-R1-F001)
• [medium] metric semantics — Use one validated metric-spec mapping with
  canonical ID, direction, score type, and bounds together. Unresolved required
  direction can enter the file-level failure ledger while known siblings remain
  publishable; genuinely unknown optional bounds can stay labeled and omitted.
  (RAV-RUN2-R1-P005; covers RAV-RUN1-R1-F005)
• [medium] metric identity — Vendor a minimal revision-pinned registry
  snapshot for known emitted metrics, deriving ID, direction, type, and bounds
  from that one source. Unknowns can remain namespaced and explicitly unresolved.
  (RAV-RUN2-R1-P006; covers RAV-RUN1-R1-F006)
• [medium] CLI provenance — Reuse one LightEval parser/default definition in
  both entry points, preserving current option aliases, and add a default/override
  parity test. (RAV-RUN2-R1-P007; covers RAV-RUN2-R1-F003)
• [low] metadata and integration — Add enum-validated deployment and
  availability overrides through the shared parser, then exercise an offline
  real-CLI publication and validate every emitted JSON semantically.
  (RAV-RUN2-R1-P008; covers RAV-RUN1-R1-F003 and RAV-RUN1-R1-F010)

Run details

• Target: PR #235 at deb0b8ddbe69fe9212b8191d7bdfe73a86d4ad0e
  (11 files, +2103/-4)
• Run ordinal: 2
• Rounds: 1/1 completed; adaptive off; material findings remain
• Mix: 3 Codex reviewers
• Focus: correctness, source accounting, credentials, publication behavior,
  simplicity, maintainability, and positive suggestion-oriented language
• Earlier review comments: 10 unique concerns; 3 fixed and 7 narrower or still
  present
• Finding counts: 0 critical, 3 high, 5 medium, 2 low, 0 nit
• Verification: exact-head tests/test_lighteval_adapter.py — 39 passed; no
  GitHub check runs were reported for this head
• Checks: concerns=5; kept=3 (narrowed=1)/ruled-out=1/lowered=1/set-aside=0
• Second check: targeted; reviewers=2; kept=10/clarified=7/set-aside=0/removed=0;
  approval changed no
• Set aside: 0 items

---

Reviewed with review-anvil.

[mrshu]

RAV-RUN2-R1-F001 [high] model routing — A bare model_name leaves developer unset here. The adapter can build the log, but publication rejects a one-component model ID without a developer route. Shared model helpers, registry resolution, and an explicit identity override can cover known and operator-supplied names; an unresolved unsafe route can remain a structured source failure.

[mrshu]

RAV-RUN2-R1-F002 [medium] task isolation — These sibling tasks share one file-level exception boundary. If a later task raises, this function never returns the valid logs already accumulated, so the result contains only the file failure. Per-task conversion can retain valid siblings and aggregate failed-task detail into the existing file-grain report.

[mrshu]

RAV-RUN2-R1-F003 [medium] CLI provenance — This entry point defaults organization to an empty string and evaluator relationship to first_party, while the top-level LightEval command uses unknown and third_party. Sharing one parser/default definition would keep identical inputs from receiving different implicit provenance.

[borgr]

Picking this up on the lighteval side rather than opening a competing PR — your three commits stay as they are, and everything below is additive on top of them (plus a merge of main, which had moved).

What's added

--include_details — converts the per-sample details parquet lighteval writes beside a run's results into an instance-level <uuid>_samples.jsonl, and points the aggregate's detailed_evaluation_results at it. Same shape and same partial-conversion semantics as lm_eval's --include_samples: a run made without save_details still publishes its aggregates, but records a per-task failure in adapter_reports/lighteval_details_failures.json and exits non-zero, so a missing details tree can't pass for a clean conversion.

New: converters/lighteval/instance_level_adapter.py, details-location helpers in utils.py, the CLI wiring, and two keys (results_file, model_name) added to the traceability dict your adapter.py already builds — the stored metadata had the parent directory but not the file, so the details tree couldn't be found from it.

lighteval extra = pyarrow>=17, to read the parquet. The core install has pandas but no parquet engine. lighteval itself stays out of the project and out of uv.lock on purpose: it needs datasets>=4 while crfm-helm pins datasets~=3.1, so locking it makes the whole resolution unsatisfiable. uv.lock is relocked for the new extra (no version churn) since CI runs uv sync --locked.

Tests + fixture. tests/data/lighteval_smoke/ was not hand-copied — scripts/upstream_smoke/lighteval_smoke.py drives lighteval's own Pipeline with their DummyModelConfig (random logprobs, fixed text, no weights, no inference), so the parquet layout, the numpy dtypes and the task-key spelling are all upstream's. It's in its own directory, deliberately not inside tests/data/lighteval/, because your test_transform_from_directory_finds_nested_results_files walks that tree recursively and counts what it finds. Two tasks, one multiple-choice (anli:r1|0) and one generative (squad_v2|0), because a fake model only exercises the metrics its task defines.

The test runs the real CLI and then the real validator at the canonical data/<collection>/<developer>/<model>/ path — which is where the semantic checks actually fire — and requires exit 0, since warning-only (exit 2) is valid locally but not merge-ready. So it asserts the output is submittable, not just that some fields hold the values we expected. pytest.importorskip('pyarrow') so it skips on the two core CI rows and runs on the two --all-extras rows.

Weekly upstream smoke (.github/workflows/upstream_smoke.yml). The committed fixture pins a past release; it can't tell us a new one renamed a field, because it keeps the old spelling and CI stays green until someone hits it in their console. This job installs the current lighteval ad hoc, has the fake model produce a run, converts it, validates it. Not on pull_request — an upstream release shouldn't redden an unrelated PR, and a scheduled failure names the version that broke it. --refresh on the script regenerates the fixture; the docstring says to read the diff first, because it will just as happily record a regression as an intended change.

Two things running it against real output turned up

Both were mine, in the new instance-level code, and both would have been invisible against a hand-written fixture:

1. doc.choices means two different things. lighteval's own Doc docstring: it holds "all options" for a multiple-choice task but "reference answers" for a generative one. Publishing the generative case as input.choices would claim the model was handed the answer to pick from. doc.sampling_methods is the discriminator (LOGPROBS vs GENERATIVE); input.reference carries the gold either way, and metadata.lighteval_sampling_methods records which kind of row it was. There's a response-shape fallback (one logprob per choice ⇒ the model was asked to rank them) for runs predating that field.

2. numpy integers are not Python integers. doc.gold_index is a plain int for a single gold and a numpy.ndarray of np.int64 for a list. isinstance(np.int64(0), int) is False (checked on numpy 1.26 and 2.4 — np.float64 does subclass float, which is what makes this easy to miss), so an isinstance(v, (int, float)) guard silently dropped every element and left reference empty on squad_v2. Every numeric check now unwraps the scalar first.

One thing I left alone deliberately: the SECRET_MODEL_CONFIG_KEYS / _SECRET_KEY_SUFFIXES redaction in utils.py. LiteLLMModelConfig.api_key is a plain str, so a converted record could otherwise carry a live credential — that guard is load-bearing.

Scope, stated plainly

Shape, not semantics. A metric that switches from percent to proportion upstream passes all of this, as does a changed prompt template. What it does catch is a converter that stops finding the details file, stops resolving the gold, or starts emitting records the datastore would reject.

There's no ConverterCase entry here — #244 owns tests/converter_cases.py, so this ships a self-contained test module and can fold into the shared harness once that lands.

Locally: 513 passed with --extra lighteval, 456 passed / 20 skipped on the core install, ruff check clean, and uv sync --locked --all-extras verified.

[mrshu]

⚒️ review-anvil report

Review decision: COMMENT — The new sidecar and smoke coverage are valuable,
but two sample-integrity failures and several focused metadata issues remain.
Result: The focused tests and all four CI matrices pass. Six new suggestions
survived verification, and nine earlier findings remain present without new
inline threads.
Scope: Adds LightEval aggregate and instance conversion, CLI publication,
real upstream fixtures, and scheduled release-smoke coverage.
Checks: 4 concerns checked; all 4 confirmed.
Second check: targeted, 2 Codex reviewers; 6 findings kept and 5 fix paths
clarified.

Earlier review comments

Earlier review comments (13 unique findings)

• Still present at high priority: RAV-RUN1-R1-F009 can publish raw nested
  generation credentials; RAV-RUN2-R1-F001 cannot publish unresolved bare
  model names; RAV-RUN1-R1-F002 still omits a measured task when a valid
  sibling lets the file publish successfully.
• Still present at medium priority: RAV-RUN1-R1-F001 can collide for accepted
  timestamp-less filenames; RAV-RUN1-R1-F005 assumes unknown metric
  direction; RAV-RUN1-R1-F006 uses an unpinned private canonical metric map;
  RAV-RUN2-R1-F002 loses valid sibling tasks after a later task error; and
  RAV-RUN2-R1-F003 keeps different provenance defaults in the two CLIs.
• Still present at low priority: RAV-RUN1-R1-F003 has no CLI overrides for
  known deployment metadata.
• Fixed on the current head: RAV-RUN1-R1-F004, RAV-RUN1-R1-F007,
  RAV-RUN1-R1-F008, and RAV-RUN1-R1-F010 now preserve both sample counts,
  use file-grain totals, report single-file failures, and test real CLI
  publication.

What I noticed

ID	Priority	Topic	Code location	What I noticed
RAV-RUN3-R1-F001	high	instance metric linkage	every_eval_ever/converters/lighteval/instance_level_adapter.py:192	A multi-metric details row publishes only its first finite score. Neither aggregate nor instance records carry a result join ID. (inline)
RAV-RUN3-R1-F002	high	details file routing	every_eval_ever/converters/lighteval/utils.py:226	Details lookup searches every model after the requested model subtree. A missing file can attach another model's samples. (inline)
RAV-RUN3-R1-F003	medium	sample hash	every_eval_ever/converters/lighteval/instance_level_adapter.py:201	The sample hash uses default JSON whitespace, so it differs from the repository's cross-adapter canonical digest. (inline)
RAV-RUN3-R1-F004	medium	instance correctness	every_eval_ever/converters/lighteval/instance_level_adapter.py:259	Every metric uses score == 1.0 for correctness. Continuous metrics can therefore publish a false correctness claim. (inline)
RAV-RUN3-R1-F005	medium	instance score validation	every_eval_ever/converters/lighteval/instance_level_adapter.py:388	A row without a finite metric becomes a fabricated score of 0.0 and avoids the failure ledger. (inline)

Non-blocking low-priority finding

• RAV-RUN3-R1-F006 [low] documentation — The root converter table still
  says LightEval instance output is unavailable, although --include_details
  now publishes it.

Things to try

Things to try (6 suggestions)

• [high] instance metric linkage — Create result IDs on aggregate metrics,
  then fan out each sample with those exact stored IDs and verify join-set
  equality. (RAV-RUN3-R1-P001; covers RAV-RUN3-R1-F001)
• [high] details file routing — Resolve only the requested model, run date,
  and task path. Treat missing or ambiguous details as conversion failures.
  (RAV-RUN3-R1-P002; covers RAV-RUN3-R1-F002)
• [medium] sample hash — Use the documented compact JSON recipe and pin one
  expected digest. (RAV-RUN3-R1-P003; covers RAV-RUN3-R1-F003)
• [medium] instance correctness — Use a reviewed binary-metric set. Mark
  correctness not applicable in metadata for all other metrics.
  (RAV-RUN3-R1-P004; covers RAV-RUN3-R1-F004)
• [medium] instance score validation — Fail the containing details task
  when any row has no finite metric. Reuse the existing task failure ledger.
  (RAV-RUN3-R1-P005; covers RAV-RUN3-R1-F005)
• [low] documentation — Update only the root table to name
  --include_details and its save_details=True requirement.
  (RAV-RUN3-R1-P006; covers RAV-RUN3-R1-F006)

Run details

• Target: PR #235 at 927da0d470d743e9952f70f903461196ce5a28c1
  (21 files, +3457/-5)
• Run ordinal: 3
• Rounds: 1/1 completed; adaptive off; material findings remain
• Mix: 3 Codex reviewers
• Focus: correctness, blast radius, simplicity, maintainability, sidecar
  semantics, smoke coverage, and positive suggestion-oriented language
• Earlier review comments: 13 unique findings; 9 still present and 4 fixed
• Finding counts: 0 critical, 5 high, 8 medium, 2 low, 0 nit, including prior
  carry-forwards
• Verification: exact-head LightEval tests — 39 passed, 1 optional skip;
  GitHub Actions passed all four core/full locked/loose matrices
• Checks: concerns=4; confirmed=4/ruled-out=0/set-aside=0/lowered=0
• Second check: targeted; reviewers=2; kept=6/clarified=5/set-aside=0/removed=0;
  approval changed no
• Action lock: the first two-auditor launch returned no verifiable rows because
  of a dispatch error. All 5 inline requests use the exact frozen source prose,
  and the review event is forced to COMMENT.

---

Reviewed with review-anvil.

[mrshu]

Multi-metric samples lose scores and cannot join aggregate results

_primary_metric keeps only the first finite metric, and neither output carries evaluation_result_id. A sample with K metrics publishes one structured score and leaves it unlinked.

What to change

Create each deterministic evaluation_result_id on the aggregate result. Emit one instance record for each finite metric using that exact stored ID. Add a multi-metric CLI publication test that checks exact aggregate/instance join-set equality, unique IDs, and K records per sample for K finite metrics.

[mrshu]

Details lookup can attach samples from another model

find_details_file searches the full details root after the requested model subtree. A missing requested-model file can therefore select a same-task file from another model.

What to change

Resolve only the exact requested-model, run-date, and task path. Treat an absent or ambiguous match as a details-conversion failure. Add coverage with two model trees and a missing requested-model file.

[mrshu]

LightEval sample hashes differ from the cross-adapter recipe

The hash payload sorts keys but keeps default JSON whitespace. Identical input and references therefore get a different digest from other EEE adapters.

What to change

Serialize raw and the full reference list with sort_keys=True and separators=(',', ':') before UTF-8 SHA-256 hashing. Add a fixed input and expected digest as a regression vector.

[mrshu]

Continuous scores are published as answer correctness

Every metric uses score == 1.0 for is_correct. A graded metric with value 1.0 can therefore become a correct answer even when correctness is not defined.

What to change

Use a tight reviewed set of source metrics known to emit binary per-sample correctness. Keep is_correct false and record is_correct_applicable=false in instance metadata for unknown or graded metrics. Add tests for acc and em plus a continuous metric with score 1.0 that remains not correct and not applicable.

[mrshu]

Rows without finite metrics become fabricated zero scores

_primary_metric returns (None, 0.0), and _transform_row publishes that value. Missing source data becomes a valid-looking failed sample.

What to change

Fail and account the containing details task when any row has no finite metric. Include the row index and sample ID in the error. Add tests for empty and all-nonfinite metric mappings, plus a mixed mapping that still emits its finite metrics.