Skip to content

fix(eval): render metrics that report a value but reach no verdict - #528

Draft
i-yliu wants to merge 1 commit into
google:mainfrom
i-yliu:fix/informational-eval-status
Draft

fix(eval): render metrics that report a value but reach no verdict#528
i-yliu wants to merge 1 commit into
google:mainfrom
i-yliu:fix/informational-eval-status

Conversation

@i-yliu

@i-yliu i-yliu commented Sep 8, 2026

Copy link
Copy Markdown

The eval framework is gaining a metric kind that reports a value and never
passes or fails (EvalStatus.INFORMATIONAL, value 4). The efficiency
metrics — tool call count, inference call count, token usage — are reported on
every run and carry it. The Dev UI does not know that status, and its eval
views assume every metric either passed or failed, so such a metric is
presented as a failure:

  • The run summary counts it in the "passed / evaluated" ratio but never in the
    numerator, so a case whose only quality metric passed reads 1/4, not 1/1.
  • The metric chips colour anything that is not PASSED with the error colour
    and border, so a token count renders exactly like a failed assertion.
  • The chips append / <threshold> unconditionally, leaving a dangling slash
    for a metric that has none, and the run-summary row leaves a dangling colon.

Teach the UI the status and render it neutrally: a metric with no verdict gets
no verdict styling, no threshold suffix, and stays out of the ratio.

Separately, stop the run-configuration dialog from dereferencing
metricValueInfo.interval unconditionally. The interval is optional on the
wire, and a metric without one took the whole threshold form down with
Uncaught TypeError: Cannot read properties of undefined (reading 'maxValue').
The dialog now offers only the metrics it can actually configure — those that
require a threshold and carry an interval to bound the slider by. Metrics
without one are always on and are reported without the user selecting them.

MetricValueInfo.interval is now optional and MetricsInfo carries
requiresThreshold, matching what the server already sends.

Testing

Served one eval history containing INFORMATIONAL results through both
bundles, with the same backend on each side so the frontend is the only
variable.

Before

before

After

after

The ratio goes from 1/4 to 1/1, the three efficiency chips lose the failure
colour and border, and the dangling 1.00 / disappears.

Also checked the ratio across four histories, to cover both tally paths:

history before after
1 passed 1/4 1/1
1 passed + 1 failed 1/5 1/2
informational only 0/3 0/0
two invocations 2/8 2/2

The dialog fix was verified separately, by letting metrics without an interval
actually reach the form. Before, "Run All" renders an empty dialog and the
console shows the maxValue TypeError above; after, the form renders and
simply does not offer those metrics.

New unit tests cover the ratio in eval-tab.component.spec.ts and the dialog's
filtering in run-eval-config-dialog.component.spec.ts, including a regression
test that the form renders rather than throwing on a metric with no interval.

The eval framework is gaining a metric kind that reports a value and never
passes or fails (EvalStatus.INFORMATIONAL). The Dev UI assumed every metric
either passed or failed, so such a metric was presented as a failure: counted
in the "passed / evaluated" denominator but never the numerator, coloured with
the error colour and border, and given a dangling "/ <threshold>" suffix it
has no threshold for.

Render it neutrally instead: no verdict styling, no threshold suffix, and out
of the ratio.

Also stop the run-configuration dialog from dereferencing
metricValueInfo.interval unconditionally. The interval is optional on the wire
and a metric without one took the whole threshold form down. The dialog now
offers only the metrics it can configure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant