Evidence from 936 runs: an answer-format contract makes `plan_execute` output gradeable

[arthapraha]

Repository: IBM/AssetOpsBench · Observed at: e11d1c1 (current main)
Source: findings 3 and 10 of our defect report

This one is a question rather than a defect report, and it is deliberately
opening the set: what we do about the others depends partly on the answer.

The situation

plan_execute builds its final answer from a prompt asking for "a concise,
direct answer", so the answer is prose by construction. static_json passes
only on strict exact match. Correct prose against a list gold scores 0.000.

Against a sentence gold we measured a correct answer at 0.600 and the
wrong documented answer at 0.538 — six points apart. At that separation the
scorer is not distinguishing right from wrong.

There is a second consequence that only shows up at volume: without a format
contract, "declined", "wrong" and "unreadable" are indistinguishable in the
output. They are three different facts, and a scorer that merges them reports
a number that describes none of them.

What we measured

We ran the 12 scenario questions whose ground truth is derivable from the shipped
data, 39 times each, in two conditions — the scenario's own MCP server mounted,
and no servers mounted — with an explicit answer-format instruction supplied.
936 runs.

condition	correct	wrong	declined	unreadable
tools mounted	413	30	23	0
no tools	0	0	459	0

Zero unreadable outputs in 925 resolved runs. Before we supplied the
instruction, the no-tools condition returned prose that could not be graded at
all — the failure above, at scale. Same scenarios, same model, same runner:
ungradeable without a contract, cleanly gradeable with one.

The second row is the part we did not expect. With no tools, the system
declined 459 times and fabricated an answer zero times. That makes a 0% score
a real floor rather than an artefact of a model bluffing for partial credit —
which is what makes these scenarios worth grading at all.

The instrument we used

Included because the numbers above are only meaningful if you can see what
produced them — this is the instrument behind the evidence, not a proposed
design for your interface. Three variants, by expected answer shape, appended to
the existing final-answer prompt:

list:
Reply with ONLY a JSON array of strings and nothing else — no prose, no explanation, no markdown outside the JSON. Each element must be EXACTLY the name or identifier and nothing else: no descriptions, no types, no counts, no units, no parenthetical detail. Write ["Chiller 6"], never ["Chiller 6 (CHILLER, 12 sensors)"]. If you cannot determine the answer, reply with exactly "UNKNOWN" — with the quotation marks. Write "UNKNOWN", never bare UNKNOWN, never ["UNKNOWN"]. Do not guess.

count:
Reply with ONLY a bare JSON number and nothing else — no prose, no explanation, no units, no markdown outside the JSON. Write 5, never "5 assets" or 5.0. If you cannot determine the answer, reply with exactly "UNKNOWN" — with the quotation marks. Write "UNKNOWN", never bare UNKNOWN, never ["UNKNOWN"]. Do not guess.

boolean:
Reply with ONLY the bare JSON literal true or false and nothing else — no prose, no explanation, no markdown outside the JSON. Write true, never "yes", never "Yes", never "supported", never "true.". If you cannot determine the answer, reply with exactly "UNKNOWN" — with the quotation marks. Write "UNKNOWN", never bare UNKNOWN, never ["UNKNOWN"]. Do not guess.

The reserved token is what separates declined from wrong. Without one, a
model that correctly cannot answer is scored identically to one that answered
incorrectly.

Pinning its exact form is what makes that separation cheap to check, and it
is the clause we would keep if we kept only one. "UNKNOWN", bare UNKNOWN and
["UNKNOWN"] are the same intent in three shapes, and a scorer that has to
recognise all three is back to pattern-matching free text — which is the original
problem wearing a smaller hat. Requiring one spelling makes the check an equality
test. In 459 no-tools runs it produced 459 clean classifications and no
near-misses to adjudicate.

The questions that are yours, not ours

We are not proposing a specific design. These are design decisions for the
project to make — the evidence above is what we can contribute to them:

1. Should the contract ship at all, or is prose-plus-llm_judge the intended
   path and static_json simply the wrong scorer for plan_execute?
2. Where would it live — the runner's final-answer prompt, the scenario
   record, or the scorer's expectations?
3. All runners or just plan_execute? We only exercised the latter.
4. What should the reserved token be? "UNKNOWN" is ours and arbitrary.

One related observation while you are in this area: evaluate's default scorer
is llm_judge, but the folder-based scenario loader hard-codes static_json and
overrides it. A user who chose llm_judge and used the folder layout is silently
not getting it.

Offer

Happy to turn this into a PR against whatever shape you would prefer, or to
close it if the answer is "use llm_judge". We have the harness and the 936-run
comparison already, so re-running the evidence against a different contract is
cheap.

---

Findings 1–6 reached the maintainers by email on 10 August; this continues that
exchange. Six further issues follow from the same measurement — two with code
PRs and four covered by one documentation PR. This one leads deliberately,
because what a fix for the others should assume depends partly on the answer
here. Full write-up including method and limits available on request; this issue
quotes only the parts relevant to scoring.