docs(evaluation): what the public data supports — the scoreability section for #506, #507, #508, #509

[arthapraha]

Description

Four issues — #506, #507, #508 and #509 — describe the same thing from different
angles: what the public artefacts do and do not support when used for scoring.
They share a fix location rather than a root cause, so this is one section
rather than four scattered edits, as promised in #506.

It adds a "What the public data supports" section to docs/evaluation.md,
which already describes scoring and is where someone choosing a scorer is
already reading.

Changes

• Documentation / Tutorial update

One file, docs/evaluation.md, +61 lines. No code touched.

what it records	issue
that most scenario questions cannot be graded against the loaded data as shipped, and that the guideline's ground truth is an internal dataset	#506
the disposition of all 55 scenario questions — 14 derivable, 41 not, by category	#507
that the MCQA pools carry answer keys but reference no loaded entity — 0 of 2,667	#509
that scenarios 205 and 206 are answerable with no tools mounted	#508

Verification Steps

1. tests/integration does not exist at e11d1c1; nothing here is executable
   in any case — the change is documentation only, no code path is touched.
2. The numbers are derived rather than asserted:
   • dispositions — an exhaustive pass over all 55 scenario questions, with
     the executable selector recorded per derivation.
   • 0 of 2,667 — counted at dataset revision
     5e25bb7f2cd37fb68b9a9e1f99d170ca5be7ce17, under an exact match and under
     one that ignores separators so Chiller 6 and CHILLER6 count as the same
     name. Reported for every candidate denominator, not just the headline one.
   • 205 / 206 — a screen run with no MCP servers mounted before any
     measurement, excluding on a single correct answer in three.

Checklist

• I have signed off my commits (DCO).

Note

The wording is a proposal, not a position. #506 asks whether the public
artefacts are meant to be scoreable end-to-end without the internal dataset, and
that question is still open — this section deliberately states what we
measured and names the question of intent as separate, rather than answering
it on your behalf.

If the answer to #506 is "the public data is a sample, not a gradeable set",
this section should say so in your words rather than ours. Happy to rewrite
it, move it elsewhere, split it into four, or close this if you would rather
write it yourselves.