Are the public artefacts meant to be scoreable end-to-end? Documented ground truth for MAIN and the shipped data overlap by one asset

[arthapraha]

Repository: IBM/AssetOpsBench · Observed at: e11d1c1 (current main)
Source: finding 4 of our defect report · follows #503

This is a question about intent, not a defect report, and we think the answer
may well be "yes, that is expected." If so, the whole of it is a request for
one sentence in the docs. We are filing it because we spent real time
reconstructing the answer from failing runs, and a line in the README would have
saved all of it.

What we observed

docs/guideline/ground_truth_design_guideline.md documents the answer for site
MAIN at lines 655 and 1980, identically in both places:

"The assets at the MAIN site are: CQPA AHU 1, CQPA AHU 2B, Chiller 4, Chiller 6, Chiller 9, Chiller 3."

The data the default manifest actually loads, from
src/couchdb/scenarios_data/shared/iot/asset_profile_sample.json, holds five
assets at siteid: MAIN:

Chiller 6, mp_1, hyd_1, Motor_01, PUMP3

The overlap is one asset — Chiller 6.

documented for MAIN	CQPA AHU 1, CQPA AHU 2B, Chiller 3, Chiller 4, Chiller 6, Chiller 9
shipped at MAIN	Chiller 6, Motor_01, PUMP3, hyd_1, mp_1
in both	Chiller 6
documented but not shipped	CQPA AHU 1, CQPA AHU 2B, Chiller 3, Chiller 4, Chiller 9
shipped but not documented	Motor_01, PUMP3, hyd_1, mp_1

One counting note, so this does not read as our arithmetic error: the file
contains six records, not five. The sixth is AHU2, whose siteid is
NORTH. Five is the count at MAIN, which is what the documented answer is about.

The practical consequence is that an agent answering correctly from the
shipped data is scored wrong against the documented answer, and vice
versa.

Why we think this may be entirely expected

The guideline's own reference list cites an "AssetOpsBench Ground Truth
Dataset (2026 - IBM Internal)" (line 2426), and the public HuggingFace
dataset, across the 20 files we enumerated, ships questions without
ground-truth answers. So the most likely explanation is
simply that the documented answers were written against an internal dataset
that the public artefacts do not include — which is a completely ordinary
thing for a benchmark to do, and not something we are asking you to change.

We want to be clear that we are not reporting the guideline as wrong. We are
reporting that an external user cannot tell which of the two is the intended
reference, and currently discovers the difference only by running the benchmark
and reading confusing scores.

What we are actually asking for

One line, in the README or in docs/evaluation.md, saying whether the public
artefacts are intended to be scoreable end-to-end without the internal
dataset — and if they are not, which parts are.

Either answer is useful. If they are not meant to be, that largely settles this
issue and the gradeability audit filed alongside it, and it shapes the wording of
the section that covers all four. If they are meant to be, then the sample data
and the documented answers are worth reconciling, and we would be glad to help
with that separately.

Reproduce

import json
d = json.load(open("src/couchdb/scenarios_data/shared/iot/asset_profile_sample.json"))
main = sorted(r["assetnum"] for r in d if r["siteid"] == "MAIN")
print(len(d), "records;", len(main), "at MAIN:", main)
# -> 6 records; 5 at MAIN: ['Chiller 6', 'Motor_01', 'PUMP3', 'hyd_1', 'mp_1']

How this issue relates to the other three

This is the first of four issues that a single documentation PR closes. The
other three are distinct things an external user trips over, sharing a fix
location rather than a root cause — which scenario questions are gradeable
against the shipped data, what the MCQA pools do and do not exercise, and which
questions are answerable without touching the tools. The last two hold whatever
the answer to this issue turns out to be.

We filed them separately because each is its own trip point and you may well
want different answers to them. But the fix is one section, not four, and
the PR adds it to docs/evaluation.md, which already describes scoring. We will
open it once the remaining three have numbers, so it can reference all four
rather than being amended three times.

If you would rather have four smaller PRs, or the section somewhere else —
the README, INSTRUCTIONS.md, a new docs/public-data.md — say so and we will
reshape it. The content is written; its location is your call.

---

Findings 1–6 reached the maintainers by email on 10 August. This continues that
exchange, and is the second of the six issues announced in #503.