adk evals: state `equals:` assertions can never pass — `state.value` is set as an object attribute and does not survive OTel export; failure reports no `actual`

[johnf]

Summary

state assertions with equals: can never pass in connected mode: the state.value attribute is set on the state.save span as a plain JS object, which is not a valid OTel attribute type, so it does not survive export. changed: assertions on the same span work perfectly, because state.changed_keys is a string array.

The failure is also silent. None of the grader's three diagnostic branches fire, and the result carries no actual field at all, so the developer sees a bare expected: with nothing to explain it.

Versions

• @botpress/evals 2.0.3, @botpress/runtime 2.0.3, adk CLI 2.0.3
• Connected mode (adk dev running, ports 3000/3001), Linux, node 26

Reproduction

An eval asserting conversation state the handler demonstrably writes. Our handler increments state.turns and sets state.disclosed = true on the first turn:

conversation: [
  {
    user: 'What kind of solar panels do you install?',
    assert: {
      response: [{ contains: 'AI assistant' }],          // control — code-sent copy
      state: [
        { path: 'conversation.turns', equals: 1 },        // FAILS
        { path: 'conversation.disclosed', equals: true }, // FAILS
        { path: 'conversation.disclosed', changed: true },// PASSES
      ],
    },
  },
],

Run five times, three consecutively against one dev server and twice against a freshly booted one.

Result — 100% consistent, not flaky

Assertion	5 runs
changed: true (2 fields)	pass, pass, pass, pass, pass
changed: false (negative case)	pass, pass, pass, pass, pass
equals: (scalar number)	fail ×5
equals: (scalar boolean)	fail ×5
equals: (object value)	fail ×5
the same equals: via outcome:	fail ×5
response.contains control	pass, pass, pass, pass, pass

The contains controls assert copy the bot's code sends only after the corresponding state write, so the write provably happened on every run.

The silent-failure part

A failing equals assertion serialises as:

{"assertion":"state: conversation.turns equals","pass":false,"expected":"1"}

No actual key. In graders/state.ts, actual is JSON.stringify(actualValue); actualValue is undefined, and JSON.stringify(undefined) returns undefined, which JSON.stringify on the enclosing report then omits.

Critically, none of the three helpful branches fired — not No state.save span found for "conversation", not No state mutation found, not State was too large for trace-based assertion (swapped to file). So from the grader's point of view the span arrived intact and simply contained no value. That rules out the missing-span and swap-to-file explanations the messages exist to cover.

Mechanism

@botpress/runtime, in the state.save span:

const savedInline = payload.location.type === "state";
if (savedInline) {
  s.setAttribute("state.value", valueToSave);         // plain object
  if (this._lastSavedValue !== void 0) {
    s.setAttribute("state.previous_value", this._lastSavedValue);
  }
}
if (changedKeys.length > 0) {
  s.setAttribute("state.changed_keys", changedKeys);  // array of strings
}

OTel attribute values must be a primitive or a homogeneous array of primitives. state.changed_keys qualifies and arrives; state.value and state.previous_value do not.

The reader side is not at fault — tryParseJson in the runner passes non-strings straight through, so an object would have been used as-is had one arrived:

function tryParseJson(value) {
  if (value == null) return void 0;
  if (typeof value !== "string") return value;
  ...
}

Which points at the fix: JSON.stringify the value before setAttribute (the reader already parses strings), or drop equals: from the documented assertion surface.

I have not attached a captured span showing the attribute absent — the conclusion is from both sides of the source plus the grader reaching its value comparison with undefined while its span-missing branches stayed quiet.

Impact

equals: is documented in the eval assertion surface (StateAssertion accepts equals, and the skill docs use { path: 'conversation.greeted', equals: true } as the example). As it stands it cannot pass, and it fails without a diagnosis, so the natural reading is "my bot didn't write that state" when the bot did.

Two fixes worth having independently:

1. Make values survive — serialise before setAttribute.
2. Until then, fail loudly: if the span is present and state.value is absent, say "state values are not available over this transport" rather than emitting no actual.

Related

adk#36 (lightweight mode exports no spans — still reproduces on 2.0.3), adk#37, adk#38, adk#39.

Minor, found while testing this: adk evals runs --latest --format json ignores --format and prints ANSI-coloured human output. Happy to split that out if you'd prefer.