adk evals: lightweight auto-started dev server exports no spans (totalSpans=0) → waitForTurnComplete timeout; connected mode works

[AVivero]

Updated 2026-07-14 after isolation. My original report framed this as "every suite times out, even with adk dev running." That was too broad — the failure is lightweight-mode-specific. Corrected description below; the underlying bug (a lightweight dev server that exports zero spans) is real and still present in 2.0.2.

Environment

• @botpress/adk-cli 2.0.2, @botpress/evals 2.0.2
• Node 22.16.0, macOS (Darwin 25.5)

Summary

adk evals <name> times out (~30s/turn) when it runs against the lightweight dev server it auto-starts — i.e. when no adk dev is connected. The conversation handler is fine: adk chat --single "<msg>" --format json returns correct grounded replies, and when adk evals is genuinely connected to a running adk dev, the same suites pass. So the defect is in the lightweight server's span export, not the handler or the runner's collector.

Reliable repro (100%)

# no adk dev running:
adk evals <any-suite>

→ runner prints ✓ Lightweight server started (no running adk dev detected) and every turn fails with:

waitForTurnComplete timed out after 30000ms — no handler.conversation span completed;
diagnostics: snapshotReceived=true, totalUpdates=0, totalKeepalives=1, totalSpans=0,
turnSpans=0, turnSpanNames=none, turnHandlerStatuses=none; spanSource: default
    at <anonymous> (@botpress/adk-cli/dist/chunk-cr2zw32y.js:11385:25)

The tell (lightweight vs connected)

• Fails: startup ✓ Lightweight server started (no running adk dev detected), diagnostics devServerUrl: http://127.0.0.1:<random>, totalSpans=0.
• Passes: startup ✓ Connected to dev server, spans flow.

Validated matrix (real 20-turn suite, adk-cli/evals 2.0.2)

invocation	startup line	result
bare, no adk dev running	Lightweight server started	totalSpans=0 → timeout, ~32s
bare, adk dev running	Connected to dev server	1 passed, ~70s
--server <serverPort>, dev up	Connected to dev server	1 passed, ~69s
--server <botPort>, dev up	Lightweight server started	totalSpans=0 → timeout, ~31s

Root cause (from the minified dist + the matrix)

The runner is OpenTelemetry-based — it stands up an OTLP /v1/traces collector and blocks on a handler.conversation span to mark a turn complete. The lightweight dev server that adk evals auto-starts exports zero spans to that collector (totalSpans=0), so completion is never detected. A full adk dev exports them correctly, which is why connected runs pass. This points at a span-export wiring gap in the lightweight dev server specifically.

Two related gotchas

1. --server <url> overrides the ~/.adk/console.port auto-detect. If the URL isn't the dev server's serverPort (the console/health server with healthUrl …/api/health — not the botPort), the runner prints no running adk dev detected and silently falls back to lightweight, then fails as above. This is almost certainly what my original "adk dev running but still fails" observation actually was.
2. The console/serverPort is non-deterministic — it defaults to 3002 but hops (e.g. to 3015) whenever the preferred port is taken, so a hardcoded --server port is fragile.

Impact

Breaks unattended CI, since the natural CI invocation is bare adk evals with no dev server up.

Workaround

Run adk dev alongside and invoke bare adk evals <name> (no --server) — it auto-connects via console.port and spans flow. (Or drive the handler through the chat channel, which needs no spans.)

Questions for maintainers

• Is handler.conversation span export expected to be wired when adk evals auto-starts its lightweight dev server? It currently exports none.
• Could the lightweight fallback at least warn instead of silently degrading when --server points at a non-serverPort (or when it can't reach the given URL)?

[johnf]

Confirming this on adk CLI 2.0.1 (linux-x64 release binary, Ubuntu; also GitHub-Actions-shaped usage) — so it predates 2.0.2 and isn't a regression there. Byte-for-byte the same signature and diagnostics (snapshotReceived=true, totalUpdates=0, totalSpans=0, turnHandlerStatuses=none).

Scale data point: a full-suite run of 99 evals with no adk dev running finished passed: 0, failed: 99 — one identical 30 s timeout per eval, 53 min wall-clock, zero connection errors (the auto-started server stayed reachable throughout; it accepts the SSE trace stream but never emits a span). Also observed: the lightweight "server" runs in-process inside the adk evals pid itself (one pid holds both listener ports; no child process), consistent with your span-export-wiring theory.

One more reason to prioritise: the docs present this path as supported — the adk-evals skill (botpress/skills, references/testing-workflow.md) says "If adk dev isn't running, adk evals auto-starts a lightweight server automatically — fine for one-off runs…". Until the wiring is fixed it may be worth a fail-fast (or at least a loud warning) instead of the silent 100%-timeout run, since bare adk evals is the natural unattended-CI invocation.

[johnf]

Re-confirmed on adk 2.0.3 (originally reported against 2.0.1) — this survived the release.

With no dev server running, a single eval against the auto-started in-process server:

ERROR: waitForTurnComplete timed out after 30000ms — no handler.conversation span completed;
diagnostics: snapshotReceived=true, totalUpdates=0, totalKeepalives=1,
lastEvent=keepalive 523ms ago, totalSpans=0, turnSpans=0, turnSpanNames=none,
turnHandlerStatuses=none

35 s wall-clock, one eval, 0 passed / 1 failed.

The diagnostics narrow it usefully: snapshotReceived=true and a keepalive 523 ms before the timeout mean the SSE channel is connected and healthy throughout — it is span export that never happens, not the transport. totalUpdates=0 says nothing at all arrived over that live channel.

For contrast, the identical eval passes in ~7 s against a connected adk dev on the same machine, same CLI version, same bot.