adk evals: a dev-server rebuild mid-run fails the in-flight turn instead of retrying, and reports it as an opaque 30s totalSpans=0 timeout

[johnf]

Environment: adk CLI 2.0.1 (linux-x64 release binary); @botpress/evals 2.0.2, @botpress/runtime 2.0.2; observed on Ubuntu (kernel 6.17) locally and on GitHub Actions ubuntu-latest.

Running a full eval suite (~100 evals, ~20 min) with adk evals against adk dev reliably risks the dev backend becoming unreachable mid-run, after which every remaining eval fails with Unable to connect. Is the computer able to access the url? (devServerUrl: http://localhost:3001). The bot HTTP port (3000) often stays up while the trace listener (3001) is gone — ss -ltn shows 3001 missing, 3000 listening.

Two distinct triggers observed:

(a) Any hot-reload rebuild mid-run. Editing any watched file while evals are in flight (Changes detected, rebuilding) drops the trace endpoint (socket hang up) and it never comes back. Reproduced twice on 2026-07-10.

(b) Event-loop degradation under load — no file edits, no watcher. Reproduced on GitHub Actions ubuntu-latest with adk dev --non-interactive --no-watch (so hot reload is ruled out): the dev server's NDJSON log shows escalating event-loop warnings (backend <dir> blocked for 5577ms, then near-continuous blocks of 0.5–9 s for five minutes) until ~9 minutes into the suite the backend stops answering entirely — no fatal error, no exit line in the log; the process just goes dark to clients and evals cascade with Unable to connect. Locally the same mode was triggered intermittently by a single eval whose turn fires two concurrent alert-email actions: backend blocked for 56784ms → Lost connection to DevConsole → reconnect failures forever (2 of 3 runs).

Expected: the trace endpoint survives a rebuild (or comes back after one), and the dev backend either keeps serving under eval load or fails loudly. A run-killing silent hang is the worst case for CI: the gate can't distinguish a dead server from a broken bot.

Repro (mode b, the CI-relevant one):

1. A bot with a non-trivial eval suite (ours: ~99 evals, multi-turn, LLM judges).
2. adk dev --non-interactive --no-watch (fresh boot, no file edits afterwards).
3. adk evals --format json > report.json.
4. Watch the dev server log for event-loop … blocked for warnings; mid-suite, evals start failing with Unable to connect against devServerUrl and never recover.

Mode (a): start any eval run, touch a watched source file mid-run, observe Changes detected, rebuilding → socket hang up → every subsequent eval fails.

Workaround: restart adk dev before every eval run; never edit files mid-run; and/or run the suite in tag-sized chunks (< ~5 min each) against a fresh server per chunk, merging the JSON reports.

[johnf]

Re-measured both halves of this report on adk 2.0.3. Neither reproduces as filed, and the real defect is narrower — retitling rather than closing, since one genuine problem remains.

Half 1 — "dies under sustained eval load": does not reproduce

104 evals unchunked against a single adk dev, 1605 s, zero runner errors, both ports still listening and the process alive at the end. Failures by run-order quartile were 6/11/4/1, so there was no gradual degradation either — the last quartile was the cleanest.

The original observation was on 2.0.1 against a bot whose turn path we have since re-engineered, so I can't say whether it was ever a CLI fault or ours. It isn't reproducible now, and we have deleted the chunked-runner workaround it justified.

Half 2 — "hot-reload rebuild also permanently drops it": also does not reproduce

Deliberate experiment: server booted with watching, a 5-eval run started, and a source file edited 18 s in, mid-run.

The trace listener never dropped. Sampling both ports once a second across the whole run:

16:56:15  3000=up   3001=up     (baseline, 9 samples)
16:56:24  3000=down 3001=up     (one sample — the bot restarting)
16:56:25  3000=up   3001=up     (recovered, 25 samples to the end)

Port 3000 bounced for ~1 s; port 3001 was up continuously. The opposite way round from the report. The run continued and the server was alive at the end.

What actually goes wrong

The eval in flight during the rebuild dies, and only that one:

PASS products-heatpump-sizing     8.6s
PASS kill-switch-redirect         7.9s
FAIL pricing-no-model-figure-sbr  31.0s  ERROR: waitForTurnComplete timed out after 30000ms
                                         — no handler.conversation span completed; totalSpans=0
PASS safety-selfharm              6.0s   ← after the rebuild
PASS comp-hopeless-waiting        7.5s   ← after the rebuild

Timing is exact: the edit landed at 16:56:20.8 and that eval started ~16:56:20.

The server knows precisely what happened and says so:

{"ts":"...T14:56:20.859Z","event":"reload","message":"Agent project reloaded"}
{"ts":"...T14:56:22.098Z","event":"log","message":"Changes detected, rebuilding"}
{"ts":"...T14:56:24.161Z","level":"error","event":"request","fatal":false,
 "message":"A request reached the dev bot while it was rebuilding/restarting (POST /) and failed;
             it will succeed on retry."}

it will succeed on retry — but adk evals does not retry. It blocks for the full 30 s idle timeout waiting for a span that cannot arrive, then reports totalSpans=0, which is indistinguishable from #36. Nothing in that error tells you a rebuild happened, even though the server logged a non-fatal, explicitly retryable failure two seconds earlier.

Suggested fix

Have the eval runner treat a request that failed against a rebuilding/restarting dev bot as retryable, as the server's own message says it is — or at minimum surface the rebuild in the error, so a 30 s totalSpans=0 timeout distinguishes "the server was rebuilding" from "spans never export" (#36).

Close this if you consider losing an in-flight turn on rebuild to be by design; the retry and the error message are the parts I would still call bugs.