Add the agent eval harness - #5
Merged
Merged
Conversation
odh-cli has had one since v0.3.0; pitstop had no evals directory at all, which is why the mcp 2.0.0 breakage was found by hand rather than by CI. Same layout, filenames, JSON field names and "ok - " output style as the sibling repo, so the two read alike. python3 replaces jq. An eval suite for a stdlib-only runtime should not require installing a tool to run it; assert.py takes the same (label, file, expression) shape jq -e had. 31 live assertions over the contracts an agent actually depends on: the envelope's provenance and quality accounting, that median_basis is on every price, that outlier appears only when true, that an unscreened price carries no median to compare against, GeoJSON [lon, lat] ordering, and that the table path names its source too. Split across the two workflows, following the rule odh states in docs/evaluation.md: the live half is a weekly canary because a contributor can do nothing about MIMIT or Overpass being down. What runs per-PR is the offline half — every command string in recipes.json and tasks.json is parsed against the shipped argparse parser, so a recipe can never name a command the CLI cannot run. odh shipped exactly that bug for two releases. Overpass reuses the canary's policy rather than inventing a second one: the same retry ladder, and an empty answer is a warning while a dishonest envelope is a failure. Empty is classified on count alone, since Overpass also sheds load with a valid 200 carrying no elements and no error at all. 403 joins 429 as upstream's fault — it is how overpass-api.de blocks a shared client IP, not a bad query.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
odh-cli has had an agent eval harness since v0.3.0; pitstop had no
evals/directory at all. That gap is why themcp2.0.0 breakage was found by reading rather than by CI.Same layout, filenames, JSON field names and
ok -output style as the sibling repo, so the two read alike.python3replacesjq— an eval suite for a stdlib-only runtime shouldn't require installing a tool to run it.What it checks
31 live assertions over the contracts an agent actually depends on: envelope provenance, the
qualityblock's screened/unscreened accounting being self-consistent with the prices returned,median_basison every price,outlierpresent only when true, an unscreened price carrying no median to compare against, GeoJSON[lon, lat]ordering, bilingual comune resolution, and the table path naming its own source.Where it runs, and why it's split
Per odh's stated rule in
docs/evaluation.md, the live half is a weekly canary — a contributor can do nothing about MIMIT or Overpass being down, so it must not block a PR.What runs per-PR is the offline half: every command string in
recipes.jsonandtasks.jsonis parsed against the shipped argparse parser, so a recipe can never name a command the CLI cannot run. odh shipped exactly that bug for two releases, so the guard is in from day one.Overpass
Reuses the canary's policy rather than inventing a second one — same retry ladder, empty answer warns, dishonest envelope fails. Two corrections found in review:
countalone. Overpass also sheds load with a valid HTTP 200 carrying no elements and noremark, which leaveserrorunset — that was classifying as usable and running the data assertions against nothing.403joins429as upstream's fault. It's howoverpass-api.deblocks a shared client IP, not a rejected query.Review corrections worth noting
--fuelsubstring caveat only warned in one direction. Verified live:--fuel Gasolioalso returns Gasolio Premium, speciale, Alpino and artico — 8 distinct names in BZ alone, so a premium blend gets ranked against ordinary diesel.coordinate_suspectdefinition was wrong: the comparison is against the median of the comune's own stations, with the ISTAT reference point only as fallback.