Run the Vally project-plan eval on MSBench against a real extension build - #1689
Open
Alex Weininger (alexweininger) wants to merge 3 commits into
Open
Conversation
Alex Weininger (alexweininger)
changed the base branch from
main
to
meganmott/happy-hedgehog
August 24, 2026 19:51
Copilot started reviewing on behalf of
Alex Weininger (alexweininger)
August 24, 2026 19:53
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an MSBench evaluation path that tests the shipped extension through a real VSIX, with documentation and manual CI execution.
Changes:
- Builds, validates, stages, and submits the extension evaluation.
- Adds benchmark configuration and VSIX-derived agent assets.
- Documents local and CI workflows.
- Adds a manual GitHub Actions workflow.
Show a summary per file
| File | Summary | Final review comments |
|---|---|---|
evals/msbench/run.sh |
Builds, stages, and submits the VSIX evaluation. | Moderate (3 votes): --skip-build may use the wrong VSIX, and an empty build glob may retain an older artifact. |
evals/msbench/README.md |
Documents local usage, CI setup, and troubleshooting. | Nit (3 votes): Documents Azure CLI and Node but omits Python 3.10+ and unzip prerequisites. |
evals/msbench/assets/user-overrides.yaml |
Defines MSBench configuration and assertions. | No final comments. |
evals/msbench/assets/extensions/.gitkeep |
Preserves the VSIX staging directory. | No final comments. |
.github/workflows/msbench-evals.yml |
Provides manual CI execution and artifact upload. | No final comments. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Suppressed comments (3)
evals/msbench/README.md:25
- The rerun command has the same working-directory problem as the quick start: from the repository root,
./run.sh --skip-builddoes not resolve to this file. Use the repository-relative path or add an explicitcd evals/msbenchinstruction.
rebuilding the extension using `./run.sh --skip-build`.
evals/msbench/assets/user-overrides.yaml:36
- This comment still attributes the setting to the Vally executor, but this YAML is used by the MSBench harness. That is misleading when diagnosing why tool confirmations are auto-approved; refer to the MSBench harness instead.
evals/msbench/assets/user-overrides.yaml:16 - This comment gives a command that only works after changing into
evals/msbench; from the repository root (the documented invocation context)./run.shis not found. Keep this reference consistent with the public command,./evals/msbench/run.sh.
- Files reviewed: 5/5 changed files
- Comments generated: 2
- Review effort level: Lite
Comment on lines
+27
to
+29
| `run.sh` is self-contained: it provisions a Python 3.10+ virtualenv, installs | ||
| `msbench-cli` plus the `vscode` special agent from the internal feed, builds and | ||
| stages the VSIX, and submits. The only prerequisites are Azure CLI and Node. |
Comment on lines
+104
to
+109
| BUILT_VSIX="$(ls -t "${REPO_ROOT}"/vscode-azureresourcegroups-*.vsix 2>/dev/null | head -1 || true)" | ||
| if [ -n "$BUILT_VSIX" ]; then | ||
| mkdir -p "$(dirname "$VSIX_DEST")" | ||
| cp "$BUILT_VSIX" "$VSIX_DEST" | ||
| fi | ||
| [ -f "$VSIX_DEST" ] || die "No VSIX at ${VSIX_DEST}. Run without --skip-build." |
Alex Weininger (alexweininger)
force-pushed
the
alexweininger-msbench-project-plan-eval
branch
2 times, most recently
from
August 25, 2026 02:17
68e3e92 to
e756bcc
Compare
Alex Weininger (alexweininger)
changed the base branch from
meganmott/happy-hedgehog
to
feat/CoR
August 25, 2026 02:17
The Vally suite exercises the azure-project-plan agent through the Copilot SDK with an MCP stand-in for the extension's tools, so it can verify the agent's contracts but never the code that ships. This runs the same seven contracts inside real VS Code with the extension installed from a VSIX. The extension registers an in-process MCP server exposing open_requirements_view and friends, which are the exact tool names the Vally tool-calls graders assert on, so the graders port across unchanged while being satisfied by the shipping code path rather than a test double. Avoids publishing a benchmark instance. user-overrides.yaml is the last of three config layers and merging is a shallow Object.assign, so our promptSteps wholly replace those of whichever instance supplies the container image. That means no Docker image to publish and no PR into vscode-copilot-evaluation. The agent definition is unpacked from the VSIX rather than cloned from GitHub, so the instructions cannot drift from the build under test. run.sh provisions the virtualenv, installs msbench-cli and the vscode special agent, builds and stages the VSIX, and submits, so the only prerequisites are Azure CLI and Node. It also fails fast on the three non-obvious ways this breaks: special agents are discovered via PATH rather than imports, vsce package ships no dist/ without a prior npm run build, and unzip piped to grep -q trips pipefail with SIGPIPE. Verified end to end from a clean virtualenv: run 2026082468156047, 7/7 assertions passed, resolved: true. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 617f531f-88d8-4179-8fb5-a0c6ec52ff92
Two jobs. `build` packages the VSIX and runs the same guards run.sh uses locally; it needs no credentials, so it runs on PRs and catches the failure that has cost the most time here — a VSIX missing dist/extension.bundle.js — without spending an MSBench run. `eval` submits to MSBench and is workflow_dispatch only, because CES identifies callers by Entra client id and that identity has to be allowlisted by the MSBench team first. Scheduling it before then would only ever be red. Move the build ahead of the Azure preflight in run.sh and add --build-only. A build break is the likeliest failure and diagnosing it should not require credentials, nor should a token sit around for the several minutes a cold build takes. Also filter --skip-build out of the arguments rather than assuming it is $1, so the workflow can forward --output and --data_dir. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 617f531f-88d8-4179-8fb5-a0c6ec52ff92
Adds run-analysis URLs so the verified runs are actually reachable instead of being bare IDs, and points the README at photo-app-requirements in evals/project-plan/eval.yaml as the source of truth for the ported contract. Documents the RATE_LIMIT failure mode: back-to-back runs get throttled by the Copilot API mid-run. Because the agent then produces nothing, this surfaces as failed artifact assertions while the negative assertions pass trivially, so it reads like a behavioral regression and is not one. Cost two misdiagnosed runs. Also drops cor-requirements.vscode.agent.yaml, the earlier draft that assets/user-overrides.yaml replaces. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 617f531f-88d8-4179-8fb5-a0c6ec52ff92
Alex Weininger (alexweininger)
force-pushed
the
alexweininger-msbench-project-plan-eval
branch
from
August 25, 2026 02:43
e756bcc to
d3b8b9b
Compare
Alex Weininger (alexweininger)
changed the base branch from
feat/CoR
to
meganmott/happy-hedgehog
August 25, 2026 02:43
Alex Weininger (alexweininger)
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1683 — base is
meganmott/happy-hedgehog, so the diff here is just the MSBench work.This takes the Vally contracts added in #1683 and runs them on MSBench against a real build of the extension.
What this does
Runs the
project-planrequirements eval on MSBench against a real VSIX build of this extension, so the agent under test is the shipping product — its actual MCP tools, its actualresources/agents/instructions — not a stand-in.That is the whole interface. It provisions a venv, mints the feed token from your existing
az login, builds and stages the VSIX, and submits the run. No prerequisites beyond the self-serviceMSBench Userrole.Verified
All 7 assertions passed,
resolved: true, on every run below — including one against this PR's actual base,feat/CoR:20260824675247592026082468156047202608247109577820260824786369532026082479418416feat/CoR2026082509500207meganmott/happy-hedgehog(Links need Corpnet/Azure VPN.)
It is not passing vacuously. The recorded tool call is
mcp_copilot_azure_open_requirements_view— the extension's real in-process MCP tool — andpatch.diffcontains a 236-line.azure/requirements.jsonwith the correct schema.Scope, stated honestly: this ports 1 of the 7 stimuli in
evals/project-plan/eval.yaml(photo-app-requirements— its six graders plus thereject_toolsconstraint become seven assertions). It proves the harness end to end, not suite coverage. The other six are the immediate follow-up.No new benchmark instance
The cheap part.
run-agent.shmerges three config layers with user-overrides last, andmergeTestConfigsis a shallowObject.assign, sopromptStepsandassertionsare replaced wholesale whileinstallExtensionsconcatenates. So we borrowvscbench.say_hellopurely for its container image and override the test. No Docker image to publish, no dataset row, no PR intovscode-copilot-evaluation.Agent instructions are unzipped from the VSIX we just installed rather than
git cloned, so they cannot drift from the build under test.About the CI half — please read
The workflow is split into two jobs by what needs credentials:
build— runs on every PR, no credentials, builds the VSIX and runs both guards. Green on this PR.eval—workflow_dispatchonly, and it will not work until one-time setup lands. I did not want to quietly ship something that cannot authenticate.MSBench runs on CES, which identifies callers by Entra client id, so CI needs a real Azure identity. Two tokens are involved, not one: the ADO feed token (minted by
run.sh) and the CES token (minted internally bymsbench-cli). Both are satisfied by anything that authenticates the az CLI, which is whyazure/login+ OIDC is the right shape.Outstanding, tracked, none of it blocking this merge:
AzCodeFeedAuthMI— currently blocked by a subscription-wideReadOnlylock.msbenchGitHub environment — needs repo admin.Scheduling the eval job before those land would just produce a permanently red check. Details in the README's Running in CI section.
Gotchas worth knowing (each cost a failed run, all now fail fast)
npm run packageis onlyvsce packageand there is novscode:prepublishhook, so without a priornpm run buildthe VSIX ships withoutdist/extension.bundle.js. The extension host then silently fails to activate, which remotely looks like every assertion failing for no reason. Good VSIX ≈ 7 MB, broken ≈ 1 MB.PATH, so callingvenv/bin/msbench-cliby absolute path reports every plugin as "not installed".unzip -l | grep -qunderset -o pipefailreturns 141 (SIGPIPE).output/error.jsonfor"type": "RATE_LIMIT"before believing a red result, and leave a few minutes between runs.Next
Remaining three single-turn stimuli plus the real
programvalidators asexec:assertions, then the multi-turn ones viapromptSteps, then expand gates and graders.vally-evals.ymlstays as the fast PR gate; this is the slower, higher-fidelity one.Relationship to #1683
This also deletes
evals/msbench/cor-requirements.vscode.agent.yaml, the earlier draft in #1683 thatassets/user-overrides.yamlsupersedes.evals/project-plan/eval.yamlremains the source of truth. This port is a hand-translation of one stimulus, so changes there need mirroring here until the remaining six are wired up — worth knowing before they drift.