Record ordinary judge call provenance - #2136
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review Unable to check for correctness in cebafba. This PR refactors judge call provenance with new typed structures and a schema version bump. An unresolved review comment identifies a bug where structured output calls are recorded without their You can customize Macroscope's approvability policy. Learn more. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e90dc26940
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0fb9a3e337
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0dbd0844d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8e9cfa58d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf9b15b471
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c3400c469
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f451009546
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb637f402e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7ae41181d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
9193c6b to
3bc3d2b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d53c20d9b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cebafba. Configure here.
1 participant
Overview
Make ordinary LLM judges more robust and observable with compact typed evidence while retaining the existing SDK client behavior.
Details
Trace.judge_calls; each record carries judge identity, response/refusal/parser evidence, and one existingModelCallfor model, sampling, endpoint, timing, usage, and provider errors.ProviderErrorpath.ModelCall.samplingfor exact replayability and parity with agentic judges.Note
Record judge call provenance in traces with structured
JudgeCallmodelJudgeCallin trace.py to record each judge invocation with a nestedModelCall(timing, usage, endpoint, sampling) plus optional text, refusal, and error fields; appended totrace.judge_calls.Judge.completein judge.py to build and populate aJudgeCallper invocation, capturing provider errors viaError.from_exceptionand raising on refusals, content filter terminations, and truncated structured outputs.extra_usagefromTraceand bumpsTRACE_VERSIONto 5; dashboard, replay, and push logic updated to read fromjudge_callsinstead.JudgeCallfrom the top-levelverifiers.v1package.judge_callsand dropextra_usage; existing trace consumers expecting version 4 schema will need updating.Macroscope summarized cebafba.
Note
Medium Risk
Breaking trace schema (v5, dropped
extra_usage) and changed judge HTTP/structured-output behavior affect scoring, serialization, and any consumer of old judge metadata; core paths are covered by updated unit and e2e tests.Overview
Judge LLM usage and outcomes move from opaque
trace.info["judge"]blobs plus a separateextra_usagelist to first-classtrace.judge_calls: each entry names the judge class, nests a normalModelCall(model, sampling, endpoint, timing, usage, provider errors), and carries response text, refusal, and parse/validation errors without storing full prompts on the trace.Judge.completenow goes through the standard chat completions path (including optionalresponse_format), appends aJudgeCallbefore the provider call, mapsOpenAIErrorvia the existing provider-error path, and treats refusals, content filters, and length-truncated structured output as judge failures recorded on the call.RubricJudgeroutes structured and plain JSON grading through that singlecomplete+parseflow.TRACE_VERSIONbumps to 5 (extra_usageremoved). Replay clearsjudge_callswhen re-scoring, skips any trace withokfalse, the eval dashboard sums judge tokens/cost from nested usage, and platform push copies judge calls into sampleinfo.Error.from_exceptioncentralizes error capture for interceptionModelCallrecords and tracecapture_error.Reviewed by Cursor Bugbot for commit cebafba. Bugbot is set up for automated code reviews on this repo. Configure here.