evaluate: 64 evaluation(s) (run 20261006T192524Z)

[fedcourtsai-dev]

Automated evaluations for run 20261006T192524Z.

predictor	case	event
claude-judge	scotus/73281687	evt-petition-disposition
claude-judge	scotus/73281693	evt-petition-disposition
claude-judge	scotus/73281694	evt-petition-disposition
claude-judge	scotus/73281699	evt-petition-disposition
claude-judge	scotus/73281702	evt-petition-disposition
claude-judge	scotus/73281703	evt-petition-disposition
claude-judge	scotus/73286453	evt-petition-disposition
claude-judge	scotus/73287447	evt-petition-disposition
claude-judge	scotus/73291758	evt-petition-disposition
claude-judge	scotus/73292884	evt-petition-disposition
claude-judge	scotus/73292885	evt-petition-disposition
claude-judge	scotus/73299074	evt-petition-disposition
claude-judge	scotus/73300246	evt-petition-disposition
claude-judge	scotus/73302615	evt-petition-disposition
claude-judge	scotus/73303792	evt-petition-disposition
claude-judge	scotus/73309407	evt-petition-disposition
claude-judge	scotus/73318133	evt-petition-disposition
claude-judge	scotus/73318742	evt-petition-disposition
claude-judge	scotus/73329541	evt-petition-disposition
claude-judge	scotus/73331499	evt-petition-disposition
claude-judge	scotus/73335108	evt-petition-disposition
codex-judge	scotus/73281687	evt-petition-disposition
codex-judge	scotus/73281693	evt-petition-disposition
codex-judge	scotus/73281694	evt-petition-disposition
codex-judge	scotus/73281699	evt-petition-disposition
codex-judge	scotus/73281702	evt-petition-disposition
codex-judge	scotus/73281703	evt-petition-disposition
codex-judge	scotus/73286453	evt-petition-disposition
codex-judge	scotus/73287447	evt-petition-disposition
codex-judge	scotus/73291758	evt-petition-disposition
codex-judge	scotus/73292884	evt-petition-disposition
codex-judge	scotus/73292885	evt-petition-disposition
codex-judge	scotus/73299074	evt-petition-disposition
codex-judge	scotus/73300246	evt-petition-disposition
codex-judge	scotus/73302615	evt-petition-disposition
codex-judge	scotus/73303792	evt-petition-disposition
codex-judge	scotus/73318133	evt-petition-disposition
codex-judge	scotus/73318742	evt-petition-disposition
codex-judge	scotus/73329541	evt-petition-disposition
codex-judge	scotus/73331499	evt-petition-disposition
codex-judge	scotus/73335108	evt-petition-disposition
gemini-judge	scotus/73281318	evt-petition-disposition
gemini-judge	scotus/73281388	evt-petition-disposition
gemini-judge	scotus/73281624	evt-petition-disposition
gemini-judge	scotus/73281656	evt-petition-disposition
gemini-judge	scotus/73281687	evt-petition-disposition
gemini-judge	scotus/73281693	evt-petition-disposition
gemini-judge	scotus/73281694	evt-petition-disposition
gemini-judge	scotus/73281699	evt-petition-disposition
gemini-judge	scotus/73281702	evt-petition-disposition
gemini-judge	scotus/73281703	evt-petition-disposition
gemini-judge	scotus/73286453	evt-petition-disposition
gemini-judge	scotus/73291758	evt-petition-disposition
gemini-judge	scotus/73292884	evt-petition-disposition
gemini-judge	scotus/73292885	evt-petition-disposition
gemini-judge	scotus/73299074	evt-petition-disposition
gemini-judge	scotus/73302615	evt-petition-disposition
gemini-judge	scotus/73303792	evt-petition-disposition
gemini-judge	scotus/73309407	evt-petition-disposition
gemini-judge	scotus/73318133	evt-petition-disposition
gemini-judge	scotus/73318742	evt-petition-disposition
gemini-judge	scotus/73329541	evt-petition-disposition
gemini-judge	scotus/73331499	evt-petition-disposition
gemini-judge	scotus/73335108	evt-petition-disposition

⚠️ 1 queued cell(s) uploaded nothing at all — no artifact and no status, so the cell died before it could report. Re-running collect will not recover these; they need a re-derivation, which the next scheduled round does on its own.

• codex-judge on scotus/73309407 evt-petition-disposition

🚩 Agent flags (13)

Structured notes surfaced against this run's cells, for triage — the agents' own, plus any the harness raised.

severity	category	actor	case	event	note
⚠️ warning	other	claude-judge	scotus/73281699	evt-petition-disposition	gemini-baseline's harness-captured retrieval log (call 32) shows a shell call git restore data/qp-topics/qp-topic-reference.json during its predict run. The predict prompt forbids any touch of data/qp-topics/, and AGENTS.md forbids re-materializing anything under data/ with git restore. The call is a write-back rather than a read, its result is unobserved (the log's capture coverage is 0.0), the case was still pending on 2026-09-17 so the file could not encode this petition's outcome, and the candidate's prose shows no trace of it; I therefore graded leakage not_applicable with retrieved_outcome_material false. Surfaced so a maintainer can check why a predictor cell found that file modified and whether the predict harness should refuse the path.
⚠️ warning	scope	codex-judge	scotus/73281699	evt-petition-disposition	gemini-baseline's harness log includes an attempted git restore of the prohibited labeling reference artifact qp-topic-reference.json. The call is unobserved, as are all 34 calls in that log; neither successful restoration nor a subsequent content read is established. No outcome-revealing fact appears in the reasoning, and the prediction predates the supplied resolution. Recorded as an out-of-scope artifact-restoration concern, not proven outcome exposure or a leakage exclusion. The evaluator did not access the labeling artifact. See gemini-baseline's evaluation.md.
⚠️ warning	scope	codex-judge	scotus/73299074	evt-petition-disposition	The provisioned event is stage cert, but the snapshot and question presented identify an original mandamus petition seeking enforcement of a prior mandate. All three candidates recognize the writ form. I scored codex-baseline, gemini-baseline, and claude-baseline on the authoritative denied/actual_granted=0 outcome and retained the contract-required sal-v4 baseline risk-set comparator. Its pooled cert rate is not an empirical mandamus baseline, so the recorded skill values have a population-transport limitation. No event, outcome, or prediction was relabeled or changed.
⚠️ warning	data-quality	codex-judge	scotus/73331499	evt-petition-disposition	Question 2 ends incompletely with 'the creation of DNA profiles from.' in both the supplied questions-presented text and petition text. This evaluation does not reconstruct the missing wording or assume whether the defect originated in the filing or its text representation. claude-baseline and codex-baseline identify the limitation; all evaluations remain scoreable against the explicit denial outcome.
info	data-quality	claude-judge	scotus/73281699	evt-petition-disposition	Band-sourcing observation on the scored baseline: the frozen elevated band under sal-v4 rests on distribution_count 2, but the docket shows the first distribution (for the 5/21 conference) was overtaken by the 5/12 response request and the second (for 9/28) is the routine post-response redistribution, so no relist in the ordinary sense occurred, while the response request itself is a signal the band does not read. claude-baseline raised the same point in its reasoning. The risk_set rate was applied as the contract requires; this is a note on what the band encodes, not a scoring deviation.
info	data-quality	claude-judge	scotus/73292885	evt-petition-disposition	CourtListener dockets/73292885 returns docket_number 25-1249 (matching the provisioned supremecourt.gov snapshot, Pestarino v. Pestarino) but case_name 'Roy J. Meidinger, Petitioner v. Commissioner of Internal Revenue'. The corpus id-to-docket-number link is sound; the caption mismatch is upstream in CourtListener. gemini-baseline's predictor retrieval note reported the same mismatch; this evaluator's MCP call on 2026-10-06 reproduced it. Noted so a case-name-keyed join or display does not mislabel this case.
info	scope	claude-judge	scotus/73299074	evt-petition-disposition	This cell is a Rule 20 petition for a writ of mandamus (docket text: 'Petition for a writ of mandamus filed'), minted as a cert-stage petition-disposition event with band 'baseline' under sal-v4. I scored it per the cert rules (risk_set basis, pooled reached rate 0.0512 over OT2017-OT2024), but that rate describes a cert population the petition does not belong to; mandamus petitions are denied at a far higher rate than baseline-band cert petitions, so the skill scores on this cell overstate skill for any candidate that simply priced the writ form. Two of the three candidates flagged the same mismatch in their own run. Consider whether extraordinary-writ petitions should carry a stage or population of their own.
info	other	claude-judge	scotus/73303792	evt-petition-disposition	Not leakage, noted for the predict-prompt maintainers: claude-baseline's captured log shows one shell call that globbed data/cases/scotus/*/events/evt-petition-disposition/predictions//20260916T170237Z, took the first match, and read that cell's prediction.json, tooling.json and retrieval.md as a format template. That is another case's committed prediction output from an earlier run, carrying nothing about this petition's outcome, and the candidate's own prose and retrieval note do not mention it. Graded clean. If reading the committed predictions tree is meant to be off-limits for predictors (as it is for evaluators), the predict prompt does not currently say so.
info	data-quality	codex-judge	scotus/73292885	evt-petition-disposition	gemini-baseline reports that a CourtListener lookup for docket 73292885 returned an unrelated Meidinger tax case rather than the provisioned Pestarino case. Its staged log confirms the lookup query but marks the response unobserved, so this is an unverified identifier-mismatch report, not a confirmed mapping error. Evaluation uses the supplied Pestarino outcome only; no external re-query or outcome substitution was made. Maintainer follow-up may be warranted.
info	scope	codex-judge	scotus/73303792	evt-petition-disposition	claude-baseline's captured September 16, 2026 shell query at 20:47:01.926Z selected an earlier prediction directory through a case wildcard and attempted to read prediction.json, tooling.json, and retrieval.md. This access is not described in the candidate's retrieval note. The staged log carries only a result digest/status, so the returned content and any influence cannot be determined. I did not follow the path or infer identity. Flagged for retrieval-scope review, not as evidence of target-outcome leakage: this event resolved October 5 and the outcome-leakage bit remains false.
info	data-quality	codex-judge	scotus/73318742	evt-petition-disposition	Candidate gemini-baseline's retrieval.md omits a corpus-query attempt present in its harness log: uv run fedcourts query --court scotus --decided-before 2026-09-17. The note says no retrieval beyond provisioned inputs and the statpack. The call's result is unobserved, so success, failure, and transfer volume cannot be established. This is a retrieval-disclosure discrepancy, not evidence of target-outcome leakage: the call was made September 16, before the October 5 denial.
info	data-quality	codex-judge	scotus/73331499	evt-petition-disposition	gemini-baseline's retrieval.md reports no retrieval beyond local inputs and the statpack, but its harness log records two broad corpus-query attempts at zero-based calls 12 and 13 (direct and uv-run forms, SCOTUS with era roberts). Both results are unobserved, so success, failure, and returned content cannot be established. The discrepancy is an incomplete retrieval note, not a finding of outcome retrieval; leakage remains not_applicable for the forward prediction. Zero result-capture coverage itself is not flagged as a defect.
info	data-quality	gemini-judge	scotus/73299074	evt-petition-disposition	A mandamus petition was provisioned as a cert-stage event with a baseline salience band. The base rates for regular cert petitions do not accurately reflect the grant probability of a mandamus petition.