evaluate: 48 evaluation(s) (run 20261007T150255Z)

[fedcourtsai-dev]

Automated evaluations for run 20261007T150255Z.

predictor	case	event
claude-judge	scotus/73344703	evt-petition-disposition
claude-judge	scotus/73351824	evt-petition-disposition
claude-judge	scotus/73500223	evt-petition-disposition
claude-judge	scotus/73500229	evt-petition-disposition
claude-judge	scotus/73500230	evt-petition-disposition
claude-judge	scotus/73500231	evt-petition-disposition
claude-judge	scotus/73500232	evt-petition-disposition
claude-judge	scotus/73500233	evt-petition-disposition
claude-judge	scotus/73500236	evt-petition-disposition
claude-judge	scotus/73500237	evt-petition-disposition
claude-judge	scotus/73500238	evt-petition-disposition
claude-judge	scotus/73500239	evt-petition-disposition
claude-judge	scotus/73500241	evt-petition-disposition
claude-judge	scotus/73500242	evt-petition-disposition
claude-judge	scotus/73500245	evt-petition-disposition
claude-judge	scotus/73500248	evt-petition-disposition
claude-judge	scotus/73500263	evt-petition-disposition
claude-judge	scotus/73500287	evt-petition-disposition
codex-judge	scotus/73309407	evt-petition-disposition
codex-judge	scotus/73344703	evt-petition-disposition
codex-judge	scotus/73351824	evt-petition-disposition
codex-judge	scotus/73500223	evt-petition-disposition
codex-judge	scotus/73500229	evt-petition-disposition
codex-judge	scotus/73500230	evt-petition-disposition
codex-judge	scotus/73500231	evt-petition-disposition
codex-judge	scotus/73500232	evt-petition-disposition
codex-judge	scotus/73500233	evt-petition-disposition
codex-judge	scotus/73500236	evt-petition-disposition
codex-judge	scotus/73500237	evt-petition-disposition
codex-judge	scotus/73500238	evt-petition-disposition
codex-judge	scotus/73500239	evt-petition-disposition
codex-judge	scotus/73500241	evt-petition-disposition
codex-judge	scotus/73500242	evt-petition-disposition
codex-judge	scotus/73500245	evt-petition-disposition
codex-judge	scotus/73500248	evt-petition-disposition
codex-judge	scotus/73500263	evt-petition-disposition
codex-judge	scotus/73500287	evt-petition-disposition
gemini-judge	scotus/73287447	evt-petition-disposition
gemini-judge	scotus/73300246	evt-petition-disposition
gemini-judge	scotus/73344703	evt-petition-disposition
gemini-judge	scotus/73351824	evt-petition-disposition
gemini-judge	scotus/73369988	evt-petition-disposition
gemini-judge	scotus/73452191	evt-petition-disposition
gemini-judge	scotus/73500219	evt-petition-disposition
gemini-judge	scotus/73500220	evt-petition-disposition
gemini-judge	scotus/73500223	evt-petition-disposition
gemini-judge	scotus/73500229	evt-petition-disposition
gemini-judge	scotus/73500230	evt-petition-disposition

⚠️ 13 queued cell(s) uploaded nothing at all — no artifact and no status, so the cell died before it could report. Re-running collect will not recover these; they need a re-derivation, which the next scheduled round does on its own.

• gemini-judge on scotus/73500231 evt-petition-disposition
• gemini-judge on scotus/73500232 evt-petition-disposition
• gemini-judge on scotus/73500233 evt-petition-disposition
• gemini-judge on scotus/73500236 evt-petition-disposition
• gemini-judge on scotus/73500237 evt-petition-disposition
• gemini-judge on scotus/73500238 evt-petition-disposition
• gemini-judge on scotus/73500239 evt-petition-disposition
• gemini-judge on scotus/73500241 evt-petition-disposition
• gemini-judge on scotus/73500242 evt-petition-disposition
• gemini-judge on scotus/73500245 evt-petition-disposition
• gemini-judge on scotus/73500248 evt-petition-disposition
• gemini-judge on scotus/73500263 evt-petition-disposition
• gemini-judge on scotus/73500287 evt-petition-disposition

🚩 Agent flags (16)

Structured notes surfaced against this run's cells, for triage — the agents' own, plus any the harness raised.

severity	category	actor	case	event	note
⚠️ warning	data-quality	codex-judge	scotus/73500229	evt-petition-disposition	gemini-baseline's reasoning says documents.json confirmed petition, BIOs, and QPs and that these were reviewed. The provisioned manifest lists only petition and QP text fetched July 18, before the August 24 BIOs. Its blinded log contains petition reads and general research but no identifiable BIO retrieval. The manifest attribution is unsupported and claimed BIO review is uncorroborated. All results are unobserved, so this is not proof of what general searches returned and not an outcome-leakage finding. See gemini-baseline's evaluation.md.
⚠️ warning	data-quality	codex-judge	scotus/73500231	evt-petition-disposition	The provisioned record/documents/questions-presented.txt ends mid-question 7 at 'pursuant to 5 U.S.' and omits question 8, although documents.json declares truncated=false. The full petition.txt contains both the continuation and question 8. codex-baseline expressly recovered the missing text; claude-baseline reports reading all eight questions; gemini-baseline's logged reads name the extract but not the full petition. This shared-input coverage defect does not block evaluation; no input files were changed.
⚠️ warning	data-quality	codex-judge	scotus/73500233	evt-petition-disposition	gemini-baseline's retrieval.md states no retrieval beyond provisioned inputs, but its harness log records two corpus-query attempts for granted other-acts cases (calls 12 and 13). Both results are unobserved, so successful retrieval, failure, and returned content cannot be established. The disclosure should record the attempts and uncertainty. This is not evidence of this petition's outcome leaking: the calls occurred before resolution and did not target its disposition; leakage_suspected remains false.
⚠️ warning	data-quality	codex-judge	scotus/73500236	evt-petition-disposition	gemini-baseline describes 2021–2025 as its prior-Term baseline, but its frozen context.term is 2025. The stated anchor includes its own Term and omits displayed eligible earlier Terms. Evaluation uses the matching sal-v4 baseline risk set over 2017–2024 (rendered weighted rate 0.05120250431778929). This lowers confidence in the rationale, not the mechanical outcome score; no own-case outcome retrieval is evidenced in this forward cell. See gemini-baseline evaluation.md.
info	data-quality	claude-judge	scotus/73500231	evt-petition-disposition	record/documents/questions-presented.txt ends mid-sentence inside question 7 ('...if I allege discrimination pursuant to 5 U.S.') and carries no question 8, while its documents.json manifest entry records truncated=false and pages=0. The full petition.txt carries all eight questions, so predictors could recover them (one candidate reports doing so), but the extract's manifest overstates its completeness. Not a scoring problem for this cell.
info	data-quality	claude-judge	scotus/73500232	evt-petition-disposition	gemini-baseline's retrieval.md states 'No corpus lookups executed', while its harness-captured retrieval_log.json carries one shell call invoking fedcourts query (Rule 4(a)(5) priors, --decided-before 2026-09-17), result unobserved (the engine's capture coverage is 0.0). The self-report and the captured log disagree; most likely a failed call reported as none. No leakage bearing (the query is topical and date-bounded), but the mismatch is worth knowing when weighing that engine's self-reported retrieval notes against capture.
info	other	claude-judge	scotus/73500233	evt-petition-disposition	gemini-baseline's retrieval.md states 'No retrieval beyond the provisioned inputs' and its reasoning says no web retrieval was performed, but its harness-captured retrieval_log.json carries two shell rows invoking fedcourts query --disposition granted with a topical phrase (results unobserved, capture coverage 0.0). The queries sought prior granted cases, not this docket, so this is a disclosure gap and not leakage; noted so the predictor-side retrieval note can be checked against the log on future cells.
info	data-quality	claude-judge	scotus/73500237	evt-petition-disposition	gemini-baseline's retrieval.md states 'No retrieval beyond the provisioned inputs', but its captured retrieval_log.json carries a shell corpus query (fedcourts query --court scotus --era 2020s on SG-waiver priors; result unobserved). Not leakage (forward cell, pre-resolution, shape query), but the self-reported retrieval note disagrees with the harness log.
info	other	claude-judge	scotus/73500241	evt-petition-disposition	codex-baseline's retrieval log carries a shell call whose text names data/qp-topics/ only as a find exclusion (find data -name AGENTS.md -not -path 'data/qp-topics/*'). The evaluate prompt's literal rule counts a file-search whose query names that path as outcome material; I read this one as an exclusion rather than a read and graded the forward cell clean (retrieved_outcome_material false). Recorded so a maintainer can overrule the reading if the rule is meant literally.
info	data-quality	claude-judge	scotus/73500242	evt-petition-disposition	The as-stored snapshot (2026-10-05.json, and per the candidates the 2026-09-17 snapshot too) dates the 'Petition for a writ of certiorari filed' proceeding Apr 13 2026, the same date as the extension application, while the linked PDF is stamped 2026-05-26 and the docket's DocketedDate is June 1, 2026. The event's opened_at (2026-06-01) is unaffected, but any filing-to-decision latency read off the proceedings list would be six weeks too long. No action taken; noted for whoever maintains the proceedings parser.
info	other	claude-judge	scotus/73500287	evt-petition-disposition	codex-baseline's captured log carries a shell call find data -name AGENTS.md -not -path 'data/qp-topics/*'. The forbidden path appears only as an exclusion predicate and the command lists files named AGENTS.md, so no qp-topics membership reached the predictor; I graded it as not outcome-material (forward cell, case open at prediction). Recording it because the evaluate prompt's rule reads literally on a query that 'names data/qp-topics/', and a maintainer may want the predict prompt or the log classifier to distinguish an exclusion from a read.
info	data-quality	codex-judge	scotus/73344703	evt-petition-disposition	gemini-baseline's retrieval.md says no retrieval beyond provisioned inputs, but its captured call list includes two topical fedcourts query attempts at 2026-09-16T21:44:32.987Z and 21:44:37.061Z. Both result_capture values are unobserved, so success, failure, returned content, and transfer cannot be established; do not treat the null dates as empty results. The queries were bounded before September 16 and the event resolved October 5, so this reporting discrepancy does not establish outcome leakage. See gemini-baseline's evaluation.md.
info	data-quality	codex-judge	scotus/73500232	evt-petition-disposition	gemini-baseline retrieval.md says no corpus lookups executed, but the harness log contains an attempted corpus-query command and query-help call. Results are unobserved, so successful retrieval, failure, and returned contents cannot be established. This is a self-report versus attempted-call discrepancy, not evidence of outcome leakage; see evaluation.md.
info	data-quality	codex-judge	scotus/73500237	evt-petition-disposition	gemini-baseline's retrieval.md states no retrieval beyond provisioned inputs, but its harness-captured call list records a corpus query for waiver/Solicitor General material. That call's result_capture and result_status are unobserved: whether it succeeded, what it returned, and its transfer volume cannot be established. The attempted lookup should be disclosed; this reporting discrepancy is not evidence of outcome leakage. Details appear in gemini-baseline's evaluation.md.
info	data-quality	codex-judge	scotus/73500242	evt-petition-disposition	The provisioned record/snapshots/2026-10-05.json dates the petition-filed entry April 13, 2026, while its linked petition filename and the provisioned petition's signature indicate May 26, 2026; docketing is June 1. codex-baseline identifies the same chronology discrepancy. This is a conflict within the supplied record, not a live-corpus freshness claim. No timeliness inference is made, no input is changed, and the denial outcome and evaluation scores are unaffected.
info	data-quality	codex-judge	scotus/73500245	evt-petition-disposition	The staged petition jurisdiction paragraph dates the Alabama Supreme Court decision December 19, 2026; its factual narrative and the provisioned docket identify December 19, 2025. Treated as an apparent source typo, not a later decision or outcome leakage. codex-baseline and claude-baseline explicitly noticed the discrepancy. No input was changed; all three predictions remain scoreable.