Draft: Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

[PriyeshDave]

Draft ready for review

Title: Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
Subtitle: You'll see why current agent benchmarks relying on LLM judges are systematically unreliable—even on replayed, deterministic agent task runs—and why this should discredit most leaderboard comparisons.
Pillar: benchmarks
Contrarian post: True
Generated at: 2026-09-06T12:41:30.005332+00:00

Sources consulted during topic selection

• https://arxiv.org/abs/2609.04198v1
• https://github.com/brunovicco/agentic-security-framework-lab
• https://github.com/AgentEvalHQ/AgentEval

Review checklist

• Concrete artifact present (code / real number / diagram)
• All factual claims and stats check out
• Code blocks are actually correct / runnable
• Matches blog voice (no AI-cliche phrasing left over)
• Title delivers on its promise

Merging this PR triggers publish.yml, which publishes to the site, dev.to,
and LinkedIn. Edit the file directly in this PR before merging if changes
are needed.