Skip to content

Preserve deep + flat eval runs and add a comparison toggle - #35

Merged
sadacca merged 1 commit into
mainfrom
claude/eval-chain-sensitivity-mu4kop
Jun 24, 2026
Merged

Preserve deep + flat eval runs and add a comparison toggle#35
sadacca merged 1 commit into
mainfrom
claude/eval-chain-sensitivity-mu4kop

Conversation

@sadacca

@sadacca sadacca commented Jun 24, 2026

Copy link
Copy Markdown
Owner

Previously the output filename was {article_id}{version}{model_sig}.json with no mode component, so a deep run and a flat run of the same article+models wrote to the same path — the second silently overwrote the first, making side-by-side comparison impossible.

  • bias_eval: encode mode in the output filename ({article_id}{version}{model_sig}_{mode}.json) so deep and flat results coexist on disk. Non-destructive: _already_evaluated's glob and build_index's content-based reads are unaffected; legacy files keep their names.

  • article_eval page:

    • Add a deep / flat / both mode selector. "both" evaluates each mode in sequence in the background thread, writing one file per mode, and only runs modes not already cached.
    • Replace the single-result loader with _load_results_by_mode(), which discovers every stored run for the article and keys them by mode (legacy untagged files treated as deep).
    • Completion polling now waits for all requested modes, with an "N/M modes complete" progress hint.
    • Results view gains a mode toggle (shown only when >1 mode exists), a caption showing the displayed run's mode + models, and mode-suffixed download names.

Claude-Session: https://claude.ai/code/session_01669jmknuhhZY3vBqwpDWvF

Previously the output filename was {article_id}_{version}_{model_sig}.json with
no mode component, so a deep run and a flat run of the same article+models wrote
to the same path — the second silently overwrote the first, making side-by-side
comparison impossible.

- bias_eval: encode mode in the output filename
  ({article_id}_{version}_{model_sig}_{mode}.json) so deep and flat results
  coexist on disk. Non-destructive: _already_evaluated's glob and build_index's
  content-based reads are unaffected; legacy files keep their names.

- article_eval page:
  - Add a deep / flat / both mode selector. "both" evaluates each mode in
    sequence in the background thread, writing one file per mode, and only
    runs modes not already cached.
  - Replace the single-result loader with _load_results_by_mode(), which
    discovers every stored run for the article and keys them by mode (legacy
    untagged files treated as deep).
  - Completion polling now waits for all requested modes, with an
    "N/M modes complete" progress hint.
  - Results view gains a mode toggle (shown only when >1 mode exists), a caption
    showing the displayed run's mode + models, and mode-suffixed download names.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01669jmknuhhZY3vBqwpDWvF
@sadacca
sadacca merged commit 75ffd86 into main Jun 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants