Claude/eval chain sensitivity mu4kop - #36
Merged
Merged
Conversation
Run filenames previously encoded only eval+judge (model_sig = eval_judge), so
runs differing only in triage model overwrote each other — and triage is the
model driving the Pass 0/1/3 work being tuned. Extend the signature to all
three pipeline models, in pipeline order, and surface every run for comparison.
- bias_eval / batch_eval: model_sig is now triage_eval_judge. Combined with the
mode suffix added earlier, the full run filename is
{article_id}_{version}_{triage}_{eval}_{judge}_{mode}.json — any single-axis
difference lands in a distinct file.
- ingest._already_evaluated: take `mode` and build the exact path including it,
so batch --skip-cached matches the real filename (it had silently stopped
matching once mode entered the name).
- article_eval page: replace the mode-only toggle with a run picker keyed on the
full (mode, triage, eval, judge) signature. The dropdown lists every stored
run by full signature so any mode and any model chain can be compared; running
the same mode with different models now produces a new run instead of being
skipped. Completion polling switched to a run-file count target (robust to
same-mode re-runs). Downloads carry a full-signature slug.
- scripts/migrate_run_filenames.py: one-off migration. Reconstructs the
compliant name from each record's stored models+mode and renames; deletes
legacy records whose triage/judge can't be recovered (pending re-run). Applied
to data/processed (17 renamed, 5 legacy deleted) and index rebuilt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01669jmknuhhZY3vBqwpDWvF
…sitivity-mu4kop # Conflicts: # data/processed/bias_frequency.json # data/processed/model_provenance.json # data/processed/stats.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.