WIP watch diff with upstream main branch - #6

Open

Jeronymous wants to merge 111 commits into

upstream-mainfrom

Commits on Oct 14, 2025

Merge branch
Oligou
committed
fd58a24

Commits on Oct 16, 2025

skip task if no documents
Oligou
committed
80fb9cd

Commits on Oct 23, 2025

Change default use_chat_template when loading the tokenizer fails
Jeronymous
committed
acd19f1
Take HF_HOME env variable into account (if set)
Jeronymous
committed
3cc6315

Commits on Oct 28, 2025

Fix MGSM evals
Jeronymous
committed
f0f7162

Commits on Oct 31, 2025

fix reshape bug
Jeronymous
committed
df19f29
Remove padding from response
Jeronymous
committed
646d657

Commits on Nov 20, 2025

add ruler metric and prompt
Oligou
committed
8c07847
Add RULER in metrics
Oligou
committed
ed1718b

Commits on Dec 9, 2025

make FLORES translation benchmark work with datasets v2 (parquet version of the dataset)
Jeronymous
committed
58d0ccf

Commits on Dec 12, 2025

Commits on Dec 15, 2025

Commits on Dec 16, 2025

Misc fixes in RULER evaluation
Jeronymous
committed
cb2da29

Commits on Dec 18, 2025

Change the code to make it work with more recent versions of vllm
Jeronymous
committed
82805ab
Fix vllm call in LLM as a judge
Jeronymous
committed
41dec9a

Commits on Jan 6, 2026

Fix error in logprob computation with vllm >= 0.12, because of prefix caching
Jeronymous
committed
2e968b2

Commits on Jan 20, 2026

Commits on Feb 11, 2026

Commits on Feb 17, 2026

add COMET and MetricX metrics to lighteval
Jeronymous
committed
ddce778
Add COMET and MetricX to FLORES benchmarks
Jeronymous
committed
b8532b6
Add new dependencies
Jeronymous
committed
48ee2dc
COMET/MetricX : add options for device and batch size
Jeronymous
committed
9730191
Fix MetricX
Jeronymous
committed
cb1d040
Fix serialization of metric
Jeronymous
committed
be22ae1

Commits on Feb 18, 2026

Fix corner case
Jeronymous
committed
121c6a2
Merge pull request #1 from OpenLLM-France/comet
Jeronymous
authored
ee69f36

Commits on Feb 19, 2026

Fix mix of data and pipeline parallelism
Jeronymous
committed
aafd3db

Commits on Feb 20, 2026

Commits on Mar 2, 2026

fix corner case
Jeronymous
committed
33968ce
tune generation_size for math tasks
Jeronymous
committed
7ab7fa0
larger limit for gsm_plus
Jeronymous
committed
6a5c942

Commits on Mar 4, 2026

add an option enable_thinking
Jeronymous
committed
e8ac11b

Commits on Mar 6, 2026

Commits on Mar 11, 2026

Refactor MathAlea dataset configuration and prompt generation functions
Lduignan1
committed
d42f5fd

Commits on Mar 23, 2026

add system prompts in french and english
Lduignan1
committed
ce6848f

Commits on Apr 7, 2026

Make GPQA-fr a generative benchmark, not a MCQ
Jeronymous
committed
1db696e

Commits on Apr 8, 2026

Implement MMLU pro eval, with generative style (for instruct models)
Jeronymous
committed
2d55527

Commits on Apr 9, 2026

Commits on Apr 22, 2026

Add possibility to avoid running evaluation
Jeronymous
committed
da058f2
Merge pull request #4 from OpenLLM-France/advbench
Jeronymous
authored
481d9bd
Merge upstream huggingface/lighteval main into merge_hf_main
Jeronymous
committed
d1cf663
Fix ruff style and lint after merge
Jeronymous
committed
180975c
Solve version incompatibility in project install
Jeronymous
committed
2466d64
less differences with the upstream branch
Jeronymous
committed
68494ca
Add copyright
Jeronymous
committed
9ca1f4b
less differences with the upstream branch
Jeronymous
committed
6ee2a9e
do not build doc on fork
Jeronymous
committed
d9fe736
Add safety / red-teaming benchmarks
Jeronymous
committed
379ed71

Commits on May 4, 2026

test: point style-bot to TOCTOU fix SHA (#1220)
paulinebm
authored
726807d

Commits on May 5, 2026

[CI] Bump style-bot-action to hardened SHA (TOCTOU fix) (#1222)
paulinebm
authored
8e25dcf
Fix/bump style bot sha (#1223)
paulinebm
authored
6b9c415
Fix/bump style bot sha (#1225)
paulinebm
authored
5cf8bd9
Fix/bump style bot sha (#1226)
paulinebm
authored
2c7d1c2
Fix/bump style bot sha (#1227)
paulinebm
authored
4ea6bce
Fix/bump style bot sha (#1228)
paulinebm
authored
a67c557
Fix/bump style bot sha (#1229)
paulinebm
authored
8aac2b7
[CI] Bump style-bot to hardened TOCTOU-fix SHA (#1230)

paulinebm
and
claude
authored
ba82a29
Fix/bump style bot sha (#1232)
paulinebm
authored
3e75d1f
[CI] Bump style-bot to hardened TOCTOU-fix SHA (#1233)
paulinebm
authored
3fd1526

Commits on May 20, 2026

Legal NLP tasks on Swiss data (#1032)

rolshoven
and
NathanHB
authored
8d29839

Commits on May 25, 2026

Add Pillow dependency (#1242)

authored
89f9395

Commits on May 29, 2026

Add LEXam legal exam benchmark to Swiss legal evals (#1243)

authored
9e42783
Add multilingual task loading to vLLM (#1248)

authored
4d47029

Commits on Jun 9, 2026

fix rolling perplexity on the transformers backend (#1252)
kashif
authored
78dbee2

Commits on Jun 17, 2026

fix max_tokens tuple bug in JudgeLM litellm call

Matteovanypersele
authored and
Jeronymous
committed
a7febad
support per-doc system role via Doc.specific["instruction_as_system"]

Matteovanypersele
authored and
Jeronymous
committed
b68623f
Add environment variable to possibly tune the memory usage of the judge LLM (to avoid some memory errors)
Jeronymous
committed
4ecdb69
make sure the memory of the LLM is freed (before the judge LLM is eventually called)
Jeronymous
committed
9f90fba
Add generative task variant for MathAlea

Lduignan1
authored and
Jeronymous
committed
ca639a2
keep unanswerable rows in squad_v2

Matteovanypersele
authored and
Jeronymous
committed
5946dea
Fix MixEval: For FreeForm, the judge was onloy seeing the first good options, not all the possible ones. Also increase generation_size from 100 to 1024 (for thinking models)
Jeronymous
committed
84f1717
add luciole_rag citation-aware grounded QA benchmark

Matteovanypersele
authored and
Jeronymous
committed
7dabf27
Add Exo7 benchmark

Lduignan1
authored and
Jeronymous
committed
cfb4c2d
Remove unsupported 'suite' argument from safety task configs
Jeronymous
committed
e03dd8a
Remove unsupported 'suite' argument from registry docstring example
Jeronymous
committed
eb76c0c
Restore CF/Hybrid formulations and sensitivity labels in global_mmlu
Jeronymous
committed
e0b6b4e
Add comet and metricx metrics to flores200
Jeronymous
committed
f122b15
vllm: fix Ministral on transformers v5 (mistral tokenizer_mode for tekken + max_images to skip vision profiling)
Jeronymous
committed
c36d670
judge: fix vLLM judge on transformers v5 (apply_chat_template return_dict=False to get token ids, not a BatchEncoding)
Jeronymous
committed
32707db

Commits on Jun 23, 2026

Commits on Jun 25, 2026

Safety benchmarks: Use Llama Guard 4 judge. And don't compute 'no_context' setting
Jeronymous
committed
5535c34

Commits on Jun 26, 2026

Add AyaRedTeaming benchmark
Jeronymous
committed
9ab9827

Commits on Jun 29, 2026

Only the main process writes the sample cache under data-parallel (#1271)
shipbehaves
authored
64f4f5a

Commits on Jul 2, 2026

fix estimation of refusal recall/precision (not necessarily a strict match of the instruction)
Jeronymous
committed
d38b468

Commits on Jul 10, 2026

Fix few-shot for MMLU pro (must contain CoT)
Jeronymous
committed
ac798cd
fix dataset path
Jeronymous
committed
cd0019b

Commits on Aug 11, 2026

Store provider credentials as SecretStr in model configs (#1326)

authored
932e1f2

Commits on Aug 19, 2026

Add SCoolKID benchmark
Jeronymous
committed
239171f

Commits on Aug 21, 2026

Add PCBench
Jeronymous
committed
7d07599
Merge upstream/main into merge_hf_main
Jeronymous
committed
65f9c53