-
Notifications
You must be signed in to change notification settings - Fork 6
Pull requests: OpenEuroLLM/JudgeArena
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
Meta-eval: (7/7) Estimate meta-eval tokens and OpenRouter reference cost
#107
opened Aug 15, 2026 by
ErlisLushtaku
Collaborator
Loading…
Meta-eval: (6/7) Measure held-out Elo error against annotation budget
#106
opened Aug 15, 2026 by
ErlisLushtaku
Collaborator
Loading…
Meta-eval: (5/7) Add Arena-Hard and AlpacaEval judge prompt presets
#105
opened Aug 15, 2026 by
ErlisLushtaku
Collaborator
Loading…
Meta-eval: (4/7) Register ranking metrics as the meta-eval scoring adapter
#104
opened Aug 15, 2026 by
ErlisLushtaku
Collaborator
Loading…
Meta-eval: (3/7) Judge both A/B orders with orientation-normalized rows
#103
opened Aug 15, 2026 by
ErlisLushtaku
Collaborator
Loading…
Meta-eval: (2/7) Score sampled battles with accuracy and Cohen's kappa
#102
opened Aug 15, 2026 by
ErlisLushtaku
Collaborator
Loading…
Meta-eval: (1/7) Declare meta-evaluation as a packaged task
#101
opened Aug 15, 2026 by
ErlisLushtaku
Collaborator
Loading…
Cache: WIP (7/7) Remove the unused Parquet cache store
#100
opened Aug 8, 2026 by
ErlisLushtaku
Collaborator
•
Draft
Cache: WIP (6/7) Synchronize inference caches through Hugging Face
#99
opened Aug 8, 2026 by
ErlisLushtaku
Collaborator
•
Draft
Cache: WIP (5/7) Adopt inference caching in MT-Bench
#98
opened Aug 8, 2026 by
ErlisLushtaku
Collaborator
•
Draft
Cache: WIP (4/7) Adopt inference caching in core workflows
#97
opened Aug 8, 2026 by
ErlisLushtaku
Collaborator
•
Draft
Cache: WIP (3/7) Cache inference across model providers
#96
opened Aug 8, 2026 by
ErlisLushtaku
Collaborator
•
Draft
Cache: WIP (2/7) Cache inference at do_inference
#95
opened Aug 8, 2026 by
ErlisLushtaku
Collaborator
•
Draft
Cache: WIP (1/7) Add content-addressed local inference caches
#94
opened Aug 8, 2026 by
ErlisLushtaku
Collaborator
•
Draft
(7/n) Complete declarative task runtime and shared pairwise orchestration
#86
opened Jul 21, 2026 by
kargibora
Member
•
7/9
Loading…
(6/n) Package MT-Bench as a declarative multi-turn task
#85
opened Jul 21, 2026 by
kargibora
Member
•
6/9
Loading…
(5/n) Package m-ArenaHard task families with language variants
#83
opened Jul 21, 2026 by
kargibora
Member
•
5/9
Loading…
(3/n) Introduce declarative YAML task definitions
#81
opened Jul 21, 2026 by
kargibora
Member
•
3/9
Loading…
(2/n) Organize Benchmark Execution Into Responsibility-Owned Packages
#80
opened Jul 21, 2026 by
kargibora
Member
•
2/9
Loading…
ProTip!
Exclude everything labeled
bug with -label:bug.