Single-judge LLM evaluation is noisy. Ask GPT-4 to rate two outputs and the ordering can flip on a re-run. JuryEval addresses this by running multiple LLM judges in parallel, then applying statistical significance tests to determine whether score differences are meaningful or just variance.

The problem with single-judge evaluation

LLM judges are inconsistent across temperatures, prompt orderings, and even re-runs at temperature 0. Most evaluation pipelines ignore this variance entirely, reporting a single score as if it were ground truth. This leads to misleading leaderboard rankings and false confidence in model comparisons.

How JuryEval works

JuryEval runs a configurable panel of judges (e.g. GPT-4o, Claude, Gemini) on the same outputs, aggregates their scores, and uses bootstrapped confidence intervals or Wilcoxon signed-rank tests to assess whether one model significantly outperforms another. Results include p-values and effect sizes alongside the raw scores.

JuryEval is available on GitHub under MIT license and installable via pip.