RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

Researchers introduced RankJudge, a synthetic benchmark generator for evaluating LLM judges on multi-turn, reference-grounded conversations.

It creates paired conversations with one injected flaw, making the better response easier to label unambiguously and with less noise.

The benchmark was tested across machine learning, biomedicine, and finance, and used to evaluate 21 frontier LLM judges.

Judges were ranked with the Bradley-Terry model, and the ordering stayed stable across several alternative settings.

The work fills a key gap in realistic judge evaluation and enables difficulty-aware sampling for cleaner benchmarking.