LLM as judge

An LLM judge is a language model that grades another model’s answers. Teams use it where having people grade every answer is too slow or costly.

A judge model can grade thousands of answers that people could not.
In one test GPT-4 agreed with experts as often as experts agreed with each other.
The same judge can still be swayed by answer order and by length.
Swapping the order and keeping agreeing verdicts removes part of the bias.

Try this: switch the judge to Claude-v1 and watch the “1” tiles take over.

Try it

What just happened

The paper built pairs of near-identical answers, then asked each judge to pick the better one twice, with the two answers in swapped places. A fair judge would pick the same answer both times. GPT-4 did so times (“=” tiles). In pairs it simply chose whichever answer was shown first.
The other judges are worse: Claude-v1 was consistent times and chose the first answer times. The bias is a property of the judge, not of the answers.
Tick “swap the order, keep verdicts that agree”. The dashed tiles are verdicts you would throw away. What stays is the pair of verdicts that survived both orders, which are the ones worth counting.
Open “Show what helps” for two fixes. Examples in the prompt raised GPT-4’s consistency from % to %. On math questions, giving the judge the reference answer cut its mistakes from 14 in 20 to 3.

Where you meet it

You change a prompt and want to know if answers got better. Reading 500 answers takes a day. A judge model reads them in minutes, and the paper says a strong one tracks human experts closely. The risk is trusting a score the judge’s habits can move.

The MT-bench and Chatbot Arena paper tested GPT-4 as a judge against human votes and reported over 80% agreement, the same level as between humans.
It names four limits of LLM judges: position bias, verbosity bias, self-enhancement bias and limited reasoning ability.
RAGAS uses a language model in the same way to count how many statements in an answer a passage supports. The RAG evals page covers it.

Where it breaks

A judge can prefer an answer for being longer. The paper made a test: take an answer with a numbered list, reword the list without adding anything, and put the reworded copy in front of the original. Nothing new was said. A fair judge would not prefer the longer answer. Pick a judge.

Out of padded answers, GPT-3.5 and Claude-v1 preferred the padded one times, and GPT-4 times. A judge that rewards length will push a model to write longer, not better. Check a judge for length before you trust it, for example by padding some answers yourself.

Check yourself

A judge picks answer A when A is first, and answer B when B is first. What does that tell you?

That it is choosing by position, not by quality. For this pair its verdict is meaningless. With the swap rule, the pair counts as a tie.

GPT-4 agreed with experts 85% of the time and experts agreed with each other 81%. Does that make GPT-4 as good as an expert?

On that test, it matched the typical level of disagreement between experts, which is the best any judge can be measured against. It does not show it is right where experts are all wrong, and it says nothing about other kinds of questions.

Your judge always prefers the longer answer. What do you change?

Test it on answers padded with no new content, as the paper did. Then tell the judge to ignore length, compare answers of similar length, or add a length check to the score.

Used in

Loading…

Go deeper

Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023). Every number on this page: Tables 2 to 4 and section 4.2.
Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference” (2024). The human-vote side. The next page, on pairwise ranking, uses it.