Pairwise preference and Elo
Pairwise ranking turns “which answer is better?” votes into a leaderboard. Chatbot Arena ranks language models this way.
- People find it easier to say which of two answers is better than to give one a score.
- Each vote moves the winner’s rating up and the loser’s down, by more when the result was a surprise.
- Elo updates after every vote, so the order of the votes changes the answer.
- A Bradley-Terry fit uses all the votes together and does not care about order.
Try this: press Simulate 20 votes a few times, then tick “replay in reverse”.
Try it
What just happened
- Four models have hidden true strengths, made up for this page. Random pairs meet and a vote follows the odds: a model rated 100 points higher wins about 64% of the time. After votes the orange bars are the Elo ratings.
- Each Elo update is K times the surprise: K × (result − expected). Here K is . Beating a much weaker model moves the ratings only a little, because the win was expected. Raise K and ratings jump with every vote.
- Replay the same votes in reverse order and the Elo ratings change, by up to points on this data, although nobody voted differently. The blue ticks do not move: the fit looks at all the votes together.
- Chatbot Arena moved its published score from Elo to Bradley-Terry coefficients. Its paper gives the reason as better suited to statistical estimation, and the order effect above is one way Elo is harder to estimate with.
Show the formulas
The 400 is a scale choice: a 400-point gap means the stronger side is expected to score about 0.91. Bradley-Terry finds the ξ values that make the observed votes most likely. The blue ticks here are those ξ values moved onto the Elo scale and centred on 1000.
Where you meet it
Scoring answers on a 1 to 10 scale gives noisy numbers, because one rater’s 7 is another’s 9. Asking “A or B?” is steadier.
- Chatbot Arena collects votes from visitors who compare two anonymous models, and the paper reports over 240,000 votes and good agreement between the crowd and expert raters.
- The same paper says its published scores moved from Elo to Bradley-Terry coefficients because they suit statistical estimation better.
- Chess rating bodies use Elo, with the K value set by how many games a player has played. FIDE uses K = 40 for new players and 10 for established masters.
Where it breaks
A rating is one number per model, so it assumes that if A beats B and B beats C, then A beats C. Votes can disagree. Here three models beat each other in a circle, 8 votes in 10 each.
Every model wins half its games, so the fit puts all three at . The leaderboard says they tie, but every pair has a clear winner. If the preferences have this shape, a ranking hides it. Look at the win rate of each pair as well as the leaderboard.
Check yourself
A model rated 1500 beats one rated 1500. With K = 32, how much does each rating move?
The expected score is 0.5, so the winner gains 32 × (1 − 0.5) = 16 and the loser drops 16.
The same 100 votes, shown in a different order, give different Elo ratings. Is that a bug?
No. Elo updates after every vote, so early votes use ratings that are still far from the truth. A Bradley-Terry fit to all the votes at once gives the same ratings for any order.
A new model has lost all of its first 5 votes. Bradley-Terry cannot rate it finitely. Why?
With no wins, the fit keeps pushing its rating down, because any lower rating makes the data more likely. This page adds a tiny prior, a fraction of a win and a loss against every other model, so the rating stays finite.
Used in
- Loading…
Go deeper
- Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference” (2024). Pairwise votes, the Bradley-Terry model, and confidence intervals on the ranking.
- Wikipedia: Elo rating system. The formulas and K-factors in real use.
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023). Compares judge-model votes with human votes.