Human agreement and kappa
People disagree even when they label the same items. Cohen’s kappa measures how much two raters agree beyond luck.
- Two raters label the same items, and the table counts each answer pair.
- Some agreement is luck, and it depends on how often each rater says yes.
- Kappa is the agreement left after removing that luck.
- It is 1 for perfect agreement and 0 for no better than luck.
Try this: pick “Almost every item is good” and compare 90% agreement with the kappa.
Try it
What just happened
- The grant readers agreed on 20 + 15 = 35 of applications, which is . That is the first card.
- Reader A said yes of the time and reader B . If each simply guessed with those habits, both would say yes of the time and both would say no of the time. Luck alone gives agreement.
- Kappa takes the agreement above luck, minus , and divides it by the most that could be above luck, 100% minus . The result is , which the usual bands call .
- Change any count and all four cards follow. The bands come from Landis and Koch (1977). Wikipedia notes they were the authors’ opinion, not an evidence-based rule.
Show the formula
Chance agreement is computed from each rater’s own yes rate, so a rater who says yes to nearly everything makes luck agree more often.
Where you meet it
You ask two people to label 100 model answers as “good” or “bad”, and a judge model to do the same. Before you trust the labels, you check how well the people agree with each other.
- The MT-bench paper measured agreement between GPT-4 and human experts at 85%, and between two experts at 81%, on pairs without ties. That is raw percent agreement, the first card here, not kappa.
- Percent agreement is easy to read and kappa corrects it for luck. Reporting both is common practice for label quality.
Where it breaks
Kappa has blind spots too. The Wikipedia article gives two cases where raters agree 60% of the time and kappa differs by a factor of two, because the raters use the labels differently. A lopsided set of labels is worse: nearly everything is “good”, the raters agree on 90%, and kappa is below zero.
In the last case chance agreement is already , so 90% is worse than luck would give. Kappa is . When one label is rare, a high percent agreement says little, and kappa can be unstable. Report the counts too.
Check yourself
Two raters agree on 70 of 100 items. Chance agreement is 50%. What is kappa?
(0.70 − 0.50) / (1 − 0.50) = 0.40.
Both raters say “good” for 95 of 100 items. They agree on 90. Why is kappa near 0 or below?
With both raters saying yes 95% of the time, luck gives 0.95 × 0.95 + 0.05 × 0.05 = 0.905. Their 90% is no better than that, so kappa is (0.90 − 0.905) / 0.095 = −0.05.
What does kappa = 0 mean?
The raters agree exactly as often as two raters with the same habits would by luck. It does not mean they disagree; it means their agreement carries no extra signal.
Used in
- Loading…
Go deeper
- Wikipedia: Cohen’s kappa. The formula, the grant example used here, the Landis and Koch bands, and the prevalence and bias effects.
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023). Agreement between a judge model and human experts, and between experts.