Exact match and F1
Exact match and F1 compare a model’s answer to a reference answer. The SQuAD reading benchmark scores answers this way.
- Exact match gives 1 when the answer equals the reference after cleaning, and 0 otherwise.
- Cleaning means lowercase, no punctuation, and no “a”, “an” or “the”.
- F1 gives partial credit for the words the answer and the reference share.
- Both compare words, not meaning.
Try this: change the reference to Austen | Jane Austen and watch Answer C reach 1.
Try it
What just happened
- Each card cleans the answer first. “Jane Austen.” becomes “jane austen”, which equals the cleaned reference, so exact match is 1 and so is F1.
- Orange words are the ones the answer shares with the reference. Answer B, “The author Jane Austen”, keeps one extra word, “author”. Exact match drops to . F1 stays high at , because every reference word was found.
- Answer C, “Austen”, has no extra words but misses one: precision 1.00, recall 0.50, F1 . Answer D buries the right name in six other words and gets .
- With several accepted answers, the best score wins. That is how SQuAD handles questions that have more than one right way to say it.
Show the formula
A word counts as shared as many times as it appears in both texts, so repeating a word does not help. F1 is the harmonic mean, which stays low if either precision or recall is low.
Where you meet it
A question-answering team compares two models on a test of short factual questions. One answers in a few words. The other answers in full sentences. The short one wins on exact match, and the sentence writer loses, even when its sentences are right.
- The SQuAD paper (2016) has more than 100,000 questions on Wikipedia articles, and reports 51.0% F1 for a baseline model against a human score of 86.8%.
- BLEU (2002) and ROUGE (2004) are the same kind of check. BLEU was made for machine translation and ROUGE for summaries. Both count shared word groups with a reference text.
- The Codex paper tested whether BLEU tracks working code. It found that programs that are not equivalent to the reference often had higher BLEU scores than programs that are, so it scored code by running unit tests instead.
Where it breaks
These metrics count words, so they cannot see meaning. A right answer in other words scores 0. A wrong answer that reuses the right words scores high. Each card shows the score next to what a person would say.
“The author of Emma” is right and scores . “Not Jane Austen” is wrong and scores . When meaning matters, people use a judge model or a human, which the next pages cover.
Check yourself
The reference is “the Eiffel Tower”. The answer is “Eiffel Tower!”. What are exact match and F1?
Both are 1. Cleaning removes the “the” and the “!”, so both texts become “eiffel tower”.
An answer has 10 words and contains the 2-word reference. What is its F1?
Shared words: 2. Precision 2/10 = 0.2, recall 2/2 = 1. F1 = 2 × 0.2 × 1 / 1.2 = 0.33. Long answers are punished even when they contain the right words.
Why might a team still use exact match when F1 gives partial credit?
When the answer is a short fixed thing, like a date or an ID, partial credit means nothing. “1813” and “1831” share no word, but a near miss is still wrong.
Used in
- Loading…
Go deeper
- Rajpurkar, Zhang, Lopyrev and Liang, “SQuAD: 100,000+ Questions for Machine Comprehension of Text” (2016). The benchmark behind these two metrics.
- Papineni, Roukos, Ward and Zhu, “Bleu: a Method for Automatic Evaluation of Machine Translation” (ACL 2002). The word-overlap metric for translation.
- Lin, “ROUGE: A Package for Automatic Evaluation of Summaries” (2004). The same idea for summaries.