Contamination and Goodhart

Benchmark contamination means a model has seen the test questions or answers in training. Goodhart’s law says a measure that becomes a target stops being a good measure.

A public benchmark is data on the internet, so a model can train on it by accident.
A model that memorised some questions scores above what it can really do.
A fresh test of the same kind, never published, shows the real level.
A small leak is hard to see, because the gap hides in the noise.

Try this: drag m up to 20% and watch the gap leave the noise.

Try it

What just happened

The model can really answer of questions like these. It has memorised of the public ones. On those it is always right, so the public score is , while a fresh test of the same kind shows .
The gap is only points, because memorising helps just on questions the model would have missed anyway. The better the model, the less the leak adds.
The last card asks the error-bars question from the other page: is the gap bigger than luck? With questions on each side, a gap of points gives z = , under 1.96. At 20% memorised the gap is points and z = .
So a small leak can go unnoticed, and a big one is easy to see once you have a clean test to compare against. That is how the GSM1k study found it.

Where you meet it

A model reports 92% on a popular math benchmark. A team writes new problems of the same kind, in the same style, and the same model scores a few points lower. That is the experiment behind GSM1k.

Zhang et al. wrote new grade-school math problems, GSM1k, to match the style and difficulty of GSM8k. Across leading models they saw accuracy drops of up to %.
They found a positive link, a Spearman correlation of , between how likely a model is to generate GSM8k examples and how far it falls on GSM1k. They read that as evidence of partial memorisation.
The same paper found that many models, especially those at the frontier, show minimal signs of overfitting, and that all models broadly generalise to new problems. A drop does not mean a model cannot do the task.
A 2024 survey of benchmark data contamination covers detection methods and alternative evaluations that cut the risk.

Where it breaks

When a benchmark becomes the number everyone is judged on, a small leak can change who is first. Two models: Y has not seen the benchmark, X scores lower on fresh problems but has memorised part of the public test. Pick how much X memorised.

At 0% or 10% memorised, Y leads on both tests. At 20% X moves ahead on the public test, and Y still leads on fresh problems. This is Goodhart’s law at work: Charles Goodhart said in 1975 that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”. Marilyn Strathern’s 1997 version is the one people quote: when a measure becomes a target, it ceases to be a good measure.

Check yourself

A model has memorised 20% of a test and really answers 50% of fresh questions. What does the public test show?

0.20 + 0.80 × 0.50 = 0.60. The gap is 10 points. A weaker model gains more from a leak, because it misses more questions that the leak can fix.

Why is a private, unpublished test set useful?

It cannot be in the training data, so it shows the real level. The price is trust: nobody can check it. A fresh test written for the purpose, like GSM1k, is a middle path, because its construction can be described and its questions kept hidden.

A model drops 3 points on a fresh test of 1,205 questions. Is that proof of contamination?

Not alone. With that many questions on each side, a 3-point gap is within noise (z about 1.6). You need more questions, a bigger gap, or other evidence such as the model’s unusually high probability of generating the public test items.

Used in

Loading…

Go deeper

Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic” (2024). The GSM1k study.
Xu, Guan, Greene and Kechadi, “Benchmark Data Contamination of Large Language Models: A Survey” (2024). Detection and mitigation.
Wikipedia: Goodhart’s law. The 1975 statement and the 1997 paraphrase.
Error bars on eval scores on this site. Why a 3-point gap needs a lot of questions.