Online evals and A/B tests
An online eval tests a change on real users: half see version A, half version B, and a metric such as thumbs-up rate decides. It is slower than an offline eval, and it is the check that counts.
- Offline evals use a fixed test set, and online evals use real users and real behaviour.
- The users needed grow with one over the lift squared, so small lifts are costly to see.
- The test must run to the planned size before you read the result.
- Looking early and stopping at the first win gives false wins.
Try this: drag the lift down to +0.5 points and watch the days needed explode.
Try it
What just happened
- You split users between A and B and watch a yes/no metric, such as thumbs-up on an answer. At a baseline of %, seeing a rise of points eight times in ten needs users in each arm.
- Both arms share the daily users, so users a day gives days. The third card turns it around: a week of traffic can only see lifts of about points or more.
- The last card quarters the lift. It needs about times as many users, close to 16, because the users needed grow with 1 over the lift squared.
- A small lift is hard to see from noise. This is the same sample-size idea as on the error bars page, applied to a rate.
Show the formula
p₀ is the baseline rate and p₁ = p₀ + lift. With significance 5% and power 80%, z_α/2 + z_β = 1.96 + 0.84 = 2.80. This is the usual two-proportion sample-size formula. It has the same shape as the one for comparing two models on the error bars page.
Where you meet it
You change a system prompt and an offline eval says answers got better. Before you roll it out, you still want to know whether real users prefer it.
- Kohavi, Tang and Xu’s book on online controlled experiments (Cambridge University Press, 2020) is the standard guide to running A/B tests well.
- Chatbot Arena is an online eval of a kind: it collects votes from real visitors who compare two anonymous models.
Where it breaks
It is tempting to check the result every day and stop as soon as it says “significant”. Each look is another chance for noise to cross the line. Below, A and B are identical, and every “win” is a false alarm. Pick how often you check.
Checking once keeps the promised 5%. Checking 10 times gives about false wins, and 20 times about . Evan Miller’s 2010 article reports a worst case of % for a test run at a nominal 5%. Fix the sample size in advance and read the result once, or use a design built for repeated looks.
Check yourself
You want to detect a lift that is half as big. Roughly how many users do you need?
About four times as many. The users needed grow with 1 over the lift squared, so halving the lift multiplies them by 2² = 4.
A test shows p < 0.05 on day 2 of a planned 14 days. Do you ship?
Not on that alone. You planned the size for a reason. Stopping at the first good-looking result is peeking, and it inflates false wins well above 5%. Keep running, or use a sequential design that allows early stops.
Both arms are the same, and you check the test 5 times. About how often will you see a false win?
About 14% of the time, in simulation, even though each single check has a 5% chance of a false win.
Used in
- Loading…
Go deeper
- Miller, “How Not To Run an A/B Test” (2010). The peeking problem in plain words.
- Kohavi, Tang and Xu, “Trustworthy Online Controlled Experiments” (2020). The practical guide to A/B testing.
- Error bars on eval scores on this site. The same sample-size reasoning for comparing two models offline.