Is my test big enough?
How many test questions prove an AI system meets a limit: exact one sided 95% bounds, the same arithmetic Plimsoll grades a run with. Free, nothing to sign up for.
The sizes at a glance.
How many distinct questions, all clean, prove a rate stays under a limit.
- under 1%
- 299at 95%
- 459at 99%
- under 2%
- 149at 95%
- 228at 99%
- under 5%
- 59at 95%
- 90at 99%
- under 10%
- 29at 95%
- 44at 99%
Distinct clean answers needed, at 95% (the default) and 99% (for tier 1 reviews) confidence.
| Confidence | under 1% | under 2% | under 5% | under 10% |
|---|---|---|---|---|
| 95%the default | 299 clean answers | 149 clean answers | 59 clean answers | 29 clean answers |
| 99%for tier 1 reviews | 459 clean answers | 228 clean answers | 90 clean answers | 44 clean answers |
One confirmed event fails a defect check outright. Floors read the same way: at least 90% right needs 29 all correct at 95% and 44 at 99%.
Plan a test before you run it.
Pick the kind of limit. The answer is the smallest number of distinct questions that can prove it.
A defect limit
For things that must almost never happen: a leak of restricted data, an action the user may not take.
For example 1 for one in a hundred.
299 distinct questions, all clean, prove a rate under 1.00%. Rewordings of one question count once. A single confirmed event fails a defect check, so the number assumes none.
A quality floor
For things that must usually go right: accurate answers, declining what it should not answer.
For example 80.
A whole number; 0 for a perfect run.
With 5 wrong answers, 50 questions prove a rate of at least 80.00%: the lower bound of 45 right out of 50 is 80.12%.
Check a test you already ran.
What a test you ran proves
Rewordings of one question count once.
A whole number, at most 500.
The line you want to prove.
With 3 events in 500, the true rate is at most 1.54% (95% confidence). A defect check fails on any confirmed event, so this run would not pass a defect limit.
Exact one sided 95% Clopper Pearson bounds, computed in your browser from the counts; nothing you type is sent anywhere. A bound is a statement about the questions asked, not about questions nobody asked.