Exact bounds, counted by cluster

A test of an AI system asks a few hundred or a few thousand questions and counts what went wrong. The count is not what a reviewer needs. A reviewer needs the worst rate the evidence still allows. Plimsoll SI reports that number for every check: an exact one sided 95% Clopper Pearson bound, computed from the counts.

Zero is not a rate

Zero leaks in 120 questions sounds clean. At 95% confidence it only shows that the true leak rate is below about 2.5%. The first committed run of deskbook, our reference system, asked exactly 120 such questions against a committed limit of 1%. Nothing leaked, and the run could not pass.

With zero events in n trials the exact upper bound has a closed form, 1 - 0.05^(1/n). Turned around, it says how many clean trials a limit needs before a plan can prove it.

Why exact

Approximate intervals behave worst exactly where AI tests live: near zero, with small counts. A normal approximation gives an interval of zero width at zero events. A percentile bootstrap reports [1.0, 1.0] when every answer is the same. The second case is not hypothetical: it is how one of our own passes was graded, and why it was withdrawn.

An exact bound comes straight from the binomial distribution and assumes nothing about sample size. In our calibration study the exact one sided upper bound covered the true rate at least 95.0% of the time at every rate tried, and between 95.00% and 95.02% at the worst rate over a fine grid. That is what exact means: it never promises more than it holds.

Why clusters

Questions are not separate trials when they share a cause. A hundred rewordings of one question about one restricted fact tend to leak together or not at all, because the fault sits in the access rule, not in the wording. Counting them as a hundred trials would shrink the bound for no reason.

Plimsoll SI groups related questions into clusters: the fact, the template, the planted injection or the forbidden tool call they test. A cluster counts once. The grade uses the largest of three bounds: the one over questions, the one over clusters, and one corrected for the design effect. The person who asks is not a cluster, so asking the same fact as more roles does not inflate the count.

In a simulation of 2,000 runs per cell, on the probe structure of an earlier deskbook corpus, ignoring clusters was badly optimistic. When every question in a cluster leaked together and the true rate was 0.3%, the bound over questions covered the true rate in only 68.8% of runs at the retrieval layer, against a promised 95%. The bound the grade uses covered it in at least 97.5% of runs in every cell.

What it costs

Counting by cluster is conservative. When the questions in a cluster are only partly correlated, the cluster bound is wider than the question rate needs, by about the cluster size (four, in the simulation). More questions do not always help either. On the corpus of the time, deskbook's answer sample stopped gaining clusters near 686, so beyond about 1,200 questions extra questions added almost nothing. The lever is more restricted facts and more even clusters, not more questions.

What this does not show

• A bound describes the questions that were asked. It says nothing about kinds of questions the plan did not contain.
• The bound counts detected events. Detector recall is measured separately and reported beside the verdict, not inside it.
• In the calibration study the accuracy and abstention questions were graded as separate questions although several ask about one fact. With full correlation the abstention lower bounds covered only 84.0% and 81.5% instead of 95%. That is the one place our bounds were found too lenient.
• Each check is a 95% one sided test with no correction across checks. Four rate checks sitting exactly on their limits gave at least one FAIL in 12.7% of simulated runs.