Most leak tests decide with a short function: does the reply contain one of these strings, does it look like a refusal. We wanted to know how often such functions agree with an access policy. So we ran the verdict code of three widely used open testing tools, offline and unmodified, on replies whose correct label we already knew.
The labelled set
2,736 unique replies, each with the user and the question that produced it. They come from three sources: a template bank assistant with 11 planted faults and a clean control, a small local model behind an open source chat front end, and one deskbook run. 8,142 transcript rows collapsed to 2,736 unique units, so a repeated reply counts once. Plimsoll SI's oracle labelled 182 replies as disclosing a value that user may not read, and 2,554 as not.
Two ways to give a tool its strings
A substring check needs the strings to look for. A tester without an access policy lists every planted value. A tester with one lists only the values this user may not read. We ran the tools both ways.
Exact substring verdicts against the oracle, all sources (the three tools gave the same counts)
- Strings given
- Every planted value
- Disclosures found
- 181 of 182
- Clean replies flagged
- 368 of 2,554: 14.4%, exact bounds 13.3% to 15.6%
- Strings given
- Only values this user may not read
- Disclosures found
- 179 of 182
- Clean replies flagged
- 0 of 2,554: upper bound 0.1%
| Strings given | Disclosures found | Clean replies flagged |
|---|---|---|
| Every planted value | 181 of 182 | 368 of 2,554: 14.4%, exact bounds 13.3% to 15.6% |
| Only values this user may not read | 179 of 182 | 0 of 2,554: upper bound 0.1% |
The 368 false alarms are users who were entitled to a value and were shown it. A string match cannot tell an authorised answer from a leak, because the difference is who asked. With the values taken from the access policy, agreement became almost complete. That agreement comes from the policy, not from the matching.
Refusal detectors
Detectors that count any reply without a refusal phrase as a successful attack did worse on this question. One flagged all 2,554 clean replies. Another, read inverted, flagged 1,187 of them (46.5%). Many correct and safe answers simply do not refuse.
Measured
2,736
unique labelled replies, 182 of them disclosures
Source: Judge audit, seed 20260930
14.4%
clean replies flagged when every planted value is a trigger
Source: Judge audit, all sources
0 of 2,554
clean replies flagged with values from the access policy
Source: Judge audit, all sources
29
agreeing calibration points a judge needs before its verdicts count (90%, at 95%)
Source: Exact lower bound, 0.9^29 ≤ 0.05
When a model judges
Some checks need judgement no string can give, such as whether an answer stays inside the assistant's job. Plimsoll SI uses a judge model only there, and only once its agreement with people is proven by an exact lower bound. The default commits 90% agreement. At 95% confidence that takes at least 29 calibration points, all agreeing, and 44 at 99%. A judge with a handful of agreeing points is not trusted, and a check that depends on an untrusted judge is not graded.
Verdicts come from exact counts against committed limits. When a run has a judge lane, the passport names it as the one place a judge is used and says how well it agreed with people.
What this does not show
- Only deterministic verdict code was run. No model based judge of any of the three tools was tested, and nothing here bears on them.
- The labels are ours, from our oracle. A reviewer who disagrees with its entitlement decisions will read the tables differently.
- 181 of the 182 disclosures come from 12 variants of one template assistant that repeats its wording, and the intervals treat unique replies as separate.
- The tools ran in specific released versions; a later release may change a detector or a phrase list. Nothing was tuned after seeing results.
- It does not say the tools are bad at what they were built for.