Our blind benchmark: the claim was not supported

We built Plimsoll SI to catch access control, carry over and injection faults that other testing tools miss. In October 2026 we tested that claim the way we ask others to test their systems: written down first, frozen, scored once. The pre registered claim was not supported. This is what the benchmark showed, and what it did not.

How it was run

• The target: an offline staff assistant for a synthetic bank, with six tools of twelve resources each and two entries, chat and an API, plus 17 clean builds.
• The faults: ten author runs, isolated from each other and from every testing tool, handed in 756 faults. After intake and blind adjudication 124 were scored, and 75 of those form the headline set: faults a text testing tool can reach on the chat entry.
• The rules: a pre registration accepted on 5 October 2026, dated amendments, then a final freeze on 8 October. The engine was frozen before authoring began.
• The comparison: eight configurations of four other open testing tools, each in a default and an informed setup, with a fixed time and call budget, judged by the same rules.
• Scored once, on 21 October 2026. The set is now spent as a holdout.

The numbers

Detection was higher than every other configuration. The next after the strongest detected 14 of 75 (19%), and every other configuration 10 or fewer. On the same faults Plimsoll SI caught 25 that the strongest other configuration missed, and that configuration caught 6 that Plimsoll SI missed (exact McNemar test, one sided p = 0.0004, a secondary view).

Why the claim failed anyway

The claim had more than one clause, and detection was not the main reason it failed.

• False alarms. Plimsoll SI flagged 3 of 16 clean builds, where six of the eight other configurations flagged none. The clean builds are small patches of one bank that was adjusted until Plimsoll SI passed it, which favours us, and the false alarm clause was still dropped. Sixteen builds cannot show a false alarm rate below 17%.
• False passes. Plimsoll SI returned PASS on 12 scored faults. The sentence promising no false PASS was withdrawn and restated with that count.
• Author groups. Against the strongest configuration the advantage held on the pooled set and in one author group, but not in each author group on its own, which the pre registration required.

Cost

A Plimsoll SI run that made calls made about 16,106 calls to the target, 1.5 times the per run budget given to the other tools (10,500). The strongest other configuration averaged 852 calls a run. In time the order reversed: 441 minutes for Plimsoll SI's 198 chat runs, against 4,788 minutes for that configuration's 119.

What changed afterwards

After scoring, fixes for three classes of mechanism were run on Plimsoll SI alone. False alarms fell to 1 of 16 and false passes to 10, at a cost of 5 detections on one clean build's parent. That is a post hoc result on a spent set, not a benchmark result. The next headline needs a fresh holdout.

What this does not show

• One synthetic target, built by us. Results on another system, another model or real documents may differ.
• The other tools were judged by one local judge model on a frozen stack. In the first scoring view a context limit truncated 4,792 judge prompts, and the strongest configuration scored 0 of 75 until those runs were replaced by reruns. We report both views; Plimsoll SI's numbers are the same in each.
• Two of the other tools generate their tests from each build's own knowledge file, so their clean build counts are upper estimates of their false alarms.
• This summary is drawn from a write up generated by the scorer. Its line by line human review was still pending when this article was published.