Controls that must fail

Zero is the result everyone wants, and the easiest one to get by accident. A scanner pointed at the wrong field, a detector with a typo in its pattern, or a probe that never reaches the system all report zero. So every Plimsoll SI check runs beside a control: the same test against a setup where the fault is known to be present. If the control does not fire, the check cannot pass.

The filter off control

deskbook removes documents the asker may not read before retrieval scores anything. Its leak suite runs every probe twice, once as shipped and once with that filter switched off. On 27 September 2026, with the filter on, 0 of 662 probes leaked at the retrieval layer. With the filter off, 634 of 662 did. The build fails if any probe leaks, and it also fails if the control finds nothing.

The committed runs keep the same pairing at a larger size. In run D11, 0 of 3,371 retrieval probes leaked with the filter on, and 3,100 of 3,371 leaked with it off.

Planted faults

A control can also be a fault we plant ourselves. Each committed deskbook run from D5 on carries five mutations, small deliberate breaks of the access path, and the run must catch all five. Every graded run from D5 to D12 recorded 5 of 5.

Which oracle judges a planted fault matters as much as planting it. We planted five realistic permission faults in deskbook's entitlement configuration. Judged by deskbook's own configuration, a stale group membership and an over broad grant were invisible: 3 of 5 caught. Judged by a separate statement of who should read what, 5 of 5 were caught. A test that asks the system who may read what can never see a fault in the answer to that question.

When the control would not fail

The multi turn family checks whether something said earlier in a conversation reaches a user who should not see it. In attempts d5 to d7 its control was deskbook's own memory with guards loosened. It leaked 0 of 269, then 0 of 285, then 0 of 300. The family could not have seen a leak, so the check was insufficient and the run INCONCLUSIVE, each time.

The diagnosis was good news for deskbook: its memory does not leak even with its output checks off, because the answering model declines to treat earlier turns as a source. Making it leak anyway would have needed a contrived setup, so that path was dropped and its code removed. From d8 the family uses a built in agent that leaks its memory on purpose; in run D11 it leaked 975 of 975 turns.

What this does not show

• A control that fires proves the test can see the kind of fault the control contains, not every kind.
• Five planted faults are five. They show the method works on those shapes; they do not estimate how often real faults are caught.
• The statement of intended access that caught 5 of 5 was written by the same developer as deskbook. It demonstrates the method; it is not a bank's source of record.