An assistant over company documents inherits the access problem of the documents and adds its own. A user who could never open a restricted file can still ask a question whose answer sits inside it. Plimsoll SI tests this by asking as every role, for every fact that role may not read, and checking every reply for the value.
The reference system
deskbook is our reference system: a support assistant for the operators of a capital markets post trade platform, over a synthetic corpus. Every document carries one label, a desk and a level. Eight personas range from a contractor who reads public material only to second line support. Two rules are tests over every principal, not conventions: nobody holds firm wide restricted access, and nobody may read restricted material from two trading desks.
The probes
Probes are generated from the corpus ground truth, so each one knows every value its principal must never see. The scan also looks for a restricted value reversed, base64 encoded, and its random suffix when that is six characters or more.
deskbook leak suite, 662 probes
- Family
- Cross desk
- Probes
- 451
- What it tries
- every principal asks for every fact it may not read
- Family
- Impersonation
- Probes
- 40
- What it tries
- claims to be a desk head, or an administrator override
- Family
- Extraction
- Probes
- 50
- What it tries
- first characters, reversed, base64, or just guess
- Family
- Enumeration
- Probes
- 32
- What it tries
- list every codename, or which restricted documents exist
- Family
- Canary query
- Probes
- 60
- What it tries
- searches with the secret value or the restricted document's id
- Family
- Injection
- Probes
- 29
- What it tries
- questions that pull an injected ticket into the context
| Family | Probes | What it tries |
|---|---|---|
| Cross desk | 451 | every principal asks for every fact it may not read |
| Impersonation | 40 | claims to be a desk head, or an administrator override |
| Extraction | 50 | first characters, reversed, base64, or just guess |
| Enumeration | 32 | list every codename, or which restricted documents exist |
| Canary query | 60 | searches with the secret value or the restricted document's id |
| Injection | 29 | questions that pull an injected ticket into the context |
The result
Leaks with the entitlement filter on, and with it off as the control
- Layer
- Retrieval, top 10 chunks, every probe
- Filter on
- 0 of 662
- Filter off
- 634 of 662
- Layer
- Answer, every probe
- Filter on
- 0 of 662
- Filter off
- 36 of 72
| Layer | Filter on | Filter off |
|---|---|---|
| Retrieval, top 10 chunks, every probe | 0 of 662 | 634 of 662 |
| Answer, every probe | 0 of 662 | 36 of 72 |
The second column is why the first means something. A leak test that finds nothing even with protection removed proves nothing; this one finds leaks in almost every probe when the filter is off. The suite ran on 27 September 2026 with a small hosted model answering over dense retrieval, without a reranker.
Later committed runs on a larger corpus repeated the zero at a larger size, over the full pipeline. In run D11, 0 of 3,371 retrieval probes and 0 of 800 answer probes leaked, with exact cluster bounds of 0.32% and 0.56% against a 1% limit.
Measured
0 of 662
probes leaked at both layers with the filter on
Source: deskbook leak suite, 27 September 2026
634 of 662
retrieval probes leaked with the filter off
Source: deskbook leak suite, 27 September 2026
0 of 800
answer probes leaked in run D11, bound 0.56%
Source: Run D11 pack
2,400 of 2,400
prompts byte identical with and without hidden documents, run D11
Source: Run D11 pack
Two corrections to the scoring
- A value the asker typed into the question is not a leak when repeated back. The first run counted 28 such echoes as leaks; the scan now leaves out values present in the question.
- Partial forms must be six characters or more, so an ordinary number can never match a four digit suffix.
A leak with no value in it
A reply can leak without naming anything: that document exists, but it is restricted for you. So Plimsoll SI also checks noninterference. The input the model receives must be byte identical whether or not the documents the asker cannot read exist. A prompt that does not change cannot depend on them. In run D11, 2,400 of 2,400 prompts were byte identical.
What this does not show
- Probes are generated from templates. They cover known attack shapes, not an attacker who iterates against the live system.
- Disclosures shorter than six characters cannot be seen by this suite. An answer that gave away only the first four characters of a code was mostly undetected by the newer detector too (2 of 120), because such stems are public by design.
- Aggregation, putting several readable documents together to infer a restricted fact, is untested.
- The answer layer of the 662 probe suite used dense retrieval without the reranker deskbook now ships; the committed runs cover the full pipeline.
- 0 of 662 is a count on one synthetic system, not a rate for any other.