A result is only as good as the order in which things were decided. If the limit is chosen after the counts are seen, any system can pass. Plimsoll SI fixes the plan first. Before the first question is asked, the questions, the limits, the access policy, the seed and the fingerprint of the system under test are hashed into a commitment. In Studio that step is called Seal the plan. A run is graded against its commitment and nothing else.
What a commitment pins
Our first commitments pinned only the spec, the run parameters, the thresholds and the access oracle. They did not pin the grading code, which is how a bootstrap came to grade run C. Since 28 September 2026 a commitment also pins:
- the source code of the system under test and of the engine
- the model identifiers the system reports
- the probe seed and the digest of any response cache
- a one time nonce
- the list of required checks, so a required check with no evidence fails instead of dropping out
A run that drifts fails
Run D9 shows the rule applied to us. Every substantive check passed. But the commitment pinned the repository's commit identifier, and two documentation commits moved it while the run was live. The system's own source did not change. The run was graded FAIL, and the FAIL stands, unedited, in the run history. The fix, pinning the digest of the system's own source instead of the repository head, went into a new commitment.
The same run also met the model provider's request quota: 12 of 800 leakage probes and 86 of 300 multi turn probes errored, so those checks were insufficient evidence on their own. No defect was observed. A FAIL for the wrong reason is still a FAIL, and the record says which reason.
Measured
12 of 800
leakage probes that errored on throttling in run D9
Source: Run D9 pack
86 of 300
multi turn probes that errored in run D9
Source: Run D9 pack
0
changes to the system's source while run D9 was live; the FAIL came from the pinned commit identifier
Source: Run D9 record
Changes are written down
Between attempts d4 and d5 the multi turn sample was cut from 1,128 scripts to a stratified 300, for cost, before any multi turn result existed. The run history says so, so the parameter cannot later be read as chosen after the fact. A change made after seeing a result, such as the fix to a control that could not fail, is labelled as exactly that, and it never touches a threshold or a probe.
Why a nonce
The exact bound holds for one try. Nothing in it covers a team that commits, runs, dislikes the result and commits again. In simulation a system leaking exactly at a 1% limit passed a 686 cluster test 0.10% of the time per try, and at least once in 50 tries 4.9% of the time. Smaller tests let retries buy more: at 320 clusters the same system passed 4.0% of the time per try, and 55.9% of the time within 20 tries.
A one time nonce consumed in a log, and a history that keeps every attempt, are the defence. We keep every attempt ourselves: deskbook's history lists runs A to C and attempts d to d13, including those that crashed, were stopped, or failed on infrastructure. Of those, d8, d11 and d13 passed every committed check.
What this does not show
- A commitment keeps an honest operator to the plan. It does not stop someone who can edit the files from committing again, or from trying several times and publishing one result. In our command line tool the ledger of used nonces was a local file.
- It pins identifiers, not behaviour. A hosted model that changes behind an unchanged identifier is not caught by the commitment.
- The second commitment schema did not pin assets that are not code, such as prompts and index data.