Run D was meant to be one committed run of deskbook on a larger corpus. It took thirteen attempts, d to d13. Three passed every committed check, one failed, and the rest stopped or ended INCONCLUSIVE. Every attempt is in the run history. Here is what stopped them, because almost none of it was about deskbook.
What actually stopped each attempt
- d: retrieval probes ran in quadratic time, because a cache file was rewritten after every miss. Fixed by saving in batches.
- d2: the leak detector recompiled its patterns for every secret, 0.19 seconds a scan. Now 0.0056 seconds, with 0 differences from the old code over 1,000 comparisons.
- d3: the cloud session token expired at probe 509. The run could not resume, because the engine source changed afterwards and the commitment pins it.
- d4: too slow. The answering model's request quota on our account was 10 a minute; at about 6 multi turn turns a minute the full set needed about 9 more hours. It was stopped with no result graded.
- d5 to d7: the multi turn control could not leak, so the family could not have seen a leak. That story has its own post.
- d9: FAIL, because the commitment pinned the repository's commit identifier and documentation commits moved it mid run, and because throttled probes left two checks short.
- d10: a transient error from the embedding service was not retried, and the crash had already changed a pinned cache. Every restart was refused. Nothing was graded.
The pass that hid a problem
Attempt d8 passed all 13 committed checks. It also served 4,092 of its 4,100 replies from a response cache warmed by the attempts before it, so the quota that later broke d9 was never tested. The cache was faithful to the prompts, and the pack said so, but a live rerun with a fresh seed was the honest next step. That was d9, and then d11, which passed live with 0 errors and 0 retries.
The rules we kept
- Estimate live calls and hours from the quota before launching, and write the estimate in the run record.
- Rate limit at the model call, not at the probe: one probe can make several model calls.
- Pin the digest of the system's own source, never the repository head.
- Commit only after every source change is merged, because the source is pinned.
- Sweep errored probes after the pass, so a late throttle does not become a permanent error.
- A rerun with the same seed shares its evidence with the failed attempt and is recorded that way. A fresh seed is new evidence.
None of this is glamorous. All of it is the difference between a verdict and a guess.