Gate a release from CI with an API key

A release gate asks one question of every build: does the system still pass the plan it passed before? With a sealed plan and an API key, your pipeline can start a run, wait for it, and stop the release unless the verdict is PASS.

1. Seal the plan once, in Studio

Build and seal the plan in Studio as in the first tutorial. Note its plan id from the plan's page. A run needs a sealed plan: an unsealed one is refused, because a run must use a commitment made before it.

2. Create an API key

Open Settings, then "API keys". Give the key a name, for example CI pipeline, and choose "Create key". Only owners and admins can create or revoke keys. Copy the key once and store it as a secret in your pipeline; only its prefix is shown again.

3. Start the run and wait

Every request carries the key as Authorization: Bearer pk_api_.... Start a run with POST /api/v1/plans/{plan_id}/runs, then read GET /api/v1/runs/{rid} until its state is done (or error, paused or expired), and read verdict.

import json, os, sys, time, urllib.request

BASE = os.environ["PLIMSOLL_API"]      # your Plimsoll SI address
KEY = os.environ["PLIMSOLL_API_KEY"]   # pk_api_..., a pipeline secret
PLAN = os.environ["PLIMSOLL_PLAN_ID"]  # a sealed plan

def call(method, path):
    req = urllib.request.Request(
        BASE + path,
        method=method,
        data=b"{}" if method == "POST" else None,
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req) as r:
        return json.load(r)

run = call("POST", f"/api/v1/plans/{PLAN}/runs")
rid = run["id"]
while run["state"] not in ("done", "error", "paused", "expired"):
    time.sleep(30)
    run = call("GET", f"/api/v1/runs/{rid}")

verdict = run.get("verdict")
print(f"run {rid}: {run['state']}, verdict {verdict}")
sys.exit({"PASS": 0, "FAIL": 1}.get(verdict, 2))

The exit codes match the command line tool: 0 for PASS, 1 for FAIL, 2 for anything else. INCONCLUSIVE is not a pass, so a pipeline that only accepts 0 never ships on it by accident.

4. Read the result

• A FAIL opens findings in Studio, each with its smallest example; the run id in the log leads straight to them.
• A paused run usually means the organisation reached its credit limit; a new run cannot start until there are credits again.
• Each run uses credits and, for an attested system, the gentle testing limits, so gate the builds that matter rather than every commit.

5. Reports for your pipeline

A finished run gives its checks in the two report formats most pipelines display, from the same API key. The default is the test results XML most CI views read; add ?format=sarif for the code scanning format.

curl -s "$PLIMSOLL_API/api/v1/runs/$RUN/report" \
  -H "Authorization: Bearer $PLIMSOLL_API_KEY" > plimsoll.xml

For runs made with the Plimsoll command line tool, plimsoll report writes the graded checks from the run record in the two report formats most pipelines display. It reads the record and does not regrade it; for an authoritative regrade against the commitment use plimsoll grade.

plimsoll report evidence/plimsoll-NAME.json --junit evidence/plimsoll.xml --sarif evidence/plimsoll.sarif

A pass becomes a plain test case, a FAIL a failure, and an insufficient check a failure typed INCONCLUSIVE, since most pipelines have no third state. The exit code follows the record's verdict.