Grading rubric

How verdicts work

This page is the published rubric behind every spck.dev merchant verdict — what each verdict means, what it takes for a run to go green, and the ways the suite refuses to overstate what it proved.

The four verdicts

Verdict	Meaning
pass	The server satisfied the requirement, and the check proved it could have caught a violation: the same check must both clean-pass a known-good reference server and flag every defect injected for it. A check that can't demonstrate both is not allowed to report pass.
deviation	The server violated a requirement quoted from the spec. The report shows the requirement text, its citation into the pinned spec source, and the actual response observed — expected next to observed, nothing summarized away.
inconclusive	The check ran but could not validate its own soundness (for example, a validation oracle was unavailable). Inconclusive is never counted as a pass.
not-tested	The check did not run — the capability wasn't declared by the server, a config value it needs wasn't supplied, or it needs a surface the run couldn't reach. Reported explicitly so absence of a failure is never mistaken for a pass.

What it takes to go green

The run verdict is computed over the black-box-testable RFC-2119 MUST requirements applicable to your server, the capabilities it declares, graded over REST today. Only a MUST deviation fails the run; the handful of SHOULD and MAY checks that run are reported as advisories and never change the exit code. Airtight means the MUST / MUST NOT / REQUIRED / SHALL obligations at spec pin cd78fb38; SHOULD-class obligations are censused report-only and never fail a run (the census is published with the coverage matrix).

• Requirements come from a register, not intuition. Every graded requirement is a register row scraped from the UCP specification, quote-verified against SHA-pinned spec sources. The check register lists every covered requirement with its citation; the coverage matrix shows what is and isn't covered yet.
• Checks are validated in both directions. Before a check may pass anything, it has to clean-pass a known-good reference implementation and catch each defect injected for it. A check that misses an injected defect is downgraded, never trusted.
• Versions are pinned. Results are graded against tagged spec versions, auto-detected from your server's discovery profile — not against a moving main branch.
• Coverage is disclosed. A pass reflects only the checks that ran. The report says how many applicable MUSTs were validated, and everything else is listed as not-tested or not-applicable rather than silently omitted.

Exit codes in CI

The CLI and the GitHub Action gate builds with the same semantics: exit 0 when no MUST deviation was observed, exit 2 on any MUST deviation, exit 1 when the target could not be tested at all (for example, discovery failed); command-line usage errors also exit 2. The JUnit report names every entry with its register check id; deviations carry the citation of the requirement graded at your spec version, or state that no register row is graded there.

What this is not

A green result is not a certification and not an endorsement — it reflects the checks run against your server at that moment, no more. The suite is unofficial and not affiliated with the UCP project; where the specification is ambiguous, the register records the ambiguity rather than inventing a requirement.