Grading rubric

How verdicts work

This page is the published rubric behind every spck.dev result — what each verdict means, what it takes for a run to go green, and the ways the suite refuses to overstate what it proved.

The four verdicts

Verdict	Meaning
pass	The server satisfied the requirement, and the check proved it could have caught a violation: the same check must both clean-pass a known-good reference server and flag every defect injected for it. A check that can't demonstrate both is not allowed to report pass.
deviation	The server violated a requirement quoted from the spec. The report shows the requirement text, its citation into the pinned spec source, and the actual response observed — expected next to observed, nothing summarized away.
inconclusive	The check ran but could not validate its own soundness (for example, a validation oracle was unavailable). Inconclusive is never counted as a pass.
not-tested	The check did not run — the capability wasn't declared by the server, a config value it needs wasn't supplied, or it needs a surface the run couldn't reach. Reported explicitly so absence of a failure is never mistaken for a pass.

What it takes to go green

The run verdict is computed over the RFC-2119 MUST requirements applicable to your server — the capabilities and transports it actually declares. Only a MUST deviation fails the run; the handful of SHOULD checks that run are reported and never fail it.

• Requirements come from a register, not intuition. Every check cites a requirement row scraped from the UCP specification, quote-verified against SHA-pinned spec sources. The check register lists every covered requirement with its citation; the coverage matrix shows what is and isn't covered yet.
• Checks are validated in both directions. Before a check may pass anything, it has to clean-pass a known-good reference implementation and catch each defect injected for it. A check that misses an injected defect is downgraded, never trusted.
• Versions are pinned. Results are graded against tagged spec versions, auto-detected from your server's discovery profile — not against a moving main branch.
• Coverage is disclosed. A pass reflects only the checks that ran. The report says how many applicable MUSTs were validated, and everything else is listed as not-tested or not-applicable rather than silently omitted.

Exit codes in CI

The CLI and the GitHub Action gate builds with the same semantics: exit 0 when every applicable MUST that ran passed, exit 2 on any MUST deviation, exit 1 when the target could not be tested at all (for example, discovery failed). The JUnit report names every entry with its register check id; deviations carry the full spec citation.

What this is not

A green result is not a certification and not an endorsement — it reflects the checks run against your server at that moment, no more. The suite is unofficial and not affiliated with the UCP project; where the specification is ambiguous, the register records the ambiguity rather than inventing a requirement.