Ship
Wire CI
Let the verdict drive the build — nothing else.
/evalglass ci (an alias of setup --ci) copies a CI workflow that runs the
vendored runtime and exits on the core’s ci_should_fail — and nothing else. It adds
no verdict logic of its own: only the Verdict
Engine decides, and it blocks the build only after the host has approved a gate. Read the
signal from the process exit code (or the typed verdict) — never by
parsing report text. 0 proceeds, 1 fails the build, 2 is an
infrastructure error.
The Verdict Engine already decided the
outcome; the copied workflow’s job is only to act on it. CI consumes the typed
VerdictPayload — it never recomputes quality. A fresh, informational run exits
0 and does not fail the build; that is correct, not a loophole.
Copy the workflow
Run /evalglass ci to copy the CI workflow into your repo. The workflow runs the
vendored runtime; non-zero exit fails the job automatically:
bash
# What the copied workflow runs — non-zero exit fails the job automatically.
PYTHONPATH=evals python -m _evalglass.harness.cli run \
--config evals/evalglass.yaml \
--format ci
--format ci renders GitHub annotations from the typed verdict and status data. The
exit code is the gate; the annotations are for humans reading the log. The workflow itself contains
no pass/fail logic — it only relays the runtime’s exit.
The exit-code contract
| Verdict | Exit class | Code | What CI should read it as |
|---|---|---|---|
| informational | zero | 0 | Evidence only — no active gate ran. Build proceeds. |
| pass | zero | 0 | Active gates met their approved thresholds. Build proceeds. |
| fail | nonzero_fail | 1 | An active gate was validly below threshold. Build fails. |
| blocked | nonzero_blocked | 1 | An active gate could not be claimed honestly. Build fails. |
| setup / infrastructure error | infrastructure_error | 2 | The run itself could not complete — not a quality verdict. |
Exit 2 is deliberately distinct: a crashed config load or missing file is not a
failing quality gate, and conflating the two would let infrastructure flakiness masquerade as a
regression. Infrastructure errors have no Scorecard, so the CLI returns code 2
directly on its setup path.
Same code, different meaning
Exit 0: informational vs pass | Exit 1: blocked vs fail |
|---|---|
informational means nothing gated — the run is evidence, not enforcement. pass means a real gate was active and met. Both exit 0, so exit 0 does not mean quality was checked. |
blocked means a gate could not be measured or claimed; fail means it was measured and fell short. Both exit 1, but one says “we don’t know” and the other “we know, and it’s short.” |
If you need real enforcement, require pass
Exit 0 covers both pass and
informational — so a pipeline that treats every 0 as
“quality checked” will green-light a run that gated nothing. When you need enforcement, inspect
the verdict explicitly:
bash
PYTHONPATH=evals python -m _evalglass.harness.cli run --config evals/evalglass.yaml
code=$?
if [ "$code" -eq 2 ]; then echo "infra error"; exit 2; fi
verdict=$(python -c "import json;print(json.load(open('evals/reports//scorecard.json'))['verdict']['verdict'])")
[ "$verdict" = "pass" ] || { echo "no enforced gate passed (verdict=$verdict)"; exit 1; } report.md and the terminal text are renderings, not the source of truth. Grepping
them for “pass” couples your gate to wording and can read authority that the typed Scorecard
never granted. Branch on the exit code or read scorecard.json.