CI gates for agentic apps

You are shipping an AI feature, and you want the same safety net you already have for ordinary code: a check in CI that catches a regression in the pull request, before it reaches a user. So you wire your eval into the pipeline. The wiring is the easy part. The hard part is the question underneath it — when should this check merely inform the PR, and when should it actually block the merge? Treat that question as automatic and you reach for the most natural default in all of CI: green means ship. On day one that default is wrong, and wrong in a way that manufactures exactly the false confidence an eval is supposed to remove. This guide is about the deliberate path from a check that informs to a gate that blocks — who walks it, and how the build reads the result honestly along the way.

The reason builders gate at all is well established. OpenAI's evaluation guidance recommends continuous evaluation that runs on every change and an eval set that grows over time¹ — the point of CI is to make the check automatic so it does not depend on someone remembering to run it. And Anthropic's engineering team is blunt about the cost of skipping it: without evals, debugging is reactive and teams cannot distinguish real regressions from noise.² A regression you catch in the PR is a line in a diff review; the same regression you catch in production is an incident. That is the whole case for wiring evals into CI. It is also where the trouble starts, because the mechanics of CI quietly invite a claim the eval has not earned. The sibling guide on eval-driven agentic development covers the change moments worth a check and ends exactly where this one begins — the line between an informational first run and an approved gate.

Why wire an eval into CI at all

Start by being precise about what a CI eval is for, because the failure mode lives in the gap between what you want it to do and what a passing run can support. What you want is a tripwire: a change of a kind that can move behavior — a reworded prompt, a swapped model, a re-indexed retriever, a new tool — runs against your project's examples automatically, and a real regression surfaces as a failed check on the PR rather than as a support ticket next week. That is a sound, ordinary engineering instinct, and EvalGlass is built to serve it: the CI workflow runs the vendored runtime on every change and hands the build a typed verdict.

The instinct goes wrong the moment it borrows the unit-test reflex wholesale. A green unit suite does license “these exact assertions held, so this case is correct.” A green eval does not license the analogous claim, because the things it measured are sampled behavior, scored by imperfect metrics, against thresholds someone chose, on examples someone supplied — and on a fresh install, against thresholds nobody has approved yet. So the eval can do everything you wanted of the tripwire while still not supporting the sentence “green, therefore safe to ship.” Holding those two apart is the entire discipline of gating an agentic app.

The "green = ship" trap

Here is the trap stated exactly, because it is the most common way CI gating for AI goes wrong. You wire the eval in on day one, the pipeline goes green, and you — or your pipeline's configuration — read that green as proof the change is safe. It is not, and the reason is structural rather than a matter of degree. EvalGlass starts every install with empty authority: a fresh setup has validated no dataset and approved no threshold, so it is gating on nothing. Its first run is therefore informational — it reports what it measured and what it could not, and it exits cleanly. The build is green because the run decided nothing, not because the run cleared a bar.

That would be a harmless detail except for one fact about exit codes that makes it dangerous: exit 0 covers both an informational run and a pass. The shell cannot tell you which one you got. A pipeline that treats every 0 as “quality was checked” will wave through a run that gated nothing, with full green confidence, forever — it is reading enforcement off a signal that only ever meant “nothing failed.” This is the same point the sibling guide on what a green AI eval actually means makes about exit 0 not being verification, applied to the specific moment you put the eval in CI.

The mistake is reading a green pipeline as a decision. On a fresh install it is the absence of a decision — an informational run that gated nothing, wearing the color of a pass.

Note what EvalGlass is not doing here. It is not exiting 1 out of the box to force you into a gate, because a tool that failed a build on data no human had validated, against a threshold no human had approved, would be manufacturing confidence from nothing — the precise failure no false green exists to refuse. The empty-authority default is deliberate. It makes the honest state — “evidence, not enforcement” — the starting state, and leaves the decision to gate where it belongs.

From informational run to activated gate

Moving from an informational check to a real gate is not a flag you flip; it is a sequence of deliberate, host-owned acts, each one of which is you taking responsibility for a specific claim. A metric gates only when four conditions all hold: the dataset is validated, a threshold is approved, the metric is set to gating, and — for a judge-based metric — the judge is calibrated against your own labeled examples. Miss any one and the metric stays informational: it reports, it does not gate. The promote a gate doc walks each condition in full; the point here is the shape of the boundary.

1. lane A · evidence

   Fresh install — empty authority

   No dataset validated, no threshold approved. The setup gates nothing by construction.

2. lane A · evidence

   Informational run

   Reports what it measured and could not, then exits 0. Evidence, not enforcement — green-lights nothing.

3. authority point · host

   Four host-owned acts

   Validate the dataset, approve a threshold, set the metric to gating, calibrate the judge if it is a judge metric. A host YAML edit, referencing approval records.

4. lane B · enforcement

   Active gate

   Now resolves a real outcome — pass (exit 0), fail (exit 1), or blocked (exit 1) — on the dimension you approved.

The crossing between the lanes is the whole subject of this guide, and it is worth stating in the flattest possible terms: there is no promote, gate, approve, or certify verb anywhere under /evalglass for the agent — or the run — to call. Activation is a host-owned YAML edit, setting metric_status: gating and threshold_approval: approved against a validated dataset, where each of those keys is honored only because it references a host-owned approval record that the config cannot mint for itself. Your coding agent can type that edit at your explicit direction — it is faster at YAML than you are — but typing it does not grant the authority behind it. The authority comes from the records you reviewed, which is to say from you. An agent operating the keyboard does not make the gate more authoritative, only faster to write down.

This is the same boundary as everywhere else in EvalGlass — the actor that creates evidence cannot authorize it to gate — landed on the one act builders are most tempted to automate. A CI gate is a promise you make and can defend: that this dataset is sound, this threshold is right, this judge is calibrated. Promote one gate at a time, deliberately, and only for a metric whose promise you would stand behind in a postmortem. (A practical note on the data: EvalGlass has no synthetic-data generator — datasets are host-supplied and host-validated, never machine-minted — so “validated dataset” always means examples a human reviewed.)

How CI reads the verdict

Once a gate is active, the build needs to read its result, and the one rule that keeps that reading honest is: consume the typed verdict, never the prose. EvalGlass resolves exactly one outcome per run in a single Verdict Engine, and the vendored CI workflow adds no pass/fail logic of its own — it relays the runtime's exit code. CI consumes the typed VerdictPayload — the exit code, or the verdict field in scorecard.json — and never parses report.md or terminal text, because grepping a rendering for the word “pass” couples your gate to wording and can read an authority the typed scorecard never granted.

The callout the table exists to make is the top two rows: exit 0 spans two meanings. informational and pass both exit 0, so a green exit on its own does not tell you whether anything gated. If you need enforcement — a build that proceeds only on a real pass — you read the verdict field explicitly rather than trusting the bare exit code, and you treat exit 2 as a distinct infrastructure signal so a flaky run never masquerades as a regression. The full shell recipe and the precedence rule (blocked outranks fail outranks pass outranks informational) live in the CI exit-code contract; this is the reader's-eye summary.

One honesty note on resolution, because a gate can only read what was actually measured. Today the finest grain EvalGlass reports is per-call detail — view --by-call groups scores by explicit subject identity. A finer per-source-function score view is planned, not built, so a gate reads the per-call ceiling EvalGlass actually has, never a resolution it does not yet.

Who actually blocks the merge

There is a last boundary that builders routinely collapse, and getting it right is what keeps the whole arrangement honest: EvalGlass does not block your merge. It reports a verdict and sets an exit code. Whether that exit code stops a pull request from merging is a separate, host-owned decision that lives in GitHub, not in the eval tool. GitHub Actions sets a check run's status from the step's exit code — 0 succeeds, any non-zero fails the job³ — but a failing check does not prevent a merge until the repo owner makes it a required status check in branch protection. Required checks must pass before collaborators can merge, and that requirement is configured by the repository owner in the branch's protection rules.⁴

So the chain that actually blocks a merge has three distinct owners, and EvalGlass owns only the first link. EvalGlass reports the verdict and exit code. GitHub Actions translates the exit code into a check-run status. The repo owner's branch-protection config enforces the block by making that check required. The tool surfaces a measurement; the host's configuration — the YAML edit that activated the gate, and the branch-protection rule that makes the check required — is what turns the measurement into an enforced block. A CI gate, properly understood, is a promise the host makes and can defend, not an automation the tool performs.

Sources

Field context for why builders wire evals into CI, and for the GitHub mechanisms that actually translate an exit code into a merge block. None of these sources describe EvalGlass or endorse auto-gating; the empty-authority default, the four host-owned gate conditions, and the no-self-approval boundary are EvalGlass's own design choices, presented as such. A passing gate is evidence the host can defend, never proof of correctness.

1. OpenAI — Evaluation best practices: set up continuous evaluation to run on every change and grow the eval set over time. Context for why builders gate; not a claim that a passing gate proves correctness.
2. Anthropic — Demystifying evals for AI agents (2026): absent evals, debugging is reactive and teams cannot distinguish real regressions from noise — context for catching a regression in the PR rather than in production.
3. GitHub Docs — Setting exit codes for actions: a step's exit code sets the check run status — exit 0 is success, any non-zero exit fails the job. The basis for the exit-code contract EvalGlass relies on.
4. GitHub Docs — About protected branches: required status checks must pass before collaborators can merge — the repo-owner-configured mechanism that turns a failing check into a merge block, outside the eval tool.