Trust

Your coding agent runs the evaluation. It never decides what the scores are worth.

EvalGlass stays honest by design: local-first, host-owned truth, bounded scorecards, and no false green — a pass is a bounded claim, never proof your AI is correct.

Letting an agent set up your evaluation is only safe if it can’t quietly inflate what a score means. EvalGlass is built so it can’t — not by policy, but by construction. Six promises hold whether a human or an agent is at the keyboard.

Human or agent

An honest evaluation

no false confidence

You decide what countsA score gates CI only after you approve the bar — never the agent.

Green never overclaimsA pass means only what the run checked — not that your output is correct.

AI judges earn the gateAn LLM grader can’t fail your build until it’s calibrated to your labels.

Every score shows its evidenceEach result traces to records you can inspect — never just prose.

It measures, never editsIt reads your outputs; it never changes your prompts, models, or code.

Nothing hidden, nothing hostedVendored in your repo — no platform, no keys; it outlives the agent.

Each promise — and where it’s enforced

• You decide what counts. Your agent can draft reference data, metrics, and config, but no score gates CI until you approve a threshold. The decision with consequences stays yours — that’s human oversight, by construction. How approval works →
• Green never overclaims. A fresh run scores your outputs but gates nothing — no threshold is approved yet. It measured; it never claims your output is correct, only what it actually checked. On the epistemic core a green is measured against more than a point: the default gate reads a metric’s lower confidence bound, so a perfect 3/3 fails the default gate — it passes only under a named point-smoke policy. experimental Why a perfect score can fail → · What a score means →
• AI judges earn the gate. When an LLM grades your outputs, EvalGlass treats it as unproven: it can report, but it can’t fail your build until you’ve calibrated it against your own labelled examples. No AI grades AI on trust alone. On the epistemic core that floor becomes structural: authority is capability-typed — a fake or uncalibrated judge can never gate — agreement with your labels is a computed JudgeAgreementStudy (Cohen κ), not a promise, and a content-addressed JudgeInstrument means a swapped judge reads drifted. experimental Judge integrity → · How calibration works →
• Every score shows its evidence. Each result carries the evidence and provenance behind it in typed records. Authority lives in that data, never in report prose — so you can audit a result, not just trust it. On the epistemic core each score also carries its confidence interval, and a load-time recompute rejects a tampered record — a hand-edited scorecard fails closed rather than being trusted. experimental Confidence & intervals → · How provenance works →
• It measures, never edits. EvalGlass reads the outputs your app already produced. It can hand a scorecard to other tools, but it never touches your prompts, models, or code. Where the line is →
• Nothing hidden, nothing hosted. It runs from vendored source in your repo and in CI — no platform, no keys, your data stays put. Every line is yours to read, and the runtime keeps working after the agent and the plugin are gone — the same whichever runtime installed it. How it’s vendored →