Threat model
The adversary is false confidence.
Not a malicious outsider — the well-meaning agent, score, or report that would quietly claim more trust than a run actually earned. Here is how each path to that overclaim is closed by construction.
The threat actor
There is no attacker breaking in. The threat is the eager helper.
It is the coding agent that would approve its own gate to make the build green. It is the AI judge that grades on vibes and calls it measurement. It is the report sentence that sounds like proof. It is the regression flagged against runs that were never comparable. Each of these inflates trust with good intentions — and each is the thing EvalGlass is built to refuse. The defenses below are not policies you opt into; they are how the core is shaped.
Attack → defense
Every row is a way trust gets inflated, and the structural reason it cannot land here.
| Attack | How EvalGlass prevents it |
|---|---|
| An agent self-approves a gate to turn the build green. | Only a human-approved threshold can gate. Agent-proposed values stay informational until you validate them. Evidence & approval → |
| An AI judge grades on trust alone and gates on its own opinion. | A judge cannot gate until it is calibrated against host labels. Uncalibrated judges stay advisory. Judge calibration → |
| Report prose overclaims what the run proved. | Authority is typed data, and reports render from the Scorecard only. Prose cannot carry a claim the data does not. Authority → |
| A regression is claimed without comparable runs. | Baseline comparability is a typed claim. A delta between non-comparable runs is reported as not comparable, not as a regression — and when the scheduled watcher (evalglass watch) finds none, that is not a clean bill of health: a no-comparable-regression result never means quality is fine. next Baseline comparability → |
| A scorecard record is hand-edited so a failing run reads as passing. | The record is content-addressed and re-derived on load: a load-time recompute rejects a tampered record rather than trusting the altered numbers, and authority is bound to the digest. experimental Confidence & intervals → |
| The calibrated judge is swapped for a different model or prompt after approval. | A judge is a content-addressed JudgeInstrument: swap it and the instrument reads drifted, so the earlier calibration and its authority do not transfer. experimental Judge integrity → |
| A perfect score over a handful of examples is used to clear a gate. | The default DecisionPolicy gates on the lower confidence bound, not the point — so a perfect 3/3 (lower bound ≈ 0.44) fails the default gate; it passes only under a named point-smoke policy. experimental Why a perfect score can fail → |
| Generated reference data or proposed thresholds quietly become authority. | Scaffolded assets start informational and stay that way until you validate them. Generation never confers authority. Evidence & approval → |
| Verdict logic is forked into the CLI or the report. | Only the Verdict Engine decides, and the CI exit derives solely from ci_should_fail. There is no second place a pass can be minted. Verdict Engine → · CI exit codes → |
| An optional lane becomes silently required. | Lanes are isolated, opt-in, and deletable. No required path imports one, and deletion leaves the core tests green. Extension lanes → |
Local-first as a data defense
The other quiet threat is data leaving your control. EvalGlass closes that by not having a place to send it. There is no hosted platform, no account, no provider keys, no telemetry — your evidence, datasets, and verdicts never leave the repo. Everything runs on your machine and in your CI from vendored source, and the runtime keeps working after the install skill is removed. Nothing phones home because there is no home to phone.
Out of scope — loudly.
EvalGlass defends against false confidence in evaluation. It does not assert that your model is safe, it does not red-team your application, and it does not replace a security review. A green Scorecard is an honest statement about the evidence behind one evaluation run — nothing more. This is not a security product, and reading it as one would be exactly the kind of overclaim it exists to prevent.
See it in the contracts
The single decision point is the Verdict Engine. Read how it turns typed authority and score state into one outcome — and why nothing else is allowed to.
Related