Use case · Model switching
You switched the model. Did the product get better?
A new model or provider can lift one behavior and quietly drop another. Instead of eyeballing a few prompts, ask your coding agent to evaluate the switch against your project-specific checks and read the deltas — but only where the two runs are comparable.
Evaluate my AI application after the model switch. Show what improved and what regressed.
What it checks
faithfulness · answer quality · tool use · latency · cost · refusal rate · workflow policy · edge cases. A delta is shown only when the two runs are typed comparable; if the dataset, prompt, or config also moved, EvalGlass reports not_comparable rather than a false regression.
The scorecard
Verdict informational — deltas are comparable, but no gate is approved. Illustrative example, not a measured result.
experimental Comparable is the
floor, not the finish. On the epistemic core each metric carries a confidence interval
and an effective sample size, and a paired delta counts as a regression only when its interval
clears zero — otherwise it is within_noise. A small delta may be noise, not a real
move; the interval is what tells the two apart.
What it will not claim
A green number is not “the new model is better.” It is “these checks moved this much, where the runs were comparable.” EvalGlass never edits your model or prompt, and nothing gates until you approve a threshold in your own config. No false green →
Ask your coding agent.
Evaluate my agentic app using EvalGlass.
Related