How it works

From AI behavior to project-specific checks.

EvalGlass turns real AI behavior into project-specific checks your coding agent runs locally — Ask, Observe, Shape, Run, Read, Cluster the failures, Watch for drift, Improve.

You don’t wire up evaluation tooling by hand. You install the plugin once, ask your coding agent what you want to know, and it evaluates your AI application — turning real behavior into checks, running them locally, and handing back a scorecard. A handful of steps, and you decide what counts.

The check loop

1 · Ask

Give your coding agent a normal product question in natural language: evaluate this app, compare this model, inspect this drift.

2 · Observe

EvalGlass reads repo context, prompts, model calls, examples, tools, release changes, and exported traces (OpenTelemetry / OpenInference / local JSON — no SDK, no network).

3 · Shape

Your agent helps author project-specific checks — metrics, examples, rubrics, scenarios, judges, baselines. Authored and generated cases land proposed, never validated reference data.

4 · Run

The vendored runtime evaluates behavior from the supplied evidence, locally, inside your repo. This is not a hosted platform.

5 · Read

The scorecard explains values, deltas, diagnostics, authority, and the unknowns — a non-scored metric is never written as 0.0.

6 · Cluster the failures next

Group the run’s failing and non-scored items by their shared Diagnostic.code, per metric, so a flat number (“faithfulness 0.82”) becomes an actionable failure mode. A non-scored item is grouped by cause, never coerced to 0.0.

7 · Watch for drift next

Re-run on a schedule and compare each run to your baseline — a typed drift.json written on each pass, cron or CI, not a daemon. A regression shows only when the runs are comparable and the interval clears zero; a no-comparable-regression result never means quality is fine.

8 · Improve

Fix prompts, switch models, add examples, review labels, tune metrics — or wire CI on a threshold you approve.

Your agent proposes the metrics

next Today the agent discovers candidate call sites and helps you shape checks. Next, it goes further: it reads your repo, prompts, output shapes, and exported traces to propose a catalog of metrics anchored to the ways your app actually fails — across a quality taxonomy and three tiers (deterministic, reference, judge). You still decide: everything it proposes lands proposed and informational — it suggests what to measure, it never approves it. This is built in the framework and not yet in the version the plugin installs — tracked on the ahead-of-code register.

The journey, in verbs

Each step maps to the /evalglass verbs your agent runs — or you can run them explicitly.

setup → connect. Discover candidate call sites, vendor the runtime into your repo, and import exported traces — or open an opt-in connect --live <platform> lane experimental for langfuse / phoenix / langsmith, where a live pull lands proposed and cannot gate.
run → view → explain. Produce a scorecard, read per-metric status and values (or view --by-call for per-call detail), and hear why each number is or isn’t trustworthy. A traces: route can set unit: trajectory|session|step next to grade the whole agent run, not one call — informational, it cannot gate.
compare → ci. Show a delta only when two runs are comparable, then let CI act on a gate you approved.
watch. next Re-run on a schedule and compare to your baseline — a typed drift.json, cron or CI, not a daemon. Drift is evidence appended after the verdict: it adds no exit class and never promotes the baseline.

Every verb & prompt, in the docs →

What runs where

A fixed chain of authority — the agent does the work, the repo owns the truth, one engine decides.

Coding agent. Reads, scaffolds, runs commands, explains. Does not grant authority.
EvalGlass runtime. Computes typed scorecards and verdicts. Does not edit your application’s behavior.
Host repo. Owns examples, traces, metrics, thresholds, calibration, baselines. Does not get silently overwritten.
Extensions. Bring behavior in or send scorecards out. Do not create false authority.

The plugin is just for setup.

The plugin discovers, scaffolds, and explains — then steps back. The vendored runtime keeps working after the plugin and the agent are gone, and a fresh run is informational: it reports, it doesn’t fail your build. What a scorecard means →

Learn

the mental model behind the loop

Use cases

the moments that trigger a run

Scorecards

the artifact you get back

Get the plugin

two commands in Claude Code

Docs

the detailed reference