Start here

Quickstart

Install the plugin, then just talk to your coding agent.

EvalGlass is agentic-native: you operate it entirely through your coding agent — Claude Code or Codex — in plain language. No CLI to memorize, no keys, nothing runs on a server. This is the fastest path from install to a first, honest Scorecard.

Five things to say, in order. The first result is not a green “pass” — it is an informational Scorecard, by design. That is exactly the right first thing to see: EvalGlass measuring honestly and claiming only what the evidence supports.

1 · Install the plugin

One time, in your coding agent — add the marketplace, then the plugin:

in your coding agent

/plugin marketplace add Syntelesis-Lab/evalglass
/plugin install evalglass@syntelesis-lab

No keys, no telemetry. Codex is supported as a second runtime. For more detail — and the direct, non-agent CLI if you prefer it — see Get the plugin.

Want to see it work first? (about 10 seconds) Before touching your repo, ask your agent to “Run the EvalGlass quickstart.” It renders a real informational Scorecard from a bundled sample — no setup, no keys — so you can read a result before adopting anything. (/evalglass run --example quickstart)

2 · Set up your app

say to your agent

Set up EvalGlass for my app.

A read-only, consent-gated scan finds candidate LLM call sites, vendors the runtime into your repo as plain code you can read, and scaffolds a proposed dataset, metrics, and CI. Nothing gates yet — it is a reviewed-by-you starting point, never a promise it found every call. (/evalglass setup)

3 · Bring your behavior in

say to your agent

Connect our exported traces.

Your exported OpenTelemetry / OpenInference or local trace JSON is imported — no SDK, no network — so checks run over how your app actually behaved. (/evalglass connect)

4 · Discover what to measure

say to your agent

Discover metrics for my app.

Your agent deep-reads your code, output schemas, prompts, and traces — across any agentic framework — and drafts a proposed MetricCatalog of what to measure, anchored to how your app actually fails. No traces yet? Your output schema alone drafts a contract-tier structural floor. Everything lands proposed / informational. (/evalglass discovering-metrics)

You’re not blocked before this lands Step 2 already scaffolds a proposed starting set of metrics today. Metric discovery is the deeper, domain-meaningful layer — built in the framework and the next canonical release. How discovery works →

5 · Run & read

say to your agent

Run the evaluation and show me the Scorecard.

You get a Scorecard with per-metric values, validity, and one honest verdict. A fresh run gates on nothing, so it is informational, not pass — it exits 0 because nothing has been approved to enforce, and a metric that could not be scored is marked non_evaluable, never 0.0. (/evalglass run → view)

Your first result is informational — that’s correct A fresh install that showed “pass” would be manufacturing confidence from data no human reviewed. To make a check actually enforce CI, you validate a dataset and approve a threshold — there is no gate, approve, or certify verb, so that authority is always yours. Read a Scorecard → · Promote a Gate →

Next steps

Operate it

Agent commands

Everything to say, and every verb underneath.

Build checks

Discover metrics

How your agent derives what to measure.

Run and read

Read a Scorecard

Understand every field in the result you produced.

Ship

Promote a Gate

Turn one informational metric into real enforcement.