“Set up EvalGlass in this repo.”
you getcandidate call sites discovered and the runtime vendored into your repo as plain code — nothing gates yet.
verbs: /evalglass setup → connect
Get the plugin
Install the EvalGlass plugin in your coding agent.
Install EvalGlass as a Claude Code plugin (Codex supported as a second runtime) and operate it through your coding agent — it grants no gate, approve, or certify authority of its own.
Two commands
Claude Code + Codex
No keys · no telemetry
Apache-2.0 · pre-alpha
Install in Claude Code
One install, then plain language. Your coding agent derives project-specific checks you own as code — your AI’s real spec.
/plugin marketplace add Evalglass/evalglass-core /plugin install evalglass-core@evalglass
evalglass-core is the plugin; evalglass is the publisher catalog. Prefer not to use the marketplace? Clone the repo and run claude --plugin-dir .
Then ask your coding agent, in plain language:
Evaluate my agentic app with EvalGlass.
See it work before you install anything
The bundled quickstart runs with no setup and no host repo — real, populated non-reference
signal and an honest informational verdict, so you can read a result before wiring
it into your own code.
/evalglass run --example quickstart
First things to say
Once it's installed, describe what you want in plain language — the right verb runs underneath. For instance:
You
Evaluate my agentic app with EvalGlass — I just switched the model. Did anything my app actually cares about regress?
Your coding agent · Claude Code plugin
Done — I ran the checks I derived for your app against your last baseline.
Retrieval faithfulness +0.07, your refund-policy guardrail held, cost −18%
— but workflow-policy answers −0.11. Deltas show only where the runs are
comparable. The verdict is informational: no threshold is approved, so nothing gates CI yet.
Want the one-line config to make it gate, for you to approve?
A few more to start with:
“Evaluate my agentic app after the model switch — what improved, what regressed?”
you geta scorecard with per-metric deltas and an honest verdict — informational until you approve a gate, so nothing fails the build yet.
verbs: /evalglass run → view
“Explain why each number is or isn’t trustworthy.”
you getthe authority and diagnostics behind every metric — a missing field is reported as missing, never invented as 0.0.
verb: /evalglass explain
“Set up a CI gate for the checks we’ve reviewed and approved.”
you geta CI workflow that blocks only on a gate you approved — no auto-approval, no false green.
verb: /evalglass ci
Also supported — the direct CLI
The plugin is an additive convenience layer; the framework also works as a plain CLI with no agent. Vendor the runtime, then run against it:
# vendor the runtime into your repo python -m evalglass.installer install --root . # run an evaluation against it PYTHONPATH=evals python -m _evalglass.harness.cli run --config evals/evalglass.yaml
Host evaluation always runs the vendored _evalglass runtime — never the plugin — so it keeps working after the plugin and the agent are gone. Everything here is plain files you can read.
On Codex now
EvalGlass packages for Codex from the same repository and skills tree — ask your agent to evaluate your app and the same umbrella routes the work. A public Codex marketplace listing isn’t published yet; until then, the direct CLI above is the way in on Codex. How the second runtime works →
The plugin grants no authority.
Anything it scaffolds starts as a draft: a proposed dataset, an informational
metric, an uncalibrated judge. Your first run reports a Scorecard, but no score gates CI until
you approve a threshold in your own config — there is no gate verb to do it for you. The
vendored runtime keeps working after you remove the plugin, on either runtime.
How trust works →
Next
How it works
the journey, end to end
The /evalglass verbs
the whole surface, with honesty notes
How it's vendored
what setup writes, and what it never touches