Reference
CLI Reference
Every command, flag, and exit code.
/evalglass <verb> umbrella the plugin exposes.
Each verb is an instruction to issue an exact command and read back typed
artifacts — the plugin never computes a verdict. Under it sit two CLIs: the host's
vendored runtime (host evaluation) and the integration installer
CLI (discover / plan / install / re-vendor).
The /evalglass verbs
| Verb | What it does |
|---|---|
/evalglass | Honest status overview; runs nothing. |
setup | Discover candidate call sites, vendor the runtime, scaffold a proposed dataset/metrics/CI. Flags --scan-only, --dry-run, --ci, --upgrade. |
connect | Import exported OTel/OpenInference or local trace JSON (no SDK, no network), or scaffold a dataset. |
connect --live <platform> experimental | Opt-in lane that scaffolds a live Langfuse/Phoenix/LangSmith pull in evalglass.yaml: credentials are env-var names (a literal secret is rejected), data_policy defaults to unknown (egress refused until you permit it), no provider SDK is imported, a missing extra is a clean SKIPPED, and a live pull is proposed and cannot gate. Off the required path. |
discovering-metrics next | Deep-read code, output schemas, prompts, and traces (any agentic framework) → draft a proposed MetricCatalog; the output schema alone drafts a contract-tier structural floor with no traces. Everything proposed/informational. Discover metrics → |
run | Run via the vendored runtime; read back the verdict. run --example quickstart is the pre-install demo. |
view | Per-metric status/validity/values; --by-call groups by explicit subject identity, never list order. |
explain | Why a number is or isn't trustworthy, from typed authority/diagnostics only. |
compare / baseline | Delta only when comparable; baseline promotion is deliberate. |
ci | Copy the CI workflow (alias of setup --ci); exits on ci_should_fail only. |
watch next | One evaluation → compare to the baseline → write a typed drift.json → exit (cron/CI, not a resident daemon). A regression only when the runs are comparable and the paired interval clears zero; a no-comparable-regression result never means quality is fine. |
add-metric · add-judge · calibrate | v1.1 authoring — scaffold proposed/uncalibrated assets; never authority. |
gate/approve/certify/promote-gate verb. Activating a gate is a host YAML edit (metric_status: gating, threshold_approval: approved), guided by the promoting-a-gate skill — never a command.
Host evaluation — the vendored runtime
Host evaluation always runs the host's vendored _evalglass runtime, never the framework package and never via import — this is what keeps a run working after the plugin and the agent are gone:
bash
PYTHONPATH=evals python -m _evalglass.harness.cli run --config evals/evalglass.yaml [--out <dir>] [--format terminal|ci]
PYTHONPATH=evals python -m _evalglass.harness.cli baseline update --from <runrecord.json> --to <baseline.json>Baseline promotion is explicit only — an ordinary run never writes a baseline. (When the framework is pip-installed, the evalglass console script is an alias for python -m evalglass.harness.cli.)
Integration-time — the installer CLI
Discovery, planning, install, and re-vendor run the bundled framework. The portable form (every runtime, incl. Codex) is python -m evalglass.installer; on Claude Code the plugin's bundled launcher ${CLAUDE_PLUGIN_ROOT}/bin/evalglass-launch runs the same code:
bash
python -m evalglass.installer discover --root . # read-only candidate report
python -m evalglass.installer plan --root . # non-authoritative install plan
python -m evalglass.installer install --root .
python -m evalglass.installer revendor --root . [--dry-run] [--confirm]Each emits typed JSON to stdout; a refused clobber goes to stderr and exits 1. (When pip-installed, the evalglass-install console script is an alias.)
Exit codes
| Code | Meaning |
|---|---|
0 | informational or pass — build proceeds. |
1 | fail or blocked — build fails. |
2 | Setup / infrastructure error — not a quality verdict. |