Run and read

Compare baselines

Claim a regression only when the runs are comparable.

/evalglass baseline promotes a baseline explicitly; /evalglass compare requests a comparison and shows a delta only when the typed comparability claim is comparable. EvalGlass claims a regression only when the run and baseline fingerprints match on the gating dimensions; otherwise it reports missing_baseline or not_comparable and makes no delta. “Improved” and “regressed” are scoped to your project-specific checks, never an absolute quality claim.

A dropped number is meaningless if the dataset, metric, or config also changed. So comparability is a typed state you read, not a subtraction you trust.

1 · Promote a baseline

Run /evalglass baseline to promote a baseline. Promotion is deliberate and explicit, never automatic — and separate from run, so an ordinary evaluation never writes a baseline. Under the hood the verb runs:

bash

PYTHONPATH=evals python -m _evalglass.harness.cli baseline update \
  --from reports/run/runrecord.json \
  --to   baselines/main.json

2 · Request a comparison

Run /evalglass compare to point the run at the baseline and request the comparison. The Scorecard’s baseline_state then resolves to one of — and a delta is shown only for comparable:

State	Read it as
comparable	Gating dimensions match — a delta is a supported claim.
`not_comparable`	A baseline exists but a gating dimension changed — do not compare.
`missing_baseline`	Comparison requested, but no baseline was found.
`comparison_not_requested`	No comparison asked for.

Comparability can gate If a metric is configured to require a baseline, anything but comparable resolves its authority to blocked — the gate refuses to compare across mismatched runs.

Next steps

Trust Model

Baseline Comparability

The four states and the decision tree.

Ship

Promote a Gate

When a baseline becomes a required gate.

Run and read

Read a Scorecard

Where baseline_state appears.