Trust Model
Measurement State
Why a missing measurement is not zero.
Most eval tools collapse everything into one float, so “the judge timed out” and “the answer
was wrong” both become 0.0. EvalGlass refuses that collapse. A measurement that
could not honestly run carries no value and keeps its reason, because invalid or
missing measurement is not low quality — it is the absence of measurement.
- enters aggregation (scored + valid)
- excluded · counted (with a diagnostic)
0.0.Status — does a number exist?
| Status | Meaning | In aggregation |
|---|---|---|
| scored | The metric produced a meaningful value. | Included only when validity is valid. |
| blocked | A required metric could not honestly run. | Excluded from math; blocks active gates. |
non_evaluable | The example lacks meaningful evidence for this metric. | Excluded; blocks if required. |
| skipped | The metric was intentionally not run. | Excluded; may block if required. |
| error | The evaluator or evidence parser failed unexpectedly. | Excluded; blocks active gates. |
Validity — can the number be trusted?
Status and validity are independent. A result can be scored yet not trustworthy:
valid— the value is a sound measurement.invalid— a value was produced but cannot be trusted.not_measured— no measurement was attempted.not_applicable— the metric does not apply to this example.
0.0 rule
Blocked, non-evaluable, skipped, and errored measurements are never encoded as
0.0. The Score type enforces it: a non-scored status must carry
value = null, and a scored status must carry a finite number — no NaN,
no ±inf, no booleans. A missing measurement is not a low score.
Two pairs people confuse
| blocked vs fail | non_evaluable vs skipped |
|---|---|
| blocked is a status: the metric could not be measured, so there is no value. fail is a verdict: the metric was validly measured and came in below its approved threshold. Both can fail CI, but one means “we don’t know” and the other means “we know, and it is short.” | non_evaluable means the evidence for this example is absent — e.g. a reference metric with no reference. skipped means the metric was deliberately not run. The first is about the data; the second is about the plan. |
A non-perfect state is never silent: each Score keeps structured diagnostics and evidence refs, so the “why” survives into the Scorecard and the Verdict Engine turns a present block into a blocked run rather than a quiet pass.