Runtime & Integrations · Evaluation Core
Score Semantics
One metric result — value separated from meaning.
Score is one metric result: value, status,
validity, diagnostics, evidence refs, evaluator version, and
provenance. Only a scored + valid
Score carries a number; everything else carries value=null.
The type enforces the cardinal no-0.0 rule directly. A non-scored status must carry
a null value, and a scored status must carry a finite number — the constructor rejects
NaN, ±inf, and booleans. This is the code-level enforcement behind
Measurement State.
The shape
json
{
"metric": "exact_match",
"status": "scored",
"validity": "valid",
"value": 1.0,
"evaluator_version": "exact_match@1",
"example_id": "datasets/grounded.jsonl#1",
"unit_id": "datasets/grounded.jsonl#1",
"diagnostics": [],
"evidence_refs": []
}
Score is public — it appears in RunRecord.scores[]. Its
aggregatable() helper is true only when status is scored and
validity is valid; Aggregation includes only those.
Subject identity
Two optional fields record which subject a Score measured:
example_id (str | None) and unit_id
(str | None). They are additive provenance — pointing at the Example / EvalUnit the
Score came from — and do not change its meaning, authority, or aggregation
eligibility; the no-0.0 rule and the fingerprints are untouched.
The engine / harness stamps identity from the current Example / EvalUnit, so built-in and host
evaluators never need to know about it. Both fields are emit-when-present (like
diagnostics and provenance): an identity-less Score serializes exactly as before, and
older runrecord.json artifacts still parse.
/evalglass view --by-call, which groups Scores by
that identity rather than by list order. Mapping a Score back to its source function is a
separate, unbuilt extension planned — it needs a trace↔call-site
correlation that does not exist yet.
Batches
An evaluator may return a single Score or a ScoreBatch of related
Scores (each independently statused). A batch lets one evaluator emit several declared metric
names from one pass without merging their meanings.
scored Score can still be invalid — measured, but not trustworthy.
Status answers “does a value exist?”; validity answers “can it be trusted?”. Only when both pass
does the value enter the math.