Trust Model
Metric Type
Reference vs non-reference metrics.
A metric’s lens is either reference — it compares the output
against host-owned gold — or non-reference — it inspects the output on its own.
Choosing the lens is the first decision when you add a metric, because it determines what
evidence the metric needs and when it cannot run.
The two lenses answer different questions. A reference metric asks “does this match the answer we expected?” A non-reference metric asks “is this output well-formed / present / shaped correctly on its own terms?” Both are useful, and they fail in different ways — which is exactly why running them together is informative.
The two lenses
| Reference | Non-reference |
|---|---|
Compares output to a host-owned reference (gold). |
Inspects the output alone — no gold needed. |
Built-ins: exact-match (binary), set-overlap (Jaccard, 0–1). |
Built-ins: field-presence (0–1), structural-shape (binary), trajectory-shape (aggregate). |
With no reference, the result is non_evaluable — never a score of 0.0. |
Runs without gold, so it stays available even on un-referenced traces. |
Divergence is signal, not noise
Because the lenses test different things, disagreement between them is meaningful. An output can be structurally valid yet wrong, or it can match the gold while drifting in shape. Reading both types together tells you how a result is good or bad:
- Shape high, reference low — well-formed but incorrect. The output looks right and isn’t.
- Shape low, reference high — correct content in a malformed wrapper. Often a parsing or formatting bug.
- Both high — well-formed and matching, within what those metrics measure.
Type does not equal authority
Picking a lens decides what a metric measures, not whether it can gate. A
reference metric on a proposed dataset still stays
informational until the dataset is validated — see
Authority.