Build checks
Datasets
Host-owned examples, with an honest validity status.
A dataset is host-owned examples — input, output, and an optional reference —
read as JSONL. Its status defaults to proposed, and only
you can move it to validated. That status is what keeps a
fresh metric informational: a reference metric can’t gate on gold
nobody reviewed.
Datasets are evidence, and EvalGlass is careful about how much a piece of evidence is allowed to claim. So the distinction that matters is not “example vs. dataset” but proposed vs. validated — and where a row came from.
The shape of an example
One JSONL row per example; a reference is optional and drives the reference lens:
json
{"input": "What is the capital of France?", "output": "Paris", "reference": "Paris"}
{"input": "Summarise the refund policy.", "output": "…", "reference": null}
A row with no reference is fine — for it, a reference metric is
non_evaluable (counted, excluded from the math), and a
non-reference metric still applies. Connect or scaffold a starter dataset with
/evalglass connect; import exported behavior as
traces.
Evidence status
| Status | What it means | Can a reference metric gate on it? |
|---|---|---|
| proposed | Scaffolded or supplied, not yet reviewed. | No — keeps the metric informational. |
| validated | A host reviewed and approved it as domain truth. | Yes — once the threshold is approved too. |
| generated | Produced by a (future) scenario workflow. | No — forced to proposed, can never enter as gold. |
Generated data never enters as validated gold
There is no generator shipping today, and the governance is deliberate: any generated or synthetic
case is forced to proposed and must be reviewed before it can
back a gate. See the optional lanes
for how that scenario workflow is bounded.