A scorecard your team can actually read.

An example scorecard

the same run, as scorecard.json

{
  "verdict": { "verdict": "informational", "ci_should_fail": false },
  "metrics": [
    { "metric": "retrieval_faithfulness", "value": 0.91, "status": "scored", "validity": "valid" },
    { "metric": "workflow_policy", "value": 0.74, "status": "scored", "validity": "valid" }
  ],
  "status_counts": { "scored": 2, "blocked": 0, "non_evaluable": 0, "skipped": 0, "error": 0 },
  "authority": { "retrieval_faithfulness": "no_approved_threshold" },
  "baseline_state": "comparable"
}

The verdict is informational because no threshold is approved — the numbers are earned, the authority to gate on them has not been granted. That distinction is the whole point. The precise, typed version →

The HTML report next

The same typed scorecard, rendered as a shareable report.html written into your repo beside scorecard.json and report.md — a verdict hero, KPI tiles, per-metric confidence-interval bands, a delta vs. the previous run, and a panel spelling out what the run does not claim. It is a rendering of the record, never a second verdict.

Violet marks the 95% interval, the dot is the point value, the vertical tick is the approved threshold. A gate reads the lower end of the interval, not the point — why a perfect score can still miss the bar →. This is an illustrative example, not a measured result.

What a scorecard shows

• Metrics & values. Each project-specific check, its score, and the evidence behind it.
• Status & validity. Whether a metric scored, was blocked, was non-evaluable, skipped, or errored — a non-scored metric is never written as 0.0.
• Deltas & baseline. What improved, regressed, or drifted since a run you promoted — shown only when the two runs are comparable.
• Authority & verdict. Whether a gate is approved, and the one typed outcome the build keys on.

Those project-specific checks come from metric discovery: it reads your code, output schema, and prompts — across any agentic framework — to draft metrics anchored to how your app fails, and can seed a structural floor straight from your output schema, so a first scorecard shows real signal even before you have a single trace. That floor is contract-tier: it catches a malformed output, never proof the answer is good. Discovered metrics land proposed; you validate before any of them gate.

The four outcomes

A scorecard resolves to exactly one verdict. Green means a pass on an approved gate — nothing more. A fresh run is informational, never “passing.”

• informational. Measured and useful, but not an approved gate. The default for a fresh run.
• pass. An approved gate met its threshold — e.g. faithfulness ≥ 0.85, approved. Never a bare claim of correctness.
• fail. An approved gate was validly measured and missed its threshold.
• blocked. EvalGlass cannot make an honest quality claim — evidence, comparability, calibration, policy, or a required measurement is missing. It holds the gate open rather than guessing.

Regression & comparison

A score change is a regression only when the two runs are comparable. If they aren’t, EvalGlass says so and names the dimension that differs — it does not call the delta a regression. That is what makes a model-switch or prompt-change verdict trustworthy. On the epistemic core, a paired delta counts as a regression only when its interval clears zero; a change inside the interval is within_noise, not a regression.

Confidence, not just a number experimental

A score is not a bare number. On the evolving epistemic core, each metric carries an Estimate — a point value, a confidence interval, and an effective sample size (n_effective), computed with Wilson or Student-t. A number with no sample size and no interval is not evidence — it is an anecdote with a decimal point.

A Decision Policy gates on the lower confidence bound (LCB), not the hopeful point — the upper bound for lower-is-better metrics. It reads the edge of the interval that could hurt you.

The escape hatch is explicit and named: a policy that sets decision_statistic: point — a point-smoke check for dev — decides on the point instead. A deliberate, recorded choice, never the silent default.

Authority is capability-typed: an uncalibrated or fake judge produces evidence, not authority, so it can never move a gate. And the judge is content-addressed — a swapped judge reads drifted until it is re-approved against the exact rig. A pass you can trust is a pass the evidence actually earned.

The version the plugin installs today still gates on the point threshold (now); the interval / lower-bound rule here is the experimental layer being hardened in the framework — surfaced so the rule is legible before it ships, never presented as today’s default. A companion trust page, Statistical honesty, goes deeper.

Confidence & intervals, in the docs →

Cluster the failures next

When a metric fails, the scorecard groups its failing and non-scored items by their shared Diagnostic.code, per metric — so “workflow-policy 0.74” becomes an actionable failure mode: the cases that failed share one cause. A non-scored item is grouped by cause, never coerced to 0.0. This groups failure instances by cause — a different axis from the still-planned metrics explorer’s call-identity view.

Where scorecards go

• Local report & repo artifact. A Markdown report and the typed scorecard.json, written into your repo. now
• Shareable HTML report. A rendered report.html — verdict hero, KPI tiles, interval bands, and a “what this run does not claim” panel — written beside the Markdown and JSON. next
• CI annotation. The verdict drives a green or red check on the threshold you approved. now
• Dashboard sink. An optional, outward export of the scorecard — local file export today; a hosted view is a deletable follow-up. No telemetry; the repo stays the source of truth. experimental