Statistical honesty experimental
Why a perfect score can fail.
A perfect score can still fail the gate: on the epistemic core a metric is a point, a confidence interval, and an effective sample size, and the default gate decides on the lower bound — so a lucky 3/3 does not pass.
“No false green” stops being a slogan and becomes a mechanism here. A number with no sample size and no interval is not evidence — it is an anecdote with a decimal point. This is the epistemic core: built in the framework, not yet in the version the plugin installs, and never shown executing as if it shipped.
A number is not evidence
Three correct answers out of three is a point of 1.0 — but three observations rule almost
nothing out. On the epistemic core, a score stops being a bare number: each metric carries an
Estimate — a point value, a Wilson/Student-t confidence interval, and
an effective sample size (n_effective). The interval is the honest width of what you
actually know.
The gate reads the lower bound
A DecisionPolicy compares the lower confidence bound to
your threshold, not the hopeful point — for a higher-is-better metric; the upper bound for
lower-is-better. It reads the edge of the interval that could hurt you. So a perfect 3/3 fails
the default gate: three lucky samples have not earned the claim.
The point is a perfect 1.00, but the 95% lower bound is ≈ 0.44 — below the 0.80 threshold (the tick). The default gate reads the lower bound, so this does not pass. Gather more evidence and the interval tightens until the lower bound clears the bar. Illustrative example, not a measured result.
The escape hatch is explicit and named: a policy that sets
decision_statistic: point — a point-smoke check for dev — decides on the
point instead. It is a deliberate, recorded choice, never the silent default.
A fake judge can never gate
Authority is capability-typed: an uncalibrated or fake judge produces
evidence, not authority, so it can never move a gate. A real one earns calibrated only
through a computed JudgeAgreementStudy (Cohen κ recomputed from
the confusion table, not a promise) and a human approval bound to the exact rig. A content-addressed
JudgeInstrument means a swapped judge reads drifted, and a
digest-bound AuthorityGrant ties that approval to the instrument it was granted for.
A tampered record fails closed
Authority lives in typed, content-addressed data, never in report prose. A scorecard is re-derived on load: a load-time recompute rejects a tampered record — hand-edit the numbers and it fails closed rather than being trusted. You can audit a result, not just believe it.
A small delta may be noise
The same discipline governs comparison. A paired baseline delta counts as a
regression only when its interval clears zero; a change inside the interval is
within_noise, not a regression. It is the rule the scheduled drift watcher
(evalglass watch next) applies too — and a
no-comparable-regression result never means quality is fine.
Green means what the evidence earned — no more.
The trust object is the scorecard, never the AI. A pass is a bounded claim the interval, the sample size, and the calibrated authority actually support — not a verdict that your application is correct. That is what makes a green mechanical instead of aspirational. What green does not mean →
The three epistemic homes
This page is the trust story; the same facts live in two companions — read whichever fits your question.
Related