Runtime & Integrations · Evaluation Core
Core Contracts
The input shapes, and how parsing fails closed.
The core accepts six typed input shapes. Each parses through a fail-closed
from_dict: a missing, NaN, infinite, boolean-as-number, or overflowing field raises
ContractError — never a silent coercion. Fail-closed parsing is the
framework’s most important bug-class defence.
Malformed input is the easiest way to fake a clean result. So the core refuses to guess: a value that is not exactly what a contract requires stops the parse with a typed error, which the Runtime Harness turns into a setup diagnostic rather than a misleading score.
The six contracts
| Type | What it carries |
|---|---|
TraceEnvelope | Vendor-neutral host behavior + a required DataPolicy; from_dict requires data_policy (no silent unknown). |
EvalUnit | unit_id, kind, trace_id, locator, members; MVP scores call, with richer units explicit. |
Example | The convergence shape: input, output, unit, optional reference, context, metadata, provenance. reference=None is meaningful — it drives non_evaluable. |
Diagnostic | code, severity, message, location, details, cause, evidence_refs; Severity is info/warning/error. |
EvidenceBundle | The effect seam: references, sources, judge evidence, verifier evidence, runtime errors, trace fragments. Evaluators receive this data, not adapters. |
JudgeEvidence | Evidence, not authority. JudgeEvidenceStatus is ok/timeout/provider_error/malformed/missing; only ok may carry a value — a timed-out judge is not 0.0. |
One public error
ContractError is the only public symbol of the validation layer (internal helpers
live in _validation.py). When you see it, parsing refused to invent data — fix the
input, don’t loosen the contract.