Runtime & Integrations · Runtime Harness
Runner
The composition root — every route becomes one scored run.
run_config(config, root) → RunRecord reads every dataset and trace through a
port, converges them to one Example list, runs replay and
judge collection, builds a MetricPlan per metric with typed authority inputs and
provenance dimensions, and calls run_evaluation.
This is where Input Convergence actually happens. It also enforces three trust properties before the core ever runs.
Three properties the runner enforces
- Incomplete input can’t pass a gate. A malformed record becomes a run-level
__evalglass_route_error__example that scores error. - Worst-source authority. Across all sources,
_run_authoritytakes the weakest dataset status and least-permissive data policy — a proposed source can’t be averaged away by a validated one. - Egress before effects. Data policy is checked (
_EGRESS_OK = {permitted, redacted}) before any subprocess or judge call, so forbidden evidence never leaves the boundary.
Only a core Example ever crosses to an evaluator (example_from_trace);
richer units group same-trace_id units into a trajectory or session Example. The
framework dimension is stamped as evalglass@<version>.
Where host Python runs
load_evaluator(ref, root) resolves a built-in by name or a host
path.py:function — the one place the Harness executes host code. Baselines are loaded
for comparison but only written by the explicit baseline update command.