Runtime & Integrations · Evaluation Core
Unit ladder
The unit: route key that grades a whole run. next
unit: call | step | trajectory | session
(a UnitKind) to choose the slice it grades. call is one model call; a richer
kind groups every unit sharing a trace_id into one aggregate scored by
trajectory_shape@1. Absent unit: is byte-identical to call.
Built in the framework; not yet in the version the plugin installs.
This is one route through the existing TraceEnvelope → EvalUnit → Example spine — no new
score meaning and no second verdict path. The human-first walkthrough is
Trajectory & session grading; this page is the typed contract.
The route key
TraceConfig carries a kind: UnitKind (YAML key unit), parsed
with the same _coerce_enum pattern as format / data_policy.
The key is additive and fail-closed: absent ⇒ call (every pre-P1 config unchanged), and
a present-but-bogus value is a setup error, not a silent fallback.
unit | Grades | Selection |
|---|---|---|
call | One model call — its input and output. | The default. One Example per call; the run fingerprint is byte-identical to a config with no unit:. |
step | Calls sharing a trace_id. | Grouped by trace_id like trajectory (finer step locators are out of scope). |
trajectory | An ordered run — all calls sharing a trace_id. | Grouped into one aggregate Example with unit_id = "trajectory:<trace_id>". |
session | A whole session. | Grouped the same way, unit_id = "session:<trace_id>". |
Reachability is two-part: the key parses at the config layer, and trajectory_shape is
registered in the harness evaluator loader so a metric may declare
evaluator_ref: trajectory_shape@1. The TRACE_SOURCE optional lane stays
call-only — aggregate grading is reachable through the built-in traces:
route.
How selection works
select_units(units, kind=…) is the selector. CALL delegates to the
unchanged per-call path; a richer kind groups call-level TraceUnits by
trace_id (insertion order preserved) into one aggregate Example whose
output is the per-member output list and whose unit.members lists the
sub-unit ids. trajectory_shape@1 reads that members list and reports the
fraction that produced a non-null output — one shape signal, never a claim the agent succeeded. The
same trace_id split across two trace files yields two aggregates (grouping is per-read).
Worst-of-members egress
Collapsing N members into one aggregate drops the per-unit egress bool the runner relies on.
_load_trace_units(units, kind) resolves it: the aggregate’s egress is the
worst of its members — permitted iff every member is, so a single
forbidden / unknown /
missing member blocks the whole aggregate (fail-closed). A forbidden
trace can never leak into a replay or judge egress through an average of the permitted parts.
The honesty guards
trajectory_shape returns non_evaluable —
not_aggregate on a call unit (a metric whose granularity
disagrees with the produced unit guards itself), no_members on an aggregate with no
sub-units, and output_all_null when no member produced an output. That last case is the
empty-trajectory rule: 0/N = 0.0 would read as “0% complete quality” when the
honest state is “no evidence”, so it is non_evaluable, never a fabricated
0.0.
And it cannot gate. Traces carry no validated gold, so an aggregate run inherits proposed status and resolves informational. An approved gating threshold over proposed trace data does not activate a gate — it stays informational, never a laundered pass.
An illustrative aggregate
One aggregate row for a three-member trajectory where two members produced output:
json
{
"unit_id": "trajectory:trace-42",
"metric": "trajectory_shape",
"point": 0.67,
"verdict": "informational",
"status_counts": { "scored": 1 }
}Illustrative example, not a measured result. The shape is 0.67 — a description of the recorded run, not a pass and not proof the agent succeeded.