Build checks
Trajectory & session grading
Grade the whole agent run, not one call. next
unit: trajectory (or session / step) on a
traces route and EvalGlass groups every call sharing a
trace_id into one aggregate and scores its shape. The verdict is
informational and cannot gate. Built in the framework; not yet in the version the
plugin installs.
The tagline to hold on to: grade the trajectory; an aggregate green is still not proof the agent succeeded. Whole-run grading widens what a Scorecard speaks for — it does not lower the bar for what counts as proof.
A call green is not a run pass
Every individual call can look fine while the run loops, mis-orders its tools, or never reaches the
goal. A unit declares which slice a result speaks
for. The default call scores one model call at a time; a richer unit grades the run as a
whole:
unit: | Grades | Read it as |
|---|---|---|
call | One model call (the default). | A statement about that call only. Absent unit: is byte-identical to this. |
step | Calls sharing a trace_id, grouped. | Finer-grained inspection of a run’s moves. |
trajectory | All calls sharing a trace_id, as one aggregate. | Did the whole ordered run behave? |
session | A whole session, grouped the same way. | End-to-end behavior across a task. |
Declare it on a traces route
The key is additive and fail-closed. Absent unit: ⇒ call, so every existing
config is unchanged; a present-but-bogus value is a setup error, not a silent fallback.
yaml
traces:
- path: runs/agent.jsonl
unit: trajectory # group each trace_id into one aggregate run
Grading a trajectory wires the built-in trajectory_shape@1 aggregate over the run’s
members — one of the six built-ins (see the metrics guide). It reports the
fraction of members that produced a non-null output: a shape signal, never a claim the agent
succeeded.
Egress is worst-of-members (fail-closed)
Collapsing N calls into one aggregate raises a question the call path never faced: if one member of the trace is forbidden from leaving the machine, is the aggregate? EvalGlass resolves it as the worst of its members — an aggregate is egress-OK iff every member is, so a single forbidden or unknown member blocks the whole trajectory. A forbidden trace can never leak into a replay or judge through an average.
- call-level members
- aggregate · informational
Informational — it cannot gate
Traces carry no validated gold, so an aggregate run inherits proposed dataset status and resolves informational. An approved gating threshold over proposed trace data does not activate a gate — it stays informational, never a laundered pass. Below is one illustrative aggregate row:
json
{
"unit_id": "trajectory:trace-42",
"metric": "trajectory_shape",
"point": 0.67,
"verdict": "informational",
"status_counts": { "scored": 1 }
}Illustrative example, not a measured result. Two of three members produced output, so the shape is 0.67 — a description of the recorded run, not a pass and not proof the agent succeeded.
non_evaluable, never 0.0
If no member produced an output, there is no behavior to aggregate. Reporting 0/N = 0.0
would read as “0% complete quality” when the honest state is “no evidence”.
trajectory_shape returns non_evaluable
(output_all_null) instead — the same never-a-fabricated-zero rule the rest of the
framework holds. A call-unit aggregate is non_evaluable too
(not_aggregate), never a made-up value.
unit: ladder and config-reachable whole-run grading are
next: built in the framework, not yet in the version the plugin
installs. The trajectory_shape@1 evaluator existing is not the same as the config-reachable
whole-run grading shipping — this page describes the route once it lands, never today’s installed
default.