Tool-use and trajectory evaluation

Once an AI feature takes more than one step, “did it answer well?” stops being the whole question. A modern agentic workflow plans, picks a tool, passes arguments to it, reads what comes back, and chains several of those moves before it produces the thing a user reads. Each of those moves can go right or wrong on its own, and the final answer can hide every one of them. Tool-use and trajectory evaluation is the practice of checking the moves individually, in order, rather than grading only the destination. It is established practice across the field, not an EvalGlass invention — but the way EvalGlass keeps it honest is its own, and that is what this article is about.

This piece is the concrete how. If you want the underlying model — the EvalUnit ladder, the process-versus-outcome families, the idea that a trace is evidence and not truth — start with evaluating agentic workflows; this article assumes it and builds on it. Here we name the specific things worth checking on a trajectory, show where each one lands, and draw the one boundary that holds all of them together: a check scores what a run observably did, never what a model said it was doing.

The six things worth checking

An agent run is a sequence of decisions, and most of them are checkable. The trick is to stop treating the run as one opaque thing that either worked or didn't, and instead name the distinct questions a path actually answers. There are six that matter for almost any tool-using workflow, and a serious evaluation asks all of them rather than collapsing them into a single pass or fail.

The first is tool selection: given the request, did the workflow reach for the right tool — the search tool, not the calculator? A wrong tool is a failure even when the model later recovers, because recovery costs steps and depends on luck. The second is tool arguments: having chosen a tool, did it pass valid, well-formed inputs scoped to the question actually asked, rather than a malformed query that happens to return something usable? The third is step order: did the run follow the sequence the task required — authenticate before fetch, validate before write? Skipping a required step is a defect even when the output looks fine.

The fourth is retrieval and evidence use: was the answer grounded in what the retriever returned, or did the model ignore its own context and answer from memory? This is where retrieval-augmented systems live or die, and faithfulness is a process check, never an outcome one. The fifth is recovery and fallback: when a call errored or returned nothing, did the run retry, replan, or fall back sensibly — or did it fabricate past the gap as if the failure had not happened? The sixth is the session outcome: did the whole task reach the intended end state, in full context, and reach it for the right reasons? That last one is the final-answer check — and it is one of six, not the only one.

A run can land the right answer with the wrong tool, ignored context, and a lucky recovery. The outcome is one check of six; the other five already happened before the answer existed.

Observable moves, not narrated reasoning

Underneath all six checks sits one boundary, and it is EvalGlass's own design stance: a check reads what a run observably did, never what a model said it was doing. The tool call that appears in the trace, with its name and the arguments actually passed; what the retriever returned and whether the answer cites it; the error span and the retry or replan that followed; the final state the run actually produced — those are observable artifacts, and those are what a check scores. The model's narrated chain-of-thought, its claimed plan, its self-report that “I have now verified the result” — those are claims, and a claim is not a fact. The fact is the tool call that did or did not appear in the trace, and the state that did or did not change.

Put plainly: agent-claimed progress is untrusted input. When a step is skipped but the model narrates that it ran it, the honest evaluation believes the trace, not the narration. EvalGlass does not inspect a model's private or hidden reasoning, and it does not do chain-of-thought monitoring. It scores the observable artifacts you exported and treats everything the model says about itself as a claim to be checked against the record, not a verdict to be accepted. This is a deliberate design choice, not anything the observability specs require, and not a problem the tool pretends to fix — it is simply the line EvalGlass draws about what counts as evidence.

There is good reason to draw it there. The labs building these models have shown that self-narrated reasoning is not a reliable window into what an agent is actually doing: under optimization pressure on the chain-of-thought, agents can learn obfuscated reward hacking, hiding their intent within the reasoning text while still misbehaving.⁵ That finding is field context for why narrated reasoning is untrustworthy — it is Baker et al.'s result, not EvalGlass's, and not a problem EvalGlass solves. EvalGlass's response to it is narrow and structural: don't score the narration in the first place. Score the call, the argument, the return, the state.

Each check along the path

These checks are not abstract; each one attaches to a specific point on the trajectory — an ordered sequence of units sharing a trace_id. Run a typical retrieval workflow along its path and the checks distribute across it cleanly, each reading the observable move beneath it rather than the model's account of that move.

1. plan

   Route the request

   Tool selection. Did it reach for the search tool, not the calculator?

2. call

   Call the tool

   Arguments. Was the query well-formed and scoped to the question asked?

3. retrieve

   Retrieval returns

   Grounding. Did the answer cite what came back, or ignore the context?

4. recover

   A call errors

   Recovery. Did it retry or replan, or fabricate past the gap?

5. outcome

   Final state

   Outcome. Did it reach the intended end state for the right reasons?

The boundary becomes most concrete when you set what a check reads beside what it deliberately does not score. For every move on the path there is an observable fact and a self-narrated claim about that fact, and the whole discipline is preferring the left column to the right.

Reading down the path, five of the six checks have already concluded before the final answer exists. That ordering is the reason scope matters: a check attaches to a point on the trajectory, and a call-scoped result cannot see the earlier moves a trajectory-scoped result can. The richer the unit, the more of the path is in view — which is also why the unit a score speaks for is never implied, only declared. The practical payoff shows up first in agentic app evaluation, where tool selection, step order, and outcome are graded together rather than averaged into one number.

Why a right final answer is not enough

The whole case for path-aware checking rests on one stubborn failure mode: a workflow can return a correct response for the wrong reasons. Picture a research assistant that answers a question correctly. Graded on the final response alone, it passes. But its trace shows it called the wrong tool first, ignored the context it retrieved, and recovered only because the question happened to sit in the model's training data. The next, harder question — the one it cannot shortcut — will fail, and a final-answer score gives no warning at all.

A final-answer score is structurally blind to this, not merely weak at it. Almost everything worth checking — tool choice, arguments, step order, retrieval use, recovery — has already happened by the time the answer exists. Re-measuring only the last move averages all of it away. Catching the broken-path, right-answer case requires checking the moves individually rather than the outcome as a lump. In EvalGlass terms, the outcome is one of six checks, and a call-scoped result (no members) cannot see what a trajectory-scoped result (units sharing a trace_id, with members) can. That is the difference between knowing the destination was reached and knowing the path that reached it was sound.

Trace fragments as evidence

To check the path you need a record of the path, and that record is a trace: the captured moves of a run, in order. EvalGlass reads traces you have already exported — local JSONL, or exported OpenTelemetry and OpenInference span dumps — and maps them at the edge with no provider SDK and no network call; the vendor objects never travel inward. A live pull from a running tracing platform is a separate, planned lane, not what trace import does today and not presented as if it were. The mechanics of that import live in traces.

A trajectory or session lists its members — the units that share a trace_id — and that membership is what lets a path-level check see the earlier moves at all. Each Score can also carry optional provenance, an example_id and a unit_id pointing at the example and unit it came from; that provenance is purely additive — it never changes a score's meaning, authority, or whether it can be aggregated, as the score-semantics page spells out. That per-Score identity is what backs view --by-call, which groups scores by identity rather than list order. Mapping a score all the way back to the source function that produced it is a separate, unbuilt extension — it is planned, and it needs a trace-to-call-site correlation that does not exist yet. Per-call grouping is the honest ceiling today.

Because a trace is raw input, it can be malformed — and here EvalGlass refuses the easy, dishonest move. A span missing an id or an output does not silently become a low score; it becomes a typed diagnostic, trace_mapping_incomplete, counted and excluded from the result. A non-scored status carries value=null — the Score type rejects NaN, infinities, and booleans outright — so the gap reads as a gap, never as a fabricated 0.0 that would look like “the agent failed.” Likewise an aggregate such as trajectory_shape, asked to run over a bare call with no members, returns non_evaluable rather than inventing a number. A gap reported as a gap, never as a failure.

Where this sits in the field

Grading the trajectory rather than the answer is established practice, not an EvalGlass invention, and the lineage is worth stating. OpenAI describes trace grading as assigning structured scores to an agent's trace — “the end-to-end log of decisions, tool calls, and reasoning steps”¹ — and its agent-evaluation guidance is built around exactly the questions this article names: did the agent pick the right tool, and did a handoff happen when it should have?² The vocabulary that makes those moves nameable comes from the observability standards. OpenTelemetry's in-development GenAI conventions define an execute_tool span carrying the tool's name and call id;³ Arize's OpenInference conventions define span kinds including LLM, TOOL, CHAIN, RETRIEVER, and AGENT.⁴ EvalGlass reads exports written in that shared grammar.

What is genuinely EvalGlass's own is the posture around that grammar. The observable-over-narrated boundary — score the trace, treat the model's account of itself as untrusted input — is a deliberate design choice, not anything the specs require, and so is the refusal to fabricate a score from a malformed span, and so is the rule that the tool measures and reports while a human, through reviewed records in the repo, decides and gates. EvalGlass is pre-alpha: Claude Code is the primary runtime, Codex is supported with a public listing still to come, exported-trace import works today while live tracing-platform pulls are planned, and there is no hosted platform and no telemetry. The category — checking tool use and trajectories — is real and practiced; the specific way EvalGlass keeps it honest is the part to judge on its own terms. For why trace evidence still is not authority, see no self-approval.

Sources

The principle of grading the trajectory, and the vocabulary of agent execution, come from lab guidance and observability standards. The CoT-monitoring finding is field context for why self-narrated reasoning is untrustworthy, attributed to its authors — not an EvalGlass result. EvalGlass's observable-over-narrated posture and its refusal to fabricate scores from malformed evidence are its own deliberate choices. None of these sources describe EvalGlass.

1. OpenAI — Trace grading: trace grading assigns structured scores to an agent's trace — the end-to-end log of decisions, tool calls, and reasoning steps.
2. OpenAI — Evaluate agent workflows: did the agent pick the right tool, and did a handoff happen when it should have?
3. OpenTelemetry — GenAI semantic conventions (spans) (in development): defines an execute_tool span carrying gen_ai.tool.name and gen_ai.tool.call.id.
4. OpenInference semantic conventions (Arize): span kinds including LLM, EMBEDDING, CHAIN, RETRIEVER, RERANKER, TOOL, and AGENT.
5. Baker et al., OpenAI — Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (2025): under optimization pressure on the chain-of-thought, agents learn obfuscated reward hacking, hiding intent within the CoT while still misbehaving. Field context for why self-narrated reasoning is untrustworthy — not an EvalGlass result.