From traces to agent evals

You have done the hard part already: you captured what your agent actually did. Maybe your framework writes OpenTelemetry spans, maybe a tracing layer emits OpenInference, maybe you log your own runs as JSON lines. Either way you now have files full of recorded behavior — model calls, tool calls, retrievals, handoffs, in order — and a reasonable question: how does any of that become an eval? How does a pile of spans turn into evidence for the project-specific checks you care about, the ones that actually decide whether this agent is good enough to ship?

The instinct most people bring is that the trace is the answer — that having the log means having the verdict, and the job is just to read it. That instinct quietly breaks three ways. It assumes the trace is automatically the truth about whether the agent behaved correctly; it assumes a malformed or partial trace means the agent failed; and it assumes a tool that “supports OpenTelemetry” connects live to your tracing backend. None of those hold. This article walks the path from exported behavior to a scorecard, and is careful at each step about what that evidence proves and what it does not. It assumes the model in evaluating agentic workflows — the EvalUnit ladder, process versus outcome, traces as evidence — and makes the import concrete.

You captured behavior — now what?

Start from what a trace actually is. A trace is the captured behavior of one run: its model calls, tool calls, retrievals, and handoffs, in the order they happened. OpenAI's own framing is the same — a trace “captures the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run.”² That record is rich, and it is tempting to think richness is the same thing as judgment. It is not. A trace is raw material. The step from raw material to an eval is the step from what happened to was what happened right — and only a check you wrote, scoped to a unit you declared, can take that step.

So the real work has a shape. First the recorded behavior has to be read in — imported — and normalized into something an evaluator can run on. Then your checks run over that normalized shape, scoped to the slice of behavior you said they speak for. Then a scorecard comes back that says exactly what the run established and no more. The rest of this piece follows that shape and names the honesty constraint that lives at each joint, because every joint is a place where a measurement can quietly start claiming more than it earned.

A trace is a subject, not a verdict

The single most important idea here is also the easiest to lose: a trace is the subject a check runs on, not a verdict to be believed. It is input that must be validated, not a result to be trusted. The trace records what the workflow did — it called this tool, passed these arguments, retrieved these documents, produced this final state. It says nothing at all about whether any of that was correct. Correctness is not a property the trace carries; it is a property only your project-specific checks, run against the examples and thresholds you own, can assess. A perfectly well-formed trace of a completely wrong agent run is still a perfectly well-formed trace.

A trace records what the workflow did, never whether what it did was right. Mapping it cleanly proves it was readable — not that the agent behaved.

This is why “the trace imported successfully” and “the agent passed” are two entirely different statements that beginners routinely collapse. A successful import means the spans were well-formed enough to reconstruct into evaluable units. Whether those units then meet your bar is a separate question, answered only after your checks run. Holding those apart is the whole discipline: the trace is the thing under evaluation, never the evaluation itself. EvalGlass treats it exactly that way — as a validated subject that the checks you authored then measure, on the EvalUnit you declared.

Import is not a live integration

Before the trace can be a subject, it has to get in — and here the second misconception waits. “OpenTelemetry support” sounds like an integration: point the tool at your collector, or hand it your Langfuse / Phoenix / LangSmith credentials, and it pulls your traces live. That is not what import does today, and the distinction is load-bearing rather than pedantic. EvalGlass reads files you have already exported yourself — local JSONL, or exported OpenTelemetry and OpenInference span dumps. It runs locally, with no provider SDK and no network call on the import path. You hand it a file; it does not reach out to anything.

A live pull from a running tracing platform — connect --live from Langfuse, Phoenix, or LangSmith — is a separate, experimental, opt-in lane, off the required path. EvalGlass badges import-now versus live-experimental rather than blurring them, because the blur is exactly how a project would manufacture a capability it does not have. The connectors page documents each lane with its own, more conservative status, per cell; traces documents the import that works now. If you want trace import today, you export the files and point EvalGlass at them. That constraint is also a feature: nothing leaves your machine, no key is required, and your recorded behavior stays in your repo.

One spine: how a trace becomes evaluable

Three export formats, one evaluator. The way EvalGlass reconciles that is to converge every source through a single vendor-neutral spine and drop the vendor objects at the edge. A local JSONL run, an exported OTel span dump, and an exported OpenInference span dump all map, at the edge, into the same typed shape — TraceEnvelope (vendor-neutral, carrying metadata, data policy, and provenance) → EvalUnit (the declared slice: call, step, trajectory, or session) → Example. Past that edge, nothing downstream knows or cares which vendor the spans came from; the metrics see only a typed Example. That convergence is what lets the checks you author run identically regardless of where the behavior was recorded.

1. 01 · exported behavior

   Files you exported

   OTel JSON, OpenInference JSON, or local JSONL — not a live feed. Mapped at the edge, with no SDK and no network call.

2. 02 · one spine

   Vendor-neutral

   TraceEnvelope → EvalUnit → Example; vendor objects dropped at the edge. Your checks run on it.

3. 03 · scorecard

   Scoped to the unit

   A score speaks only for the EvalUnit measured. A malformed span → a typed diagnostic, excluded — not a 0.0.

The scorecard at the end of that spine speaks only for the EvalUnit it actually measured. If every example you imported was a single call, a green run says those calls met their thresholds — it does not say the multi-step agent behaves correctly end to end, however much the rich trace tempts that reading. A call-scoped pass says nothing about the trajectory or session it sat inside. Mapping a score all the way back to the exact source function that produced it is a separate, unbuilt extension; per-call grouping (view --by-call) is the honest ceiling today. The unit a result speaks for is declared, never implied.

A malformed span is a gap, not a failure

Because a trace is raw input, it can be malformed — a span missing an id, a unit missing its output, a fragment that simply will not reconstruct. The third misconception is that a broken or partial trace means the agent failed, that the gap should land as a low score. Here EvalGlass refuses the easy, dishonest move. A span it cannot map does not silently become a low score; it becomes a typed diagnostic, trace_mapping_incomplete, counted and excluded from the result. A non-scored status carries a null value, never a fabricated 0.0 that would read as “the agent failed” when the truth is “the evidence was incomplete.”

The distinction is the difference between two very different claims: the agent did badly and we could not measure this part. The record reconstructs faithfully or it refuses; it never quietly shrinks, and it never invents a number to fill the hole. A gap is reported as a gap. This matters most in the case that looks reassuring: a run that comes back green with some spans excluded. That green is not proof the excluded behavior passed — it is silence about it. The excluded units were not measured, so the scorecard says nothing about them, and reading the green as “all clear” would be exactly the false confidence the diagnostic exists to prevent. The mechanics of that mapping, and the typed diagnostics behind it, live in traces.

Evidence is still not authority

Suppose the import is clean, the units map, your checks run, and the scorecard is green on validated evidence. There is one more boundary, and it is the deepest. Trace evidence, even validated trace evidence, is still evidence — it is not authority. A measured path is a fact about what happened; whether that fact is allowed to gate a build is a decision, and decisions belong to a human host owner, not to the artifact itself. The reason is the same one that stops a function from grading its own output: the actor that produced the trace cannot also be the actor that authorizes it. Self-produced evidence that approves itself is not evidence at all.

EvalGlass enforces this structurally. It measures the path, surfaces what it found, and returns a scorecard scoped to the EvalUnit it actually measured — and there it stops. A fresh setup gates nothing; the first run is informational and exits cleanly, because gating out of the box would mean trusting data no human had reviewed. Turning a check into a gate is a host edit you make: a human host owner is where someone validates the imported data, approves the thresholds, calibrates any judges, and promotes a baseline before anything is allowed to fail a build. The trace can inform that decision; it can never make it. For the full argument underneath this, see no self-approval.

Where this sits in the field

Turning traces into agent evals is established practice, not an EvalGlass invention, and the lineage is worth stating. OpenAI describes trace grading as assigning structured scores to an agent's trace — “the end-to-end log of decisions, tool calls, and reasoning steps”¹ — and its agent-evaluation guidance is built around starting from traces while you debug behavior, asking whether the agent picked the right tool and whether a handoff happened when it should have.² The vocabulary that makes those moves nameable comes from the observability standards. OpenTelemetry's in-development GenAI semantic conventions define spans for agent operations and tool execution;³ Arize's OpenInference conventions define span kinds including LLM, TOOL, CHAIN, RETRIEVER, RERANKER, AGENT, and EVALUATOR.⁴ Those specs are the shared grammar an export is written in; EvalGlass reads exports written in them. None of them describe EvalGlass.

What is genuinely EvalGlass's own is the posture around that grammar. The exported-only, no-SDK, no-network import stance is a deliberate design choice, not anything the specs require. So is convergence through one vendor-neutral spine with vendor objects dropped at the edge; so is the refusal to fabricate a score from a malformed span; and so is the rule that the tool measures and reports while a human, through reviewed records in the repo, decides and gates. EvalGlass is pre-alpha: Claude Code is the primary runtime, Codex is supported with a public listing still to come, exported-trace import works today while live tracing-platform pulls are planned, and there is no hosted platform and no telemetry. The category — turning recorded agent behavior into evidence for checks you own — is real and practiced; the specific way EvalGlass keeps it honest is the part to judge on its own terms.

Sources

The principle of grading the trace, and the vocabulary of agent execution, come from lab guidance and observability standards. EvalGlass's exported-only, no-SDK import posture — its convergence through one vendor-neutral spine, and its refusal to fabricate scores from malformed evidence — are its own deliberate choices, not requirements of these specs. None of these sources describe EvalGlass.

1. OpenAI — Trace grading: trace grading assigns structured scores or labels to an agent's trace — the end-to-end log of decisions, tool calls, and reasoning steps — rather than only the final output. Field context for grade-the-path; not an EvalGlass result.
2. OpenAI — Evaluate agent workflows: a trace captures the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run; start with traces while debugging, asking whether the agent picked the right tool and whether a handoff happened when it should have.
3. OpenTelemetry — GenAI semantic conventions (in development): defines agent-operation and tool spans — the vendor-neutral grammar an export is written in. The spec does not describe EvalGlass.
4. OpenInference semantic conventions (Arize): span kinds including LLM, TOOL, CHAIN, RETRIEVER, RERANKER, AGENT, and EVALUATOR. The spec does not describe EvalGlass.