What is built, and what is not

What the statuses mean

• now Shipped and running on your machine and in CI today.
• next Committed and short-horizon — not shipped yet.
• planned Intended, but not near-term.
• experimental Exists as a stub or deletable lane; not core-reliable.

Maturity

Read this first. EvalGlass is pre-alpha and should be treated as such.

• Pre-alpha. The shape and the contracts are settling. Expect change.
• No 0.1 release yet. Version is 0.1.0. There are no tagged releases. next
• Only main is supported. Vendor from the current main branch; older states are not maintained.
• Not on PyPI. EvalGlass is not published to PyPI. It is vendored into your repo from source. next

Built now

These pieces exist and run on your machine and in CI. The framework milestones M0–M5 are complete, and the plugin series (P0–P3) plus the score-identity (F1) and authoring (A1) follow-ups have all merged to main.

• Claude Code plugin & the /evalglass verbs. The umbrella surface — setup, connect, run, view, explain, compare, baseline, ci — with an always-on honesty guardrail. Install in two commands. now
• Evaluation Core. The effect-free center that owns score meaning, authority, and the single verdict. now
• Runtime Harness. Local JSONL datasets and traces, Markdown reports, and CI exit codes. now
• Replay, baselines, CI annotations. Subprocess replay, structured baseline comparability, and CI annotations. now
• Setup — discovery & vendoring. Discovery of candidate call sites, vendoring, manifest and lock, and host-owned scaffolds. now
• Authoring tier. add-metric, add-judge, calibrate — every scaffold lands proposed/uncalibrated, never authority. now
• Judges with calibration. Judge metrics that gate only once you calibrate them. now
• Score subject identity & view --by-call. Per-call grouping by explicit example_id/unit_id — never list order. now
• Optional extension lanes. Opt-in, isolated, and deletable add-ons outside the required path. now

Ahead of code — every unshipped capability, with its status

Documented but not shipped to you. Each carries a precise status; none is shown executing as if it ran. We will not name a date we cannot keep.

• A 0.1 release. A first tagged version once the contracts hold steady. next
• PyPI publishing. An installable package alongside vendoring. next
• A public Codex marketplace listing. EvalGlass already runs on Codex from the same canonical skills tree; the public listing is a maintainer step that isn’t published yet. next
• Live tracing connectors (connect --live). Opt-in, deletable lanes for Langfuse, Phoenix, and LangSmith exist in the framework and are reachable with the connect --live <platform> verb, which scaffolds the lane in evalglass.yaml: credentials are env-var names (a literal secret is rejected, never echoed), data_policy defaults to unknown so egress is refused until you permit it, no provider SDK is imported, and a live pull is proposed data that cannot gate. Importing exported traces stays the shipped, now path; the live lanes are built in the framework but not yet in the version the plugin installs. A connector is evidence, never authority — removing the lane changes no verdict. experimental
• Agent-driven metric discovery. Beyond finding candidate call sites, your coding agent reads the repo, prompts, output shapes, and exported traces to propose a catalog of failure-mode-anchored metrics for your app, across a quality taxonomy and three tiers. Everything it proposes lands proposed and informational — it suggests what to measure, it never approves it. Built in the framework; not yet in the version the plugin installs. next
• HTML scorecard report (report.html). Alongside the JSON and Markdown, a single self-contained HTML report: the verdict, per-metric values with their intervals, a “what this run does not claim” panel, and the delta from your previous run. The JSON stays the machine source of truth. Built in the framework; not yet in the version the plugin installs. next
• Epistemic core — honest estimates & lower-bound gating. A score stops being a bare number: each metric carries an Estimate — a point value, a Wilson/Student-t confidence interval, and an effective sample size (n_effective) — and a DecisionPolicy gates on the lower confidence bound, so a perfect 3/3 fails the default gate (its 95% lower bound ≈ 0.44) and passes only under a named point-smoke policy. Authority is capability-typed — a fake or uncalibrated judge can never gate — earned through a computed JudgeAgreementStudy (Cohen κ); a content-addressed JudgeInstrument means a swapped judge reads drifted, a digest-bound AuthorityGrant ties approval to the exact rig, and a load-time recompute rejects a tampered record. Paired baseline deltas flag a regression only when the interval clears zero, else within_noise. Built in the framework; not yet in the version the plugin installs. experimental
• Drift watch (evalglass watch). Make drift detection continuous: one run → compare to your baseline → write a typed drift.json → exit (scheduled on cron or CI, not a resident daemon). A regression is flagged only when the runs are comparable and the paired interval clears zero, else within_noise; not_comparable or a missing baseline is reported as exactly that. It adds no exit class and never promotes the baseline — and a no-comparable-regression result never means quality is fine. Built in the framework; not yet in the version the plugin installs. next
• Trajectory & session units (unit: trajectory|session|step). A traces: route can grade the whole agent run, not one call, wiring the trajectory_shape@1 aggregate; egress is worst-of-members (fail-closed). It is informational and cannot gate, and an empty trajectory is non_evaluable, never 0.0. Built in the framework; not yet in the version the plugin installs. next
• Diagnostic clusters (Scorecard.clusters). Group a run’s failing and non-scored cases by their shared cause instead of reporting one flat number — “the 18% that failed are all missing-citation cases.” A non-scored case is grouped by cause, never coerced to 0.0. Built in the framework; not yet in the version the plugin installs. next
• Command judge. An opt-in subprocess judge — the host provides a command EvalGlass invokes to score. It lands uncalibrated → informational and cannot gate until calibrated to your labels. Built in the framework; not yet in the version the plugin installs. next
• OpenAI-compatible judge lane. A generic OpenAI-compatible transport for judge metrics that imports no provider SDK; rubrics are host-injected and the lane is authority-free and deletable. Opt-in, uncalibrated → informational. Built in the framework; not yet in the version the plugin installs. experimental
• Metrics explorer. A separate, browsable per-call view of deltas and distributions across runs, grouped by call identity — a different axis from the diagnostic clusters above. planned
• Hosted dashboard sink. A local file export of the scorecard ships as a deletable lane; a hosted view to browse scores is a future outward sink — no telemetry, the repo stays the source of truth. next
• Prompt-optimizer hand-off. Send a copy of a scorecard to an external optimizer. EvalGlass never auto-tunes or edits your prompts. next
• Annotation workflow. Route uncertain examples to human review; the governance to return validated truth exists, a polished UI does not. experimental
• More worked examples. End-to-end walkthroughs for common host setups. planned

Non-goals & source of truth

• Not a hosted platform. EvalGlass stays local-first. There is no SaaS and no telemetry.
• It will not auto-tune your prompts. It measures and reports honestly; it does not change your application.
• Synthetic-data generation (connect --synth). The governance exists so any generated case is forced to proposed and can never enter as validated reference data — but no generator is built, by design. A decided non-goal, not a backlog item.
• Per-source-function score attribution. Mapping a score back to the exact source function that produced a call would need trace↔call-site correlation that does not exist; inventing one would be false confidence. This is a decided non-goal, not a backlog item — per-call view --by-call ships today.
• The repository is the source of truth. This page may lag the code. When they disagree, the repo wins. GitHub milestones →