What is built, and what is not

The two status axes

Capability maturity — how far the code has actually reached:

• now Shipped and running on your machine and in CI today.
• next Committed and short-horizon — not shipped yet.
• planned Intended, but not near-term.
• experimental Exists as a stub or deletable lane; not core-reliable.
• nongoal A decided non-goal — not planned, by design.

Offer availability — whether a customer can obtain it under a real offer, shown by a separate squared badge, never a maturity chip:

• open source Freely obtainable today — this is Core.
• private preview A private, gated engagement ahead of general availability — Discovery and Intelligence. They exist and are offered privately, with no public price.
• generally available Openly obtainable under a standing offer — no product is here yet.

Maturity

Read this first. EvalGlass is pre-alpha and should be treated as such.

• Pre-alpha. The shape and the contracts are settling. Expect change.
• Tagged releases; track main. The latest tag is v0.2.1 — see the changelog. Vendor from the current main branch; older states are not maintained.
• Not on PyPI. EvalGlass is vendored into your repo from source, not installed from PyPI.

Built now

These pieces ship in v0.2.1 — they run on your machine and in CI today.

• Claude Code plugin & the /evalglass verbs. The umbrella surface (setup, run, view, explain, compare, ci…) with an always-on honesty guardrail, installed in two commands. now
• Evaluation Core. The effect-free center that owns score meaning, authority, and the single verdict. now
• Runtime Harness. Local datasets and traces, JSON / Markdown reports, and CI exit codes. now
• Replay, baselines & CI annotations. Subprocess replay, structured baseline comparability, and CI annotations. now
• Setup — call-site scan & vendoring. A read-only, mechanical scan of candidate LLM call sites (installation support only), vendoring, and a manifest and lock. now
• Authoring tier. add-metric, add-judge, calibrate — every scaffold lands proposed, never authority. now
• Judges with calibration. Judge metrics that gate only once you calibrate them to your labels. now
• Score subject identity & view --by-call. Per-call grouping by explicit id — never list order. now
• Optional extension lanes. Opt-in, isolated and deletable add-ons outside the required path. now
• HTML scorecard report (report.html). A self-contained report: the verdict, per-metric intervals, a “what this run does not claim” panel, and the delta from your last run. now
• Epistemic core — honest estimates & lower-bound gating. Every score carries a confidence interval, and the gate reads the lower bound — so a lucky 3/3 fails the default gate. How it works → now

Ahead of code — every unshipped capability, with its status

Documented but not shipped to you. Each carries a precise status; none is shown executing as if it ran, and we name no date we cannot keep.

• Live tracing connectors (connect --live). Opt-in, deletable Langfuse / Phoenix / LangSmith lanes; a live pull is proposed and cannot gate. Importing exported traces is the shipped path. experimental
• Drift watch (evalglass watch). Continuous drift: run → compare to baseline → write drift.json → exit (cron or CI, not a daemon). next
• Trajectory & session units. Grade the whole agent run, not one call; informational, cannot gate. next
• Diagnostic clusters. Group a run’s failing cases by their shared cause instead of one flat number. next
• Command judge. An opt-in subprocess judge; uncalibrated → informational until calibrated. next
• OpenAI-compatible judge lane. A generic transport for judge metrics — no provider SDK, authority-free and deletable. experimental
• Metrics explorer. A browsable per-call view of deltas and distributions across runs. planned
• Hosted dashboard sink. An outward view of scores; no telemetry — the repo stays the source of truth. next
• Prompt-optimizer hand-off. Send a scorecard to an external optimizer; EvalGlass never auto-tunes your prompts. next
• Annotation workflow. Route uncertain examples to human review; the governance exists, a polished UI does not. experimental
• Opt-in synthetic scenario generator (connect --synth). An opt-in Core generator; every case is forced proposed, never validated data. No generator exists yet. planned
• More worked examples. End-to-end walkthroughs for common host setups. planned

Non-goals & source of truth

• Not a hosted platform. EvalGlass stays local-first. There is no SaaS and no telemetry.
• It will not auto-tune your prompts. It measures and reports honestly; it does not change your application.
• Per-source-function score attribution. Mapping a score back to the exact source function that produced a call would need trace↔call-site correlation that does not exist; inventing one would be false confidence. This is a decided non-goal, not a backlog item — per-call view --by-call ships today.
• The repository is the source of truth. This page may lag the code. When they disagree, the repo wins. GitHub milestones →