How Core compares
How EvalGlass Core compares to other AI eval tools
EvalGlass Core is a different shape of tool from most AI-evaluation products: its checks are derived from your own system — code, schemas, prompts, traces — and owned as code in your repo, where the usual starting point elsewhere is a generic metric catalog you assemble yourself. Core is local-first, open-source (Apache-2.0), and operated through your coding agent — no hosted platform, no SDK to wire in by hand.
This page helps you choose honestly, not win a checkmark race. Core is pre-alpha; several tools below are mature, generally-available products that do things Core does not — we say so plainly, and end with where Core is not the right fit. Discovery finds what to test; Intelligence explains why it failed — different jobs from Core's execution, excluded here until they are generally available and can be compared factually.
Factual comparison
Only objectively verifiable, stable properties are compared. We deliberately do not score, rate, or rank these tools, and we make no performance or accuracy claims about any of them. Categories overlap: some are observability platforms that also run evals, some are test frameworks. Check each vendor's own docs for current details.
| Tool | License | Deployment | Primary interface | Maturity | Source |
|---|---|---|---|---|---|
| EvalGlass Core | Apache-2.0 (open source) | Local-first — runs on your machine and in CI; no hosted platform | Coding-agent plugin (Claude Code; Codex as a second runtime) | Pre-alpha — v0.2.1, tagged, not yet on PyPI | GitHub |
| Promptfoo | MIT (open source) | Local-first CLI; optional hosted cloud | CLI + YAML config / SDK | Generally available | promptfoo.dev |
| DeepEval | Apache-2.0 (open source) | Local Python framework; optional Confident AI cloud | Python test framework / SDK | Generally available | deepeval.com |
| Langfuse | MIT core (open source; enterprise modules licensed separately) | Self-host or managed cloud | SDKs + web UI | Generally available | langfuse.com |
| Arize Phoenix | Elastic License 2.0 (source-available) | Self-host or Arize cloud | SDKs + web UI | Generally available | Arize-ai/phoenix |
| LangSmith | Proprietary (commercial) | Managed cloud; self-host on the enterprise plan | SDKs + web platform | Generally available | langchain.com/langsmith |
| Braintrust | Proprietary (commercial) | Hosted SaaS; hybrid self-host on enterprise (control plane stays hosted) | Web platform + SDKs | Generally available | braintrust.dev |
Facts checked 2026-06-06 · see docs/design/seo-geo-alternatives-review-checklist.md
On provider keys and telemetry
EvalGlass needs no provider API keys on its run path and sends no telemetry: an LLM-as-judge metric is an optional, host-owned lane, not a requirement of the core run. Most eval tools call provider APIs to compute model-graded metrics, and several collect usage analytics by default — but those depend on each tool's configuration and version, so we do not assert them per tool here. Check each vendor's own docs for its current posture. Read EvalGlass's own stance on Local-first and No false green.
When EvalGlass is not the right fit
EvalGlass is deliberately narrow and early. Choose a different tool when:
- You want a hosted dashboard, managed storage, or a team web UI today — EvalGlass keeps project truth in your repo and ships no hosted platform.
- You need a mature, stable release right now — EvalGlass is pre-alpha (v0.2.1, tagged, not yet on PyPI), and its contracts are still settling.
- You want a large prebuilt metric or benchmark library out of the box — EvalGlass helps you and your agent author project-specific checks, not a packaged metric suite or leaderboard.
- Your team does not work through a coding agent (Claude Code or Codex) — the primary workflow assumes one.
- You need always-on live production tracing through vendor SDKs today — EvalGlass imports exported traces (OpenTelemetry, OpenInference, or local JSON); a live pull is an experimental, opt-in lane (the
connect --liveverb), never always-on production tracing.
Where EvalGlass does fit
- You already work through a coding agent and want evaluation to live in that workflow, not a separate platform.
- You want evaluation to stay local-first and repo-native, with your prompts, traces, and gold data staying in your project.
- You care about no false green — bounded scorecards that never imply more than the run actually measured.
- You want CI gates you approve yourself in host-owned config, rather than authority granted by the tool.
More context: Trust model · Local-first · Honesty charter · Roadmap & status · What it is / isn't · Vocabulary · Source on GitHub.