Vendoring · M3

EvalGlass Skill

Install EvalGlass without granting authority.

The EvalGlass Skill is now delivered as part of the EvalGlass plugin. It discovers a host repo read-only, proposes a non-authoritative plan, installs by vendoring the managed runtime into evals/_evalglass/ and scaffolding host-owned starter assets, and supports safe re-vendoring via setup --upgrade. It vendors only core, harness, and adapters — never the plugin's own component directories. It grants no authority and is not needed at run time.

Adoption must not create false confidence. So the Skill is conservative at every step: it inspects without executing, plans before writing, scaffolds non-authoritative assets, preserves host files, and leaves a validation checklist. To get hands-on, see Install the plugin; this page explains the mechanism.

How it works

INSPECT

read-only · writes nothing

discover

plan (non-authoritative)

install

INSTALL

vendor + scaffold

managed runtime

manifest + lock

host assets

authority.json empty

stop

VALIDATE

you decide · grants authority

validate gold

approve thresholds

calibrate judges

run

RUN

informational by default

✗grants_authority = false

Skill (integration-time)
human validation (grants authority)
first run (informational)

Discover and plan write nothing; install vendors the managed runtime and scaffolds non-authoritative starter assets; then it stops for you to validate. Only host validation grants authority — and the runtime keeps working after the Skill is gone.

The four commands

Command	Effect
`discover --root`	Read-only AST scan over source text — no exec, no LLM SDK — producing a `HostDiscoveryReport` and data-policy prompts.
`plan --root`	A non-authoritative `InstallPlan`; preserves host assets; `grants_authority` is structurally `false`.
`install --root`	Vendors managed files (rewriting imports to `_evalglass`, injecting the pinned version), writes the manifest + lock, and scaffolds host assets.
`revendor --root [--dry-run] [--confirm]`	Updates managed files only — see Revendor Safely.

What discovery finds

discover walks the repo read-only (honouring ignore files) and reports what it sees — it never executes host code or imports a provider SDK:

Signal	How it’s found
Python project	A `pyproject.toml` / `setup.py` or any `.py` file.
LLM call sites	An AST pass flags calls that look like a model call (known SDK names + `.create`/`.generate`/`.complete`, or hint substrings). Surfaced for your review — never run.
Trace candidates	`.jsonl` files in trace-shaped directories or named like a trace.
CI configs	YAML under `.github/workflows/`.
Existing eval assets	Every host file under `evals/` (outside the managed dir) — recorded as truth to preserve.

For each recorded data source with no declared policy, discovery emits a DataPolicyPrompt (an open question — permitted / redacted / forbidden), and it raises open questions such as “no validated gold found; reference metrics stay informational until you supply and validate gold.” It assumes nothing.

What install writes

Install does two things, and reports created vs preserved so nothing is silently overwritten:

1 · Vendors the managed runtime

Copies only the managed runtime — core, harness, and adapters — into evals/_evalglass/ with every import rewritten to the _evalglass namespace and the pinned version injected, then writes vendor-manifest.json (every managed file + hash) and evalglass.lock (framework_version + installed_features).

Plugin component directories are never vendored The plugin's own delivery surface stays in the plugin and is never copied into the host: skills/, commands/, hooks/, plugin-docs/, assets/, bin/, and .claude-plugin/ (and .codex-plugin/). Only core, harness, and adapters land in evals/_evalglass/. The vendored runtime is just the framework — none of the agent-facing machinery comes with it.

2 · Scaffolds host-owned starter assets

Seven files under evals/ — each skipped if it already exists:

File	Purpose
`evalglass.yaml`	Commented starter config — informational by design.
`datasets/sample.jsonl`	Sample gold (proposed, not validated).
`traces/sample.jsonl`	A sample recorded trace.
`evaluators/answer_nonempty.py`	A host-evaluator template.
`authority.json`	An empty `AuthorityRecord` → first run is informational.
`README.md`	The validation checklist.
`ci/github-actions.yml`	A CI snippet wired to the exit code.

Scaffolds stay non-authoritative

Scaffolded gold lands proposed; thresholds proposed; judges uncalibrated.
authority.json starts empty, so the first run is informational.
The Skill never overwrites host files and reports conflicts instead of guessing.

Typed contracts

Every Skill output is a typed, JSON-serialisable artifact (a JSON contract, not a JSON Schema file):

Contract	Produced by
`HostDiscoveryReport` · `DataPolicyPrompt`	`discover`
`InstallPlan`	`plan`
`VendorManifest` · `ManagedFileRecord` · `EvalglassLock`	`install` / `revendor`
`AuthorityRecord`	scaffold (empty) — the host fills it to grant gating
`InstallerError`	any refused or malformed operation

No runtime dependency on the plugin — across runtimes The runtime never imports the Skill (guarded in CI). Install, then delete the plugin and the agent context — your evaluation still runs. This holds across runtimes: the vendored runtime is identical whichever runtime installed it (Claude Code or Codex) and keeps working after the plugin and agent are gone. To pick up a newer framework version, re-vendor with /evalglass setup --upgrade.

Next steps

First Run

Install the plugin

The hands-on discover → install → run walkthrough.

Vendoring

Host Layout

Managed vs host-owned on disk.

Guide

Revendor Safely

Upgrade managed runtime.

The installer lives in src/evalglass/installer/ — discovery.py, plan.py, vendor.py, scaffold.py, revendor.py (ADR 0010–0013).