Ship

Watch for drift

One honest drift check per run, then exit. next

evalglass watch runs one evaluation, compares it to the configured baseline, writes a typed drift.json, and exits. “Continuous” means a scheduled re-invocation — a nightly cron or a CI job — not a resident daemon. Built in the framework; not yet in the version the plugin installs.

Quality rarely fails at one moment; it erodes — a provider updates, a prompt tweak lands, retrieval degrades. The watcher answers “is it worse than baseline?” on a schedule, and answers it honestly. Its one non-negotiable: a no-comparable-regression result never means quality is fine — only that this comparison found no comparable regression.

One run, then exit

Each invocation is a single, bounded cycle: run → load the baseline → compare → write drift.json → exit. There is no polling loop and no long-running process — the cadence lives outside EvalGlass, in your scheduler. This keeps effects bounded and stays true to the local-first identity: no resident process, no hosted monitor.

Schedule it with cron or CI

Because “continuous” is scheduled re-invocation, you wire it into whatever already runs on a cadence. The watcher writes the artifact; a later step reads it back. Drift never sets the exit code, so reading drift.json is how a pipeline learns what changed.

bash

# nightly cron entry — one drift check per invocation, then exit (never a daemon)
0 3 * * *  cd /srv/app && evalglass watch --config evalglass.yaml

# a scheduled CI job does the same, then reads the artifact back
evalglass watch --config evalglass.yaml
jq '.comparability, .comparison.deltas' reports/*/drift.json

What licenses a regression

The watcher reuses the epistemic core’s paired comparison and its interval-licensing rule — it never re-implements the math. A metric is called a regression only when the two runs are comparable and the paired confidence interval on the delta clears zero on the worse side. A delta whose interval still straddles zero is within_noise. A metric with no declared direction is skipped and recorded, never a crash.

Comparability	What it means	Drift reported
`comparable`	The runs share a comparable baseline; a paired per-metric comparison ran.	Per-metric `regression` / `within_noise` / `improvement`.
`not_comparable`	The current run is not comparable to the baseline (fingerprints differ).	Reported as-is — no regression claim, never laundered into “no regression”.
`missing_baseline`	No baseline file yet (a legitimate first run before any promotion).	Reported as-is — drift simply not evaluated.

Drift informs; it never decides A DriftResult carries no verdict, no exit class, and no authority. Drift surfaces as the typed drift.json artifact plus an explanatory Diagnostic (INFO / WARNING) appended after the verdict. The exit code still derives only from the run’s own verdict — the same 0 / 1 / 2 taxonomy, unchanged. If a regression should fail CI, that flows through the Verdict Engine on a comparable baseline with an approved gate, not through a new drift exit.

It reads the baseline; it never moves it

The watcher reads the promoted baseline to run the item-paired comparison — it never writes it. Drifting does not move the bar. Promotion stays the deliberate, separate baseline update act, documented under Compare baselines.

An illustrative drift.json

A comparable check, with one metric flagged and one inside the noise band. It carries no verdict field at all — the artifact is informational by construction:

json

{
  "comparability": "comparable",
  "comparison": {
    "baseline_run_id": "run-2026-06-30-abc123",
    "deltas": {
      "faithfulness": {
        "metric": "faithfulness",
        "n_paired": 24,
        "delta": -0.08,
        "outcome": "regression",
        "interval": { "method": "student_t", "level": 0.95, "lower": -0.14, "upper": -0.02 }
      },
      "helpfulness": {
        "metric": "helpfulness",
        "n_paired": 24,
        "delta": -0.01,
        "outcome": "within_noise",
        "interval": { "method": "student_t", "level": 0.95, "lower": -0.05, "upper": 0.03 }
      }
    }
  }
}

Illustrative example, not a measured result. faithfulness is a regression only because the interval on its delta clears zero; helpfulness moved too, but its interval still straddles zero, so it is within_noise. Even a clean result is informational — a no-comparable-regression drift never certifies quality.

Next steps

Reference

Drift artifact

The typed drift.json + paired-comparison contract.

Reference

Baseline comparability

When two runs can honestly be compared at all.

Guide

Compare baselines

Promote the bar the watcher reads — a separate, deliberate act.

The drift watcher lives in src/evalglass/harness/drift.py and the watch command in src/evalglass/harness/cli.py; the paired comparison it reuses is src/evalglass/core/comparison.py (ADR 0048).