Skip to main content
Eval is where you define what “good” means for a capability and measure it — in batch runs that compare models and prompts side by side, and continuously on production traces as they arrive. The same evaluators serve both. The Evaluations area of the Console has three surfaces: Runs, Eval sets, and the Eval library.

​
Evaluators

An evaluator is one reusable, versioned scorer — editing one writes a new version row, so past runs keep meaning what they meant. Five kinds exist:
A model grades the response against a rubric you write in plain language. The rubric is compiled into a checklist of atomic, weighted yes/no questions; on a boolean evaluator an item can be a gate that fails the verdict regardless of the rest. Judges return reasoning and per-item sub-scores. The managed Correctness and Faithfulness judges use a proportional mode instead: they enumerate the output’s claims and score the share that holds. Judge responses are cached content-addressed, so re-scoring identical inputs is free.
Every evaluator declares a scope (final_output, turn, step, trajectory, sample, or dataset), a score type (numeric, categorical, or boolean) with a pass_threshold, an evidence requirement (model_output, harness_artifact, trajectory, or reference), and a surface — model, harness, or any — that says where it may run: a model grader never runs on live trace scoring, a harness grader never runs on a generate variant. If a run cannot supply the required evidence, the evaluator abstains or is marked not-applicable instead of producing a number.

​
Claims

Every evaluator also carries a claim: what it asserts and at which grain. The claim is what trace scoring uses to compose one execution score out of many verdicts. The grain is the slice a claim is judged on: unit (one turn or step), trajectory (the path through the behaviour), terminal (the deliverable), or session (the whole conversation). Grains land in three weighted phases — steps, trajectory, output — and no single evaluator vetoes the composite.

​
Behaviour bindings

An evaluator can be bound to one of the capability’s behaviours with a role: an outcome evaluator judges the behaviour’s deliverable, a step evaluator judges one anchor segment of its route. The Eval library groups evaluators by behaviour (the Console labels it “Task”), shows each behaviour’s coverage (outcome plus every step), and lists the rest under Unassigned. GET /api/behaviours/coverage/?capability={id} returns the same rollup. Judges are authored behaviour-scoped from the behaviour’s group in the Eval library or through the upsert_evaluator MCP tool: POST /api/evaluators/author/ takes behaviour, behaviour_role (outcome or step), and — for a step judge — the anchor_segment, validated against the behaviour’s contract. GET /api/behaviours/{id}/authoring-context/ grounds the authoring with the behaviour’s latest contract: anchor sequence, steps, tool set, and terminal. A judge whose applicable_roles include trace_scoring must be bound to a behaviour, and eval_set plus eval_set_role attach the new judge to an eval set in the same call. PUT /api/evaluators/{id}/author/ re-authors an existing judge.

​
Authoring

Author from the Eval library by hand, or describe what you want to test and let Overmind generate the evaluator (POST /api/evaluators/generate-prompt/). The generated prompt template references the agent’s output as {{output.<path>}} and dataset fields as {{reference.<path>}}, and everything stays editable before saving. overmind scan push preloads the capability’s Default eval set for every capability it created, remounted, or whose card changed, in tiers: Tier 0, rule-based checks compiled from the capability’s card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. When Tier 1 authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability’s evidence profile — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply.
Eval library grouped by behaviour

The Eval library grouped by behaviour, with each behaviour's outcome and step coverage.

Evaluators are platform-managed templates, project-wide, or scoped to a single capability.

​
Eval sets

An eval set is a named group of a capability’s evaluators. Each member carries a role:
  • generative members score outputs produced during eval runs and optimisation runs.
  • trace_scoring members run on every production trace, per task execution, writing verdicts and the execution score into Observability. The mechanics are on Trace scoring.
The same evaluator can hold both roles. One set per capability is active (POST /api/eval-sets/{id}/activate/); it is what trace scoring, optimisation runs and training runs default to. POST /api/eval-sets/{id}/members/ adds members.

​
Eval runs

An eval run is one experiment: one data source × N variants × M evaluators.
  • Data source — dataset (the version is used and frozen, so the run stays reproducible as the dataset evolves) or trace_filter, drawing directly from captured traffic. max_items caps how many rows are drawn.
  • Variants — the columns of the experiment. A variant’s mode is generate (re-run a model over each row) or existing (grade the recorded output as-is). Each pins a model (a catalogue model, a registered endpoint, or one of your trained models) and optionally a prompt; one variant is the baseline (is_baseline) every other is measured against.
  • Evaluators — attached as frozen snapshots at creation, so later library edits cannot change what a completed run measured.
Creation refuses a run whose eval set has no live generative LLM judge, and a run over a closed-form labelled dataset whose eval set carries no comparator against the gold labels. In the Console, an eval dataset version feeds two consumers from its cell: Run the optimiser, and Use in a training job, where it scores the job before and after training. Eval runs themselves are created through the API and the run_evaluation MCP tool.
Execution fans out server-side: sample preparation (generation), then one scoring task per sample × evaluator, then aggregation into the run summary. A run’s status moves pending → running → completed, or ends failed or cancelled. For recorded conversations, a generated variant can set params.generation_strategy to per_assistant_turn. The model produces one decision for each recorded assistant turn, using the conversation recorded before that turn as context. Tool calls are evaluated as decisions without executing their tools. Each subsequent turn uses the recorded history, keeping its context independent of the candidate’s previous decisions. Such a run attaches a per-turn judge automatically, and params.replay_dimensions names what it grades. This also applies to conversations containing a single recorded assistant turn; a tool-only decision does not require a final answer. A sample that exceeds the generation worker’s time limit records an error and is excluded from scoring. Scoring starts after sample preparation finishes, so a run can be generating while its score count is still zero.

​
Reading the report

Eval run report with results, distributions and operations

A finished run: per-evaluator pass rates and mean scores per variant, the score distributions, and cost and latency per model.

The report shows per-variant rollups (mean score, pass rate) per evaluator, the score distribution per evaluator and variant, cost and latency per model — generation cost and eval cost separated — and a per-datapoint table. Opening a datapoint shows each evaluator’s score with its reasoning; the Compare runs tab lays one run’s results next to another’s.
Per-datapoint evaluator scores with reasoning

Datapoint drill-down: each evaluator's score with reasoning. A check that could not be applied is not-applicable and does not enter the averages.

  • Outcomes are typed. Every score’s outcome is scored, abstained, not_applicable, skipped, or error — and only scored results feed averages and pass rates.
  • Samples can be degraded. Each trace is normalised into a canonical transcript before scoring; if it cannot be reconstructed, the sample is marked degraded with a degraded_reason and excluded from aggregates. Failure scores also carry a failure_role — root_cause or propagated — so one upstream failure does not read as five independent ones.
Scores accumulate into a per-capability history (GET /api/evaluators/score-history/) that powers the trend chart on the capability page.

​
Calibrating judges

Samples accept annotations — human labels with an optional note, written through the annotate_evaluation_sample MCP tool — giving you ground truth to compare judge behaviour against. When a judge and your annotations disagree consistently, tighten the rubric; the compiled checklist shows which criterion is drifting.