Skip to main content
The Optimiser takes an agent you can already measure and edits the agent itself. Candidates are real git diffs against your repository — rewritten prompts, sharpened tool descriptions, adjusted control flow — each replayed over your dataset and scored with your evaluators. Only a change that beats the best score so far survives; the winning diff stays on the candidate for you to apply in your repo. The same loop compares models: a backtest replays the dataset through each candidate model instead of each candidate diff.

​
Prerequisites

1

A capability from your code

Candidates are edits to source, so the capability must be synced from your repo with a known entrypoint in overmind.toml. See Agent & Capabilities.
2

An eval dataset

The run replays the agent over the rows of an eval dataset’s active version, which must fit both eval contracts; the inputs must be a shape the agent can be invoked with. See Datasets.
3

An eval set

Candidates are graded by the eval set you pick, pinned at creation so later edits can’t change an in-flight run’s scoring. See Eval.
4

A coding agent in the repository

Your agent executes in your environment, not Overmind’s. The loop is driven from your checkout by the overmind CLI, and the candidate diffs are written by your coding agent. Optimiser → New run in the Console hands you the prompt to paste — Harness for prompt and code optimisation, Backtesting for model comparison — which the /overmind optimise and /overmind backtest commands run end to end.
New optimiser run dialog with Harness and Backtesting tabs

New run: the Harness and Backtesting prompts the coding agent runs.

The run’s budget is the number of iterations (default 5) and candidates per iteration (default 3); patience — consecutive iterations without improvement before early stop — is 3.

​
How a run works

The client owns codegen, the smoke test, and every datapoint run; the server holds the dataset version, scores posted outputs, and records the winner. Each step is one CLI verb — set-template, run-smoke, run-baseline, add-candidate, run-iteration, complete — and overmind optimise next tells the coding agent which one comes next. See the CLI. A run opens with a smoke test — before spending anything, the first datapoint runs through the command template to verify the entrypoint can be invoked. Then it measures the baseline: your agent exactly as it is, scored across the whole dataset as iteration 0. Each subsequent iteration takes a batch of candidate diffs from the coding agent, applies each in its own git worktree, replays every datapoint through it (one shared pool of workers across candidates and datapoints), posts the outputs, and waits for the platform to score them with the same eval pipeline used everywhere else — every candidate links to a full eval run you can open. A candidate becomes the new best only if it beats the best score so far. Runs end by exhausting the budget or early-stopping after consecutive stalls, and can be cancelled at any point.

​
Modes

A backtest ranks its candidates coverage-fair: by score, then by the share of the eval suite’s claims the candidate’s outputs covered, then by fewest excluded commands — so a model that scored well by skipping hard datapoints does not win. When every candidate ties and the suite left card claims uncovered, no winner is selected and the run says so. Model calls in a backtest can run on Overmind credits through the platform gateway or on your own OpenRouter key.

​
Following a run

Optimiser run page with score chart

A finished run: baseline to best over five iterations, with the score-per-iteration chart and the iteration log.

The header carries the baseline score, the best tested harness or model, the overall winner, iterations and candidates completed, patience, elapsed time, when it started, and the entrypoint under test. Each iteration expands into its candidates, and each candidate into its patch — the literal diff, copyable and downloadable — plus a link to its eval run:
Optimiser iterations with expanded candidate patch

An expanded iteration: the winning candidate's patch is a reviewable diff, and every candidate links to the eval run that scored it.

​
The result is a scored diff

The winning candidate keeps its diff on the record: the patch on the run page. Apply it in your repository with your normal git workflow; the coding agent that drove the loop already has it staged in the candidate’s worktree. Because candidates were scored with the same evaluators you use everywhere else, the headline score is directly comparable to your eval runs and live trace scores.

​
API

The loop’s own calls — POST .../template/, .../add-iteration/, .../results/, .../evaluate/, .../complete/ — are what the CLI posts; you do not call them by hand. Over MCP, check_optimizer_readiness validates the dataset, eval set, model ids, and credits, start_optimizer schedules a run in any mode and returns the CLI command to drive it, and inspect_optimizer_result reads iterations, candidate patches, the winner, and the next action.

​
Good to know

  • You measure what you optimise. If your evaluators don’t capture what you care about, the Optimiser will faithfully improve the wrong thing. Invest in Eval first.
  • Model costs are yours. The loop runs your agent for real — iterations × candidates × datapoints model calls under your provider keys (or Overmind credits for a backtest through the gateway). The budget is the cost lever; the judge calls that score outputs are metered on the platform.
  • The dataset defines the situations explored. A narrow dataset produces improvements narrow to match; refresh it with recent traces periodically.
  • Plateaus point at the model. When runs stop improving, the remaining gap is usually in the model rather than the code around it — the run page points you to Training, and by then you already have the dataset and eval set it needs.