Skip to main content
Training turns an agent’s accumulated data into a model of your own. You pick a training dataset and launch one or several experiments in parallel on GPU infrastructure — Modal or Baseten, whichever the platform is configured for. Loss curves stream live, and every finished model is judged with your own evaluators against the model the capability runs today — before anything changes in production.

​
What a run needs

The train/eval separation is checked, not assumed: an overlap check compares the two datasets’ source traces (GET /api/finetuning-jobs/dataset-overlap/), the wizard shows the count, and any row of the training set that also appears in the eval set is excluded from training and logged on the job, so the benchmark never scores data the model saw. The job uses and freezes the exact version it trains on, and validation runs before launch with the same rules the contract applies, so problems surface as fixable findings up front instead of a failed run an hour in.
Training wizard with dataset setup and model picker

The training wizard: datasets and graders on the left, the recommended models with their estimates on the right.

​
Configuring experiments

The wizard recommends configurations from your dataset’s statistics and cost/duration estimates, via the same endpoints you can call directly:
A single run trains multiple experiments in parallel — each with its own base model and settings — and the benchmark decides between them. Every recommended setting stays editable per experiment: epochs, learning rate, and batch size, plus the training method — adapter training (LoRA) with rank, alpha, and dropout exposed, or full fine-tuning where the model supports it. The context length is derived from your longest row and shown, not edited. Over MCP the same steps are get_model_catalog, check_finetune_readiness, estimate_finetune, and start_finetune.

​
Supported models

The catalog holds about forty open-weight instruct models across the Qwen 3.5, 3.8, 3.6, 3 and 2.5 families, Llama 3, Gemma 4, LFM2.5, Antares, GPT-OSS, Muse Glimmer, and Nemotron 3.5 — from a 230M-parameter compact model up to large mixture-of-experts. Every one serves through the same OpenAI-compatible API and produces downloadable weights. Not every model takes both training methods: the 27B-and-up dense models and every mixture-of-experts train as LoRA adapters only, and the picker hides the method toggle where full fine-tuning is unavailable. Three things the list doesn’t show, all enforced at launch. A model’s max training context is often shorter than its inference context window, so a model whose context is shorter than your longest row is dropped from the recommendation and refused at submission. Tool calling varies by model, so if your agent calls tools, only the models that support them are offered. And a backend must be configured — Modal or Baseten — or the launch fails at submission with the reason. The in-product picker is the live source of truth for exactly what’s available to your project.

​
Watching a run

Training run monitor with loss curves and configuration

A finished experiment: train and validation loss per step, the learning-rate schedule, token accuracy, gradient norm, and the run configuration.

The monitor streams per-step metrics for every experiment — train loss, validation loss, the learning-rate schedule, token accuracy (train and validation), and gradient norm — with progress, ETA, tokens processed, and credits used. The same data is available raw:
A run going wrong is visible in the curves long before it ends — validation loss climbing away from train loss — and any experiment can be cancelled individually and retried later (POST .../cancel/, POST .../retry/).

​
The benchmark: your trained model vs. your production model

Each run spawns judge evaluations at two points, both scored on your chosen eval dataset with your chosen graders, over up to 100 rows:
  1. Baseline — the capability’s production model, snapshotted at submit time. When the capability has no resolvable production model, the untouched base model is deployed and scored instead, so there is always a reference.
  2. Final — your newly trained model.
The headline number is the delta of the aggregate score — the mean over the graders — with the per-grader rows underneath. The comparison is deliberately against the model the capability actually runs, because that’s the comparison a switch decision needs.
Training evals table with baseline and final scores

The Evals table under the metrics: Baseline and Final rows per experiment, scored with the same graders.

When the eval dataset’s references are labels — a classifier, a router — the benchmark also reports per-class precision, recall, and F1 with their macro, micro, and weighted aggregates, and a confusion matrix for the final model:
Per-class metrics and confusion matrix on the training page

Classification metrics for a routing capability: per-class F1 and the confusion matrix.

​
After training

A succeeded experiment deploys its model automatically — there is no separate publish step. The model appears in Inference with a copyable reference (shaped like ft-<job>-<base>) and in the capability’s Models section, and its weights are downloadable (overmind model download-checkpoint <deployment-id>). Everything stays linked: a served model traces back to its training run, dataset, and the eval scores that justified it. When the benchmark says the trained model wins, there are two ways to switch:
  • Make live on the model sets it as the capability’s active_model. Anything that calls the capability alias overmind/<capability-uuid> gets the new model on its next request, with no repository change.
  • Copy prompt on the training run (GET /api/finetuning-jobs/{id}/model-swap-prompt/, or the get_model_swap_prompt MCP tool) hands your coding agent the edit that retargets the capability’s code at the new model reference; ?pin=true writes the concrete model id instead of the alias. The switch stays in your normal review flow, never automatic on the server.

​
Good to know

  • Model quality comes almost entirely from the data. Get the active version to fit the train contract with a clean capability report before spending on a run; a few hundred clean, trace-sourced rows beat thousands of noisy ones.
  • The run uses a version; keep curating in the workshop and the next run picks up the new active version.
  • Each experiment reports its final credit cost; estimates are shown before you commit.