FinetuningJob with its own base model and hyperparameters, on the configured GPU backend (FINETUNING_BACKEND, default modal). Loss curves stream while a job runs, and every finished model is scored with the eval set you picked against the model the capability runs today.
What a run needs
Train and eval must not share source traces.
GET /api/finetuning-jobs/dataset-overlap/?dataset=&eval_dataset= counts rows whose trace_id appears in both active versions, and the wizard shows the count under Eval Dataset. Submitting a job whose train and eval versions share a trace is rejected (400, “Split them before training”). When the job runs, it recomputes the overlap against the eval dataset’s active version at that moment and drops any overlapping rows from training, logging the count as a job event. The job uses and freezes the exact version it trains on, and POST /api/finetuning-jobs/validate-dataset/ runs the validator before launch.
The training wizard: datasets and graders on the left, the recommended models with their estimates on the right.
Configuring experiments
The wizard ranks the catalog for your dataset and estimates cost and duration through the same endpoints you can call directly:group_id and the Training page lists them as one run (GET /api/finetuning-jobs/runs/). Each experiment has its own Epochs, Learning rate and Batch, and where the model supports both methods a LoRA adapters switch; with LoRA on, Rank, Alpha and Dropout are exposed. The panel header states the dataset’s task type, row count, token total, longest row and whether it contains tool calls. Over MCP the same steps are get_model_catalog, check_finetune_readiness, estimate_finetune and start_finetune.
Supported models
The training wizard lists these 38 open-weight models. Modal and Baseten share the same catalog rows. Every trained model serves through the same OpenAI-compatible API and its weights are downloadable. Dense models of 27B and above and every mixture-of-experts model train as LoRA adapters only; the picker shows “LoRA fine-tuning” with no switch for them. Three constraints are enforced at launch, not shown in the list:- Training context. A model’s maximum fine-tuning context (
finetuning.context_length, per training method where the catalog splits it) is often shorter than its inference window. A model whose context does not cover the dataset’s longest row plus 256 tokens of headroom is dropped from the recommendation and refused at submission. - Tool calling. When the dataset contains tool calls, only models with
supports_tool_callingare offered. - Backend.
FINETUNING_BACKENDismodalby default;basetenandtogetherremain configurable. Modal trains the same catalog rows as Baseten.
Watching a run
A finished experiment: train and validation loss per step, the learning-rate schedule, token accuracy, gradient norm, and the run configuration.
queued → preparing → running → deploying → succeeded, or ends failed or cancelled. The monitor polls per-step metrics — train loss, validation loss, learning rate, token accuracy (train and validation) and gradient norm — with trained steps, ETA, tokens processed and credits used:
POST /api/finetuning-jobs/{id}/cancel/ stops a running job and its judge evals. POST .../retry/ re-queues a failed or cancelled job.
The benchmark
Each job with an eval dataset and graders runs two judge evals on that dataset, capped at 100 rows by default (hyperparameters.eval_max_items, at most 200):
- Baseline — the capability’s production model, snapshotted on the job as
baseline_modelat submit time. A model that is one of your ready deployments is scored through the Overmind gateway; a model OpenRouter serves goes through OpenRouter; a catalog base OpenRouter does not list is deployed on Modal for the eval. When the capability has no resolvable model, the untouched base model is scored instead. A model with no route lands anunavailablerow. - Final — the trained model, scored through its own deployment once it is ready.
baseline_delta: the final aggregate score (the mean over the graders, gate-only evaluators excluded) minus the baseline’s, with per-grader scores underneath. Each row links to its eval run.
The Evals table under the metrics: Baseline and Final rows per experiment, scored with the same graders.
expected_output values are labels, each row also carries class_metrics: per-class precision, recall, F1 and support, accuracy with macro, micro and weighted aggregates, and a confusion matrix. The Console shows the per-class table with macro and weighted rows and the matrix.
Classification metrics for a routing capability: per-class F1 and the confusion matrix.
After training
A job that finishes training entersdeploying and registers its model automatically; it reads succeeded once the deployment is ready. On the Free plan a deploy past the plan’s limit is skipped and the job records why. The deployed model appears on Inference with the reference ft-<first 8 chars of the job id>-<base model, lower-cased> (for example ft-9635f347-qwen3-4b) and in the capability’s Models tab. Its weights are downloadable from the model page or with overmind model download-checkpoint <deployment-id>.
Two ways to switch the capability to the new model:
- Make live on the deployed model sets it as the capability’s
active_model. Calls to the capability aliasovermind/<capability-uuid>reach the new model on the next request, with no code change. - Copy on the training run (
GET /api/finetuning-jobs/{id}/model-swap-prompt/, or theget_model_swap_promptMCP tool) returns a prompt for your coding agent that retargets the capability’s LLM client at the alias;?pin=truewrites the deployment’s concrete model id instead. Available once the job issucceededand its deployment is ready. The edit lands in your repository through your normal review.
Good to know
- Model quality comes from the data. The active version must fit the train contract before a run can start.
- A run freezes the version it trains on; keep curating in the workshop and the next run picks up the new active version.
- Each experiment reports its credit cost when it completes; estimates are shown before launch.