Inference

Your trained models are served behind an OpenAI-compatible API at https://api.overmindlab.ai/api/v1 (or your own host), authenticated with the same ovr_ key you use everywhere else. Frontier models are callable through the same gateway, so comparing your trained model against a hosted one is a change of the model field, not a second integration.

Deployment lifecycle

Deployment starts automatically when a training experiment succeeds. A model moves through Queued → Deploying → Warming → Ready, and only Ready models accept traffic; a failed deployment can be retried without retraining. Undeploy stops serving and scales the GPU pool to zero; the row reads Not deployed and the weights are preserved, so Deploy brings the same model back. Delete has the same effect on the serving side and keeps the archived checkpoint too — a deleted model is retried back into service the same way, or downloaded with overmind model download-checkpoint. How a model is served depends on how it was trained. A dense model trained as a LoRA adapter on Modal is served as an adapter on a shared BF16 base: every adapter of that base shares one container pool, so a second one deploys in seconds and a cold start is amortised across them. Everything else — full fine-tunes, mixture-of-experts, Baseten-trained models — is merged and FP8-quantised into a private checkpoint with its own pool.

Inference page with deployed models table — The Inference page: every deployed model with its base model, capability, training run, copyable reference, serving state, and usage.

The Status column is the live serving state: Live when the model served a call in the last 90 seconds or has warm workers, Warming during a cold boot, and Dormant when it is deployed but idle — a dormant model still accepts traffic; the first call after a pause pays the cold start. Opening a model shows its lineage (base model, capability, training run — each linked), live metrics (requests, tokens, throughput in tokens/s, model-inference latency, cost, cost this month), the savings against the capability’s original model, downloadable weights, the API snippet, and the danger zone.

Deployed model detail with metrics charts and weights download — A served model: requests, tokens, activity, throughput and latency over time, and the weights download.

Latency here measures model inference time only, as a median with a p95 tail. Requests that hit a cold model are tracked separately as cold_start_ms and excluded from the warm figures. A genuine cold boot takes two to seven minutes; while the model warms, the gateway keeps the connection alive with an idle ping every 15 seconds, on streaming and non-streaming requests alike — a non-streaming completion therefore arrives as a chunked response, and a failure that happens after the first ping arrives as an {"error": ...} body under HTTP 200.

Calling your model

Use any OpenAI SDK pointed at the base URL, with your model’s reference (copyable from the models table) as the model name — or the capability alias, overmind/<capability-uuid>, which resolves to whichever deployment the capability has made live, so a model switch never touches the calling code:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.overmindlab.ai/api/v1",
    api_key="ovr_your_key_here",
)

response = client.chat.completions.create(
    model="overmind/<capability-uuid>",   # or the model reference, e.g. ft-9635f347-qwen3-4b
    messages=[{"role": "user", "content": "Classify this ticket: ..."}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

import overmind

client = overmind.Client()  # reads OVERMIND_API_KEY / OVERMIND_API_URL

response = client.chat.completions.create(
    model="ft-9635f347-qwen3-4b",
    messages=[{"role": "user", "content": "Classify this ticket: ..."}],
)
print(response.choices[0].message.content)

for m in client.models.list().data:   # also: client.models.get(id), client.models.delete(id)
    print(m.id, m.status)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.overmindlab.ai/api/v1",
  apiKey: process.env.OVERMIND_API_KEY,
});

const response = await client.chat.completions.create({
  model: "ft-9635f347-qwen3-4b",
  messages: [{ role: "user", content: "Classify this ticket: ..." }],
});
console.log(response.choices[0].message.content);

curl https://api.overmindlab.ai/api/v1/chat/completions \
  -H "Authorization: Bearer ovr_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ft-9635f347-qwen3-4b",
    "messages": [{"role": "user", "content": "Classify this ticket: ..."}]
  }'

The request takes the OpenAI fields you would expect — messages, temperature, max_tokens, top_p, frequency_penalty, presence_penalty, tools, tool_choice, response_format, stream, and stream_options.include_usage — plus reasoning_effort, include_reasoning, and chat_template_kwargs for models that reason. A deployed model gets its family’s reasoning defaults filled in, and max_tokens is raised to the family floor when it is too small to hold a reasoning trace.

Endpoints

Method	Path	Purpose
`POST`	`/api/v1/chat/completions`	Generate a completion; streaming via `stream: true`
`GET`	`/api/v1/models`	List your deployed trained models, the capability aliases, and the callable frontier models. Filter with `?status=` — `ready` (default), `queued`, `deploying`, `warming`, `failed`, `deleting`, `deleted`, or `all`
`GET`	`/api/v1/models/{model_id}`	One model’s status and metadata (any status)
`DELETE`	`/api/v1/models/{model_id}`	Stop serving a trained model; the archived checkpoint stays. An alias cannot be deleted

The gateway routes by model id: your ft-... references and overmind/... aliases go to Overmind’s serving infrastructure; frontier ids (openai/..., anthropic/..., google/..., x-ai/..., deepseek/..., meta-llama/..., qwen/...) go to OpenRouter under the platform’s key and are metered as inference. Every call appears in the model’s usage metrics. The models list is the live source of truth for what you can call; the Model Library covers the base models behind your trained ones, with parameters, context windows, and pricing.

Errors worth handling

Code	Meaning	What to do
`400`	Malformed body, non-numeric `temperature` or `max_tokens`, or an unknown `?status=`	Fix the request
`402`	Insufficient credits (hosted)	Top up under Settings
`404`	Model not found, not in this key’s project, or an alias with no live model	Check the model reference and the key
`409`	Deleting a model that is already deleting or deleted	Nothing to do
`429`	Upstream rate limit (frontier models)	Standard SDK retry logic works — the status is passed through
`502`	Transient backend/provider error	Retry with backoff
`503`	Model not ready — still deploying or warming, or frontier inference not configured on a self-host	Poll `GET /api/v1/models/{id}` and retry

Closing the loop

Point your traced application at a deployed model — or at the capability alias — and its calls flow back into Observability like any other provider’s: tokens, latency, and live eval scores included. A model switch is therefore measurable in production with the same evaluators that justified it offline. For the switch itself, Make live on the model retargets the alias, or copy the model swap prompt from the training run into your coding agent to edit the code.

Overview

Tracing

Core

Agent Testing

Models

Agents & CLI

Platform

Deployment lifecycle

Calling your model

Endpoints

Errors worth handling

Closing the loop

​Deployment lifecycle

​Calling your model

​Endpoints

​Errors worth handling

​Closing the loop

Deployment lifecycle

Calling your model

Endpoints

Errors worth handling

Closing the loop