> ## Documentation Index > Fetch the complete documentation index at: https://docs.overmindlab.ai/llms.txt > Use this file to discover all available pages before exploring further. # Inference > Call your fine-tuned models through an OpenAI-compatible endpoint — deployment lifecycle, the capability alias, request fields, error codes, and the Console Inference page. Every successful [training run](/models/training) registers a deployment and serves it from Overmind's GPU pool through an OpenAI-compatible chat completions endpoint. You call it with your existing client and an Overmind API key; the model id is either the deployment's reference or a capability alias that follows whichever model you have made live. ## How a model is served A deployment moves through `queued → quantizing → deploying → warming → ready`, or ends `failed`, `deleting`, `deleted`. Two shapes: * A dense model trained on Modal with LoRA is served as an **adapter on a shared BF16 base**; deployments of the same base share one process and skip the `quantizing` stage. * Everything else — full fine-tunes, mixture-of-experts models, models whose family has no adapter support — is **merged and quantised into a private checkpoint** (FP8 where the base supports it) and served from its own vLLM process. Each process scales to zero when idle: two minutes without traffic for most models, fifteen for the largest. The first request after that cold-boots the model in two to seven minutes; the response is held open with an idle ping every 15 seconds so the connection survives. ## Calling your model Point any OpenAI-compatible client at `https://api.overmindlab.ai/api/v1` with your API key (`Authorization: Bearer ` or `X-Api-Key: `). The model page's **View API snippet** shows this for each model. ```python theme={"system"} from openai import OpenAI client = OpenAI(base_url="https://api.overmindlab.ai/api/v1", api_key="") response = client.chat.completions.create( model="ft-9635f347-qwen3-4b", messages=[{"role": "user", "content": "Classify this ticket: ..."}], ) print(response.choices[0].message.content) ``` The SDK ships the same surface without the OpenAI dependency: `overmind.Client()` reads `OVERMIND_API_KEY` and `OVERMIND_API_URL`, and exposes `chat.completions.create(...)` (streaming with `stream=True`), `models.list(status=)`, `models.get(id)` and `models.delete(id)`. Two ids work in `model`: * **The deployment reference** — `ft--`, the deployment's `model_id` on its page — pins one model. * **The capability alias** `overmind/` resolves to the capability's `active_model`. **Make live** on a deployment retargets it; the next request hits the new model. Model ids under `anthropic/`, `deepseek/`, `google/`, `meta-llama/`, `openai/`, `qwen/` and `x-ai/` are frontier models the same endpoint forwards to OpenRouter, so a repository can switch between frontier and fine-tuned models by changing the id alone. They are listed only when the API has an OpenRouter key configured. ### Request fields `POST /api/v1/chat/completions` accepts `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `frequency_penalty`, `presence_penalty`, `tools`, `tool_choice`, `response_format`, `stream`, `stream_options.include_usage`, `reasoning_effort`, `include_reasoning` and `chat_template_kwargs`. Unknown fields are ignored. * **Reasoning.** For families that emit chain-of-thought, `include_reasoning` defaults to `false` and `reasoning_effort` to the family's default; pass them explicitly to override. `chat_template_kwargs` is merged over the family's defaults and forwarded to the chat template. * **`max_tokens`.** Some families spend part of the budget on a preamble that cannot be turned off; a value below the family's floor is raised to it so the answer is not empty. * **Streaming.** `"stream": true` returns SSE `data:` events; the idle ping is an SSE comment line (`: `). * **Non-streaming to a deployed model.** The `200` status line and a newline ping are sent before the model answers, and the JSON body follows as a chunked response. An upstream failure after that point arrives as `{"error": {...}}` inside the `200` body — check for an `error` key, not only the status. ### Error codes | Status | Trigger | | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `400` | Malformed body: not a JSON object, `model` or `messages` missing, numeric fields of the wrong type, an alias that is not `overmind/`; on `DELETE`, a frontier id or an alias | | `402` | Insufficient credits | | `404` | No deployment matches the id in your projects; the alias's capability has no `active_model`; `GET` on a frontier id | | `409` | `DELETE` on a deployment already `deleting` or `deleted` | | `429` | OpenRouter rate limit on a frontier model | | `502` | OpenRouter or the serving backend returned an error | | `503` | The deployment or the alias's target is not `ready`; a frontier id was requested and no OpenRouter key is configured | ## Managing deployments ```bash theme={"system"} curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/v1/models" # ?status=ready by default; ?status=all curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/v1/models/{model_id}" curl -X DELETE -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/v1/models/{model_id}" ``` `GET /api/v1/models` lists `ready` deployments by default; `?status=` accepts any deployment status or `all`, and anything else is `400`. With `ready` or `all` the list also carries one `overmind/` row per capability whose active model is ready, and the OpenRouter frontier models when a key is configured. `GET .../{model_id}` accepts a deployment reference or an alias. `DELETE` on a deployment marks it `deleting`, removes it from the serving backend, marks it `deleted` and returns `{"id": ..., "object": "model", "deleted": true}`; the row stays, and a failed backend removal leaves it `failed` with a `502`. The Console uses the project API, `/api/deployed-models/`, for the rest: | Action | Effect | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST .../{id}/deploy/` | Re-runs registration for the deployment's training job | | `POST .../{id}/retry/` | Resets a `failed` or `deleted` deployment to `queued` and re-runs registration; the training job must be `succeeded` or `deploying`. The **Retry** / **Deploy** button on the model page's banner | | `POST .../{id}/undeploy/` | Removes the model from the serving backend and marks it `deleted`; weights are kept, so `retry/` brings it back | | `DELETE .../{id}/` | Same teardown from the Console's **Delete model** | | `GET .../{id}/metrics/` | Request count, prompt and completion tokens, median and p95 warm latency, median throughput, average cold-start latency, cost this month and total, and the savings against the capability's original model priced at its OpenRouter rate | | `GET .../{id}/activity/` | Requests, tokens, median latency and throughput per `minute`, `hour` or `day` bucket | | `GET .../{id}/live/` | Live serving signal: recent activity, warming, and the backend's runner counts | | `GET .../{id}/checkpoints/` | A presigned download link for the trained weights, valid for one hour | **Make live** is `PATCH /api/capabilities/{id}/` with `active_model` set to the deployment. The CLI's `overmind model download-checkpoint ` fetches the weights through `checkpoints/`. ## The Console Inference page Inference page listing deployed models The list shows **Model**, **Base model**, **Capability**, **Training job**, **Reference**, **Status**, **Requests**, **Tokens**, **Throughput**, **Latency** and **Last active**; latency and throughput are medians over warm calls. Status shows the deployment status until it is `ready`; a ready model shows its serving state from `live/` — **Live** when it served a request in the last 90 seconds or the backend reports a running worker, **Warming** while a cold boot started within the last ten minutes is still in progress, **Dormant** otherwise. The model page carries the same badge next to the reference, **View API snippet**, and **View traces** into Observability filtered to the model. The overview shows **Requests**, **Tokens**, **Throughput** and **Latency**, the context length, GPU and deployment time, the lineage (**Base model**, **Capability**, **Training job**) and **Make live** or the copyable model-swap prompt. Below it: **Activity**, **Throughput** and **Latency** charts from `activity/`, the **Weights & checkpoints** disclosure with **Download weights**, and **Delete model** under Danger zone. A `failed` or `deleted` model shows a banner with **Retry** or **Deploy**. Inference model detail page ## Good to know * Every request through the endpoint is recorded as an inference call and traced under the project, so a live fine-tuned model feeds the datasets for its next round of training. * Quantisation adds a `quantizing` stage to the first deployment of a merged model; adapter deployments skip it. * A retry after `failed` starts from the checkpoint download and re-runs the merge and quantisation.