# Capabilities and repository evidence Source: https://docs.overmindlab.ai/explanation/capabilities How a source map relates to observed behaviour without replacing it. The Agent is the project's map of the application. A capability is a purpose the application serves, such as finding evidence or routing a ticket. Internal roles, tools and model calls describe how that purpose is implemented. Ask this in the project conversation after creating the relevant [tutorial or workflow](/tutorials/index) results. Explain how this project’s capability map relates to its recorded traces in Overmind. Use a real capability to show the connection between repository code, observed executions and evaluation coverage. A local scan combines an AST chassis with coding-agent interpretation. It records entry points, prompts, tools, conditions and transitions in `overmind.toml`. Python and TypeScript/JavaScript source can participate in the chassis. ## A map is not an execution A trajectory map describes source-backed possibilities. An available tool is not proof that the model selected it. A prompt instruction is not equivalent to a code-enforced branch. Missing conditions remain unmapped until the source establishes them. Anchor reachability checks establish structural consistency. They do not guarantee semantic correctness, exhaustive coverage or behaviour in production. Traces supply observed evidence for particular executions. The saved repository snapshot identifies the checkout at scan time. Sync time is separate: uploading an old scan later does not make it a fresh analysis. Older snapshots without provenance remain unknown, and live traces can come from a different revision. ## Stable identity Telemetry binds through `overmind.capability.id`, using a known UUID or slug. A display name is not the binding key. Ingest stores unknown identities as unbound rather than creating a capability from an arbitrary span. A scan that omits a capability makes it leftover and keeps its data. An explicit deletion removes it from current identity lookup while preserving historical evidence. Stable slugs allow display-name changes without changing identity. ## Optional context Capabilities help tie code, traces and task criteria together. They are optional for uploaded datasets, data-first training and standalone decision comparisons. The absence of a repository is not missing evidence when the task itself begins with data. For the procedure, see [Map an agent repository](/how-to/map-agent). # Data lineage and held-out evidence Source: https://docs.overmindlab.ai/explanation/data-lineage Why original sources, immutable cells and group identity matter. A dataset is a retained source and the published steps derived from it. A cell identifies a concrete version of rows. A recipe revision identifies the code used to create an output, and a run receipt records its execution against exact inputs. Ask this in the project conversation after creating the relevant [tutorial or workflow](/tutorials/index) results. Trace one prepared fraud example back to its original source in Overmind. Explain what changed along the way and which saved preparation and dataset version a training run would use. ## Original evidence and prepared data Original bytes remain available after landing. Document extraction retains page and region evidence, parser/OCR facts and known limitations. Extracted text is a representation of a source, not proof that every visual detail was recovered. Preparation changes data through retained code. Publication checks structure, lineage and declared requirements. It cannot establish that an ambiguous numeric column means calibrated probabilities or that a generated answer is correct. Preserve uncertainty in the output and supporting explanation. Technical compatibility and semantic suitability answer different questions. ## Versions belong to consumers A training or evaluation consumer pins exact cells. A later preparation cannot silently change what an earlier result measured. Correcting a recipe produces a new revision and output; historical inputs stay reproducible even when the Console shows the corrected preparation. A displayed version number is a navigation aid. Cell IDs and fingerprints identify the actual data. Exporting a cell reads its rows; it is not the same action as freezing a consumer's inputs. ## Related observations are not independent samples Duplicate observations, multiple turns from one conversation, shared source content and synthetic descendants can carry related information. Splitting them across fitting and final evaluation can reveal the answer indirectly. Saved partitions preserve inherited identities and declared groups. They retain duplicate observations rather than erasing their frequency or weights. Exact overlap checks cannot rule out paraphrases or pretraining contamination. Development data supports choices during fitting. Calibration data supports fitted calibration. Final data supports a later comparison after those choices are fixed. Reusing final outcomes to choose a recipe changes what that comparison can establish. ## Execution attribution A platform run links its package, revision, source and output. An external import validates supplied lineage and records its producer. Those are different claims: attributed external execution is not independently verified execution. See [Prepare data](/how-to/prepare-data), [Partition data](/how-to/partition-data) and [Pipeline reference](/reference/pipelines). # Decision models and Jev Source: https://docs.overmindlab.ai/explanation/decision-models Distinguish typed probability prediction, task training and expectation scoring. A decision model receives state, a question and a set of options. Its output is a probability for each option in the supplied order. This differs from generating answer text and then extracting a label. Ask this in the project conversation after creating the relevant [tutorial or workflow](/tutorials/index) results. Explain when a decision model is useful for fraud detection and when a chat model would be a better fit. Use a transaction example to compare their inputs, outputs and evaluation needs. ## Preserve the supervision you have A probability distribution can retain disagreement or uncertainty that an argmax label removes. A hard categorical observation can map to a one-hot target when the label and option mapping are known. A mean-only ordinal observation does not imply a unique distribution of votes. For ordinal means, retain the declared option scale and target mean. For distributions, retain every option probability, ties and example weights. The coding agent must establish their meaning from source evidence rather than from the numeric shape alone. ## Training and inference The qualified Clef training path uses a separately pinned decision runtime, a backbone adapter and full-precision decision head. Verification checks saved components and probability behaviour. It establishes serialization/execution properties, not task quality. A decision deployment accepts typed questions and returns probabilities. Chat messages, output-token budgets and chat aliases are separate contracts. Model catalogue visibility also differs from launch qualification; D3 and D1 paths remain disabled until qualified. ## Jev as an evaluation engine The platform uses Jev through its dedicated decision transport to score `llm` expectation rows. This scoring role is distinct from training a model on your dataset. A generative model supplies explanations for low scores; it does not silently replace failed Jev decisions. Use [Compare decision models](/how-to/compare-decisions) for your probability-model benchmark and [Evaluation reference](/reference/evaluation) for the platform's matrix scoring contract. # How evaluation works Source: https://docs.overmindlab.ai/explanation/evaluation The relationship between live scores, frozen comparisons and decision benchmarks. Overmind uses explicit evaluation definitions to turn evidence into measurements. The definition, the evidence it receives and the population it scored determine what the result means. Ask this in the project conversation after creating the relevant [tutorial or workflow](/tutorials/index) results. Explain the difference between checking answer format and measuring fraud-detection accuracy. Use examples from our Overmind evaluation to show what each result establishes and what evidence is still missing. ## Live and controlled evaluation Enabled eval matrix rows score matching spans. A deterministic row runs a constrained Python check; an `llm` row states an expectation judged by Jev. Finished traces can also receive trajectory measurements on their root span. Scores append history and remain inspectable with their reasons and errors. A controlled evaluation freezes a matrix and a dataset cell. Each target gets its own run in a shared group. Recorded targets use saved outputs; generated targets request new outputs. Comparison pairs rows both sides scored. A dataset row represents the capability's answer, so answer-level checks apply even when their live `span_name` names the function producing that answer. Tool-only checks do not automatically become checks of the final dataset answer. ## What a score does not establish Abstention means a check did not produce a judgement. A technical error means judging failed. A pending row has not finished. None is a quality zero or a pass. The same principle applies to generation: a token-limited response can be transport-successful while incomplete. Read coverage and degraded outcomes before comparing means. A high mean over the few completed cases can hide a weak overall result. Jev supplies the expectation decision. A separate generative model can explain a low score. Selecting an explanation judge does not replace the scoring engine, and a provider failure does not silently switch to a generative grader. ## Decisions need their own measurements A probability model predicts over options supplied with the question. Full distributions, hard observations and ordinal means carry different information. Their metrics must preserve that distinction. Native comparisons freeze suites and optionally fit calibration before final scoring. Raw and calibrated results remain separate. A calibrated score on calibration data is in-sample evidence, not final generalisation performance. Model speed is another experiment: it depends on input shape, concurrency, batching, warm state and measurement boundaries. Quality results alone cannot establish serving throughput or an all-in cost saving. ## Improvement is a comparison claim Keep the task, data roles, model identities, checks and coverage visible. Development gains can guide another experiment. A defensible final result requires an appropriate held-out population and an honest account of missing or incompatible outputs. # Training, benchmarking and serving Source: https://docs.overmindlab.ai/explanation/model-lifecycle Why a trained artifact, a ready deployment and a live route are different states. Training creates a model artifact from pinned data and a recorded recipe. Deployment prepares that artifact for its supported inference runtime. Evaluation measures task behaviour. Activation selects a verified chat deployment for an application alias. Ask this in the project conversation after creating the relevant [tutorial or workflow](/tutorials/index) results. Inspect our model in Overmind and explain whether it has finished training, has a ready deployment and is being called by the application. Show the evidence for each state and any remaining steps. These states are related, but none substitutes for the others. | State | Evidence it provides | | - | - | | Prepared | Model-specific inputs were built and validated | | Trained | The recorded training attempt finished and retained its artifact | | Reload verified | Saved weights and required components can be loaded under the qualified contract | | Deployment ready | The runtime passed its readiness verification | | Worker warm | A worker is currently available or recent traffic establishes warmth | | Benchmarked | A comparison measured the artifact on its pinned evidence | | Active | A capability alias selects the deployment after activation verification | | Application connected | A successful application request reached that alias | Successful training automatically starts the matching chat or decision deployment. An unsuccessful deployment does not invalidate the fact that training finished, and a retry does not require repeating training. A capability's benchmark model is independent of its active serving model. Changing the benchmark default changes future comparisons; it does not reroute application traffic. Activation preserves the previous selection until the new deployment verifies. Workers can later scale down, so being active does not guarantee a permanently warm worker. A direct request to a pinned deployment does not prove the application uses the alias. Native decision artifacts include a decision head and probability contract. They deploy through that runtime, not the chat-completions contract. Costs also have distinct boundaries. A forecast is an estimate with assumptions and missing components; a worker receipt records measured usage; a ledger charge records metered spend. None should be presented as a complete provider invoice unless the evidence actually covers it. # Working with a native coding agent Source: https://docs.overmindlab.ai/explanation/native-agent How responsibilities are shared between your agent, CLI, MCP and Console. Overmind's workflows span your coding environment and the platform. The coding agent interprets the task, reads source material, writes code and explains the evidence. Overmind retains data and recipes, executes supported operations and records their outcomes. Explain what you handle in my coding environment and what Overmind handles when we train a fraud model. Walk through data preparation, training and evaluation using our project as the example. ## How one request becomes a workflow A user request describes the outcome and constraints. The coding agent discovers the connected project, reads tool schemas and resources, resolves the relevant data, and carries out the dependent operations. It can write a preparation package locally, publish it through MCP, inspect the resulting cell and use that exact cell in the next step—all in one conversation. The agent returns evidence you can review: what it changed, the saved data or run, measured results and remaining decisions. A prompt can start a long-running job; the job continues on the platform even when the chat disconnects. In a later conversation, supply its link and ask the agent to inspect or continue it. A plain-language prompt in these docs is a request to your coding agent. A **native MCP prompt**, such as `author-dataset-transformation`, is a named workflow supplied by the server. Clients expose those prompts differently; ordinary chat requests work without memorising slash commands. The agent can use a native prompt where supported and follow the connected tools and resources otherwise. ## Four interfaces, one project | Interface | Responsibility | | - | - | | Coding agent | Task interpretation, repository edits, transformation authoring and evidence-based decisions | | CLI | Local bytes, repository scanning/sync and local application execution | | MCP | Project-scoped platform tools, workflow prompts, resources and durable job inspection | | Console | Visual inspection of sources, cells, traces, results, deployments and progress | MCP is a first-class platform interface sharing domain services with REST and the Console. A plugin can package its connection and skills, but the connected tool catalogue supplies the current contract. Installed prose can lag a deployment. Initial account login, credential entry and client permission grants still use the host's supported flows. Membership or billing controls without an MCP operation remain in their management surface. The agent should name the required action when it reaches one. Local filesystem work stays in the coding environment. MCP does not upload a local file simply because the agent knows its path; the CLI transfers its bytes with a separately verified connection. ## Starting with data A project can begin with a brief and source files. There is no need to invent a repository, capability or trace history. The useful chain is a retained source, an explicit preparation, exact consumer inputs and a measured result. Data Workshop has no platform chat agent. Landing does not start semantic preparation. A coding agent authors a retained Python package; the isolated runtime executes it and publishes cells. Semantic conclusions remain attributable to the evidence and authoring work that support them. ## Starting with an application A local scan describes the application's capabilities and source structure. Traces describe observed executions. Evaluations relate those observations or controlled outputs to explicit checks. An optimiser runs candidate application changes in the local environment; model training executes through the configured provider. These paths can meet: traces can become source data, and data-first models can later serve an application. Neither path requires pretending that source data or model outputs are verified answers. ## Boundaries that matter A deployment address identifies where the platform request goes, not where all compute runs. A local server can use remote model and GPU providers. The coding agent also has its own model provider and funding. A job receipt makes asynchronous work recoverable across chats and disconnects. It records what was requested, what ran and what remains unknown. This is why a reconnecting agent should inspect existing work before deciding to submit more. # Compare decision models Source: https://docs.overmindlab.ai/how-to/compare-decisions Compare foundations and trained probability models on frozen suites. Use this workflow for models that return probabilities over supplied options. A training job or repository scan is not required. Use the trained model and held-out data from your [fraud workflow](/how-to/train). The agent checks the connected catalogue for an eligible Jev comparison target. Benchmark our trained fraud decision model against an eligible Jev model in Overmind using held-out examples from [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Compare fraud detection, false alarms and probability quality, and explain any gaps in the results. **What you get:** A saved probability comparison, retained predictions and metrics for the covered labels. The steps below describe the workflow your agent follows and the results to verify. 1. Prepare typed decision inputs and separate calibration/final cells. Preserve complete distributions and mean-only ordinal references without converting either to invented labels. 2. Call `list_decision_models` and inspect eligibility for each foundation, trained artifact or supported external probability adapter. A catalogue entry does not establish measured hardware fit. 3. Save a draft with `create_native_evaluation`: provide participants, an explicit baseline, `final_cell`, optional `calibration_cell`, inference settings, seed and request key. 4. Call `prepare_native_evaluation`. Inspect the frozen suites, compatible rows and any unavailable measurements. Preparation does not submit inference. 5. Call `launch_native_evaluation` for the saved draft when its execution and cost scope are accepted. Follow its job and operation receipts. 6. Use `inspect_native_evaluation` to page retained inputs, references and predictions. Use `view="labels"` for per-label precision, recall, F1 and coverage where hard references and a fixed benchmark option order qualify. 7. Download the checksum-bound report through its returned path on the same authenticated server. Compare per-benchmark results, paired coverage, raw/calibrated scores and missing, invalid or incompatible outputs. Calibration is fitted before final scoring and has its own in-sample results. Temperature fitting requires probability-labelled references; omit calibration for a mean-only suite. Mean-only ordinal references do not imply classification accuracy or a target probability distribution. ## Partial or interrupted work Follow the receipt's recovery state. Completed predictions are reused; a missing acknowledgement is not permission to resubmit provider work. Explicit native collector recovery requires the exact recorded call ID. A paused local scoring stage can rebuild a report from saved predictions without invoking the provider again. A completed comparison can have issues. Report expected versus scored counts for each participant. Pairwise results use shared scored inputs and can be biased by missing coverage. ## Measure serving performance separately Use `measure_decision_performance` with an explicit saved workload: input shapes, sample count, repetitions, concurrency and questions per request. Inspect complete client latency and valid throughput, including cold/warm distinctions and unknown timing. Prediction reuse for a quality comparison is explicit. Independent repetitions and performance measurements are not silently replaced with cached answers. Neither quality scoring nor performance measurement activates production routing. # Connect your coding agent Source: https://docs.overmindlab.ai/how-to/connect Connect MCP and local file transfer to the same account, project and deployment. Connect your coding agent to your Overmind account to use platform tools and upload files from your coding environment. These steps use Overmind Cloud; you do not need to deploy the platform. Give your agent the [Overmind quick-start instructions](https://console.overmindlab.ai/setup.md). It follows the setup for your coding environment; complete login or credential entry when needed. Read [https://console.overmindlab.ai/setup.md](https://console.overmindlab.ai/setup.md) and follow the instructions to set up Overmind in my coding agent. **What you get:** A verified account connection and project, with MCP and file upload access checked. The Console setup screen offers **Copy prompt**. Paste the prompt into your coding agent. Use the attached arrow to choose Claude Code, Codex, Cursor or OpenCode. For Claude Code, Codex and Cursor, **Open in** then starts that agent's desktop app with the prompt filled in. OpenCode uses the copied prompt. The prompt includes the selected project when one exists. After connection, the progress card shows completed setup work. The continuation card's prompt asks the coding agent to install what it lacks, check live onboarding milestones through MCP, skip completed steps, and finish the remaining setup. If authentication is pending, the agent reports the blocker instead of restarting setup. On mobile browsers, sign-in and sign-up use a redirect instead of a popup and preserve the return destination, including MCP consent. The setup screen offers only **Copy prompt** on a phone. The steps below describe the workflow your agent follows and the results to verify. ## 1. Establish the MCP connection The [quick-start instructions](https://console.overmindlab.ai/setup.md) cover MCP configuration, the Python package and client setup. The Overmind Cloud MCP endpoint is `https://api.overmindlab.ai/api/mcp/`. For a repository, install the CLI and initialise your client: ```bash theme={"dark"} uv tool install "overmind>=0.1.106" overmind init --ide codex --api-url https://api.overmindlab.ai ``` Replace `codex` with `cursor`, `claude` or `opencode` as appropriate. `init` installs skills and client configuration; it does not by itself prove authentication or a completed repository scan. `init` writes the MCP entry without a key and prints the client's sign-in step. Sign in through the client's OAuth prompt; an API-key header is optional. Keep credentials out of prompts and committed files. Reload the client after changing MCP configuration. ## 2. Verify the destination Ask the connected agent to call `list_projects`. Check `connection.mcp_url`, choose the intended project and keep its UUID. Account connections supply `project_id` on every project operation. IDs from another deployment cannot identify this project's resources. Read `overmind://interface/current` for the connected contract and catalogue fingerprint. Refresh the client's tool list when it has cached an older schema. ## 3. Configure and check file transfer MCP authentication does not automatically give the CLI access to local bytes. With an authorized account key already available locally, configure the endpoint-bound account connection: ```bash theme={"dark"} overmind connection configure \ --api-url https://api.overmindlab.ai --project-id PROJECT_ID --json overmind connection check \ --api-url https://api.overmindlab.ai --project-id PROJECT_ID --json ``` Configuration can update global Codex MCP and the account transfer connection; reload that client if changed. It does not globally select a project or grant host network permission. Run the check from the directory and execution environment that will upload files. For repository-free work, also check from an empty directory without injecting an alternate credential. A workspace's configuration may otherwise hide a broken account connection. ## 4. Choose the next task For a dataset-to-model workflow, choose [How to train a decision model](/tutorials/first-dataset) for typed probabilities or [How to fine-tune a language model](/tutorials/fine-tune-language-model) for generated responses. For an application repository, [map its capabilities](/how-to/map-agent). Data-first projects can proceed without a scan. ## Follow setup in the Console While your agent onboards a repository, it reports each of the ten setup stages with `report_onboarding_progress`. A project without capabilities shows that progress in place of the setup prompt. A project with capabilities keeps its capability grid and shows the agent and current stage in one row above it; **Dismiss onboarding progress** hides the row. Progress belongs to one project: signing in once does not mark other projects as connected. ![Onboarding progress row above the capability grid](https://mintcdn.com/overmind-b84ae13c/jOsLwNjkhuLAOlvs/images/platform/onboarding-progress.jpg?fit=max\&auto=format\&n=jOsLwNjkhuLAOlvs\&q=85\&s=3563ec5861e8f0d7fcd09045c9f71a75) A project with capabilities shows the reporting agent, the current stage and the completed count above its capability grid. The notification bell lists what agents and the CLI did in the project: connections, requested jobs and changes to datasets, evaluations and capabilities. Job rows update as the jobs run, and the pages they reference refresh without a reload. ## Using a self-hosted deployment If you run the open-source platform yourself, use the API address of that installation in place of the Cloud URL throughout this guide. Its MCP endpoint ends in `/api/mcp/`. Follow [Self-host Overmind](/how-to/self-host) for deployment instructions and [Deployment configuration](/reference/configuration) for server settings. ## If connection fails * **Wrong endpoint:** correct the connection to the intended deployment before sending its credentials. Do not reuse a credential against a different address. * **MCP works, upload fails:** check the CLI's resolved account connection and project scope. * **Sandbox network denial:** request scoped host network permission in the coding environment, then repeat the same connection check. * **Missing tool:** refresh the catalogue and inspect the connected contract before concluding that a feature is unavailable. See [MCP reference](/reference/mcp) and [CLI reference](/reference/cli) for authentication and command details. # Run an evaluation Source: https://docs.overmindlab.ai/how-to/evaluate Freeze a dataset and checks, compare targets and inspect paired results. Use this guide to measure recorded answers or generated model outputs. For typed probability outputs, use [Compare decision models](/how-to/compare-decisions). Select the two chat models and a fraud eval set in the conversation. For probability outputs, use [decision-model comparison](/how-to/compare-decisions). Compare the two models we selected on a held-out evaluation set from this fraud dataset: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Use Overmind to show their accuracy, which fraud cases they miss and where their predictions disagree. **What you get:** Frozen comparable runs, paired results and examples explaining their differences. The steps below describe the workflow your agent follows and the results to verify. ## 1. Pin the data and checks Inspect an eval dataset and choose its exact cell. Confirm that model inputs exclude the reference answer. Create or select eval matrix rows, then a named eval set if needed. Use `create_eval_matrix_row` for a deterministic script or a one-sentence `llm` expectation. Use `create_eval_set(rows=[...])` to group row IDs. Existing runs retain their frozen matrix when definitions change. ## 2. Define the comparison Build `targets` for `check_evaluation_readiness` and `run_evaluation`: * `mode="recorded"` scores the dataset's recorded outputs. * `mode="generate"` generates outputs with `model_name` or `model_ref`. * Mark one target `is_baseline=true` when comparing changes. Use consistent data, reference handling, generation parameters and explanation judge across the comparison. The `judge_model` field selects the model that explains low scores; it does not replace Jev's expectation scoring. ## 3. Check and launch Call `check_evaluation_readiness` with `dataset`, `cell`, `eval_set`, `targets`, optional `judge_model` and `project_id`. Resolve technical incompatibilities. Review context and cost estimates; context warnings are advisory and do not change the selected model. Call `run_evaluation` with the same configuration, a name and deliberate `max_items`/`sampling` values. This starts work and returns one run per target in a shared group. Poll the returned jobs; submitting again creates another evaluation. Standalone evaluations are launched through MCP or REST. The Console displays results; it is not the standalone launch path described here. ## 4. Inspect results before choosing a winner Use `compare_evaluations` with the candidate run and its baseline. Read the paired count, wins, losses, ties and metric deltas alongside coverage and technical failures. Open individual run rows to inspect input, reference, model output and score reasons. A truncated generation or grading failure is not a quality zero. A high average over a small surviving subset does not establish a successful comparison. If a criterion is wrong, correct its source and start a new run. Do not reinterpret an old frozen run as if it used the replacement check. See [Evaluation reference](/reference/evaluation) for target fields and row contracts. # Import traces from another provider Source: https://docs.overmindlab.ai/how-to/import-traces Inspect source boundaries, confirm capability mapping and import retained traces. Overmind supports tracing connectors for Langfuse, LangSmith, Braintrust and Galileo. You need a destination project and read credentials for the source provider. Replace `Langfuse` with your supported provider. Enter provider credentials through the CLI or its secure configuration. Import the last 24 hours of traces from our Langfuse project into Overmind. Check the integration and show me what arrived, including any mapping gaps or import failures. **What you get:** A reviewed mapping, an import receipt and evidence that the selected trace boundaries landed. The steps below describe the workflow your agent follows and the results to verify. 1. Create the connector with `overmind connector add PROVIDER --project-id PROJECT_ID --json`, supplying the intended `--api-url`. Provider credentials come from environment variables or a terminal prompt. Alternatively, use Integrations in the Console. Do not paste provider keys into MCP calls. 2. Call `inspect_connectors` with `include_source_projects=true` and select the source project or stream. Use its preview options when examining source observations. 3. Inspect `suggested_boundaries`, alternatives, `unmapped_roots`, observation shapes and available capability mappings. The selected boundary names describe trace roots; selecting an ancestor drops nested child boundaries. 4. Call `configure_connector` with the source, time range, proposed `capability_mapping` and explicit polling settings. Leave `confirm_mapping=false` to stage the proposal. Inspect the returned pending mapping and unmapped roots. 5. Present that staged mapping to the human responsible for the application. After their reply, call `configure_connector` again with `confirm_mapping=true` to confirm the unchanged pending mapping. A first call that proposes a new mapping cannot also confirm it, even if the flag is true. Verify `mapping_pending=false`. 6. Call `sync_connector` to queue the configured import. Follow its returned job and polling guidance rather than submitting another sync to refresh status. 7. Inspect representative traces and capability attribution through the returned Console link or MCP trace resources. Check that the imported boundary captures the intended application input and output. If the source or mapping boundaries changed since the last import, sync can recarve existing connector spans. Inspect the returned `recarving` state and verify the result. Keep unmapped roots visible when the evidence does not support an assignment. Enable ongoing polling only when wanted. Disconnecting a connector keeps already imported traces. Counts and available history reflect the provider's retained data, which can change over time. See [CLI reference](/reference/cli#connectors) for credential environment variables and [Investigate a failure](/how-to/investigate) for the next step. # Instrument an application Source: https://docs.overmindlab.ai/how-to/instrument Add trace boundaries, capability identity and useful input/output evidence. Use this guide to make a real request inspectable in Overmind. Begin with a connected project and, for capability attribution, a synced capability ID or slug. Use this in your application repository after [mapping its capabilities](/how-to/map-agent). Add Overmind tracing to this application so I can follow a request through its main steps. Capture useful inputs, outputs and failures, then verify the setup with a local example. **What you get:** Tracing edits, local checks and an explicit run to verify the captured evidence. The steps below describe the workflow your agent follows and the results to verify. ## Python 1. Install `overmind[tracing]` into the application's environment. 2. Call `overmind.init()` at process startup with the deployment URL, service name and capability identity. Provide the API key through protected runtime configuration. 3. Wrap the request's entry point in `overmind.run()` or an entry-point decorator. Capture the application input and delivered result. 4. Trace meaningful tools, retrievals and workflows. Use provider instrumentation for model calls; avoid recording the same call twice. 5. Add source-level `expectations=["…"]` to the relevant supported decorators when the code declares evaluation criteria. Sync after changing these declarations. 6. Run a representative request with a unique conversation ID, flush, and inspect the actual trace. The installed `/overmind ensure-tracing` workflow can apply these changes. `get_instrumentation_plan` returns Python placement tickets; `verify_instrumentation` inspects supplied spans without writing another trace. Check nesting, capability binding, input/output values, errors, model identity and measured usage. Decorator presence alone is not verification. See the [Python SDK reference](/reference/python-sdk). ## TypeScript, JavaScript and other languages Use your application's existing OpenTelemetry setup. There is no current Overmind JavaScript SDK, and Python placement tickets are not supplied for JavaScript anchors. For Overmind Cloud, set an explicit trace endpoint to avoid exporter path ambiguity. For self-hosting, use your installation’s API base with the same `/api/v1/traces` path: ```bash theme={"dark"} export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.overmindlab.ai/api/v1/traces" export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" ``` Configure the exporter header `x-api-key` through your application's secret mechanism. Stamp `overmind.capability.id`, a meaningful span type and the actual input/output evidence. Add `conversation.id` for a multi-turn session. Use the [OTLP reference](/reference/otlp) for the wire contract. ## Missing or incomplete traces Check the exporter destination, project credential, process flush and request boundary. Unrecognised capability identities remain unbound; they do not create capabilities. A short-lived process can exit before buffered spans are sent. Inspect payload capture before enabling it on sensitive application paths. `capture="none"`, ignored arguments and explicit formatting let you control which values enter telemetry; automatic secret-key redaction is not a complete data policy. # Investigate a failure Source: https://docs.overmindlab.ai/how-to/investigate Follow a score, error or latency signal back to recorded evidence. Start with a project, a time window and the symptom you need to explain. Select the project and capability in your conversation. This needs application traces; a Hugging Face dataset cannot substitute for observed failures. Investigate why this capability has been failing over the last 24 hours in Overmind. Find the main failure patterns, show representative traces and suggest what I should investigate in the code. **What you get:** A bounded diagnosis with trace links, coverage and a proposed experiment. The steps below describe the workflow your agent follows and the results to verify. 1. Call `inspect_capability_health` for the capability and period. The same observations can also be inspected in the Console's Traces and Sessions views. Compare error rate, latency and scoring coverage as separate measurements. 2. Use `query_failures` for scored failures or `query_traces` for errors, session IDs, models and time filters. Keep the trace IDs that support the finding. 3. Open the trace resource and inspect its span tree. Read the input, output and relevant tool or retrieval evidence before inferring a cause. 4. Identify whether the result is a low quality score, abstention, technical grading error or pending work. Inspect the row definition and recorded reason. Missing scores are not passes or zeros. 5. Compare with successful requests from the same task and conditions. Check whether the apparent pattern is multiple independent cases or repeated observations of one conversation. 6. Record a bounded finding: affected cases, the measured symptom, evidence and unresolved hypotheses. For recurring observations, use `list_signals` to inspect the catalogue and `set_signal_subscription` to configure the relevant project or capability subscription. Model-backed plan drift is opt-in; a signal is a reason to investigate, not a confirmed diagnosis. ## Turn evidence into an experiment Select relevant traces and create a dataset, then [prepare its inputs and references](/how-to/prepare-data). A recorded response is evidence of what happened, not automatically a correct answer. Use [evaluation](/how-to/evaluate) for an output comparison or [the optimiser](/how-to/optimise) for changes that must execute in the repository. Keep cases used to develop the change separate from final assessment. See [How evaluation works](/explanation/evaluation) for the relationship between live scoring and frozen runs. # Manage project access and API keys Source: https://docs.overmindlab.ai/how-to/manage-access Choose credential scope, issue keys and keep account and project boundaries explicit. Projects own capabilities, traces, datasets, runs, connectors and deployments. Choose the project before changing membership or credentials. Use the selected project and keep the returned key in protected configuration. Your agent will guide secure storage if the client needs a separate step. Create a project-scoped API key for my application in Overmind that expires in 30 days. Help me store it securely and explain which project it can access. **What you get:** A scoped credential and verified access through the appropriate secure setup flow. The steps below describe the workflow your agent follows and the results to verify. ## Issue a key Use the Console's API key dialog or MCP `create_api_key`. Supply a name, description and supported expiry: 30, 90, 180 or 365 days, or no expiry. Omit `project_id` to request an account key; pass it for a project key. An account key reaches authorized projects, while a project key remains restricted to one. A project key cannot mint an account key. Store the plaintext returned at creation in a secret manager or protected local configuration. It is shown once; later listings return metadata. Do not commit it to source control or paste it into an agent prompt. ## Share project access Use the project's membership controls. Account membership determines which projects an account connection can discover and use. Keep each client operation explicitly project-scoped; another chat's choice does not select a project for this one. Use the deployment's supported invitation flow for a new member. Self-hosted email/password authentication and hosted Clerk sign-in are different login configurations; API keys are the common programmatic path. ## Rotate or revoke Create and install the replacement credential in the systems that need it, verify their access, then revoke the old key through the key management surface. Inspect expiry and last use when diagnosing a suddenly failing client. For repository setup, the CLI protects credential-bearing files and refuses to write a key into a tracked file. Keep the manifest's identity metadata separate from `.overmind/credentials.toml`. See [API authentication](/reference/api#authentication) for headers and [Connect your coding agent](/how-to/connect) for endpoint-bound local configuration. # Map an agent repository Source: https://docs.overmindlab.ai/how-to/map-agent Discover capabilities, review the source-grounded map and sync it to a project. Use this guide when you have application code and want Overmind to relate its structure to traces and evaluation criteria. A data-only project does not need this step. Open the application repository and [connect Overmind](/how-to/connect) first. Map this repository’s capabilities in Overmind. Explain what the agent can do, where each capability lives in the code and which parts already have evaluation coverage. **What you get:** A source-backed capability map and the repository provenance attached to it. The steps below describe the workflow your agent follows and the results to verify. 1. [Connect your coding agent](/how-to/connect) from the repository root and verify the destination project. 2. Ask the agent to run the installed Overmind setup workflow. On clients with slash commands, use `/overmind setup`. 3. Review the discovered product purposes. A capability should describe work such as routing a support request; internal workers or model roles do not automatically need separate capabilities. 4. Review the source-backed entry points, prompts, tools, branches and expected outcomes. Repair unmapped transitions from the code before syncing. 5. Review `overmind.toml` and the endpoint, project and metadata to be uploaded. The workflow runs `overmind chassis`, authors the map and runs `overmind sync`. It does not upload a repository archive, but captured prompts and source metadata are included. 6. Verify the successful sync and read `overmind://project/current?project_id=PROJECT_ID` or inspect the Agent page. Check capability IDs and repository provenance. The chassis supports Python and TypeScript/JavaScript. Structural verification confirms source anchors and reachability; it does not prove that the application always follows the intended path. Keep capability slugs stable when changing display names. A capability absent from a later scan becomes a leftover; its data remains. Commit the manifest. `.overmind/credentials.toml` stays out of version control through `.overmind/.gitignore`. After changing source code, refresh the scan. If the checkout changed during scanning, repeat it so the retained provenance describes the code actually inspected. Next, [instrument the application](/how-to/instrument). A scan alone does not emit traces. # Monitor training and inspect checkpoints Source: https://docs.overmindlab.ai/how-to/monitor-training Read saved progress, development checks and failed stages without starting more work. Use the job ID returned by training or explicit preparation. Read `get_job` and follow its operation resource; `inspect_training_progress` provides detailed development evidence. Use the job returned by [training](/how-to/train). In a new conversation, provide the job link or ID. Check our latest fraud-model training run in Overmind. Show its current stage, recent validation results and any failures or stalled progress, then explain what the recorded results tell us so far. **What you get:** A report from stored progress and checks, without starting another job. The steps below describe the workflow your agent follows and the results to verify. ## Identify what is happening Read the current stage and its measured counters. Transfer, runtime initialisation, weight loading, adapter setup, tokenizer verification, dataset construction, training, validation and checkpoint verification can report separate progress. Compare three clocks: * **Observation:** when the platform last read a state. * **Heartbeat:** when the worker last reported being alive. * **Forward progress:** when the worker last advanced measured work. A fresh heartbeat alone does not establish that weight loading or an optimizer step advanced. Missing loader counters cannot produce a measured percentage. ## Inspect development evidence Call `inspect_training_progress(job=...)` and follow the returned check or checkpoint identities. Use `check` for retained examples and `probe` for frozen sample identities. When an overview returns `collections`, pass its `field` JSON Pointer with `offset` and `limit` to retrieve the exact entries. `check`, `probe` and `field` are mutually exclusive inspection modes; send only one per call. Read loss, coverage, invalid predictions, check failures and checkpoint reload verification separately. For classification, inspect the majority-label comparison on the same scored subset. Missing predictions for a represented label are evidence to investigate; they are not a diagnosis of the cause. Detailed examples may still be collecting after a check completes. A pending evidence download does not mean the trainer is still running. Inspection reads stored receipts and does not wake a worker. ## Decide whether to stop Use `cancel_finetune` to record cancellation intent before the provider request. Inspect acknowledgement and terminal state. Retained checks and checkpoints remain useful evidence even if training is cancelled. Do not assume that a reloadable adapter supports full optimizer continuation. Use only the recovery capabilities exposed by that job and runtime. If provider acknowledgement is unknown, inspect its saved call identity instead of launching another trainer. A completed training job can still have a failed deployment or incomplete final comparison. Report each outcome separately. See [Training reference](/reference/training) and [Recover interrupted work](/how-to/recover). # Improve prompts, code or model choice Source: https://docs.overmindlab.ai/how-to/optimise Run a bounded repository experiment against frozen data and checks. Use the Optimiser when a candidate must execute your application. You need a synced capability, an eval dataset, appropriate checks and a repository that can run one datapoint reproducibly. Use your application repository, a synced capability and an eval set. To practise with the [example dataset](/reference/example-dataset), use a fraud-classification application that can run one transaction reproducibly. Improve this capability’s prompt using our Overmind evaluation set. Compare the proposed changes with the current version and show which cases improved, regressed or still fail. **What you get:** A bounded experiment with the baseline, candidate results and a reviewable diff. The steps below describe the workflow your agent follows and the results to verify. 1. Call `check_optimizer_readiness` with the capability, dataset/cell, eval set and intended mode. Inspect prerequisites and the provider costs of running your own application. 2. Choose `optimize` for prompt/code changes, `model_comparison` for model candidates or `hybrid` for combinations. Set iteration and candidate limits deliberately. 3. Create the experiment with `start_optimizer`, or use the Console's copied coding-agent prompt. Copying a prompt does not launch execution. 4. Attach the CLI to the returned experiment: ```bash theme={"dark"} overmind optimise start --experiment EXPERIMENT_ID overmind optimise next ``` 5. Follow the next-action response. Write a command template that accepts one datapoint and prints its result; register it with `set-template`. Run the smoke test, then the unmodified baseline before submitting candidate diffs. 6. Let `run-iteration` execute candidates in separate local worktrees and submit outputs for platform scoring. Inspect candidates and their evaluation runs with `inspect_optimizer_result`. 7. Stop at the declared budget or completion condition. Review the retained winning diff and apply it through the repository's normal review process; a model comparison requires an explicit configuration change. The platform retains the dataset version and frozen matrix. Your coding environment authors changes and executes the application. Provider calls made by your application use its credentials and can incur costs even during the smoke test. ## Assess the result Read coverage, failed outputs, score deltas and regressions alongside the winner. Repeated development comparisons can overfit the examples; use independent held-out cases before rollout. Recorded tool replay answers a narrower question than running the live application. It cannot establish how a new tool choice would behave in an environment that was never executed. The optimiser does not deploy your application repository or authorize production activation. Use [evaluation](/how-to/evaluate) and [serving](/how-to/serve) for those separate checks. # Separate training and held-out data Source: https://docs.overmindlab.ai/how-to/partition-data Save group-preserving train, development, calibration and final roles. Use a saved partition when repeated observations, conversations or related source examples must stay together. Begin with an inspected, prepared source cell and a known task contract. Start with the [fraud example dataset](/reference/example-dataset) from [Prepare data](/how-to/prepare-data). The agent inspects related transactions and chooses an appropriate split. Create training, development and held-out evaluation splits in Overmind for this fraud dataset: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Check for overlap that could make the evaluation misleading, then explain the split sizes and label balance. **What you get:** A saved partition recipe, actual role counts and exact cells without shared groups. The steps below describe the workflow your agent follows and the results to verify. 1. Identify related records: source groups, conversations, repeated inputs and synthetic descendants. Declare additional grouping fields from evidence rather than guessing from column names. 2. Choose role fractions and a fixed seed. Use **train** for fitting, **development** for recipe/checkpoint decisions, **calibration** for fitting calibration and **final** for the independent comparison. Leave unused roles out of the recipe. 3. Call `create_data_partition` with `name`, `request_key`, `source_cell` and `recipe`. The recipe accepts `seed`, `fractions`, `group_by`, optional `stratify_by` and explicit `holdouts`. 4. Follow the returned job through construction. Inspect actual role counts and group assignment; group sizes may prevent exact requested fractions. 5. Verify the output cells against their intended consumer. Train/development roles receive training intent, while calibration/final roles receive evaluation intent. Native decision rows in calibration/final members are projected into input-only decisions and separate references. Other row formats are retained; changing intent alone does not convert them. 6. Save the role cell IDs and use them explicitly in later jobs. Do not deduplicate repeated observations merely to achieve a ratio. Preserve their weights and provenance. Missing declared group fields are an error to resolve, not a reason to silently drop the grouping rule. Use `retry_data_partition` only after inspecting a failed construction and the supported recovery action. It resumes the saved recipe rather than inventing another split. For a data-first modelling project, retain the task definition and exact role cells in `save_training_plan`. Final answers must not guide recipe selection. See [Data lineage and held-out evidence](/explanation/data-lineage). # Prepare data for a task Source: https://docs.overmindlab.ai/how-to/prepare-data Retain original sources, author a reproducible transformation and verify the published output. Use this guide to turn files or traces into data for exploration, evaluation or training. Your coding agent owns interpretation and transformation code; Overmind retains the sources, executes saved packages and publishes versions. Use the [fraud dataset](/reference/example-dataset) from the tutorial, or attach your own source in the conversation. Prepare this dataset for training a fraud detector in Overmind: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Use the transaction details to predict fraud, check the data for problems, and show me the prepared examples and any limitations. **What you get:** Published examples, retained code, a version for the consumer and a summary of checks. The steps below describe the workflow your agent follows and the results to verify. ## 1. Define the intended result State the task, prediction unit, permitted inputs and intended consumer. Inspect a named capability if one exists. Clarify what labels, scores, blank values and groups mean before changing them. Create a brief-only draft with `start_dataset`, or upload an original source directly. Use explicit intent `explore`, `eval` or `train`; omitted intent remains pending. ## 2. Land and verify the source Check file boundaries, record counts and nested values locally. Upload original bytes with the CLI; the draft's returned upload arguments already bind the destination project and dataset. ```bash theme={"dark"} overmind dataset upload source.jsonl \ --project-id PROJECT_ID --api-url API_URL --intent explore --wait --json ``` For multiple files, supply them in the intended order in one command. For a top-level JSON wrapper, use `--json-rows-field FIELD` to select its row array explicitly. Follow `dataset_transfer` for bytes/publication and `dataset_run` for extraction. Inspect the source cell, fingerprint and extraction metadata. Compare landed counts and representative values with the original. For documents, inspect page/region evidence and OCR limitations. Resolve mismatches before preparation. ## 3. Author and retain the transformation Inspect project recipes with `inspect_dataset_workbench`. Read retained code before reusing it; similar column names do not establish the same target meaning. For a new transformation, ask the agent to use `author-dataset-transformation` and the current manifest from `overmind://dataset-upload`. Write a Python package with explicit, meaningful steps. Preserve source identities, declared groups, full probability targets, option order and weights. Do not remove duplicates or blank values without a task-specific reason. Run local checks against independently expected row members and boundary cases. Then upload and register the package **before** full processing: ```bash theme={"dark"} overmind dataset pipeline-upload ./preparation \ --project-id PROJECT_ID --api-url API_URL --json ``` Call `save_dataset_pipeline` with the returned package ID, name, dataset and stable request key. Saving retains code and associates the recipe with the preparation; it does not execute it. ## 4. Validate, preview and publish Call `validate_dataset_pipeline` against the source cell and fingerprint. Preview with `run_dataset_pipeline(mode="preview")`, using the exact revision and a request key for that preview. Inspect the retained preview evidence. A bounded prefix can miss rare cases; absolute row-count checks remain deferred until publication. Verify branch coverage independently. Publish with `run_dataset_pipeline(mode="publish")` and a distinct request key. Poll its exact `dataset_pipeline` receipt. Successful publication appends the step cells atomically; failure publishes no partial chain. ## 5. Verify the final output Inspect counts, schema, sampled values, target semantics and lineage across the intended source families. Check the final consumer contract and any unresolved review metadata. Record the scope of verification; format readiness alone does not prove label correctness. Hand the exact final cell ID to [partitioning](/how-to/partition-data), [evaluation](/how-to/evaluate) or [training](/how-to/train). ## Correct or reuse a recipe For a correction, save a new revision in the same family with `expected_revision`, then run it against the original source. The selected preparation shows the corrected process; existing consumers retain their pinned cells. Use `derived_from` for an attributed variant. For genuine external results, use `import_dataset_version` with exact source bindings and parent identities. Do not upload prepared outputs as unrelated new sources. External attribution records the producer; it does not prove execution or semantic correctness. See [Pipeline reference](/reference/pipelines) for manifests, branching, batching, bindings and recovery constraints. # Recover interrupted work Source: https://docs.overmindlab.ai/how-to/recover Resume from receipts without duplicating uploads, transformations or provider calls. Start with the last returned job or request key. Read its current state before submitting anything else. Provide the last job link, ID or saved receipt if this is a new conversation. Check the interrupted Overmind job from our last session. Explain what completed, what is still running and whether it can safely resume, then continue using the recorded recovery path where available. **What you get:** An evidence-based recovery of the existing work and its resulting state. The steps below describe the workflow your agent follows and the results to verify. | Situation | Next action | What to verify | | - | - | - | | Upload disconnected | Repeat the identical CLI upload command | Same ordered files, destination and request key; recorded byte offset | | Bytes published but landing incomplete | Read `get_job(kind="dataset_run")` | Extraction stage, failure details and explicit import recovery action | | Source import lost its started worker | Inspect, then use `resume_dataset_import` where offered | Fenced attempt and retained source; do not start an unrelated upload | | Pipeline failed or expired | Inspect its exact receipt and retained code before a new run | No partial cells published; new execution needs a new request identity | | Pipeline request response was lost | Retry the identical request key and inputs | Existing receipt, not another execution | | Training launch response was lost | Retry the same launch request key and recipe | Existing job and any saved provider call | | Native comparison paused | Inspect then `resume_native_evaluation` | Exact stage; native collection requires its recorded call ID | | Deployment failed | Inspect then `retry_deployment` | Successful training artifact retained; no retraining | | MCP inference disconnected | Read the existing `inference_request` or retry its identical key | Same receipt and result, including unknown acknowledgement | ## Read state at the right layer `dataset_transfer` reports byte transfer and publication. `dataset_run` reports landing. `dataset_pipeline` reports transformation execution. Completing one does not complete the others. Follow `poll_after_seconds` or the returned polling guidance while work is active. A queue wait does not prove a worker is stuck. Read terminal receipts once and follow their next actions rather than continuing to poll indefinitely. An observation timeout does not establish remote cancellation. If a provider might have accepted a request, use its saved identity for reconciliation. Do not infer from a disconnected client that paid work never started. ## Cancel with the owning control Use the exact pipeline-run cancellation for a transformation, `cancel_dataset` for its supported dataset work, and `cancel_finetune` for training. Cancellation can prevent publication or record provider intent without instantly terminating remote computation. After recovery, verify the output and its identities before declaring success. Preserve previous successful versions and frozen consumer inputs. # Self-host Overmind Source: https://docs.overmindlab.ai/how-to/self-host Configure the local Compose deployment and its external provider dependencies. This guide starts a new local development deployment from the platform repository. If it is already running, inspect its configuration and health instead of starting a second stack. Run this from the platform checkout with Docker available. Self-hosting uses local shell and configuration work before MCP can connect. Set up Overmind locally from this checkout and connect my coding agent to it. Verify the services and connection, then show me how to open the local Console and start using a project. **What you get:** A checked local stack and an agent connection to the intended deployment. The steps below describe the workflow your agent follows and the results to verify. ## 1. Prepare the configuration Install Docker with Compose and clone the repository: ```bash theme={"dark"} git clone https://github.com/overmind-core/overmind.git cd overmind cp .env.example .env ``` Fill in the required values described in [Configuration reference](/reference/configuration). Current startup validation requires the model provider, Modal training/serving, checkpoint archive and sandbox configuration. The supported training backend is Modal. A local API does not imply local GPU execution. Deploy and configure the required provider workers before launching provider-backed features; inspect their execution and cost settings. Keep `CLERK_API_SECRET_KEY` blank for local email/password login. Configure the frontend overrides: ```bash theme={"dark"} cp frontend/.env.example frontend/.env.development.local ``` Check `VITE_API_BASE_URL=http://localhost:8000`, `VITE_SELF_HOSTED=true` and a blank `VITE_CLERK_PUBLISHABLE_KEY`. ## 2. Start and check the stack ```bash theme={"dark"} docker compose up -d --build docker compose ps curl --fail http://localhost:8000/health ``` Compose includes the API, frontend, Postgres, Redis, six Celery worker profiles, beat, the isolated sandbox and Grafana. The frontend is served at [localhost:5173](http://localhost:5173); a separate frontend development server is not required for this Compose setup. If a service fails, inspect its logs, for example: ```bash theme={"dark"} docker compose logs --tail=100 api ``` Fix the named configuration or dependency failure before proceeding. ## 3. Create a local account and connect Open the local Console and sign in with an email and password. Local login creates an account on first use without email verification. Create a project. Configure your client for this local deployment: ```bash theme={"dark"} overmind init --ide codex --api-url http://localhost:8000 ``` Replace `codex` with your supported client. Use `http://localhost:8000/api/mcp/` as its MCP endpoint and sign in through the client's OAuth prompt, which uses this installation's login. Outside `DJANGO_DEBUG`, set `MCP_SERVER_URL` to the deployment's MCP URL. Check the endpoint returned by `list_projects` and keep `--api-url http://localhost:8000` explicit on sync and transfer commands. Use the [connection checks](/how-to/connect) to verify MCP and local file transfer access. Do not launch training merely to test connectivity. ## Operate and update Retain persistent database and source storage, back up before migrations, and deploy the worker releases required by the API. Running jobs retain their pinned runtime; changing local source does not change an already deployed provider worker. The platform repository's `AGENTS.md`, `.env.example` and deployment files are the operational source of truth. Its [licensing guide](https://github.com/overmind-core/overmind/blob/main/LICENSING.md) describes the AGPL platform and MIT SDK scope. # Serve and activate a model Source: https://docs.overmindlab.ai/how-to/serve Verify a trained deployment and connect an application to its supported inference contract. Successful training automatically starts deployment through the matching inference runtime. Start from the returned deployment ID and inspect its status. For an older successful job without a deployment, use `deploy_finetune`; deployment failure does not require retraining. Use the deployment returned by the completed job. For the fraud example, supply a transaction in the conversation. This smoke request uses the serving provider. Test our trained fraud model in Overmind on a few transactions and explain its predictions. Show me a working application request using the deployed model’s actual input and response format. **What you get:** A recorded inference result and an application integration snippet for that deployment. The steps below describe the workflow your agent follows and the results to verify. ## 1. Verify the deployed contract Read the deployment resource and operational timeline. A chat model accepts messages. A native decision model accepts typed decisions and returns option probabilities. Use the contract recorded on the deployment. For a smoke request, call `run_inference` with the deployment and a stable `request_key`: * Chat: supply `messages` and an intentional output budget. * Decisions: supply `decisions` with `state`, `question`, `kind` and ordered `options`; omit chat generation parameters. Poll the `inference_request` receipt. Check the output, usage and errors. For chat, inspect `finish_reason` and `truncated`; for decisions, verify the option/probability mapping. ## 2. Connect the application Use the deployment's verified API snippet or [Inference reference](/reference/inference). A pinned chat model ID addresses one deployment. A capability alias follows the capability's selected chat model. For native probabilities, use the typed inference endpoint or returned deployment-specific request contract. A decision deployment is not a chat model and does not become one by using a chat alias. ## 3. Activate a chat alias when needed When you want the selected capability's application alias to use this chat model: Make this chat model active for the selected capability in Overmind. Verify that activation completed and show me how to confirm my application is calling it. If the application uses `overmind/CAPABILITY_UUID`, call `set_active_model` with that capability and ready deployment. Follow the returned `model_activation` job. Activation wakes and verifies the deployment before switching the alias. The previous selection continues serving until verification succeeds. Read the failure stage before retrying. Send a real application API-key request through the alias after activation. A verification probe or a request pinned directly to the model does not confirm that your application adopted the alias. ## 4. Monitor the deployed workload Inspect request success, latency, throughput and measured cost with a stated time period and traffic source. Deployment readiness, worker warmth, active routing and application connection are separate states. For a confirmed failed deployment, use `retry_deployment` after inspecting its error. For an uncertain remote submission, reconcile the existing attempt first. See [Recover interrupted work](/how-to/recover). ## Serve on a dedicated GPU A dedicated endpoint serves one model on reserved GPUs instead of shared serverless capacity. On the model page, **Configure GPU** sets the GPU type, the replica range, scale to zero and the idle timeout. **Start** shows the GPU type and its rate in credits per GPU-hour before anything starts; **Stop** keeps the configuration for the next start. While replicas start, the endpoint shows the stage the worker reports. **Spend & usage** on the same page shows dedicated credits, billed GPU hours and token counts for the selected period. **Dedicated API snippet** gives the request for the dedicated endpoint. Coding agents use `list_dedicated_gpus`, `start_dedicated_deployment`, `update_dedicated_deployment` and `stop_dedicated_deployment`. Starting reserves paid capacity: confirm the GPU and rate with the user first. # How to train a model Source: https://docs.overmindlab.ai/how-to/train Fine-tune a language or decision model, follow training and assess the result. Begin with a prepared training cell, a task definition and separate development data where required. A capability is optional. Decide whether the output is conversational text/tool calls or typed probabilities before selecting a model. ## Choose the model for your output Use a **language model** when you need text responses or tool calls. Use a **decision model** when you need typed probabilities over defined outcomes. Both paths use prepared data and a recorded training recipe, but their input formats, evaluation and serving interfaces differ. ### Language models For a complete preparation-to-benchmark walkthrough, see [How to fine-tune a language model](/tutorials/fine-tune-language-model). Give your agent examples of the responses you want the model to produce. This example uses the [fraud dataset](/reference/example-dataset) to train a language model that responds with a classification label. For another task, supply examples of the text or tool-call behaviour you need. Use Overmind to fine-tune a language model to classify transactions as fraud or not\_fraud using this dataset: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Prepare the training examples, help me choose a suitable model, explain the expected cost, and compare the trained model with its base on held-out data. ### Decision models Give your agent the inputs and the outcomes you want to predict. For a complete preparation-to-benchmark walkthrough, see [How to train a decision model](/tutorials/first-dataset). Train a decision model in Overmind to predict fraud using this dataset: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Help me choose a suitable model, explain the expected cost, and follow the run through to its validation results. Both requests include training. Your agent checks data and model compatibility, and brings back decisions that need your input. **What you get:** A training run, progress updates and a model you can evaluate. The steps below describe the workflow your agent follows and the results to verify. ## 1. Inspect eligibility and readiness Use `get_model_catalog` for chat training or `list_decision_training_models` for native decision training. Inspect the model's supported method, context and runtime constraints. Call `check_finetune_readiness` with the exact dataset/cell and planned validation/evaluation inputs. Resolve format and model compatibility failures in Data Workshop. Readiness does not certify the labels or expected task performance. For native decision training, the qualified path is Clef with Modal LoRA on one GPU. D3 and D1 catalogue paths remain launch-disabled until their runtimes qualify. Preserve runtime options, target distributions, weights and declared ordinal means. ## 2. Choose development and final assessment Pin train, validation and eval cells deliberately. For chat comparisons, select the eval set and the before/after targets. All four chat evaluation flags can be disabled; native jobs use a separate probability evaluation plan. Freeze the monitoring policy with the recipe. Declare classification labels, exact matching, a JSON Schema or JSON field pointers when generation checks need them. Keep development measurements separate from final held-out evaluation. For an evidence-linked programme, save a training plan with task meaning and exact partition identities. A full experiment can launch directly. Use a development-only pilot and measured selection when you want to compare recipes first; a pilot is optional. Exploratory jobs remain available separately. ## 3. Inspect preparation and forecast Use `estimate_finetune` for the selected model and recipe. Inspect hardware, rate fetch time, evidence, missing cost components and duration uncertainty. An unmeasured total is unknown, even when a GPU hourly rate is known. Explicit `prepare_training_data` creates a visible setup job and starts model-specific preprocessing. It does not admit GPU training. The preparation pins tokenization and source selections; a matching launch promotes that same job. For multiple candidates, use `create_training_experiment`, then `prepare_training_experiment`. Review the saved forecast before `launch_training_experiment`. A bounded `create_training_profile` can measure a recipe with missing runtime evidence; it is still paid execution. ## 4. Launch the accepted recipe Use `start_finetune` or the decision-specific `start_decision_job`. Supply exact cells and a stable `request_key`. Omitted hyperparameters use the platform's supported planning path; explicit settings constrain the recipe and unsupported settings are rejected. Example launch `hyperparameters` object: ```json theme={"dark"} {"training_type": {"type": "Lora"}} ``` For `start_decision_job`, omit chat capability, eval-set, benchmark-model and group selections, and keep all four chat evaluation flags false. Its held-out probability comparison uses the [decision comparison workflow](/how-to/compare-decisions). Inspect the returned job, requested and effective recipe, runtime identity and input pins. Reuse the same key with identical inputs after an uncertain response. A changed recipe under the same key conflicts. ## 5. Follow completion and assess the result Follow [Monitor training](/how-to/monitor-training). Successful training starts deployment through the matching chat or decision runtime. Deployment readiness and benchmark completion remain separate outcomes. Compare the selected final results, coverage and failures before claiming improvement. Download weights through the deployment handoff when available. [Activation](/how-to/serve) is a separate decision from training and benchmarking. # Overmind overview Source: https://docs.overmindlab.ai/index Understand your AI application, prepare data, evaluate changes and train models from your coding agent. Overmind is a platform for building and improving AI applications and models. It brings together observability, data preparation, evaluation, model training and inference, so you can follow an improvement from the source data or observed failure through to a measured result. You can start with an existing application or a dataset and a task. Connect Overmind to your coding agent, describe what you want to achieve, and work through the results in the same conversation. ## What you can do with Overmind > [card] [Understand your application](/how-to/investigate) > > Map what your agent does, trace its executions, and investigate failures, quality, latency and model usage. > [card] [Prepare data](/how-to/prepare-data) > > Turn files, datasets or recorded traces into reproducible training and evaluation inputs. > [card] [Evaluate and improve](/how-to/evaluate) > > Measure outputs against explicit checks, compare models, and test prompt or code changes. > [card] [Train and serve models](/how-to/train) > > Fine-tune language or decision models, benchmark their results, and call ready deployments from your application. These workflows connect: a failure found in a trace can become an evaluation example, and a prepared dataset can support training and a held-out comparison. You can also use each workflow on its own. ## How you use it Your **coding agent** is where you describe the task. Through **MCP**, it discovers your Overmind projects and uses the platform's tools to prepare data, run evaluations, launch training and inspect results. It also writes code and uses the CLI for local files and repository work. The **Overmind Console** gives you a visual view of the same project: datasets, traces, evaluations, training progress and deployments. Use it to inspect what the agent has created and follow running jobs. A single prompt can cover a complete workflow. The agent works out the steps, carries the project context forward and brings back the results or decisions that need your input. Overmind retains the data, recipes and run records so you can inspect or continue the work later. [Learn how the interfaces work together](/explanation/native-agent). ## Get started 1. **Connect your coding agent.** Paste the setup prompt below into your coding environment and follow the account and client setup steps. 2. **Choose a project and starting point.** Bring a dataset and describe the output you need, or open your application's repository to work from its code and traces. 3. **Ask for an outcome.** Request data preparation, an investigation, a model comparison or a training run. Your agent checks the proposed execution and expected cost before launching training. Read [https://console.overmindlab.ai/setup.md](https://console.overmindlab.ai/setup.md) and follow the instructions to set up Overmind in my coding agent. The [connection guide](/how-to/connect) explains setup and verification. Once connected, choose a task: > [card] [Train a decision model](/tutorials/first-dataset) > > Learn the full preparation, training and benchmarking workflow for predictions with typed probabilities. > [card] [Fine-tune a language model](/tutorials/fine-tune-language-model) > > Train a model for text responses or tool calls using examples of the behaviour you want. > [card] [Connect an application](/how-to/map-agent) > > Map a repository's capabilities, then add tracing and evaluate its behaviour. > [card] [Explore the tutorials](/tutorials/index) > > Follow a worked example of training, tracing or evaluation. For a specific task, use the how-to guides. Use [Reference](/reference/mcp) for tools and formats, or [Explanation](/explanation/native-agent) for the concepts behind the workflows. # REST API map Source: https://docs.overmindlab.ai/reference/api Authentication, schema discovery and the current resource families. Overmind Cloud uses `https://api.overmindlab.ai` as its API base. For self-hosting, use your installation’s API address. Each server publishes its own OpenAPI schema at the paths below. Use this after [connecting Overmind](/how-to/connect). Supply or select the named resource in your conversation; the agent reads the contract rather than guessing IDs. Show me how to inspect the dataset we prepared using the Overmind REST API. Explain the endpoint, required authentication and response fields, with an example request that uses an environment variable for the key. | Path | Content | | - | - | | `/api/schema/` | Machine-readable OpenAPI | | `/api/docs/` | Interactive Swagger explorer | | `/api/redoc/` | Reference explorer | | `/health` | Unauthenticated health check | The deployed schema is authoritative for request/response fields. The endpoint index below names resource families, not every custom action. MCP uses the same domain services through purpose-built tools. ## Authentication API keys use `X-Api-Key`, `Authorization: Bearer` or `Authorization: Api-Key`. Account scope reaches authorized projects; project scope is restricted to one. Select the project using each endpoint's declared query/body field. The hosted Console uses Clerk when configured. Local email/password login is `POST /api/auth/local/`, with create-on-first-use and no email verification. JWT token/refresh endpoints also exist. MCP OAuth tokens are specific to MCP and are not ordinary REST/CLI credentials. API keys are managed under `/api/auth/api-keys/`, with calling-key inspection at `/api/auth/api-keys/current/`. Plaintext keys are returned only at creation. Guest sessions are read/claim-only unless an endpoint explicitly permits the requested guest write. ## Core and observability | Resource | Path | | - | - | | Projects | `/api/projects/` | | Memberships and invitations | `/api/projects/{project_id}/memberships/`, `/invites/` | | Agent map | `/api/agent/` | | Capabilities | `/api/capabilities/` | | Repository sync | `GET` / `POST /api/v1/sync` | | Traces | `/api/traces/` | | Sessions | `/api/sessions/` | | OTLP ingest (protobuf or JSON) | `POST /api/v1/traces` | | Signal definitions/settings | `/api/signal-subscriptions/` | | Observed signals | `/api/signal-events/` | | Tracing connectors | `/api/connector-credentials/` | Trace list/detail are backed by spans, not a separate Trace table. The current API has no task-execution, behaviour or verdict resource families. ## Data and preparation | Resource | Path | | - | - | | Datasets | `/api/datasets/` | | Durable local transfers | `/api/dataset-transfers/` | | Upload storage | `/api/uploads/` | | Retained packages | `/api/dataset-pipeline-packages/` | | Recipe revisions | `/api/dataset-pipelines/` | | Bindings | `/api/dataset-pipeline-bindings/` | | Exploration | `/api/data-explorations/` | | Group-preserving partitions | `/api/data-partitions/` | Dataset actions expose source/cell inspection, rows, export, workbench and pipeline execution. New recipes require a retained package. Use the CLI for local byte transfer rather than manually reconstructing its resumable protocol. ## Evaluation, training and serving | Resource | Path | | - | - | | Eval matrix | `/api/eval-matrix/` | | Eval sets | `/api/eval-sets/` | | Eval runs | `/api/eval-runs/` | | Run rows | `/api/eval-run-rows/` | | Score history | `/api/eval-scores/` | | Optimiser experiments/candidates | `/api/optimizer-experiments/`, `/api/optimizer-candidates/` | | Training preparations | `/api/training-preparations/` | | Training jobs | `/api/finetuning-jobs/` | | Decision job identity | `/api/decision-jobs/` | | Decision catalogue | `/api/decision-models/` | | Task training plans | `/api/training-plans/` | | Saved training experiments | `/api/training-experiments/` | | Native comparisons | `/api/native-evaluations/` | | Decision performance workloads | `/api/decision-performance/` | | Deployments | `/api/deployed-models/` | | Durable deployment inference | `/api/deployed-models/{deployment_id}/inference/` | | Inference receipts | `/api/inference-requests/{request_id}/` | | Chat completions/model catalogue | `/api/v1/chat/completions`, `/api/v1/models` | | Typed system-one inference | `/api/v1/systemone` | Read the operation schema to distinguish saving, preparation and launch. A saved training experiment or native comparison draft does not start provider work. ## Errors and pagination DRF lists generally return `count`, `next`, `previous` and `results`; custom evidence endpoints may use offset/cursor contracts. Follow the returned pagination rather than assuming the first response is complete. Typical status classes are `401` for authentication, `403` for permission, `404` for absent or inaccessible resources, `409` for identity/revision conflicts and `402` for insufficient credits where billing gates apply. Field validation and workflow errors include details to inspect before retrying. Use a stable request key where an operation supports it. Do not assume a repeated POST is harmless on operations without that contract. # CLI commands Source: https://docs.overmindlab.ai/reference/cli Account connection, source transfer, packages, repository sync and optimiser commands. The `overmind` command ships with the Python package. `overmind --version` reports the installed version; `overmind COMMAND --help` gives that version's exact options. Use this after [connecting Overmind](/how-to/connect). Supply or select the named resource in your conversation; the agent reads the contract rather than guessing IDs. Check that my Overmind CLI is connected to my account and help me upload the data file I have selected. Confirm which project receives it and show me the resulting dataset when the import finishes. ```bash theme={"dark"} uv tool install overmind ``` Install `overmind[tracing]` in the application environment for Python instrumentation. The CLI alone does not require tracing dependencies. ## Connection Replace `API_URL` with `https://api.overmindlab.ai` for Overmind Cloud, or your installation’s API address for self-hosting. Replace `PROJECT_ID` with the project selected from your connected account. ```bash theme={"dark"} overmind connection configure --api-url API_URL --project-id PROJECT_ID --json overmind connection check --api-url API_URL --project-id PROJECT_ID --json ``` The saved account connection is `$XDG_CONFIG_HOME/overmind/connection.toml`, defaulting to `~/.config/overmind/connection.toml`, with mode `0600`. API address and account credential are bound together. Repository credentials and explicit settings can take precedence; use the connection check from the actual working environment. Never copy a saved credential to a different URL to work around a connection error. A network permission denial is resolved in the host execution environment. ## Repository setup and sync | Command | Purpose | | - | - | | `overmind init --ide CLIENT --api-url API_URL` | Install workflow skills, seed missing configuration and add the MCP entry | | `overmind chassis --root PATH` | Deterministic Python and TypeScript/JavaScript source inventory | | `overmind sync` | Upload and pull the repository manifest | | `overmind sync up` | Upload the snapshot and write assigned identities back | | `overmind sync down` | Pull the server snapshot into the local manifest | | `overmind skills list --verbose` | Installed/available workflow skills | | `overmind skills sync NAME... --ide CLIENT` | Refresh selected skills | Clients include `cursor`, `claude`, `opencode` and `codex`. An explicit `--api-url` pins the destination. `--env local` is for development of a self-hosted installation; ordinary Cloud setup does not require it. `/overmind setup` is a coding-agent workflow, not a shell subcommand. It runs discovery, authors the manifest and syncs it. There is no `overmind scan` command. `sync down` changes the local manifest; review local edits before pulling. ## Dataset upload ```bash theme={"dark"} overmind dataset upload FILE1 FILE2 \ --project-id PROJECT_ID --api-url API_URL --intent explore --wait --json ``` | Option | Meaning | | - | - | | `FILE...` | Ordered source files; one transfer and source publication | | `--project-id` | Explicit destination project; otherwise use repository configuration | | `--dataset` | Attach sources to an existing draft or append a version | | `--brief` | Task description for a new dataset | | `--intent` | `train`, `eval` or `explore`; omission stays pending | | `--capability` | Optional capability UUID | | `--json-rows-field` | Explicit top-level JSON row array; single-file selection | | `--request-key` | Explicit stable transfer key; default derives from bytes and destination recipe | | `--wait` | Wait for terminal landing; fail if landing fails | | `--split`, `--split-position` | Train/eval split share and `head`, `tail` or `random` placement | | `--api-url`, `--api-key`, `--path` | Explicit connection/configuration overrides; avoid secrets on shell history | | `--json` | Machine-readable result | Attachment uses the existing dataset's metadata; do not combine it with new-dataset brief, intent, capability or split settings. For explicit train/development/calibration/final roles use [saved partitions](/how-to/partition-data). Transfers bind SHA-256, byte count, file order and destination recipe. Repeat the identical command after interruption. Multi-file limits are 100 files and 2 GiB combined; per-type limits still apply. Confirm current limits through `overmind://dataset-upload`. `state_scope=at_publication` is a publication snapshot. `--wait` reports `observed_after_landing`. Use the returned current job to distinguish those states. ## Dataset export and source download ```bash theme={"dark"} overmind dataset export DATASET_ID --cell CELL_ID \ --format jsonl --output rows.jsonl --api-url API_URL --json overmind dataset export DATASET_ID --source SOURCE_SHA256 \ --output original.pdf --api-url API_URL --json ``` Formats are `jsonl` and `csv`; `--cell` accepts a cell UUID or displayed version, defaulting to the active version. `--source` downloads retained original bytes and verifies their checksum. Existing output paths are refused. Export does not count as consumer use. ## Retained Python packages ```bash theme={"dark"} overmind dataset pipeline-upload ./preparation --project-id PROJECT_ID --api-url API_URL --json overmind dataset pipeline-download PACKAGE_ID --project-id PROJECT_ID --output preparation.zip ``` Upload accepts a directory or package archive and returns the package identity for `save_dataset_pipeline`. Directory uploads omit Python bytecode caches; unsupported files still fail. Read [Pipeline reference](/reference/pipelines) before authoring the manifest. ## Connectors ```bash theme={"dark"} overmind connector add langfuse --project-id PROJECT_ID --api-url API_URL --json ``` | Provider | Credential environment variables | | - | - | | Langfuse | `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`; optional `LANGFUSE_HOST` | | LangSmith | `LANGSMITH_API_KEY` | | Braintrust | `BRAINTRUST_API_KEY` | | Galileo | `GALILEO_API_KEY` | The command can prompt in a terminal. It returns the connector identity; inspect source boundaries, confirm their capability mapping and start the import through MCP or the Console. ## Optimiser | Command | Purpose | | - | - | | `overmind optimise start -e EXPERIMENT_ID` | Attach to an existing experiment | | `overmind optimise next` | Get the next local action | | `overmind optimise set-template FILE` | Register the single-datapoint command template | | `overmind optimise run-smoke` | Run one datapoint to check the command | | `overmind optimise run-baseline` | Execute the unmodified application over the pinned data | | `overmind optimise add-candidate --diff FILE` | Queue a candidate diff | | `overmind optimise run-iteration` | Execute candidates and submit outputs | | `overmind optimise status --json` | Inspect progress | | `overmind optimise complete` | Close the experiment | Pass `-e` to select the experiment explicitly. Otherwise the CLI uses local `.overmind/optimise_state.json`. The local dataset cache is version-bound. See [the task guide](/how-to/optimise). ## Checkpoints ```bash theme={"dark"} overmind model download-checkpoint DEPLOYMENT_ID --output weights.zip --api-url API_URL --json ``` This downloads the deployment's archived checkpoint. The archive is written after deployment starts, so a ready deployment can have none yet. It is not a generic download of every intermediate training checkpoint. The output path must not already exist. # Deployment configuration Source: https://docs.overmindlab.ai/reference/configuration Server settings and operating boundaries for self-hosted Overmind. This reference is for operators deploying the open-source platform. Overmind Cloud users can [connect their coding agent](/how-to/connect) without configuring these services. The platform repository's `.env.example`, `overbae/settings.py`, `docker-compose.yml` and `frontend/.env.example` define the current deployment contract. Set secrets through protected configuration; do not commit filled environment files. Use this in your platform checkout when [self-hosting Overmind](/how-to/self-host). Review the configuration for my self-hosted Overmind deployment. Check the required services, authentication and public URLs, and explain anything missing without displaying secret values. ## Local development endpoints These addresses apply when running the platform locally with Compose. A remote self-hosted deployment uses its configured addresses. | Service | Default | | - | - | | API | `http://localhost:8000` | | MCP | `http://localhost:8000/api/mcp/` | | Console | `http://localhost:5173` | ## Required configuration Current API startup validation requires values for the following services, even if your first task does not launch training: | Variables | Purpose | | - | - | | `SANDBOX_URL`, `SANDBOX_TOKEN` | Isolated document and retained-package runtime | | `OPENROUTER_API_KEY` | Routed model operations | | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | Provider training/serving access | | `INFERENCE_API_URL`, `INFERENCE_API_KEY` | Deployed serving endpoint and shared secret | | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_BUCKET_NAME` | Checkpoint archive | | `HF_TOKEN` | Required base-model access | `FINETUNING_BACKEND` must be `modal`. Set `MODAL_ENVIRONMENT` explicitly and deploy the immutable worker releases expected by the API. Local hot reload does not deploy changed provider code. ## Identity and URLs | Variable | Purpose | | - | - | | `DJANGO_SECRET_KEY` | Signing secret | | `DJANGO_DEBUG` | Development mode | | `DJANGO_ALLOWED_HOSTS` | Accepted API hosts | | `DJANGO_CORS_ALLOWED_ORIGINS` | Allowed Console origins | | `FIELD_ENCRYPTION_KEY` | Encryption of stored connector credentials | | `FRONTEND_URL` | Console links returned by the platform | | `MCP_SERVER_URL` | Canonical MCP URL and OAuth resource identity | | `CLERK_API_SECRET_KEY` | Hosted sign-in; blank selects local auth configuration | | `VITE_API_BASE_URL` | Frontend API destination | | `VITE_SELF_HOSTED` | Local Console email/password mode | | `VITE_CLERK_PUBLISHABLE_KEY` | Hosted frontend sign-in; blank for local auth | Local frontend overrides belong in `.env.development.local`; the repository's environment-specific files take precedence over a generic `.env`. ## Workers and sandbox Compose has six Celery worker/queue profiles: `control`, `io`, `io_traces`, `batch`, `landing` and `interactive`. Trace scoring is isolated from file landing and interactive preparation. Waiting jobs do not imply that their execution attempt has started. Document parsing and retained Python packages execute through the isolated sandbox. It has no platform credentials, shared application media mount or outbound network. The landing worker streams retained bytes and receives progress; unavailable parsing does not fall back to application-process execution. `WORKSHOP_PROJECT_CONCURRENCY` bounds per-project interactive execution. Shared admission and observed capacity govern landing. A nonzero `DATASET_IMPORT_MAX_QUEUE_SECONDS` expires imports that wait longer; the default `0` never expires them. Do not change queue topology merely to clear a slow job without understanding its resource limits. ## Model pacing and analytics `JEV_REQUESTS_PER_MINUTE` and `JEV_TOKENS_PER_SECOND` control account-wide pacing through shared capacity. Configuration failure is a technical scoring error, not permission to substitute a generative judge. `POSTHOG_PROJECT_TOKEN` enables optional server product analytics according to deployment settings. Local analytics remains off unless configured. Training evidence is retained independently of analytics delivery. ## Metering and billing With `STRIPE_SECRET_KEY` configured, remaining-credit and seat/plan controls apply. Without it, usage can still be metered without the remaining-credit gate. Native coding-agent usage belongs to its external environment; model evaluations, training and serving have their own execution costs. Published rates, worker measurements and recorded ledger charges have different scopes. A local API address or a missing platform credit gate is not a promise of free provider execution. # Datasets, sources and consumer formats Source: https://docs.overmindlab.ai/reference/datasets Identities, file limits, query bounds and training/evaluation row shapes. Use this after the relevant [tutorial](/tutorials/index), or substitute a resource from your own project. Inspect the fraud dataset we prepared in Overmind. Explain its source, available versions and intended uses, then identify the version ready for training and any unresolved preparation issues. ## Identities | Entity | Meaning | | - | - | | Dataset | Project-scoped brief, intent, optional capability and source/preparation history | | Source | Retained original file or trace selection with extraction facts | | Cell | Exact published rows, schema and fingerprint | | Active version | Current default for a new consumer when no cell is supplied | | Consumer pin | Exact cell retained for an evaluation, optimiser or training job | | `source_row` | Source identity, independent of current display position | | `_overmind_parent_rows` | Declared contributing source identities for externally imported derived rows | Intent is `train`, `eval`, `explore` or pending when not selected. A capability is optional. Once a version has been used, the dataset’s intent and capability association are fixed. Selecting a different active version does not rewrite frozen consumers. ## Inventory and preparation The Console lists one row per source workspace: the source dataset and the outputs derived from it. **Status** shows current work (upload bytes when both counts are known, queued work, transformations, previews and imports). **Role** shows Training, Development, Calibration, Evaluation or Source from completed partitions and recorded consumers. **Source** shows the uploaded file or the source kind. ![Dataset inventory with status, role and source columns](https://mintcdn.com/overmind-b84ae13c/jOsLwNjkhuLAOlvs/images/platform/datasets-inventory.jpg?fit=max\&auto=format\&n=jOsLwNjkhuLAOlvs\&q=85\&s=5a039ec8e088efabd9c12972a5cb06d1) The dataset inventory with an open preparation on one workspace. A coding agent groups its preparation work with `start_dataset_preparation` and closes it with `update_dataset_preparation`: `completed` with published output cells, `paused` or `cancelled`. Jobs inside an open preparation appear under one notification. An open preparation labels the dataset but does not block editing. **End preparation** on the dataset page cancels it, and Overmind cancels a preparation after 24 hours without activity. ## Source formats and limits Read `overmind://dataset-upload` for the connected deployment's supported formats and numeric limits. Tables include CSV, TSV, JSON, JSONL/NDJSON and Parquet. Supported document/image landing retains original bytes and extraction metadata. Current document limits are 100 MiB and 2,000 PDF pages. A transfer accepts up to 100 ordered files and 2 GiB combined, still subject to each file's limit. `--json-rows-field` explicitly selects a top-level JSON array for a single file. PDF text recovery checks pages missed entirely by the primary parser before OCR. Tesseract OCR defaults to English; `update_dataset(ocr_languages=[...])` on a draft sets languages for subsequent imports, and `[]` restores the default. Retried imports retain their original settings. Extraction records engines, languages, orientation and page regions where available. Font-encoding artifacts remain flagged rather than semantically repaired. OCR does not guarantee visual-table reconstruction or complete reading order. ## Inspection and query `inspect_dataset` pages cells and source inventories independently. `source_limit` defaults to 10 and is at most 20; follow `source_page.next_cursor`, including when a response budget shortens a page. `query_dataset` reads one exact cell through SQL. Current response bounds are 100 rows, 32 KiB JSON and 200 columns. Oversized results fail without clipping values. Project only required columns, aggregate or export the cell for larger work. Mixed scalars and precision-sensitive integers are retained without lossy float conversion. Do not infer that text resembling JSON has become a structured value unless its schema declares that shape. ## Chat training A trainable conversation retains message roles, content and tool-call structure, including supervised assistant targets. For example: ```json theme={"dark"} { "messages": [ {"role": "user", "content": "Classify: reset my password"}, {"role": "assistant", "content": "account"} ] } ``` Exact model preprocessing validates the selected tokenizer/template and supervised tokens. Format acceptance does not verify that the assistant response is correct. ## Evaluation Generated-answer evaluation separates input from its reference: ```json theme={"dark"} { "input": "Classify: reset my password", "expected_output": "account" } ``` For `mode="recorded"`, the scorer uses a terminal assistant message when present, otherwise the row's retained `expected_output` as the recorded answer. It does not assume an arbitrary `output` column is the answer. A recorded answer cannot serve as independent evidence of its own correctness; use checks with valid evidence and inspect the pinned row contract. ## Typed decision training ```json theme={"dark"} { "decision": { "state": "The account owner cannot sign in.", "question": "Which queue should receive this ticket?", "kind": "choice", "options": ["delivery", "account"], "target_probabilities": [0.1, 0.9] } } ``` `choice` selects among options, `noul` uses false/true order and `score` uses ordered options. Distributions retain one finite probability per option and sum to one; positive example weights retain their contribution. Options can vary by example. Mean-only ordinal supervision uses `target_mean`, increasing `option_values` and `target_semantics="ordinal_mean"`. It does not invent a probability distribution. Evidence-backed target semantics and provenance belong in retained code and metadata. Final pipeline consumers are `chat_train`, `model_eval`, `decision_train` and `decision_eval`. Declare a nested decision column as `object`. Native evaluation projection keeps references outside `input.decision`. ## Trace sources Trace selections retain recorded input/output, transcript, usage and score together. A source capability filter takes precedence over destination association. A unique matching invocation scopes the evidence; repeated matching invocations fail before publication. Unmatched sources retain whole-trace evidence without invented single-capability attribution. Observed tool arguments do not replace missing wrapper input/output, and recorded answers do not become verified labels automatically. # Evaluation matrix and runs Source: https://docs.overmindlab.ai/reference/evaluation Row definitions, deterministic span fields, targets and result interpretation. Use this after the relevant [tutorial](/tutorials/index), or substitute a resource from your own project. Inspect our latest Overmind evaluation. Explain which data, models and checks it used, how to interpret the scores and whether any examples failed to produce usable results. ## Matrix rows | Field | Contract | | - | - | | `name` | Row name | | `type` | `deterministic` or `llm` | | `source` | Python `evaluate(span)` script, or one-sentence expectation | | `capability` | Optional capability scope; otherwise project-wide | | `span_type` | Blank or `llm_call`, `tool_call`, `retrieval`, `workflow` | | `span_name` | Optional exact live span name | | `enabled` | Whether the saved row participates | | `definition_source` | `stock`, `code`, `config`, `custom` or `unknown` | `create_eval_matrix_row` creates a row. The MCP contract does not edit row definitions in place: delete and create a replacement. Deletion removes current set membership; existing results and frozen run snapshots remain. The Default set can include source decorator expectations, valid TOML catalogue checks, legacy declared expectations and seeded checks. Source declarations update through sync. A missing declaration is not reconstructed from a display label. ## Deterministic scripts A check defines `evaluate(span)` in import-free Python. It sees read-only fields and bounded related spans. Supported globals include limited `json` and `re` helpers; arbitrary imports, files and network access are unavailable. Core span fields are `span_id`, `parent_id`, `trace_id`, `span_type`, `name`, `tool`, `status`, `input`, `output`, `expected`, `duration_s`, `usage` and `attributes`. Access them as attributes, such as `span.output`. Related evidence includes `parent`, `children` and `siblings`. ```python theme={"dark"} def evaluate(span): if not span.output: return None return float(span.output in ("delivery", "account")) ``` Return a boolean, a score from 0 to 1, a two-item `(score, reason)` tuple/list or `None` to abstain. Script failures are errors, not failed quality checks. The executor applies AST, CPU, memory and wall-time bounds. Stock checks include JSON validity, tool selection, exact match, contains, regex, required keys, field conformance, goal state, tool arguments, model-call count, latency and cost budgets. Templates needing a reference, pattern or budget stay disabled until configured. ## LLM expectations Jev evaluates whether the output meets the expectation using the available input and evidence. A generative model supplies reasons for low scores. `judge_model` selects that explanation model; blank uses the configured default. Jev failures do not fall back to generative grading. Live span, trajectory and plan-drift work have bounded attempts; exhausted attempts remain error metrics. Live scores reside in `Span.eval_attributes` and append `EvalScore` history. ## Eval sets `create_eval_set` requires `name` and `rows` (1–200 row IDs or names), with optional `capability` and `activate` (default false). Creating a set does not launch an evaluation. ## Readiness and launch Shared selection fields are `dataset`, optional `cell` or displayed `version`, `eval_set` and `judge_model`. Readiness defaults to the capability's active set or applicable enabled rows when a set is not supplied. `run_evaluation` requires a name and 1–6 `targets`. Each target accepts: | Field | Contract | | - | - | | `label` | Display name | | `mode` | `generate` or `recorded` | | `model_name`, `model_ref` | Supported generated-target model identity | | `params` | Supported generation configuration | | `is_baseline` | Marks the group's baseline | | `output_tokens` | Planned output reservation | `max_items` defaults to 100, bounded to 1–100,000. `sampling` is greater than zero and at most one. Launch returns one run per target with a shared group ID and the pinned cell. Dataset answer scoring uses matrix rows with blank or `llm_call` span type, regardless of a live `span_name`. Runs freeze the selected matrix and explanation judge. The Console displays results; use MCP or REST for standalone launch. ## Comparison and health `compare_evaluations` accepts a run and optional baseline, defaulting to the completed baseline in its group. It returns paired count, wins/losses/ties, per-metric deltas and trust information. Scores, abstentions, technical errors, pending work and degraded generation remain distinct. Paired results use only rows both runs scored. Inspect missing coverage before interpreting the overall delta. Context estimates include reserved output for generation and explanations. Warnings are advisory; unknown capacity is not a confirmed overflow. At execution, rejected requests and exhausted grading attempts remain technical failures. Output-token exhaustion is inspectable and excluded from trusted quality aggregates where degraded. # Fraud example dataset Source: https://docs.overmindlab.ai/reference/example-dataset The ecommerce transaction source used in the walkthroughs. The examples use [apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl) on Hugging Face—the ecommerce fraud dataset used in our earlier model-training workflow. Explore this ecommerce fraud dataset in Overmind: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Explain the fields, what the fraud label means and how balanced the examples are. Tell me what it can help us test and its main limitations. ## Source facts The source is a CSV of synthetic ecommerce transactions. Its target is `is_fraud`. The [dataset card](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl/blob/main/README.md) describes the source and lists an MIT license. | Property | Verified value | | - | - | | File | `ecommerce_fraud_sample_10k.csv` | | Rows | 10,000 | | Non-fraud | 9,580 | | Fraud | 420 | | Revision inspected | `8314a33ce236442577ac988c104c2f72f17fda7f` | The available fields are `transaction_id`, `timestamp`, `user_id`, `transaction_amount`, `merchant_category`, `device_fingerprint`, `ip_address` and `is_fraud`. ## What preparation should establish The coding agent inspects which transaction fields are available at prediction time and how related observations should be grouped. It keeps the target out of model input and retains the original source, preparation and exact versions used by consumers. For a binary decision, the supplied hard label can become a reference over non-fraud and fraud options in an explicit order. This does not imply that the source contains calibrated probabilities. Class imbalance matters: predicting non-fraud for every row gives 95.8% accuracy on the full source. Inspect fraud precision, recall and coverage alongside aggregate accuracy. Repeated users, devices or IPs need examination before choosing a split; the agent should explain its choice. ## Reproducing the source check The original CSV was inspected at the revision above, with SHA-256 `05f63f3c49fc9a3fab5e230e8474e064052a239fb56c1aa1e10be98821e3a44c`. Pin that revision when reproducing these counts. Newer source revisions may differ. Synthetic-data results demonstrate the workflow. Assess real fraud-detection performance on suitable independently held-out operational data. # Glossary Source: https://docs.overmindlab.ai/reference/glossary Terms used in the Console, MCP, CLI and result receipts. Use this after the relevant [tutorial](/tutorials/index), or substitute a resource from your own project. Explain datasets, cells, evaluation runs and deployments using our fraud project in Overmind. Connect each term to something we have created so I can understand how the workflow fits together. | Term | Meaning | | - | - | | Account connection | Credentials with access to projects authorized for that account | | Project | Tenant boundary for capabilities, data, runs, connectors and models | | Agent | The project's application map, composed of capabilities | | Capability | A product purpose tied to source context and observed telemetry; optional for data-first workflows | | Scan | Local source discovery and interpretation, retained in a manifest | | Repository snapshot | Provenance of the checkout inspected during the scan; distinct from upload time | | Trace | Spans sharing a trace ID | | Span | One recorded operation with timing, attributes and evidence | | Session | Related traces grouped by conversation identity | | Unbound | Telemetry without a resolvable capability identity | | Signal | A recorded observation matching a configured condition; not a root-cause diagnosis | | Dataset | A task brief, intent and retained source/preparation history | | Intent | Train, eval, explore or pending until explicitly selected | | Cell | A concrete dataset version with rows, schema and fingerprint | | Fingerprint | Content identity used to bind exact data or artifacts | | Lineage | Recorded relationship between source and derived rows/cells | | Package | Retained transformation code and manifest | | Pipeline family | Identity grouping immutable revisions of a preparation recipe | | Pipeline revision | One saved recipe, package and declared flow | | Pipeline run | An execution receipt against exact source inputs | | Preview | Bounded execution/check evidence without publication | | Binding | A saved source-to-revision processing configuration, initially paused | | Consumer pin | Exact dataset cell retained for a run | | Eval matrix row | A deterministic check or an LLM expectation with scope | | Eval set | Named collection of matrix rows | | Evaluation target | Recorded answers or a selected model generating answers | | Eval run | One target evaluated with frozen data and matrix | | Baseline | The explicitly selected comparison target | | Paired coverage | Examples both compared targets scored | | Abstention | No judgement produced because the check cannot apply | | Grading error | Technical failure while evaluating, separate from quality failure | | Development check | Measurement used while training or choosing a recipe | | Calibration suite | Data used to fit calibration before final scoring | | Final suite | Held-out comparison data reserved from fitting and selection | | Decision | State, question and ordered options with typed probability output | | Ordinal mean | Mean observation on a declared scale, without an implied target distribution | | Training preparation | Model-specific tokenization and validation of pinned examples | | Checkpoint | Retained trained state; exact contents and resume support depend on the runtime | | Deployment | An artifact registered with its matching inference runtime | | Activation | Verify a ready chat deployment, then switch a capability alias | | Benchmark model | Default comparison selection, independent of active serving | | Warmth | Current worker availability or recent serving evidence | | Request key | Stable identity for retrying an operation with identical inputs | | Operation receipt | Retained stages, events, identities and measured progress | | Heartbeat | Worker-aliveness observation, distinct from forward progress | | Forecast | Estimate with assumptions, evidence and unavailable components | | Recorded spend | Metered usage supported by receipts; not necessarily the entire provider invoice | Read [Working with a native coding agent](/explanation/native-agent) for how these concepts connect. # Inference contracts Source: https://docs.overmindlab.ai/reference/inference Chat completions, typed decisions, durable requests and deployment state. Use this after [connecting Overmind](/how-to/connect). Supply or select the named resource in your conversation; the agent reads the contract rather than guessing IDs. Show the request format for calling our deployed model in Overmind. Use its actual deployment details to give me an example input and explain the response and readiness state. ## Chat completions The OpenAI-compatible base is `API_URL/api/v1`. Authentication accepts an authorized Overmind API key. ```python theme={"dark"} import os from openai import OpenAI client = OpenAI( base_url=os.environ["OVERMIND_API_URL"].rstrip("/") + "/api/v1", api_key=os.environ["OVERMIND_API_KEY"], ) response = client.chat.completions.create( model="YOUR_DEPLOYMENT_MODEL_ID", messages=[{"role": "user", "content": "Classify: reset my password"}], max_tokens=128, ) print(response.choices[0].message.content) ``` `model` can pin a deployment's `model_id` or use `overmind/CAPABILITY_UUID` for an active chat alias. Supported frontier models may route through the configured provider; catalogue presence and access depend on the deployment. `POST /api/v1/chat/completions` supports message input, model generation parameters, tools, structured response configuration and streaming according to the selected runtime. Read the deployed schema and model contract for supported fields. Input plus reserved output must fit the serving context. The default output reservation is 8,192 tokens; `max_completion_tokens` is an alternative to `max_tokens` and takes precedence when both are supplied. Requests do not silently shrink the output budget to fit. Serving context is separate from training sequence length. `finish_reason="length"` means the generated answer is incomplete. A successful HTTP status alone does not establish a complete result. ## Streaming and long requests SSE and long completions retain 15-second idle keepalives. Streaming failures can carry an error frame. A non-streaming failure after keepalives committed headers can terminate without complete JSON and appear as a transport error. Retain `X-Request-ID` for failure investigation. ## Durable MCP inference `run_inference` requires `deployment` and `request_key`, with exactly one input contract: | Input | Fields | | - | - | | Chat | `messages`; optional `temperature`, `max_tokens` | | Native decision | `decisions` containing `state`, `question`, `kind`, `options`; no generation parameters | The call returns an `inference_request` job. Identical retries recover the receipt; changed inputs conflict. Unknown provider acknowledgement is reconciled without blind resubmission. Job reads do not wake a worker. For chat results, `truncated` describes generation exhaustion. `content_clipped` separately describes the MCP response-size limit; follow the result resource to page the retained answer. ## Typed decision API `POST /api/deployed-models/{deployment_id}/inference/` creates a durable deployment request; `GET /api/inference-requests/{request_id}/` reads it. The system-one endpoint is `POST /api/v1/systemone`. Use its schema or the deployment's generated snippet for the exact synchronous request contract. Native decisions preserve supplied option order and return typed probabilities. They are not chat completions, and generation-temperature/token parameters do not apply. ## Deployment and routing state Deployment status, worker state, active routing and application connection are separate fields. Worker observations can be warm, warming, asleep or unknown; missing measurements are not zero containers. Compatible dense and explicitly supported MoE LoRA models can share a BF16 base. Other supported chat artifacts use their checkpoint preparation route. Native decision artifacts use their separately qualified runtime. Inspect the deployment's actual profile instead of inferring precision or cold-start time from its model family. `set_active_model` starts a durable activation, verifies the target and only then switches a chat alias. `set_benchmark_model` changes future benchmark selection independently. A successful application API-key alias call confirms application connection; internal probes and pinned model calls do not. ## Metrics and downloads Deployment resources and REST metrics accept `period=1h|24h|7d|30d|all` and `source=application|all`. Application traffic includes alias and pinned-model API-key calls; all traffic also includes internal and historical unknown-source calls. Reported latency/speed distributions depend on measured successful requests; failed calls and missing timing remain separate. Cost coverage must accompany estimated totals. `overmind model download-checkpoint DEPLOYMENT_ID` downloads the retained deployment archive once it exists; a ready deployment can have none yet. Deployment retries reuse successful training artifacts; they do not rerun training. # Licensing Source: https://docs.overmindlab.ai/reference/licensing AGPL-3.0 for the platform, MIT for the SDK and CLI, and a commercial licence for different terms. Show me which licences apply to the Overmind platform and SDK, and link to their licence files. The [overmind-core/overmind](https://github.com/overmind-core/overmind) repository ships under two open-source licences. A commercial licence is a paid alternative from Overmind Ltd. | Code | Licence | | - | - | | Platform — API (`overbae/`), Console (`frontend/`), and the rest of the repository | [AGPL-3.0](https://github.com/overmind-core/overmind/blob/main/LICENSE) | | SDK, CLI, and client libraries (`overmind/`) | [MIT](https://github.com/overmind-core/overmind/blob/main/overmind/LICENSE) | Customer applications that `pip install overmind` (or import it) stay under their own licence. The MIT package does not pull the platform's AGPL into your agent. ## Self-hosting [Run the platform yourself](/how-to/self-host) under AGPL-3.0. An unmodified copy on your own network complies. A modified copy offered as a network service must make that modified source available to its users. ## Commercial licence A paid licence from Overmind Ltd covers the same platform under different terms, for customers who cannot use AGPL. Write to [hello@overmindlab.ai](mailto:hello@overmindlab.ai) or [talk to the team](https://www.overmindlab.ai/contact). # MCP interface Source: https://docs.overmindlab.ai/reference/mcp Authentication, discovery, tool families, resources and asynchronous receipts. The Streamable HTTP endpoint is `/api/mcp/` on the selected deployment. MCP shares domain services with REST and the Console. It is not a generic HTTP proxy. Use this after [connecting Overmind](/how-to/connect). Supply or select the named resource in your conversation; the agent reads the contract rather than guessing IDs. Inspect the Overmind tools available in this chat. Explain which ones support datasets, evaluation, training and inference, and show how I can ask you to use them for a typical workflow. ## Discovery and scope | Surface | Contract | | - | - | | `list_projects` | Authorized active projects, pagination and `connection.mcp_url` | | `overmind://interface/current` | Connected contract version, permission-scoped catalogue fingerprint and interface metadata | | Project tools | Account connections pass `project_id` explicitly | | Project resources | Add `?project_id=PROJECT_ID` or follow the returned scoped link | | Project API key | Restricted to its bound project | | `get_credit_balance` | The connection owner's account balance and usage by service; see [credit reporting](/reference/mcp#credit-reporting-and-task-budgets) | Use the connected `tools/list`, `prompts/list` and resources for complete argument schemas. Tool availability depends on permissions and the deployed version; a static tool count is not a capability guarantee. ## Authentication Account connections sign in with OAuth through the deployment's login and consent flow; `overmind init` configures this. API-key connections use `X-Api-Key` or a supported bearer header. Account keys reach authorized projects; project keys retain narrower scope. MCP OAuth tokens do not authenticate ordinary CLI file transfers. Read permission exposes inspection operations; writes require the operation's permission scope. Expiry, account state and membership continue to apply. ## Tool families This index identifies the entry points; it does not replace their schemas. | Area | Inspection or preparation | Execution or mutation | | - | - | - | | Projects | `list_projects`, `get_credit_balance`, `get_credit_usage`, `get_project_overview`, `list_objects`, `search_objects` | `create_api_key`, `report_onboarding_progress` | | Observability | `inspect_capability_health`, `query_traces`, `query_failures` | Signal subscriptions through `set_signal_subscription` | | Signals | `list_signals` | `set_signal_subscription` | | Instrumentation | `get_instrumentation_plan`, `verify_instrumentation` | Repository edits happen locally | | Sources | `list_datasets`, `inspect_dataset`, `query_dataset` | `start_dataset`, `create_dataset_from_traces`, `update_dataset`, `resume_dataset_import` | | Workshop | `inspect_dataset_workbench`, `validate_dataset_pipeline` | `start_dataset_preparation`, `update_dataset_preparation`, `save_dataset_pipeline`, `run_dataset_pipeline`, `cancel_dataset_pipeline_run`, `import_dataset_version` | | Bindings | Workbench binding inspection | `save_dataset_pipeline_binding`, `set_dataset_pipeline_binding_state`, `run_dataset_pipeline_binding` | | Data roles | `explore_dataset`, workflow inspection | `derive_dataset`, `create_data_partition`, `retry_data_partition` | | Matrix evaluation | `check_evaluation_readiness`, `compare_evaluations` | `create_eval_matrix_row`, `revise_eval_matrix_row`, `delete_eval_matrix_row`, `create_eval_set`, `run_evaluation` | | Decision comparison | `list_decision_models`, `prepare_native_evaluation`, `inspect_native_evaluation` | `create_native_evaluation`, `launch_native_evaluation`, `pause_native_evaluation`, `resume_native_evaluation` | | Training | `get_model_catalog`, `list_decision_training_models`, `check_finetune_readiness`, `estimate_finetune`, `inspect_training_progress` | `prepare_training_data`, `start_finetune`, `start_decision_job`, `cancel_finetune` | | Experiments | `list_model_workflows`, `prepare_training_experiment` | `save_training_plan`, `create_training_experiment`, `launch_training_experiment`, `create_training_profile` | | Performance | Workflow/job inspection | `measure_decision_performance`, `resume_decision_performance`, explicit `reuse_evaluation_predictions` | | Serving | Deployment resources, `get_model_swap_prompt`, `list_dedicated_gpus` | `deploy_finetune`, `retry_deployment`, `run_inference`, `set_active_model`, `set_benchmark_model`, `start_dedicated_deployment`, `update_dedicated_deployment`, `stop_dedicated_deployment` | | Optimiser | `check_optimizer_readiness`, `inspect_optimizer_result` | `start_optimizer`; local execution uses the CLI | | Connectors | `inspect_connectors` | `configure_connector`, `sync_connector` | | Operations | `get_job`, `inspect_jobs`, `inspect_operation`, `inspect_mutation`, `inspect_dependencies` | Use each workflow's named lifecycle controls | Preparation tools can create records or run preprocessing. An inspection-style name is not a guarantee of zero provider cost; read the tool's impact and result contract before executing it. ## Credit reporting and task budgets Use `get_credit_balance` to read the connection owner's account balance, lifetime grants and spending, and usage by service. Account and project keys return account-wide figures; no `project_id` is required. Amounts are exact decimal strings in credits: 100 credits = 1 USD. Check my Overmind credit balance. You have a budget of 500 credits for this task. Check estimated costs before launching paid work, monitor recorded spending, and stop before starting more work that would exceed the budget. Save the initial report's `observed_at` and pass it as `since` on subsequent calls. `usage.spent_credits` reports account charges recorded since that timestamp, including charges without a project assignment. `usage.by_service` breaks down that spending. Balance, grants and lifetime spending stay account-wide lifetime totals. Top-ups do not reduce usage. Omit `since` for lifetime usage. Reporting does not reserve credits or enforce a hard task cap. Concurrent spending within the reported scope is included; unposted charges and external coding-agent costs are excluded. Recorded usage can lag work already in progress. A zero recorded total does not establish that pending work is free. When `billing_enabled` is false, the deployment meters usage without enforcing a credit balance. ## Resources | URI | Content | | - | - | | `overmind://project/current` | Project identity, scan provenance and Console URL | | `overmind://activity` | Project activity with task and asset references, and onboarding milestones (`path=agent` or `path=data`) | | `overmind://dataset-upload` | CLI handoff, format limits, package manifest and runtime requirements | | `overmind://dataset-export` | Version export guidance | | `overmind://checkpoint-download` | Checkpoint download guidance | | `overmind://connector-setup` | Provider-credential CLI handoff | | `overmind://capabilities/{capability}` | Saved capability map and related state | | `overmind://traces/{trace_id}` | Recorded span evidence | | `overmind://datasets/{dataset}` | Cells, source facts and dataset state | | `overmind://eval-runs/{eval_run}` | Frozen run state and result evidence | | `overmind://finetunes/{job_id}` | Training state, pinned inputs and monitoring | | `overmind://deployments/{deployment}` | Serving identity, readiness and metrics | | `overmind://operations/{id}` | Recorded events and separate progress clocks | | `overmind://jobs/{kind}/{id}` | Workflow-specific job state | Follow returned resource links for packages, revisions, bindings and data-first workflows. Their identifiers and pagination belong to the connected contract. ## Jobs, pagination and retries Background operations return durable identities. `get_job(kind=..., id=...)` reads the receipt; `inspect_operation` pages stored events without waking a worker. Use returned polling guidance while active and next actions after completion. Requests that accept a stable key recover the same work when the key and payload match. Changed payloads conflict. Mutations with an optional catalogue `request_key`, including `run_evaluation`, recover the original response through `inspect_mutation`. Omission creates new work. A retryable error releases the key, so retry with the same key. Any other refusal stays on its receipt; fix the cause and use a new key. Pending or unknown receipts require reconciliation before resubmission; native request-key workflows retain their own receipts. Pagination can be limited by both item count and response bytes. Follow returned cursors, offsets, `collections` descriptors and `evidence_resource` links for complete retained evidence. Tool errors include a structured code, message, retryability and field details where applicable. Authentication and authorization failures remain distinct from workflow failures. ## Prompts and local handoffs Prompts include `author-dataset-transformation`, `develop-model-from-data`, `investigate-capability`, `instrument-repository`, `prepare-evaluation`, `evaluate-change`, `finetune-capability`, `optimize-capability`, `compare-models`, `ship-model`, `upload-dataset-file`, `export-dataset`, `connect-traces` and `download-checkpoint`. The CLI transfers bytes and runs local repository work. Provider secrets enter protected local configuration or the appropriate Console flow, not MCP tool arguments. See [Connect your coding agent](/how-to/connect). ## Inventory and lineage Use `list_objects` to search names across project objects, or select `kinds`. `trained_model` includes successful chat and decision training jobs even when no deployment exists. Filter by status, capability, creation time or native request key. Follow `next_offset` until null; matching names are candidates, not proof of a unique identity. Each item carries its project, type and resource. `search_objects` searches up to eight explicit accessible projects and reports inaccessible IDs as partial coverage. `get_project_overview` returns counts by object kind and status plus recent objects. `inspect_dependencies` pages recorded direct consumers and routing; external application references and indirect pipeline lineage remain unknown. `inspect_model_lineage` returns pinned input cells, preparation, evaluation references, deployment and active capabilities. Missing pins are not replaced with the dataset's current active cell. Follow the training resource for recipe and checkpoint evidence. ## Batch monitoring and evaluator revisions `inspect_jobs` accepts up to eight job references. Pass its cursor as `after` and `timeout_seconds` from 0 to 30 to wait for a changed snapshot. Each missing or inaccessible job has its own error. Batch reads default to a compact summary; request `view="full"` for progress and details. Summary responses identify omitted fields and link to the complete job resource. Inspection does not start provider work. Job responses distinguish stage, heartbeat and forward progress, and expose saved workflow actions and polling guidance where available. Read `overmind://eval-rows/{id}` for an evaluator's definition and `updated_at`, or get `updated_at` from `list_objects` with `kinds=["eval_row"]`. Pass it as `expected_updated_at` in `revise_eval_matrix_row`, with only the fields that change. Omitted fields keep their values, and the capability cannot change. `source` replaces the whole definition, so read every `source_page` of a long row before you edit it. The edit retains earlier definitions, set membership and frozen historical run snapshots. A stale timestamp fails rather than overwriting a concurrent edit. ## Project credit attribution `get_credit_balance` reports account balance and usage by service. `get_credit_usage` pages posted charges owned by your account within the selected project, grouped by service and recorded training job. `training_experiment` filters linked training jobs; this is not a complete experiment bill. Missing job attribution remains null. Other members' charges, unposted usage and provider estimates are excluded. Neither tool enforces a spending cap. Retained intermediate checkpoints are discoverable with `list_objects(kinds=["checkpoint"])`, including checkpoints from jobs that later failed. Follow the returned checkpoint resource for its identity and training-job reference. Discovery does not certify serving readiness. # OTLP traces and attributes Source: https://docs.overmindlab.ai/reference/otlp The trace ingest protocol and identity fields. Use this after [connecting Overmind](/how-to/connect). Supply or select the named resource in your conversation; the agent reads the contract rather than guessing IDs. Check this application’s OpenTelemetry configuration for sending traces to Overmind. Explain the endpoint, authentication and capability mapping it uses, and identify anything missing from the setup. | Field | Contract | | - | - | | Endpoint | `POST /api/v1/traces` | | Transport | OTLP/HTTP protobuf or JSON | | Content type | `application/x-protobuf` or `application/json` | | Compression | Gzip supported | | Authentication | Project/account API key in a supported header | | Root | Span with no parent; spans sharing a `trace_id` form a trace | There is no JavaScript tracing SDK in the current platform. Use an OpenTelemetry exporter in JavaScript/TypeScript or any other supported language. An explicit `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` includes the complete path. A generic exporter base endpoint may append its own `/v1/traces`; verify the actual request path instead of appending twice. ## Identity and evidence | Attribute | Purpose | | - | - | | `overmind.capability.id` | The capability binding: known UUID or slug | | `overmind.capability.name` | Display label; does not bind | | `overmind.project.id` | Project attribution when required by the credential | | `overmind.span.type` | `entry_point`, `workflow`, `tool_call`, `function`, `llm_call`, `retrieval` | | `conversation.id` | Group related traces into a session | | `service.name` | Emitting service | | `code.namespace`, `code.function.name` | Source function identity | | `vcs.ref.head.revision` | Emitting revision | | `inputs`, `outputs` | Captured operation evidence where supplied by the SDK | | `overmind.unit_kind` | Run/turn metadata | | `overmind.delivery` | Delivered-result metadata | The ingest path also normalizes supported GenAI, OpenLLMetry and OpenInference attributes for messages, model identity, tools and usage, including Claude API, Claude Code and OpenRouter spans. Evidence that was not emitted remains unavailable. For Claude API traces, add a Custom destination in the Claude Console with the API base URL plus `/api` as the OTLP endpoint URL and an `X-Api-Key` request header. Claude Code traces carry the user prompt with `OTEL_LOG_USER_PROMPTS=1`; tool input and model responses require its detailed tracing (`ENABLE_BETA_TRACING_DETAILED=1`), which interactive sessions receive only for allowlisted organizations. For OpenRouter Broadcast, add an OpenTelemetry Collector destination with the API base URL plus `/api/v1/traces` as the endpoint and `{"X-Api-Key": ""}` as the headers. Span identity can override resource identity for applications serving multiple capabilities. Unknown capabilities are stored as unbound; ingestion does not create a capability. Trace ingest and stored-evidence reads do not require remaining model credits. Model-backed scoring is a separate operation. See [Instrument an application](/how-to/instrument) for setup and [How evaluation works](/explanation/evaluation) for scoring semantics. # Retained transformation packages Source: https://docs.overmindlab.ai/reference/pipelines Revision identity, execution, branching, validation and publication contracts. A new Dataset Pipeline revision requires a retained Python package. Saving a revision does not execute it. Inline operation definitions and execution of package-free historical revisions are not supported. Use this after the relevant [tutorial](/tutorials/index), or substitute a resource from your own project. Show how our fraud dataset was prepared in Overmind. Walk through the saved transformation steps, the run that produced the current version and the validation results recorded for that output. ## Package and revision A package contains `manifest.json` and its Python files. The manifest pins version `1`, the deployment-advertised immutable `sha256:…` runtime, ordered steps, parameter schema and execution limits. Read the exact manifest and script I/O contract from `overmind://dataset-upload`; runtime hashes are deployment-specific. Upload with `overmind dataset pipeline-upload`, then pass its package UUID to `save_dataset_pipeline`. | Field | Meaning | | - | - | | `name` | Revision name | | `request_key` | Stable save identity | | `package` | Required retained package UUID | | `dataset` | Optional association with the preparation | | `pipeline` | Existing recipe family when saving a new revision | | `expected_revision` | Optimistic concurrency check for a family update | | `derived_from` | Attribution for a variant derived from another revision | Returned `pipeline_id` identifies the family; returned `id` identifies the exact revision. A stale family update reports `revision_conflict`. An old save retry does not reset a newer preparation association. ## Steps and flow Packages contain 1–20 meaningful script stages. Each entrypoint exposes its substantive transformation; shared helpers hold reusable utilities. Retained-file inspection verifies checksums and pages full source with `file`, `offset`, `limit` and `next_offset`. A step can declare: * `id` and `input` naming `source` or an earlier step. * `inputs` for ordered concatenation of disjoint earlier branches. * `condition` with an expression and entrypoint line describing agent-authored routing. * Column and row requirements, including the final consumer contract. * `batch_rows` for independent row-wise work. Repeated input dependencies form forks. Conditions are declarations; scripts perform selection, and all steps execute, including empty branches. Fan-in rejects overlapping row identities and incompatible nonempty schemas rather than silently deduplicating or coercing them. It is concatenation, not a relational key join. The returned `flow` records nodes, edges, conditions and output. `flow.unconsumed_steps` identifies branches outside the final result. A declared graph does not prove that branches are complete or mutually exclusive. ## Execution request `run_dataset_pipeline` takes `dataset`, exact `pipeline` revision, `source_cell`, `source_fingerprint`, `request_key`, optional `parameters` and `mode`. | Mode | Result | | - | - | | `preview` | Bounded samples/checks retained; no cells published | | `publish` | Successful step outputs appended atomically | `preview_rows` defaults to 100 and is bounded from 1 to 1,000. A preview samples a prefix, not guaranteed coverage of all source families. Absolute `min_rows`/`max_rows` requirements remain deferred until publication; schema, lineage and row-preservation checks apply to both modes. `validate_dataset_pipeline` performs retained-code and declared-source checks without execution. Runtime execution occurs in the isolated approved container, without platform credentials, host mounts or outbound network. ## Batching and limits For row-independent work, `batch_rows` accepts 1–100,000. Batches run separately and concatenate in source order. Each expanded JSONL input/output must fit half the declared `scratch_mb`, and each batch has the declared time limit. Global sorting, grouping, splitting and deduplication must not use this batching contract. Combined-output row checks and atomic publication still apply; a failed batch publishes no partial cells. ## Receipts and recovery `get_job(kind="dataset_pipeline", id=RUN_ID)` reads one run. Receipts retain exact input/output cells and fingerprints, revision/package, execution state, stages, queue age, measured counts, checks, attribution and publication outcome. `evidence_resource` links full preview/row evidence. A failed or cancelled run preserves prior successful sources and consumers. Cancellation prevents publication without claiming remote termination. Interrupted/expired execution is not automatically replayed. Controller recovery fences lost work and cleans owned containers before accepting another execution. ## Bindings Bindings pin a revision, parameters, source, destination and explicit limits. They start paused. `set_dataset_pipeline_binding_state` enables or pauses them with `expected_version`; `run_dataset_pipeline_binding` processes one saved snapshot without enabling ongoing processing. Processing uses bounded full snapshots, including late updates and removed matches. A successful unchanged snapshot is a no-op. Checkpoints commit with publication. Trace snapshots above 10,000 traces fail rather than truncate. Revision adoption requires an explicit rebuild. ## External imports `import_dataset_version` accepts at most 2,000 inline rows/4 MiB, or a project-owned `artifact_cell` and fingerprint for larger results. Bind the original source and preserve `source_row` or declare every contributing `_overmind_parent_rows` identity. The receipt records external producer attribution and validates lineage. It does not independently verify external execution or inherit human-review claims from the source. See [Prepare data](/how-to/prepare-data). # Python tracing SDK Source: https://docs.overmindlab.ai/reference/python-sdk Initialization, capture controls, span decorators and evaluation declarations. Install the tracing extra in the application environment: Use this after [connecting Overmind](/how-to/connect). Supply or select the named resource in your conversation; the agent reads the contract rather than guessing IDs. Review this Python application’s Overmind tracing. Explain what is already captured, identify gaps in its main request flow and show the relevant SDK changes that would fill them. ```bash theme={"dark"} python -m pip install "overmind[tracing]" ``` The base package provides the CLI and inference client. Tracing names require the extra. See [Record a Python trace](/tutorials/first-trace) for an example to run after configuring the API key and project. ## Initialization This example sends traces to Overmind Cloud. For a self-hosted installation, supply its API base URL. ```python theme={"dark"} import overmind overmind.init( service_name="support-service", overmind_base_url="https://api.overmindlab.ai", capability_id="support-triage", providers="auto", ) ``` | Parameter | Meaning | | - | - | | `overmind_api_key` | Explicit key; otherwise resolve protected repository/environment credentials | | `overmind_base_url` | API base URL; otherwise environment, repository configuration or hosted default | | `project_id` | Project attribution, particularly for account-scoped credentials | | `service_name` | `service.name` resource attribute | | `environment` | Deployment environment label | | `capability_id` | UUID or slug sent as `overmind.capability.id` | | `capability` | Display label only | | `providers` | `"auto"` or selected installed provider instrumentors; omission enables none | | `redact_keys` | Additional case-insensitive dictionary keys to redact | | `export_orphan_spans` | Whether standalone function fragments may export; default false | | `debug` | Diagnostic logging | Call once at startup before provider calls. Missing credentials normally return `False` and leave helpers inactive; `OVERMIND_STRICT_MODE=true` makes missing configuration raise. Reinitialization refreshes identity without constructing another exporter. Supported instrumentors cover OpenAI, Anthropic, Google, Agno and LangChain, including LangGraph through the LangChain instrumentor. Only installed integrations can be enabled. ## Environment and credentials Identity/configuration variables include `OVERMIND_API_KEY`, `OVERMIND_API_URL`, `OVERMIND_PROJECT_ID`, `OVERMIND_SERVICE_NAME`, `OVERMIND_ENVIRONMENT`, `OVERMIND_CAPABILITY_ID` and `OVERMIND_CAPABILITY_NAME`. Repository credentials in `.overmind/credentials.toml` are checked against the manifest's endpoint/project. Explicit identity arguments do not merge a missing half with unrelated capability environment defaults. `OVERMIND_SPAN_FLUSH_INTERVAL_MS` and `OVERMIND_SPAN_MAX_EXPORT_BATCH_SIZE` configure batching. `OVERMIND_GIT_SHA` overrides detected revision; `SERVICE_VERSION` sets the service version. An incoming `TRACEPARENT` can continue a remote trace. Optional product analytics uses the separate `analytics` extra. `OVERMIND_ANALYTICS_ENABLED=false` or `DO_NOT_TRACK=1` disables it; CI also disables it. Core tracing does not require PostHog. ## Run and delivery ```python theme={"dark"} with overmind.run("answer", intent=question, conversation_id=session_id) as run: result = answer(question) run.deliver(result) ``` `run` is a context manager or sync/async decorator. It captures the boundary, intent, conversation, tags and error status, and flushes on exit. Decorator identity/intent values can be callables receiving the wrapped arguments. A return value is not automatically a `deliver` call. `deliver(payload, grounded_by=None, name="deliver", provenance="agent")` records the delivered payload on a child span. Optional grounding identifies evidence span IDs or handles. ## Decorators `observe` accepts a span name and type. Typed shortcuts are `entry_point`, `workflow`, `tool` and `retrieval`. They support sync and async functions and re-raise exceptions after recording them. ```python theme={"dark"} @overmind.observe( type="llm", capture="messages", expectations=["The answer cites only evidence present in the input."], ) def answer(messages): return call_model(messages) ``` | Option | Meaning | | - | - | | `span_name` | Name or callable; defaults to function qualified name | | `type` | `entry_point`, `workflow`, `tool_call`, `function`, `llm_call` or `retrieval` | | `capture` | `auto`, `none` or `messages` | | `ignore` | Arguments excluded from capture | | `self_attrs` | Explicit receiver attributes to include | | `format_input`, `format_output` | Explicit payload formatting functions | | `capability_id`, `capability` | Binding identity and display label | | `intent` | Declared purpose of the operation | | `expectations` | List of one-sentence criteria | | `reference_trajectory` | List of declared trajectory steps | | `provenance` | `user`, `agent`, `environment` or `harness` | | `unit` | Optional unit metadata | Expectations are lists, not one bare string. `overmind sync` reads supported source decorator declarations and creates scoped matrix rows. Removing a declaration removes its corresponding managed row on sync. Runtime span evidence and the saved matrix are related but distinct. Automatic capture scrubs common secret-key names and bounds binary-looking payloads. Use `capture="none"` or explicit formatting where values must not enter telemetry. ## Context and metadata | Helper | Purpose | | - | - | | `capability(name=None, id=None)` | Scoped capability identity; name alone is a display label | | `task(key, unit=None)` | Task metadata; `unit="turn"` groups a turn's spans | | `session(conversation_id)` | Scoped conversation identity | | `set_conversation_id(id)` | Conversation identity for subsequent spans | | `set_user(user_id, email=None, username=None)` | User metadata | | `set_tag(key, value)` | Current-span attribute | | `capture_exception(exc)` | Record an explicitly handled exception | | `start_span(name, span_type=..., attributes=...)` | Context manager for a block rather than a function | | `force_flush_traces()` | End open turn scopes and flush before process exit | `intent`, `expect`, `checkpoint`, `eval_context` and `end_conversation` emit runtime evidence events. Their presence is not proof that every declared criterion produced a score; inspect the resulting matrix/trace metrics. ## Existing OpenTelemetry setup When the application owns its `TracerProvider`, add an Overmind OTLP exporter to that provider and use `overmind.tracing.enable_tracing(providers)` for the SDK processors and selected integrations. Avoid creating competing global providers. `PromptString` retains a prompt template and its arguments for supported provider instrumentation. The `overmind.integrations.langgraph` binding helper can attach task scopes and delivery metadata to graph nodes; match these to the actual application structure. # Training inputs, monitoring and receipts Source: https://docs.overmindlab.ai/reference/training Exact preparation, frozen recipes, decision qualification and measured progress. Use this after [connecting Overmind](/how-to/connect). Supply or select the named resource in your conversation; the agent reads the contract rather than guessing IDs. Inspect our fraud-model training run in Overmind. Explain the data versions, model and settings it used, then summarise its validation results and recorded costs, including anything still unmeasured. ## Input selection Training accepts a dataset and optional exact cell/version, capability, validation dataset/cell and evaluation dataset/cell. A capability is optional. Consumer inputs are pinned atomically so later Workshop changes cannot alter the job. Exact preparation uses the selected tokenizer, template and supervised targets. Invalid tool formatting, over-limit examples and missing supervised targets fail rather than silently truncate or discard rows. Preparation preserves ordered row fingerprints, duplicates and separate validation selection. Training reuses the verified token artifact. Explicit `prepare_training_data` creates the dashboard job before dispatching preprocessing. The job remains preparing, awaiting launch or preparation failed until a matching launch promotes it. Preparation alone does not enter GPU dispatch. ## Recipe and launch identity `start_finetune` accepts `request_key`, `dataset`, `base_model`, optional exact cell selections and `hyperparameters`. Omission uses the supported planner; explicit settings constrain it. The effective optimizer `batch_size` is distinct from private worker microbatch implementation. The following is the launch `hyperparameters` object: ```json theme={"dark"} {"training_type": {"type": "Lora"}} ``` Inside launch `hyperparameters`, `training_type` must be an object. The separate `prepare_training_data` tool instead accepts a top-level `training_type="lora"` or `"full"` string, plus a required `context_length`. Native decision full-weight training and unqualified packing are unavailable. `lora_r` must be 8, 16, 32, 64, 128, 256, 320 or 512, and `lora_alpha` must be twice `lora_r`. Explicit unsupported settings are rejected, not silently clamped. Receipts bind inputs, recipe, provider intent and runtime release/environment. Reusing a launch key with the same recipe returns the same job; changing the recipe conflicts. Unknown provider acknowledgement is reconciled against saved metadata. ## Chat benchmark selection | Field | Purpose | | - | - | | `baseline_model` | Run-specific benchmark; otherwise capability benchmark default or codebase incumbent | | `eval_incumbent_before` | Include the saved incumbent comparison target | | `eval_incumbent_after` | Include the saved incumbent after target | | `eval_model_before` | Include the untouched base target | | `eval_model_after` | Include the trained target | | `eval_judge_model` | Frozen low-score explanation judge for this job | Untouched-base and trained-model comparisons are the normal defaults; all four can be off. Selected chat comparisons require compatible eval data/checks. Training labels and live routing do not silently choose a benchmark. The before/after names identify model snapshots; consult the job's stages for actual execution timing. Final comparisons are distinct from development checks. Truncated/degraded output does not produce a trusted headline improvement. ## Development monitoring Monitoring is frozen with readiness, preparation, estimates and launch. Supported schedules are adaptive, steps, epoch and off. Adaptive cadence uses measured optimization/check time, not quality-driven recipe changes. Manifest-bound samples and check identities remain inspectable. Development-loss selection and early stopping are explicit opt-ins. A reload-verified checkpoint is not proof of full optimizer resume. Generation probes support declared classification labels, exact matching, JSON Schema or `json_fields`. Field checks use 1–64 JSON Pointers, each at most 1,024 characters and 64 segments. Escape `/` as `~1` and `~` as `~0`; empty string selects the whole document. Arrays remain ordered and equality preserves types and numeric precision. Missing reference fields are unscorable; wrong/missing predictions fail. Field coverage and whole-example pass rates are separate. Field checks cannot combine with labels/schema in one probe. No tools execute during generation probes. Metered judges, independent challenge suites and declared-slice sampling are not part of this monitoring contract. `inspect_training_progress` accepts job, check/probe selection and pagination. Collection descriptors expose `field` JSON Pointers for large native metrics. The latest generated check remains visible independently of newer loss-only checks. Classification assessment compares the same scored subset with its majority-label baseline; inconsistent counts remain inconclusive. ## Native decision qualification Clef uses the qualified single-GPU Modal LoRA path with Unsloth's decision runtime. D3 and D1 paths are catalogued but launch-disabled pending runtime qualification. Complete target distributions, option order, weights and mean-only ordinal supervision are preserved. `hyperparameters.pre_training_baseline` defaults to true. False skips only the initial development baseline; checkpoint and final validation remain. A successful artifact retains the adapter, full-precision decision head, tokenizer/configuration and manifest. Fresh-process verification checks reloaded components and probability behaviour. Successful training starts a matching decision deployment whose readiness requires a probability prediction. Chat benchmarking and serving remain separate. `start_decision_job` requires a stable request key and rejects chat `capability`, `eval_set`, `baseline_model` and `group_id` selections. Native training uses its own probability validation; keep the four chat evaluation flags false and use a separate native comparison for the held-out benchmark. `DecisionJob` provides a public decision-job ID over the shared training lifecycle. Receipts can return both it and the FinetuningJob ID; these do not identify two independent trainers. ## Experiments and profiles `create_training_experiment` saves explicit variants, purpose and optional training-plan/comparison bindings. `prepare_training_experiment` builds a configuration-bound forecast; `launch_training_experiment` launches the reviewed set. A runtime profile is bounded to 1–128 optimizer steps and 60–3,600 seconds per provider attempt, without automatic provider retries. It measures execution, not model quality. Evidence-linked quality pilots use separate plan/stage and development-selection rules. Native training attempts retain the provider's 24-hour function timeout. `runtime_limit_seconds` is not a complete invoice cap. Optional planning constraints bind a reviewed quote; missing charges and unmeasured components remain explicit. ## Progress and costs Stages can report selected rows, acknowledged transfer bytes, provider materialization, tokenization, dataset construction, optimization and verification. Acknowledged file bytes are not live network throughput. Stage, observation, heartbeat and forward-progress times are distinct. Forecasts use live rate snapshots and compatible completed measurements where available. A GPU rate can be known while duration and total remain unquoted. Recorded usage, ledger charges and estimated unreported components remain separate; unknown cost is not zero. # How to fine-tune a language model Source: https://docs.overmindlab.ai/tutorials/fine-tune-language-model Prepare examples, fine-tune a language model and compare its responses with the base model. Fine-tuning adapts a pretrained language model to examples of the behaviour you want. Use it for text responses, structured answers or tool calls. Your coding agent can prepare the examples, train the model and compare its outputs with the original base model in one workflow. This walkthrough uses the same [ecommerce fraud dataset](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl) as the decision-model tutorial. Here, the model generates a text label: `fraud` or `not_fraud`. The [decision-model walkthrough](/tutorials/first-dataset) covers typed probability outputs. ## Before you begin [Connect your coding agent](/how-to/connect) and select a separate tutorial project. You do not need an application repository or a previously trained decision model. This workflow includes training and benchmark inference, which can incur provider costs. Your agent explains the proposed model and expected cost before launch. Use Overmind to fine-tune a language model to classify transactions as fraud or not\_fraud using this dataset: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Prepare the data, train a suitable model, and benchmark it against the base model on held-out examples. Show me where it improves and where it makes mistakes. The agent carries the workflow through preparation, training and benchmarking, bringing back decisions that need your input. You do not need a separate request for each stage. ## What happens next 1. **Prepare examples.** The agent turns transaction details into user messages and the supplied labels into assistant responses. It keeps `is_fraud` out of the input, retains the preparation, and separates training, development and held-out data. 2. **Define the comparison.** It prepares held-out inputs with separate reference labels and proposes checks for valid output labels and classification correctness. You review the evaluation criteria before they are used. 3. **Fine-tune a model.** It selects a compatible language model, checks the training setup and expected cost, and follows the accepted run through development validation. 4. **Benchmark the result.** It compares generated responses from the base and fine-tuned models on the same held-out examples using the same checks. The agent prepares chat-training examples for this path. See [conversation and evaluation formats](/reference/datasets#chat-training) if you want to inspect their structure. Development checks help monitor training; the final held-out comparison measures the result separately. ## Check the result At the end, the agent should show: * The completed training run, selected base model and exact data versions used. * Base-model and fine-tuned responses alongside the held-out reference labels. * Classification results, including missed fraud and legitimate transactions flagged as fraud. * Invalid responses, missing results and the number of examples scored for each model. * Whether the fine-tuned model improved on the base model, with representative mistakes and limitations. A response can use the correct label format and still be wrong. The source also contains many more non-fraud transactions than fraud transactions, so overall accuracy alone is insufficient. The [dataset reference](/reference/example-dataset) records its label counts and limitations. These are synthetic examples. A better result here does not establish performance on real transactions, and fine-tuning may not improve on the base model. ## Continue in the same conversation Show me examples where the fine-tuned language model and base model disagree. Explain the remaining classification and response-format errors, and recommend what to try next. Use [Monitor training](/how-to/monitor-training) to inspect the run, [Run an evaluation](/how-to/evaluate) for another comparison, or [Serve a model](/how-to/serve) to call a ready deployment from your application. # How to train a decision model Source: https://docs.overmindlab.ai/tutorials/first-dataset Prepare data, train a decision model and benchmark its results with one prompt. Train a decision model to return probabilities for a defined set of outcomes, then compare it with a baseline on held-out data. Your coding agent handles preparation, training and benchmarking from one request. This walkthrough uses the [ecommerce fraud dataset](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl): 10,000 synthetic transactions with a supplied fraud label. Fraud detection is the worked example; the workflow also applies to other classification and routing tasks when you supply suitable data and target definitions. For generated text responses or tool calls, follow [How to fine-tune a language model](/tutorials/fine-tune-language-model). ## Before you begin [Connect your coding agent](/how-to/connect) and select a project for the tutorial. You do not need an application repository. This workflow includes training and benchmark inference, which can incur provider costs. Your agent explains the proposed model and expected cost before launch. Use Overmind to prepare this ecommerce fraud dataset, train a decision model, and benchmark its results on held-out data: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Show me how it compares with a baseline and where it makes mistakes. The agent works out the preparation and carries the dataset, model and benchmark context through the workflow. You do not need to issue a new prompt for each stage. ## What happens next 1. **Prepare the data.** The agent inspects the source, prepares the transaction inputs and fraud labels, and separates training, development and held-out examples. It retains the preparation in Overmind so the result can be inspected and reproduced. 2. **Train a model.** It selects a compatible decision model, checks the training setup and expected cost, then follows the accepted run through validation. 3. **Benchmark the result.** It compares the trained model with a suitable baseline on the same held-out examples and explains the measured results. Data checks are part of the workflow. Preparation is not a separate stopping point; the agent continues toward the requested benchmark and brings back decisions that need your input. Follow progress in the same conversation or inspect the saved datasets and runs in the Console. ## Check the result At the end, the agent should show: * The trained model and its completed training run, including the recorded validation results. * The benchmark participants, held-out data and number of examples scored by each model. * How often the models detect fraud, miss fraud or flag legitimate transactions, with representative mistakes. * The comparison with the baseline, including any missing results or limitations. The original source has **9,580 non-fraud** and **420 fraud** transactions, so overall accuracy alone can hide poor fraud detection. The supplied `is_fraud` label is the answer to predict, not part of the model input. The agent handles these checks during preparation; [dataset details](/reference/example-dataset) explain the source and its limitations. These are synthetic examples. Improvement on this benchmark does not establish performance on real transactions, and a trained model may not outperform its baseline. ## Continue in the same conversation Explain where the trained fraud model improved over the baseline and where it still makes mistakes. Show representative held-out examples and recommend what to try next. Use [Compare decision models](/how-to/compare-decisions) for another comparison, [Monitor training](/how-to/monitor-training) to inspect a run, or [Serve a model](/how-to/serve) to try the trained result in an application. # Run your first evaluation Source: https://docs.overmindlab.ai/tutorials/first-evaluation Ask your agent to run a small check and explain the result. This lesson checks whether recorded answers use the expected fraud labels. It introduces evaluation with a deliberate failure, so you can see what a failed check looks like. ## Before you begin Use [Prepare data](/how-to/prepare-data) to prepare the [fraud example dataset](/reference/example-dataset) in a separate tutorial project. A trained model is not required. This exercise checks answer format; it does not benchmark a trained fraud model. The platform may use its configured model to explain a low score. Use examples from this fraud dataset to teach me how evaluations work in Overmind: [https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl](https://huggingface.co/datasets/apex0data/ecommerce-fraud-detection-synthetic-10k-sampl). Create a small label-format check with one deliberately invalid answer, run it, and explain the result. Your agent prepares a separate exercise dataset, defines the check and follows the evaluation through completion. ## Check the result Valid recorded answers use the labels `fraud` and `not_fraud`. An intentionally invalid answer, such as `unknown`, should fail that vocabulary check. The agent should show the evaluated answers, their scores and any errors or unscored rows. This verifies the answer format. A valid label can still be the wrong answer for a transaction. Measuring accuracy requires model predictions compared with separate reference labels on held-out examples. Keep the deliberately modified exercise separate from training and final assessment. Use a tutorial project because enabled project-wide checks can also participate in live scoring. ## Continue in the same conversation Show me the failed example, the answer it contained and the rule it broke. Explain what this result tells us about answer format and what it cannot tell us about fraud accuracy. Once you have a trained decision model, [compare it with Jev](/how-to/compare-decisions) on held-out transactions. # Record your first trace with a prompt Source: https://docs.overmindlab.ai/tutorials/first-trace Let your coding agent write, run and inspect a tiny Python application. You will record a local order lookup and inspect its span tree through MCP. The example application makes no model calls and needs no external dataset. Enabled platform scoring can still invoke models after trace ingestion. ## Before you begin [Connect Overmind](/how-to/connect), select your tutorial project and open an empty working directory with Python available. Configure its API key through your environment's secret mechanism. MCP login alone does not configure the Python SDK's telemetry credentials. Create and run a simple local Python example that looks up an order and records a trace in Overmind. Show me the captured inputs, output and timing so I can understand what tracing records. This prompt requests one local run and its telemetry upload. If credentials or host permissions are missing, complete that setup before the agent runs it. ## Inspect the code it creates The central tracing code should have this shape. The agent supplies a unique conversation ID and the verified deployment configuration: ```python theme={"dark"} import overmind if not overmind.init(service_name="order-tutorial"): raise RuntimeError("Configure an Overmind API key before running this example") @overmind.tool() def lookup_order(order_id: str) -> dict: return {"id": order_id, "status": "shipped"} with overmind.run("order-status", conversation_id="UNIQUE_TUTORIAL_SESSION") as run: result = lookup_order("A-1001") run.deliver(result) print(result) ``` The agent installs `overmind[tracing]` in an isolated environment, runs the script once and queries the session using `query_traces`. The run flushes buffered telemetry when its context exits. ## Check your result Expect the local output to contain order `A-1001` with status `shipped`. The matching trace should have an `order-status` root and child spans for `lookup_order` and delivery. Inspect the actual captured input and output, not just the span names. There should be one trace for the unique session. If none arrives, the agent checks credentials, destination and flush behaviour. A retry should be reported as another execution. No capability is declared in this lesson. The trace can remain unbound while still being inspectable. ## Try a follow-up Walk me through this trace from the initial request to the result. Explain which steps ran, how long they took and whether any failed. For your own application, continue with [Map an agent repository](/how-to/map-agent) and [Instrument an application](/how-to/instrument). # Tutorials Source: https://docs.overmindlab.ai/tutorials/index Learn dataset-to-model workflows, tracing and evaluation through your coding agent. Each tutorial starts with one prompt and ends with something you can inspect. Your agent handles the MCP calls, local files and CLI commands. Start with the [quick-start instructions](https://console.overmindlab.ai/setup.md) and select a separate tutorial project. The [connection guide](/how-to/connect) explains how to verify access. Read [https://console.overmindlab.ai/setup.md](https://console.overmindlab.ai/setup.md) and follow the instructions to set up Overmind in my coding agent. Select your project when the agent shows the available choices. It carries that context through the conversation. | Lesson | You ask for | You will inspect | | - | - | - | | [Train a decision model](/tutorials/first-dataset) | Prepare data, train a model and benchmark it | A trained model, baseline comparison and example mistakes | | [Fine-tune a language model](/tutorials/fine-tune-language-model) | Prepare examples, fine-tune and benchmark text responses | Base-model and fine-tuned outputs, scores and example mistakes | | [Record a trace](/tutorials/first-trace) | Write and run one local tracing example | One trace with a tool call and its result | | [Run an evaluation](/tutorials/first-evaluation) | Check recorded fraud labels | Valid answers and a deliberate vocabulary failure | Both model-training tutorials and the evaluation tutorial use the same [fraud example dataset](/reference/example-dataset). The trace lesson uses a local function and needs no external data. You can take it independently. Both model-training tutorials include training and benchmark inference. The trace example application makes no model calls, but enabled platform scoring may; the label-format evaluation does not train a model, although the platform may generate explanations for low scores. Your coding agent uses its own provider. Each tutorial states the work it starts.