Architecture
There is no collector, sidecar, or proxy between your agent and the platform. SDKs (and any vanilla OTel exporter) POST OTLP protobuf directly to the API, which writes spans to Postgres and fans out post-processing to background workers:200 response means your spans are stored. Re-sending the same spans is safe: rows are upserted on the span ID, so retries and duplicate exports never create duplicate data. Trace scoring is queued the moment a root span lands and runs on its own worker queue, isolated so ingest bursts can’t starve evaluation.
The OTLP endpoint
The
/v1/traces alias exists so a stock OTel HTTP exporter works with nothing but a base URL — exporters append /v1/traces themselves:
http://localhost:8000.
Getting traces in
The fastest path is the one the empty Observability page hands to your coding agent: pick the client, Copy prompt, paste. The prompt tells the agent to installovermind[tracing], call the get_instrumentation_plan MCP tool for exact placements, apply them, and then verify one real trace with verify_instrumentation — the same loop the /overmind ensure-tracing command runs. See MCP.
In a local repository, run overmind sync first so the Python SDK can reuse its saved project credential. Deployed processes and non-Python SDKs should receive OVERMIND_API_KEY through their runtime secret configuration. To instrument by hand:
- Python SDK
- TypeScript
- Any language (raw OTel)
- Connectors
init(), calls made with the OpenAI, Anthropic, Google Gemini, Agno, and LangChain client libraries are captured automatically — prompts, completions, tool calls, token usage, latency, and errors. run() brackets one agent run and deliver() marks its result, which is what trace scoring judges. See the Python SDK reference for the decorators that trace your own code.How spans map to the data model
A trace is not a separate record: it is the set of span rows sharing atrace_id, and the root span (the one with no parent) is what the trace list shows. Each stored span keeps the full OTLP payload — timing, status, resource attributes, span attributes, events, and links — plus platform fields computed at ingest: a span type, an operation name, and foreign keys to the project, capability, and conversation it belongs to.
Span classification
Each span is typed at ingest, in priority order:- An explicit
overmind.span.typeattribute (entry_point,workflow,tool_call,function,llm_call,retrieval— what the SDK decorators set) always wins;overmind.span_typeand baretypeare accepted spellings. - An OpenInference
openinference.span.kind:llmandembeddingbecome model calls,toola tool call,retrieverandrerankerretrieval,chain,agent,guardrail, andevaluatorworkflow spans. gen_ai.operation.name = "execute_tool"or atool.nameattribute marks the span as a tool call.- A span name containing
tool,function_call,function.call, orexecute_toolmarks it as a tool call. - Otherwise the span is a model call.
Attributes the ingest pipeline reads
The pipeline understands three attribute dialects — the Overmind SDK’s own keys, the OTel GenAI semantic conventions, and OpenLLMetry/Traceloop keys — checked in that priority order:
Traceloop instrumentation is normalised on the way in — scope names under
@traceloop/* are rewritten to @overmind/* — so OpenLLMetry-instrumented apps work without changes.
Capability attribution
Traces attach to a capability through one attribute:overmind.capability.id— the capability’s UUID, copied from its page in the Console. The only key ingest binds by; stable through renames. A span-level id wins over the resource-level one, so one process can serve several capabilities.overmind.capability.name— a display label shown beside raw span attributes. It never resolves a capability.
overmind.init(capability_id=...), or a capability(..., id=...) scope per request in a multi-capability process; switching capability mid-trace is a handoff that trace scoring scores as its own unit. A child span that carries no identity — a subprocess, say — inherits the trace’s capability when the trace maps to exactly one.
An id the project does not know never creates a capability. The span lands under the Unbound filter instead, and binds retroactively once the id resolves (a deleted capability restored, or a backlog rebind after a sync). Sync (or create the capability) first, then copy its id into the SDK.
Sessions
Stamp a sharedconversation.id on the traces of one multi-turn exchange — set_conversation_id("session-42"), or run(conversation_id=...) — and Overmind groups them into a session with rolled-up trace counts, tokens, cost, timespan, and a session score folded from the asks the user made and which ones were delivered. The Sessions view in Observability lists them.
Trace status
A trace islive while spans are still arriving or the root span is missing, completed once the root has landed, and interrupted when no root arrives inside the settle window (TRACE_SETTLE_SECONDS, default 600 s). The trace list, the trace header, and GET /api/traces/ (trace_status) all carry it; a live trace keeps polling in the Console.
Scores on arrival
When a root span lands, the platform carves the trace into units, binds each unit to one of the capability’s tasks as a task execution, and runs the trace-scoring members of the active eval set against each unit. Every verdict is stored as its own row, read overGET /api/verdicts/; the unit’s span keeps the composed markers in feedback_score.trace_scoring, whose _execution.score is the composite the Console shows in the Score column. Sessions get a score of their own. A trace that never sends its root is picked up by a sweep once it is interrupted. The full contract — carve precedence, binding sources, how claims compose, the marker shape — is on Trace scoring.
This makes the executions list a live quality monitor: filter to low scores over the last hour and read exactly which step failed and why.
Exploring traces in the Console
The Observability page is a filterable table with three views:- Task executions (default) — one row per scored unit: capability, task, conversation, terminal, status, duration, tokens, cost, model, time, and the execution score. Group by conversation folds a page’s rows under their
conversation.id. - Root traces — one row per end-to-end run; the only view with bulk selection for datasets.
- Sessions — one row per
conversation.idwith rolled-up counts and the session score.
Task executions with quick filters. The Score column carries the execution score; the Task column names the synced task the unit bound to.
An execution: the intent, the route through the task's anchors, and each evaluator's verdict with its reasoning.
Trace detail: the span tree with token/cost rollups, plus the selected span's input, output, and verdicts.
From traces to datasets
In the Root traces view, select rows (or Select all across pages) and Add to dataset. This is the highest-leverage workflow in Observability: real production behaviour becomes the test set your agent is measured against, one row per trace with its wire transcript, delivered output, score, and a link back to the trace. Datasets covers what happens next.
Bulk selection in the Root traces view: the selected traces about to become a dataset.
Reading traces over the API
Everything the Console shows is available over REST with the same API key:query_traces, query_task_executions, query_failures, and the overmind://traces/{trace_id} resource. See the REST API page for the full endpoint list.
Practical guidance
Initialise tracing once, at startup, before the first model call. Give each process its ownservice_name so its telemetry stays separate, and set the capability identity (capability_id) from the start. Auto-instrumentation alone produces flat model-call spans and no scoring unit: bracket every agent run with overmind.run(), call deliver() on the result, and decorate the functions capability discovery anchors on so each unit binds to its task. Spans that start their own trace outside a run boundary are dropped as orphan fragments. For short-lived processes, flush before exit — run() flushes on exit; otherwise call force_flush_traces().