> ## Documentation Index
> Fetch the complete documentation index at: https://docs.swarmd.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent telemetry

> Send an agent's own log lines and steps to Swarmd over OpenTelemetry, so each trace shows what the agent was doing beside what the platform recorded.

# Agent telemetry

The audit trail records what agents **said to each other**: every relayed
message, every policy decision, every hop. It cannot see inside an agent.
Agent telemetry closes that gap. An agent sends its own log lines and its
steps (model calls, tool calls, delegations) to Swarmd over OpenTelemetry.
Each one is stamped with the turn's correlation id, and the trace view draws
them on the same timeline as the audit events.

| | Audit trail | Agent telemetry |
| - | - | - |
| Written by | The platform (relay, gateway, registry) | The agent, through the SDK |
| Authoritative? | Yes. Hash-chained and signed | No. Diagnostic only, never in the hash chain |
| Retention | Your audit retention policy | 30 days |
| Can be switched off | No | Yes, per agent and per deployment |

<Note>
  **Preview, and opt-in at both ends.** The platform operator turns the
  telemetry service on (see [Agent telemetry for
  self-hosting](/self-hosting/agent-telemetry)), and each agent turns on what it
  sends with environment variables. Upgrading the SDK alone sends nothing.
</Note>

***

## Turn it on for a Google ADK agent

Agents built on `swarmd-google-adk` 0.4.0 or later need no code change.
`serve()` reads these variables at startup:

```bash .env theme={null}
SWARMD_TELEMETRY_ENABLED=true   # your log lines
SWARMD_TELEMETRY_STEPS=true     # your steps: model calls, tool calls, delegations
```

Restart the agent, send it a message, and open the trace from **Observe ›
Trace explorer**. Log lines appear between the audit events, and the steps
appear as a tree under the timeline. The agent's **Logs** tab also gets an
**Application logs** view over a time range.

| Variable | Default | What it does |
| - | - | - |
| `SWARMD_TELEMETRY_ENABLED` | off | Ships every standard-library `logging` record at or above `SWARMD_TELEMETRY_LEVEL`. |
| `SWARMD_TELEMETRY_STEPS` | off | Ships spans: structure, names, timing, status, model name, token counts and tool names. Prompt and response **content is stripped**. |
| `SWARMD_TELEMETRY_REASONING` | off | Also ships the content: system prompts, user messages, completions, tool arguments and tool results. **See the warning below.** |
| `SWARMD_TELEMETRY_LEVEL` | `INFO` | Lowest log level shipped. Must be a level name (`DEBUG`, `INFO`, `WARNING`, …). |
| `SWARMD_TELEMETRY_BASE_URL` | `SWARMD_BASE_URL` | Where to send. Leave unset unless you route through a proxy of your own. |
| `LOG_LEVEL` | `info` | Since 0.4.0 this sets the root logger's level, not just uvicorn's. Below `INFO`, your `logger.info()` lines are dropped before telemetry sees them. |

A switch is on only for `1`, `true`, `yes` or `on`. Anything else is off.

<Warning>
  **Leave `SWARMD_TELEMETRY_REASONING` off outside a test tenant.** Redaction of
  reasoning content is not built yet. With it on, the full text of every prompt,
  completion and tool result is stored in ClickHouse for 30 days, readable by
  anyone with `TELEMETRY:READ`. The data stays in your cluster, but it is the
  most sensitive data your agents handle. Large `call_llm` steps can also go over
  the per-record size limit, and those steps are then dropped.
</Warning>

### Where it goes

The SDK posts OTLP/HTTP JSON to `<SWARMD_BASE_URL>/telemetry/v1/logs` and
`/telemetry/v1/traces`, using the agent's existing client-credentials token.
On a self-hosted install `SWARMD_BASE_URL` is your own gateway, so
**telemetry never leaves your network**. Agents get the `TELEMETRY:WRITE`
permission automatically, including agents registered before the upgrade.

***

## What gets captured from Google ADK

ADK already opens spans for everything it does. Until something sets a real
OpenTelemetry tracer provider, those spans go nowhere. Setting
`SWARMD_TELEMETRY_STEPS` sets one, and each step is labelled with what kind of
step it is:

| ADK span | Step kind | Shown as |
| - | - | - |
| `call_llm` | `reasoning` | A model call |
| `execute_tool` | `tool` | A tool or MCP call |
| `invocation` | `invocation` | The turn itself |
| `managed_agent_interaction` | `delegation` | Handing work to another agent |
| `send_data` | `transfer` | Data sent between agents |
| context-cache spans | `cache` | Cache reads and writes |

Spans from the A2A SDK's own event-queue polling are dropped. They are noise,
and in testing they made up most of a turn's spans.

Every log record and span carries `swarmd.correlation_id`,
`swarmd.context_id` and `swarmd.task_id`. These are taken from the inbound
request and A2A ids, and are what joins them to the audit trail.

***

## Alongside Datadog, Grafana or your own collector

Telemetry is an extra destination. It never replaces yours.

* **Logs.** Swarmd uses its own `LoggerProvider` and never takes over the
  global one, so your exporter receives each record exactly once. The SDK
  never changes logger levels, handlers or `propagate`.
* **Spans.** If your process already set a global `TracerProvider`, Swarmd
  adds its processor to it and does not replace it. Your exporter sees spans
  unchanged, content included. Note that every span in that process is then
  also sent to Swarmd, except the A2A polling scope.
* **Reasoning content.** ADK publishes prompts and completions as `gen_ai.*`
  log records through the *global* logger provider. If you already set one,
  Swarmd does not take it over. It logs a warning and skips reasoning capture.
  When it does capture, it sets `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`
  only if you have not set that variable yourself.
* **Noise.** The loggers `httpx`, `httpcore`, `urllib3`, `opentelemetry` and the
  SDK's own telemetry logger are never shipped. Each export also makes `httpx`
  log one line into *your* handlers, which you can filter.

***

## It cannot take your agent down

* Export runs off the request path in bounded batches (queue 2048, every 5 s,
  10 s timeout).
* On any error a batch is dropped and counted. Nothing is raised into your
  code.
* A `401` gets one retry with a fresh token.
* Pending records are flushed on shutdown.
* If telemetry cannot start at all (no credentials, an incompatible
  OpenTelemetry pin, an unreachable endpoint), the agent logs one line and
  starts without it.

The SDK needs `opentelemetry-sdk` `>=1.30,<2`, which `swarmd-sdk` installs.

***

## Other frameworks, and wiring it by hand

The environment variables are read only by `swarmd-google-adk`'s `serve()`.
For LangChain or your own server, call the SDK directly once at startup:

```python theme={null}
from swarmd_sdk.telemetry import (
    enable_telemetry_from_runtime,
    enable_tracing_from_runtime,
)

logs = enable_telemetry_from_runtime(runtime, "invoice-checker")
steps = enable_tracing_from_runtime(runtime, "invoice-checker")  # optional
```

Both read the endpoint and token from the runtime you already built with
`create_runtime()`, and accept keyword overrides:

| Function | Useful overrides |
| - | - |
| `enable_telemetry_from_runtime` | `level`, `excluded_loggers` (added to the defaults), `logger_provider` (add Swarmd to a provider you configured), `capture_framework_events` |
| `enable_tracing_from_runtime` | `tracer_provider`, `set_global` (default `True`, only if none is set), `step_kinds` (span name → step kind), `excluded_scopes`, `capture_content` |

LangChain has no built-in spans. You get your log lines, and spans only if you
add a LangChain OpenTelemetry instrumentation yourself and pass `step_kinds`
so they are labelled. `swarmd-langchain` does not yet stamp the A2A context
and task ids, so its records join a trace by correlation id only.

***

## Reading it back

| Where | What | Needs |
| - | - | - |
| **Observe › Trace explorer**, open a trace | Log lines between the audit events, steps as a tree under the timeline. Up to 500 of each per turn | `TELEMETRY:READ` |
| Agent › **Logs** › Application logs | The agent's log lines over a time range | `TELEMETRY:READ` |
| `GET /telemetry/v1/logs`, `GET /telemetry/v1/spans` | Filter by `correlationId`, `agentId`, `contextId`, `taskId`, `startTime`, `endTime`. Spans also take `stepKind` and `reasoningOnly`. Up to 1000 per page | User token with `TELEMETRY:READ` |

Only the **Tenant Administrator** group has `TELEMETRY:READ` by default. Add
it to other groups under **Manage › Groups** if your developers need it.

When the trace shows no telemetry, it says why: you lack the permission,
telemetry is not enabled on this deployment, or the agent sent nothing for
that turn. It never shows an empty result as if the agent did nothing.

***

## Next

<CardGroup cols={2}>
  <Card title="Enable it on your cluster" icon="server" href="/self-hosting/agent-telemetry">
    ClickHouse, the two chart switches, sizing.
  </Card>

  <Card title="Environment reference" icon="list" href="/sdks/python/reference">
    Every `SWARMD_*` variable the SDK reads.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.