# Traces

The Traces view shows OpenTelemetry distributed traces for agent workflows. Each trace captures the full execution of an agent run — LLM calls, tool invocations, sub-agent calls, and any latency or error events — as a waterfall visualization.

Navigate to **Highflame Studio** → **Observatory** → **Traces**.

***

## How traces are collected

Traces are emitted by two sources:

1. **Shield SDK** — the Python and TypeScript SDKs automatically instrument LLM calls, tool calls, and guardrail evaluations as OpenTelemetry spans when the SDK is initialized
2. **Agent Gateway** — the gateway adds gateway-side spans (request received, policy evaluated, upstream call made) and propagates W3C trace context headers so that SDK spans are stitched into the same trace

If you are using only the gateway (no SDK), traces will contain gateway spans only. For full end-to-end traces including LLM calls and tool executions, instrument your agent with the Shield SDK.

***

## Trace list

The trace list shows recent traces in reverse chronological order:

| Column       | Description                                                |
| ------------ | ---------------------------------------------------------- |
| **Trace ID** | Unique identifier                                          |
| **Started**  | When the root span began                                   |
| **Duration** | Total trace duration (root span end − start)               |
| **Spans**    | Number of spans in the trace                               |
| **Agent**    | Agent identity associated with the trace                   |
| **Status**   | OK, Error, or Blocked (if a gateway or SDK block occurred) |
| **Session**  | Link to the parent session, if available                   |

Filter by time range, agent, status, or minimum duration.

***

## Waterfall visualization

Click any trace to open the waterfall visualization. The waterfall shows all spans in the trace as horizontal bars, positioned by start time and sized by duration.

### Span types

Spans are color-coded by service:

| Color  | Service                           |
| ------ | --------------------------------- |
| Blue   | Agent Gateway                     |
| Green  | Shield SDK — LLM call             |
| Teal   | Shield SDK — tool call            |
| Orange | Shield SDK — guardrail evaluation |
| Red    | Error or blocked span             |
| Purple | Sub-agent call                    |

### Span hierarchy

Spans are nested to reflect their parent-child relationships. A typical agent trace looks like:

```
[Gateway] handle_request                         ████████████████████████████
  [SDK] agent_run                                  ██████████████████████
    [SDK] guardrail: user_prompt                     ██
    [SDK] llm_call: claude-sonnet-4-6                    ██████
    [SDK] tool_call: search_documents                          ████
      [SDK] guardrail: tool_call                               ██
      [SDK] guardrail: tool_response                              ██
    [SDK] llm_call: claude-sonnet-4-6                               █████
    [SDK] guardrail: assistant_response                                  ██
  [Gateway] return_response                                                █
```

***

## Span detail panel

Click any span to open the detail panel:

### Attributes

All span attributes (key-value pairs) recorded by the SDK or gateway:

* `agent.id` — ZeroID agent identity
* `session.id` — session identifier
* `llm.model` — model name for LLM spans
* `llm.input_tokens`, `llm.output_tokens` — token counts
* `tool.name` — tool name for tool call spans
* `guardrail.policy` — policy name for guardrail spans
* `guardrail.outcome` — allow, block, or monitor

### Events

Structured log events recorded within the span (e.g., guardrail match details, tool call arguments, error stack traces).

### Links

Links to related traces (e.g., a sub-agent trace spawned from this one) or to the parent session.

***

## Trace retention

Traces are retained for 30 days by default. Contact Highflame support to configure a longer retention period.

***

## Integrating with external observability tools

Observatory traces use the OpenTelemetry standard. You can export spans to your existing observability backend (Datadog, Honeycomb, Jaeger, etc.) by configuring an OTLP exporter in the Shield SDK:

{% tabs %}
{% tab title="Python" %}

```python
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Add alongside the Highflame SDK initialization
tracer_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="https://your-otlp-endpoint"))
)
```

{% endtab %}

{% tab title="TypeScript" %}

```typescript
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";

// Add alongside the Highflame SDK initialization
tracerProvider.addSpanProcessor(
  new BatchSpanProcessor(new OTLPTraceExporter({ url: "https://your-otlp-endpoint" }))
);
```

{% endtab %}
{% endtabs %}

Observatory and your external backend will both receive the same trace data.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.highflame.ai/observatory/traces.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.