$ cat ai-engineering/llm-observability.md

Production Tracing and Observability for LLM Systems

How to trace model calls, tools, retrieval, cost, cache usage, and evaluation in LLM applications.

Jatin Bansal@blog:~/ai-engineering$ open llm-observability

Flat model-call logs cannot show which loop step produced a bad answer, what the retriever returned, or whether a cache miss caused a latency spike. An LLM trace connects the user turn, generations, retrievals, tools, and evaluators so an engineer can reconstruct the path to the final response.

Trace the application path

LLM observability captures structured telemetry for every model call, tool invocation, retrieval, and evaluator pass. One user turn produces a tree of spans linked by trace and parent IDs. Typed attributes hold prompts, completions, tool arguments, retrieved chunks, token counts, cache state, latency, and cost. The OpenTelemetry GenAI conventions and OpenInference provide portable attribute names.

Define the span tree

A working LLM trace for an agent turn has six span types. The relationships between them are what turn flat logs into a debuggable surface.

The session span: One per user conversation. Holds tenant ID, user ID, session start/end timestamps, total cost and tokens rolled up from children. The query unit when support escalates a thread.
The turn span (parent): One per user → assistant exchange. Children are everything that ran inside that exchange. Carries the user message, the final assistant message, the cumulative cost, the cumulative latency, and signal flags (error, budget_breach, compaction_triggered). Parent span of every other span in the turn.
The generation span (LLM call): The OTel gen_ai.client span. One per messages.create call to a provider. Attributes: gen_ai.request.model, gen_ai.system (anthropic/openai/…), gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read_input_tokens, gen_ai.usage.cache_creation_input_tokens, gen_ai.response.finish_reasons, latency, cost. Payload as gen_ai.prompt.{N}.content and gen_ai.completion.{N}.content events (or, in the OpenInference convention, llm.input_messages and llm.output_messages as JSON-encoded attributes).
The tool call span: One per tool_use block dispatched. Attributes: tool.name, tool.arguments (JSON), tool.result (JSON), tool.error (if any), latency, retry count. The child of the generation span that emitted the tool_use, which is in turn the parent of the next generation span that consumes the tool_result. This is the loop body of the agent loop article made visible.
The retrieval span: One per retriever call. Attributes: retrieval.query, retrieval.top_k, retrieval.documents (list of {id, score, snippet} records), latency. Child of the tool-call span when retrieval is exposed as a tool; child of the turn span directly when retrieval runs in a JIT context loop outside the model’s tool surface.
The evaluator/judge span: One per LLM-judge call. Attributes: eval.rubric, eval.score, eval.reasoning, judge model name. Linked to the turn span by the same trace ID but typically run after the turn closes; sampled traces get re-scored offline, and the eval spans append to the existing trace rather than living in a separate store. This is the bridge between observability and the eval pyramid: the judge is offline, but the trace it scored is the production artifact.

The non-obvious requirement is the parent linkage discipline. A turn span with three generation spans as siblings (no causal ordering) is much less debuggable than a turn span with a chain of generation → tool-call → generation spans, where each generation’s parent is the tool-call whose result it consumed. The chain encodes the ReAct loop’s iteration count, and lets you ask “show me turns where the second generation emitted an error”; the kind of query that catches the failure mode where the loop driver is hiding an exception three frames deep.

Choose semantic conventions

Two semantic conventions compete in 2026. Choose deliberately; they are converging but not identical.

OpenTelemetry GenAI semantic conventions are the OTel SIG’s standardization, developed since April 2024. The gen_ai.client span is stable as of late 2025; the gen_ai.agent span and a number of attribute names are still experimental in mid-2026. The convention uses log-event attachments for prompt/completion content (so payloads can be sampled separately from span attributes), which fits OTel’s existing log pipeline shape but makes naive trace UIs awkward; the message content is in events, not attributes. Vendor support: Datadog, Honeycomb, Langfuse, and increasingly Phoenix.

OpenInference is Arize’s parallel convention, originally designed for Arize Phoenix and shipped as the default instrumentation for LangChain, LlamaIndex, and many SDKs. OpenInference uses span attributes (llm.input_messages, llm.output_messages, llm.token_count.*, tool.name, tool.parameters) rather than events, which is friendlier to most trace viewers but bloats individual spans. The convention covers retrieval (retrieval.documents), agents (agent.span.kind), and embeddings (embedding.embeddings) in addition to the basic LLM call.

In practice, the two are close enough that a translation layer between them is straightforward, and OpenInference’s roadmap tracks the OTel work explicitly. The pragmatic rule: if your target backend is Phoenix or Arize AX, instrument against OpenInference; if your target is anything else (Langfuse, Datadog, LangSmith, Honeycomb, a vanilla OTel collector), instrument against OTel GenAI conventions. Auto-instrumentation libraries (the OpenInference SDKs, the OTel auto-instrumentations) handle most of the per-framework wiring; the choice is mostly about which attribute names land in your queries.

Sample traces and scrub PII

Two policies determine whether the trace pipeline is sustainable or a cost runaway.

Sampling. A naive pipeline that stores every prompt and completion runs into payload-cost economics fast. The pattern that survives: head-based sampling for cheap aggregates, tail-based sampling for forensic traces, plus 100% retention of error/anomaly traces. Concretely:

Every span is summarized into a metric row (no payload, just attributes) and shipped to the metrics store. Cheap, sub-millisecond, every turn.
A configurable fraction of spans; 1-10% in production, 100% in staging; have their payloads shipped to the trace store. Random selection by trace ID so all spans in a sampled trace are kept together.
Any trace with an error flag, a budget_breach, a judge.score below a threshold, or a user-submitted thumbs-down is force-sampled at 100% regardless of the random sample. Forensic capture for the cases you actually need to debug.
The metric store is queried for trends; the trace store is queried for investigation. A bug report arrives with a session ID; the trace store has the session’s spans; the metric store doesn’t.

The tail-based variant requires a buffer at the collector that holds spans until the trace closes, then samples based on aggregate criteria; useful when “errors” aren’t known at span start. Production teams typically combine: head-based for the baseline sample, tail-based with the OTel collector’s tail_sampling_processor for the error/anomaly augmentation.

PII. Prompts contain user data; completions contain model-generated data that sometimes reflects the user data verbatim. The minimum-viable policy: a scrubbing layer between instrumentation and export, with named-entity recognition (regex first for emails, phone numbers, credit-card patterns; an NER pass for names and addresses if the workload warrants it), a deterministic hash for fields that need to be queryable across spans without being identifying (e.g. user IDs), and a versioned scrubbing manifest checked into the repo so changes go through review. The PII detection and data privacy article walks the detection cascade and the deletion pipeline for the privacy layer the trace store is also subject to. The GDPR right-to-be-forgotten story applies here too; a trace containing a deleted user’s data is a deletion-target the same way the memory store is.

A subtler PII issue: judged eval spans append after the fact to a production trace, and the judge’s reasoning may quote the user’s content. If your scrubbing runs at instrumentation time but the judge runs at eval time, the judge’s output bypasses the scrubber. Add a separate scrub pass on judge-emitted attributes, or run the judge against the already-scrubbed payload and accept the (usually small) quality hit.

Attribute cost and cache usage

Two attributes that production teams find themselves wishing they had captured from day one: cache state and cost attribution.

Cache state. The prompt-caching article made the case for cache_creation_input_tokens vs cache_read_input_tokens being first-class. The corresponding observability requirement: every generation span tags those two counts as separate attributes, and a derived cache_hit_rate = cache_read / (cache_read + cache_creation + cache_miss_input) metric lands on the per-session dashboard. The agent harness anatomy article flagged that a harness without cache telemetry runs at 5-10× the cost it should be; invisible until the invoice arrives. The trace is the only place to see which turn killed the cache, which is the only way to fix the assembly bug at its source.

Cost. Per-span cost in dollars (computed from token counts and a pricing table) lets the trace store answer “show me the top 10 most expensive turns this week,” “what’s the cost distribution by tool,” “which tenants are heavy users.” Tagging the session span with tenant_id and the generation span with model.id and model.tier enables per-tenant per-model rollups that finance asks for once a quarter. Pricing tables change quarterly across providers; pin the pricing-table version into the trace store at the span attribute level (cost.pricing_version) so historical comparisons stay coherent across rate changes.

Code: instrumented Python harness with Langfuse

Langfuse’s Python SDK is the cleanest entry point for a self-hostable, OTel-native trace pipeline. The harness below wraps an Anthropic agent with the loop body from the agent loop article, emits the span shape above, and ships to a Langfuse backend. Install: pip install langfuse anthropic.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
# pip install langfuse anthropic
import os, time, hashlib
from anthropic import Anthropic
from langfuse import get_client

# Langfuse client picks up LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST env vars.
langfuse = get_client()
anthropic_client = Anthropic()

MODEL = "claude-sonnet-4-6"
PRICING = {"claude-sonnet-4-6": (3.00, 15.00)}  # $/MTok input, output

def scrub(text: str) -> str:
    """Naive PII scrubbing; production uses a real NER pipeline."""
    # Hash emails for queryability without identification.
    import re
    return re.sub(
        r"[\w.+-]+@[\w-]+\.[\w.-]+",
        lambda m: f"<email:{hashlib.sha256(m.group().encode()).hexdigest()[:8]}>",
        text,
    )

def run_turn(session_id: str, tenant_id: str, user_msg: str, tools: list, dispatch):
    # Turn span is the parent of every other span in this iteration.
    with langfuse.start_as_current_observation(
        as_type="span",
        name="agent.turn",
        input={"user_msg": scrub(user_msg)},
        metadata={"session_id": session_id, "tenant_id": tenant_id},
    ) as turn:
        messages = [{"role": "user", "content": user_msg}]
        total_cost = 0.0
        for step in range(10):  # step cap; see agent-loop article
            t0 = time.monotonic()
            # Generation span — OTel-style gen_ai.client attributes.
            with langfuse.start_as_current_observation(
                as_type="generation",
                name="anthropic.messages.create",
                model=MODEL,
                input=[{"role": m["role"], "content": scrub(str(m["content"]))} for m in messages],
                metadata={"step": step},
            ) as gen:
                resp = anthropic_client.messages.create(
                    model=MODEL, max_tokens=1024, tools=tools, messages=messages,
                )
                latency_ms = int((time.monotonic() - t0) * 1000)
                # Token + cache attribution.
                in_tok = resp.usage.input_tokens
                out_tok = resp.usage.output_tokens
                cache_read = getattr(resp.usage, "cache_read_input_tokens", 0) or 0
                cache_create = getattr(resp.usage, "cache_creation_input_tokens", 0) or 0
                cost = (in_tok / 1e6) * PRICING[MODEL][0] + (out_tok / 1e6) * PRICING[MODEL][1]
                total_cost += cost
                gen.update(
                    output=scrub(str(resp.content)),
                    usage_details={
                        "input": in_tok, "output": out_tok,
                        "cache_read_input_tokens": cache_read,
                        "cache_creation_input_tokens": cache_create,
                    },
                    cost_details={"total": cost, "pricing_version": "2026-05"},
                    metadata={"latency_ms": latency_ms, "finish_reason": resp.stop_reason},
                )

            # Append assistant turn.
            messages.append({"role": "assistant", "content": resp.content})
            if resp.stop_reason != "tool_use":
                turn.update(output=scrub(str(resp.content)), metadata={"total_cost_usd": total_cost})
                return resp

            # Dispatch each tool_use block as its own span.
            tool_results = []
            for block in resp.content:
                if block.type != "tool_use":
                    continue
                with langfuse.start_as_current_observation(
                    as_type="span",
                    name=f"tool.{block.name}",
                    input=block.input,
                ) as tool_span:
                    t1 = time.monotonic()
                    try:
                        result = dispatch(block.name, block.input)
                        is_error = False
                    except Exception as e:
                        result = str(e)
                        is_error = True
                    tool_span.update(
                        output=result if not is_error else None,
                        metadata={
                            "latency_ms": int((time.monotonic() - t1) * 1000),
                            "is_error": is_error,
                        },
                        level="ERROR" if is_error else "DEFAULT",
                    )
                tool_results.append({
                    "type": "tool_result", "tool_use_id": block.id,
                    "content": str(result), "is_error": is_error,
                })
            messages.append({"role": "user", "content": tool_results})

        turn.update(
            metadata={"budget_breach": True, "reason": "step_cap"},
            level="WARNING",
        )

# Flush at process exit for short-lived applications.
import atexit; atexit.register(langfuse.flush)

every span is opened with start_as_current_observation, which uses Python’s context-manager protocol to ensure parents are set correctly even on exception paths; a custom span manager that forgets this drops the parent linkage on error, which is exactly when you need the trace. usage_details carries the cache attribution as separate fields so the cache-hit-rate metric is computable per span without re-parsing the payload. cost.pricing_version is pinned at the span level; when the provider changes pricing, historical traces still reconstruct the cost they incurred at the time, not the cost they would incur today.