Implementing AI Observability: Tracing and Metrics for GenAI

Implement AI observability for generative apps with tracing, token and latency metrics, error diagnostics, and eval loops to improve reliability, cost control, and quality.

Sanket Joshi7 min read

Contents 8 sections
  1. What is AI Observability
  2. Why it Matters
  3. Architecture / How it Works
  4. Step-by-Step Example
  5. Best Practices
  6. Common Mistakes
  7. Real-World Use Cases
  8. Conclusion

Generative applications fail differently from traditional APIs. A request can return 200 OK and still be wrong, unsafe, too expensive, or too slow. Without observability, teams cannot tell whether the problem came from retrieval, prompt assembly, model behavior, tool calls, or downstream rate limits.

AI observability gives you a way to answer those questions in production, quickly and consistently.

This guide covers a practical stack for tracing, metrics, and evaluation loops so you can ship generative systems with confidence.

What is AI Observability

AI observability is the practice of instrumenting generative request paths so you can inspect:

  • where time was spent,
  • how many tokens were used,
  • what failed and why,
  • whether quality stayed acceptable,
  • how behavior changed after prompt/model releases.

In practice, this means combining three signal types:

Signal Purpose Typical examples
Traces Request-level execution graph spans for retrieval, model call, tool call
Metrics Trend and SLO monitoring p95 latency, token usage, error rate
Logs/events Rich contextual diagnostics guardrail blocks, retries, policy actions

Tip: Observability is not just for outages. It is your fastest feedback loop for quality and cost optimization.

Why it Matters

Most teams start with app logs and discover they are insufficient for GenAI operations. Key production questions require correlated telemetry:

  • Why did quality drop after a prompt change?
  • Why is token spend up 35% this week?
  • Why does p95 look fine but streaming feels slow?
  • Which tool calls cause the most errors?

AI observability answers these through structured traces and metrics tied to deployment versions, model IDs, and request paths.

Common production risk areas

  1. Debugging blind spots across multi-service agent chains
  2. Cost drift from context growth and retries
  3. Latency regression from retrieval/tool bottlenecks
  4. Quality regressions after model/prompt updates
  5. Compliance exposure from over-collecting sensitive prompts

Architecture / How it Works

A robust observability design for generative apps typically follows this shape:

Text
Client -> API Gateway -> Orchestrator -> Retrieval/Tools -> Model Provider
                              |                 |             |
                              +------ Traces / Metrics / Events ------+
                                              |
                                      Collector / Backend
                                              |
                                 Dashboards + Alerts + Eval Reports

Trace model

  • One root span per user request.
  • Child spans for each stage: retrieval, prompt build, model call, tool execution, post-processing.
  • Attributes for provider/model/version, request class, and outcome.
  • Events for meaningful milestones (first token, tool retry, guardrail decision).

Metrics model

Track both system and AI-specific metrics:

Metric family Examples Why it matters
Latency p50/p95/p99, TTFT User experience and SLA control
Token usage input/output/cached tokens Cost and context discipline
Reliability error rate by class, retry count, rate-limit hits Stability and capacity planning
Quality groundedness, relevance, safety flags, user feedback Release gating and trust

Governance model

  • Redact or hash sensitive content by default.
  • Keep full prompt/response capture opt-in and sampled.
  • Propagate correlation IDs end-to-end.
  • Attach deployment metadata on every trace (model_snapshot, prompt_version, feature_flag).

“If your telemetry cannot answer who changed what and when quality shifted, you are not production-ready.”

Step-by-Step Example

The following Python example uses OpenTelemetry-style instrumentation with a generative model call.

Step 1: Install dependencies

Bash
pip install openai opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http

Step 2: Configure tracing and metrics exporters

Python
import os
from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter

resource = Resource.create({
    "service.name": "genai-app",
    "deployment.environment": os.getenv("ENV", "prod")
})

trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=os.getenv("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT")))
)
trace.set_tracer_provider(trace_provider)
tracer = trace.get_tracer("genai-app")

metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint=os.getenv("OTEL_EXPORTER_OTLP_METRICS_ENDPOINT"))
)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
meter = metrics.get_meter("genai-app")

Step 3: Define key metrics

Python
request_latency = meter.create_histogram("genai.request.latency.s", unit="s")
time_to_first_token = meter.create_histogram("genai.request.ttft.s", unit="s")
token_usage = meter.create_histogram("genai.request.tokens", unit="{token}")
error_counter = meter.create_counter("genai.request.errors")
quality_score = meter.create_histogram("genai.response.quality.score", unit="1")

Step 4: Instrument model calls

Python
import time
from openai import OpenAI

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def ask_model(user_input: str, session_id: str):
    start = time.perf_counter()
    attrs = {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "openai",
        "gen_ai.request.model": "gpt-4.1",
        "app.session_id": session_id,
        "app.prompt_version": "v2025-03-13"
    }

    with tracer.start_as_current_span("genai.chat", attributes=attrs) as span:
        try:
            response = client.responses.create(
                model="gpt-4.1",
                input=[
                    {"role": "developer", "content": "Be accurate and cite context."},
                    {"role": "user", "content": user_input}
                ]
            )

            usage = getattr(response, "usage", None)
            input_tokens = getattr(usage, "input_tokens", 0) if usage else 0
            output_tokens = getattr(usage, "output_tokens", 0) if usage else 0

            token_usage.record(input_tokens, {"token.type": "input", "model": "gpt-4.1"})
            token_usage.record(output_tokens, {"token.type": "output", "model": "gpt-4.1"})
            span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", output_tokens)

            elapsed = time.perf_counter() - start
            request_latency.record(elapsed, {"model": "gpt-4.1", "status": "ok"})
            span.add_event("response_received", {"latency_ms": int(elapsed * 1000)})

            score = 0.90
            quality_score.record(score, {"eval": "online_lightweight", "model": "gpt-4.1"})
            span.set_attribute("app.quality.score", score)

            request_id = getattr(response, "_request_id", None)
            if request_id:
                span.set_attribute("provider.request_id", request_id)

            return response.output_text

        except Exception as exc:
            elapsed = time.perf_counter() - start
            request_latency.record(elapsed, {"model": "gpt-4.1", "status": "error"})
            error_counter.add(1, {"model": "gpt-4.1", "error.type": exc.__class__.__name__})
            span.record_exception(exc)
            span.set_attribute("error", True)
            span.set_attribute("error.type", exc.__class__.__name__)
            raise

Step 5: Add a lightweight eval loop

Python
def nightly_eval(dataset: list[dict]) -> dict:
    passed = 0
    for row in dataset:
        pred = ask_model(row["input"], session_id="nightly-eval")
        ok = row["must_include"].lower() in pred.lower()
        passed += int(ok)
    pass_rate = passed / max(1, len(dataset))
    return {"pass_rate": pass_rate, "cases": len(dataset)}

Run this against a stable benchmark set and block releases when quality or cost thresholds fail.

Best Practices

  • Define a stable trace taxonomy before broad rollout.
  • Attach deployment metadata to all traces and metrics.
  • Track tokens, cost, latency, and quality together per request class.
  • Implement dynamic sampling: retain all errors and slow traces.
  • Keep sensitive content capture minimized and policy-controlled.
  • Use evals as CI gates, not post-release reports.

Minimum alert set

  • p95 latency regression beyond threshold
  • sudden token-per-request spike
  • error-class distribution shift
  • quality score drop after model/prompt rollout

Common Mistakes

  • Recording only API status codes and missing semantic failures.
  • Monitoring averages only and ignoring p95/p99 or TTFT.
  • No end-to-end correlation IDs across app and provider telemetry.
  • Capturing raw prompts by default, creating compliance risk.
  • Separating observability from release workflow.

Anti-pattern

Text
"We shipped the new prompt because tests passed locally."

For GenAI, local spot checks are insufficient without trace-backed production metrics and eval comparison.

Real-World Use Cases

  1. RAG support assistants: isolate retrieval vs generation failure modes quickly.
  2. Agentic workflows: identify tool-call bottlenecks and retry storms.
  3. Cost governance: detect token inflation by prompt version and auto-roll back.
  4. Release management: compare model snapshots with eval pass/fail gating.
  5. Compliance evidence: prove policy checks and redaction paths executed.

Conclusion

AI observability is not optional infrastructure. It is core product capability for any team operating generative systems at scale. Tracing explains behavior, metrics expose trends, and evaluation loops protect quality during change.

Start small but structured: define a trace model, collect latency/token/error metrics, and tie releases to measured outcomes. Once that foundation exists, your team can move faster with less risk and clearer accountability.

Further Reading

Keep reading

More on Observability & evals