OpenTelemetry for GenAI: Standardizing AI Observability

Generative AI systems changed how applications behave in production. A traditional request path used to be mostly deterministic: request in, fixed business logic, response out. With LLM-powered features, one user action can fan out into retrieval calls, prompt assembly, model inference, tool invocation, safety checks, and post-processing. Latency is variable, token usage is nonlinear, and quality can drift by prompt, model, tenant, or region.

This is exactly why teams are moving toward OpenTelemetry (OTel) for GenAI observability. Instead of treating AI as a black box, OTel gives you a common language to instrument traces, metrics, and logs across your stack. You can correlate a user complaint with the exact prompt template, model version, tool chain, and token cost that produced the response.

If your GenAI feature has ever failed with “it works on my machine” but no one can explain why production quality dropped, standardized telemetry is the missing control plane.

Why GenAI observability is different from classic APM

Traditional APM is excellent at tracking service dependencies and request latency. GenAI adds new dimensions that classic dashboards do not model well by default:

• Prompt variability: Tiny prompt changes can change output quality and latency significantly.
• Token economics: Cost and performance are token-driven, not just request-driven.
• Model churn: Providers release new model versions frequently, and behavior can shift.
• Tool orchestration: Agentic workflows call many tools and external systems in one turn.
• Quality uncertainty: A response can be fast and still wrong, unsafe, or off-brand.

You need observability that captures technical performance and AI-specific semantics in the same trace.

What OpenTelemetry brings to GenAI teams

OpenTelemetry is a vendor-neutral observability framework that standardizes how telemetry is generated, transported, and analyzed. For GenAI, this means:

1. One instrumentation approach across frameworks, model providers, and runtimes.
2. Shared semantic conventions so telemetry fields have consistent meaning.
3. Interoperability with many backends (open source or commercial) without lock-in.
4. Correlation by design between application spans and AI spans.
5. Portable governance for redaction, sampling, and retention policies.

In practice, OTel lets platform teams define “how we observe AI workloads” once, then apply that standard to chat apps, copilots, document pipelines, and autonomous agents.

“You can’t optimize what you can’t compare.” OTel enables comparison across teams by making telemetry portable and consistent.

A practical GenAI telemetry model

When instrumenting GenAI, think in terms of a hierarchy:

• User operation span (for example, POST /chat)
• Orchestration span (prompt assembly, retrieval, planning)
• Model span(s) (LLM request/response boundaries)
• Tool spans (search, SQL, API calls)
• Safety and evaluation spans (content filtering, guardrail checks)

A useful baseline is to capture the following fields in every model interaction.

Different SDKs may emit slightly different key names today, but the goal is clear: normalize AI telemetry fields so dashboards and alerts stay stable even when frameworks change.

End-to-end architecture with OpenTelemetry

A common production architecture looks like this:

1. Application emits OTel traces/metrics/logs.
2. OTel Collector receives telemetry through OTLP.
3. Processors redact sensitive fields and enforce sampling.
4. Exporters route data to tracing, metrics, and log backends.
5. Dashboards join AI spans with infra and business signals.

Client -> API Gateway -> App Service
                      -> Retriever / Vector DB
                      -> LLM Provider
                      -> Tools (SQL, REST, Search)

App + libraries --OTLP--> OpenTelemetry Collector
Collector --export--> Trace backend / Metrics backend / Log backend

The Collector is where standardization becomes operational. It gives you a central place to enforce policy instead of relying on every team to implement policy correctly in code.

Step-by-step: instrumenting a Python GenAI endpoint

Below is a minimal example of instrumenting a Python service with OpenTelemetry and custom AI attributes.

1) Install packages

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
pip install opentelemetry-instrumentation-fastapi

2) Initialize tracing and exporter

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

resource = Resource.create({
    "service.name": "genai-chat-api",
    "service.version": "1.3.0",
    "deployment.environment": "prod"
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="http://otel-collector:4318/v1/traces"
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("genai.observability")

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

3) Emit a model span with AI attributes

@app.post("/chat")
async def chat(request: dict):
    with tracer.start_as_current_span("gen_ai.chat.completion") as span:
        model = "gpt-4.1-mini"
        prompt_version = "support-bot-v17"

        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.temperature", 0.2)
        span.set_attribute("ai.prompt.template.version", prompt_version)

        # Call your model SDK here (pseudo-code)
        result = {
            "text": "Order #1234 is shipped and will arrive tomorrow.",
            "input_tokens": 512,
            "output_tokens": 38,
            "finish_reason": "stop"
        }

        span.set_attribute("gen_ai.usage.input_tokens", result["input_tokens"])
        span.set_attribute("gen_ai.usage.output_tokens", result["output_tokens"])
        span.set_attribute("gen_ai.response.finish_reason", result["finish_reason"])

        return {"answer": result["text"]}

4) Add collector-side redaction and batching

receivers:
  otlp:
    protocols:
      http:
      grpc:

processors:
  batch:
    timeout: 5s
  attributes/redact:
    actions:
      - key: ai.prompt.raw
        action: delete
      - key: ai.user.email
        action: delete

exporters:
  otlp:
    endpoint: your-observability-backend:4317
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/redact, batch]
      exporters: [otlp]

This pattern keeps your app code lean while moving data governance into a centralized collector policy.

Best practices for standardizing GenAI observability

• Define a telemetry contract: Publish required span attributes for all GenAI services.
• Version prompts and tools: Always emit prompt/template version and tool version.
• Track both latency and tokens: P95 latency without token context is misleading.
• Separate raw and derived signals: Keep response quality scores in dedicated attributes.
• Use consistent operation names: For example, gen_ai.chat.completion across teams.
• Instrument retrieval and tools: Many failures happen outside the model call.
• Adopt environment-aware sampling: Full fidelity in staging, cost-aware sampling in prod.
• Correlate with user/business context: Tenant, feature flag, and request path are essential.

Tip: Start with a strict minimum schema and expand deliberately. Uncontrolled attribute growth causes noisy, expensive telemetry.

Common mistakes teams make

• Logging full prompts and outputs by default: This creates privacy and compliance risk.
• Relying on provider dashboards only: You lose cross-service context and lock into one view.
• Ignoring tool spans: Agent failures are often due to tool errors, not model quality.
• No prompt version telemetry: Regression analysis becomes guesswork.
• Mixing environments in one dashboard: Production and staging noise hides real incidents.
• Alerting on averages only: Tail latency and outliers matter much more for user trust.
• Treating evaluation as separate from observability: You need runtime quality indicators in traces.

A good litmus test: if you cannot answer “which prompt change increased cost last week?” from telemetry, your schema is not mature enough.

Real-world use cases

1) Incident triage for latency spikes

A support copilot starts timing out. OTel traces show the LLM span is stable, but retrieval spans against a vector store doubled in latency. The team fixes index warm-up and restores SLA without rolling back model changes.

2) Cost optimization at prompt level

Finance flags a sudden AI cost increase. Token metrics grouped by ai.prompt.template.version reveal a new template added verbose instructions and increased input tokens by 40%. The prompt is compressed and monthly spend drops.

3) Quality regression after model update

A model version rollout improves speed but increases hallucinations for policy questions. By correlating user feedback with model/version attributes, the team routes sensitive intents to a more reliable model tier.

4) Compliance audits and data minimization

Security asks for proof that PII is not stored in telemetry. Collector processors show deterministic redaction policies and audit logs, reducing manual compliance overhead.

5) Multi-vendor portability

An organization runs different model providers by geography. Because telemetry follows OTel conventions, SRE dashboards remain consistent even when provider routing changes.

Metrics that matter for GenAI operations

Don’t track everything first. Track what drives user value, reliability, and spend.

The key is correlation: a single metric rarely explains behavior alone. Token growth plus retry growth plus model change tells a very different story than any one signal in isolation.

Governance, privacy, and cost controls

GenAI telemetry can become expensive and sensitive quickly. Build controls from day one:

• Redact by default in the collector, then allowlist exceptions.
• Hash user identifiers when you only need cohort analysis.
• Use tiered retention (short for raw traces, longer for aggregate metrics).
• Create budget alarms tied to telemetry volume and token usage.
• Document data lineage from app span to storage backend.
• Align with legal/compliance on what qualifies as sensitive prompt content.

For many teams, observability maturity follows this progression:

1. Basic tracing for request latency.
2. AI semantic attributes for model/token visibility.
3. Policy-driven collector with redaction and sampling.
4. Runtime quality signals and evaluation correlation.
5. Automated optimization loops for cost and reliability.

Conclusion

OpenTelemetry is becoming the backbone of GenAI observability because it solves the standardization problem, not just the dashboard problem. In AI systems, reliability, quality, and cost are tightly connected. You need telemetry that can express all three across application code, model calls, tools, and infrastructure.

The practical path is straightforward: define a minimal GenAI telemetry contract, instrument model and tool spans, enforce governance in the collector, and build dashboards that correlate technical and AI-specific signals. Once this is in place, teams move faster with fewer blind spots, safer operations, and clearer cost control.

If your organization is scaling from one chatbot to a portfolio of AI products, OTel gives you the shared observability language needed to operate GenAI like a real platform rather than a set of isolated experiments.