End-to-End Observability in Azure AI Foundry Apps

What is End-to-End Observability

End-to-end observability refers to the complete visibility into every component, interaction, and transaction across your entire AI application stack. In the context of Azure AI Foundry applications, observability encompasses three core pillars: traces, metrics, and logs.

Traces capture the complete flow of requests as they move through your system, showing latency, errors, and the relationship between different components. Metrics provide quantitative measurements of system behavior—response times, token usage, API call counts, and error rates. Logs record detailed events and state changes that help you understand what happened at specific points in time.

Unlike traditional monitoring, which often focuses on infrastructure health, observability in AI Foundry applications extends to understanding the behavior of large language models, agent decisions, tool executions, and downstream API interactions. This requires a different approach to instrumentation and data collection that captures the unique characteristics of generative AI systems.

Why it Matters

Production AI applications face unique observability challenges that abstract away from traditional software systems. You need to understand not just whether your system is running, but whether your models are generating appropriate responses, whether retrieval augmented generation (RAG) pipelines are finding relevant context, and whether agents are making sound decisions.

Without proper observability, you're operating blind. Consider these critical scenarios:

• Latency debugging: A customer reports slow responses. Where is the bottleneck? Is it model inference, external API calls, or token processing?
• Quality issues: Users complain that AI responses are becoming less relevant over time. You need traces showing which documents were retrieved, which prompts were sent, and which model versions were used.
• Cost management: Large language models consume tokens that directly impact your operational costs. You need visibility into token usage patterns per user, per request, and per feature.
• Compliance and auditing: Depending on your industry, you may need to audit every decision an AI agent made, including its reasoning and the data it accessed.
• Production debugging: When something fails in production, you need detailed telemetry to reproduce and fix the issue without disturbing user experience.

Azure AI Foundry's built-in observability capabilities, powered by OpenTelemetry and Azure Monitor Application Insights, provide the foundation for addressing all these challenges.

Architecture / How it Works

Azure AI Foundry's observability architecture follows the OpenTelemetry standard, a vendor-neutral specification for collecting telemetry data. Here's how the components work together:

Data Collection Layers

Azure Foundry automatically captures server-side traces for Prompt agents, Host agents, and workflows without requiring code changes. This includes capturing execution flow, tool calls, model invocations, and response handling. For deeper insights, the Azure AI Projects SDK (Python) enables client-side instrumentation, allowing you to capture application-specific metrics and custom spans.

Storage and Analysis

All telemetry data flows into Azure Monitor Application Insights, where it's indexed and stored for 90 days by default. Within Application Insights, you can view traces in the Azure portal or query them using Kusto Query Language (KQL) from the connected Log Analytics workspace. The data follows OpenTelemetry semantic conventions for generative AI systems, ensuring standardized field names and structures across different tools and frameworks.

Components Overview

Tracing Approaches

You can instrument observability through three distinct methods:

1. Server-side traces (zero-code): Create a Foundry project and connect an Application Insights resource. Foundry automatically logs traces for common agent and workflow scenarios.

2. Client-side traces (SDK-based): Use the Azure AI Projects SDK in your Python application to add custom instrumentation and control exactly what data is captured.

3. Local development tracing: Use the AI Toolkit extension in VS Code with a local OTLP-compatible collector for real-time debugging during development without needing cloud connectivity.

Step-by-Step Example

Let's walk through implementing end-to-end observability for a customer service AI agent in Azure Foundry.

Step 1: Create an Application Insights Resource

First, create an Application Insights resource in Azure to serve as your telemetry backend:

# Using Azure CLI
az monitor app-insights component create \
  --app customer-service-ai \
  --location eastus \
  --resource-group my-resource-group

# Get the connection string
az monitor app-insights component show \
  --app customer-service-ai \
  --resource-group my-resource-group \
  --query connectionString

Step 2: Connect Application Insights to Your Foundry Project

In the Microsoft Foundry portal (ai.azure.com):

1. Open your Foundry project
2. Navigate to Agents → Traces
3. Select Connect
4. Choose your Application Insights resource (or create a new one)
5. Confirm the connection

Your project is now ready to capture traces automatically.

Step 3: Instrument Client-Side Code (Optional but Recommended)

For production applications, add explicit client-side instrumentation using the Azure SDK:

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from opentelemetry import trace
from azure.core.tracing.opentelemetry import OpenTelemetrySpan

# Initialize the project client with tracing enabled
credential = DefaultAzureCredential()
client = AIProjectClient.from_config(credential)

# Create a tracer for custom spans
tracer = trace.get_tracer(__name__)

# Execute an agent with automatic tracing
with tracer.start_as_current_span("customer_service_request") as span:
    span.set_attribute("customer_id", "cust_12345")
    span.set_attribute("request_type", "billing_question")
    
    response = client.agents.create_run(
        agent_id="my-agent-id",
        thread_id="thread_xyz"
    )
    
    # Custom spans for application logic
    with tracer.start_as_current_span("process_response") as child_span:
        child_span.set_attribute("response_length", len(response.content))
        # Your processing logic here

Step 4: Monitor and Analyze Traces

In the Foundry portal Traces tab, you'll see:

• Complete trace timeline showing all operations
• Latency breakdown by component
• Tool execution details and results
• Prompt and completion content
• Conversation history with token accounting
• Error details if failures occur

Tip: Use the search and filter capabilities to find traces by conversation ID, response ID, trace ID, or timestamp. This is invaluable when investigating issues reported by users.

Best Practices

1. Minimize Sensitive Data in Telemetry

Traces may capture user inputs, model outputs, and tool arguments that contain sensitive information. Implement data minimization practices:

• Redact PII before it enters telemetry
• Don't store credentials or API keys in prompts
• Apply consistent data masking policies
• Monitor for unexpected data patterns in traces

2. Plan Your Instrumentation Strategy

Start with automatic server-side tracing, then add client-side instrumentation only where you need additional visibility. This reduces overhead while maintaining comprehensive coverage.

3. Implement Proper Access Controls

Trace data contains execution details that shouldn't be visible to everyone:

• Assign the Log Analytics Reader role for those who need to query traces
• Use Microsoft Entra groups to manage access at scale
• Apply the same retention and security policies as your production logs
• Audit access to sensitive telemetry regularly

4. Establish Baseline Metrics

Capture baseline metrics for your normal operating conditions:

• Average latency per operation type
• Token consumption patterns
• Error rates and typical error types
• Tool execution success rates

This enables you to detect anomalies quickly and set meaningful alerts.

5. Use Conversations for Holistic Analysis

Don't analyze individual traces in isolation. Review conversation threads which capture:

• Full dialogue history
• Token accounting across the entire conversation
• All tool calls and their results
• Agent reasoning and decision points

Common Mistakes

Mistake 1: Ignoring the 90-Day Retention Window

Application Insights retains traces for 90 days by default. Don't discover this limitation when you need to investigate an incident from 6 months ago. Export important traces or use Log Analytics for longer-term archival.

Mistake 2: Treating Observability as Optional

Teams sometimes implement basic monitoring and skip tracing, expecting to "add it later." This leads to operational blind spots in production. Trace instrumentation should be part of your initial architecture.

Mistake 3: Over-Capturing Data

While it's tempting to trace everything, excessive telemetry increases costs and makes analysis harder. Focus on tracing:

• Request entry and exit points
• External API calls
• Model invocations
• Tool executions and errors
• Custom business logic

Mistake 4: Not Setting Up Alerts

Telemetry data is only valuable if you act on it. Create alerts for:

• Latency spikes above acceptable thresholds
• Error rates exceeding baselines
• Agent failures or tool execution errors
• Unusual token consumption patterns

Mistake 5: Forgetting RBAC Permissions

Many teams struggle with authorization errors when querying traces. Ensure users have appropriate roles assigned before expecting them to access telemetry.

Real-World Use Cases

Healthcare Chatbot: A healthcare organization deployed an AI chatbot to answer patient questions. Using Azure Foundry observability, they discovered the agent was retrieving outdated medical guidelines. By examining traces showing document retrieval, they identified the issue in their knowledge base indexing logic and fixed it within hours, preventing potentially harmful advice.

Financial Advisor Agent: A fintech company wanted to audit every recommendation their AI agent made for regulatory compliance. Traces captured tool calls to historical data APIs, the reasoning shown to the agent, and the final recommendation. When questioned by regulators, they could replay complete conversation flows with full context.

E-Commerce Recommender: An online retailer used observability to optimize costs. Traces revealed that their RAG agent was making multiple database calls to retrieve product information. By analyzing trace patterns, they identified redundant queries and implemented smarter caching, reducing API costs by 32%.

Multi-Agent Workflow: A manufacturing company orchestrated multiple AI agents to handle supply chain decisions. Observability traces showed the flow of decisions through the agent network, revealing a bottleneck where one agent was waiting for resource-intensive calculations. They restructured the workflow based on these insights.

Conclusion

End-to-end observability has transitioned from a "nice-to-have" feature to an essential requirement for production AI applications. Azure AI Foundry provides powerful, built-in observability capabilities through OpenTelemetry and Application Insights integration, enabling you to see exactly what's happening in your AI systems.

By implementing the patterns described in this article—leveraging automatic server-side tracing, adding targeted client-side instrumentation, monitoring conversations holistically, and maintaining strong security practices—you'll build AI applications you can confidently operate and optimize in production.

The path forward is clear: start with Azure Foundry's automatic tracing, implement client-side instrumentation for custom insight, analyze trace patterns to identify optimization opportunities, and use conversations to understand holistic behavior. This comprehensive observability foundation transforms AI from a "black box" into a transparent, auditable, and optimizable system.

Your ability to observe, understand, and optimize your AI applications directly correlates with your ability to deliver reliable, compliant, and cost-effective AI experiences to your users.