AI Platform · Case study

One gateway in front of every model

Instead of letting every application connect directly to an LLM, I built a secure AI gateway that gives teams one governed entry point for model access, routing and observability.

Role
Senior Data Engineer → Senior AI Engineer at Tieto (Since 2025)
Author
Sanket Joshi, Azure cloud architect and DevOps engineer
Stack
  • LiteLLM
  • Langfuse
  • LLMs
  • AI gateway
  • Observability
  • Model routing
  • Enterprise AI

The problem

As organizations experiment with more AI models, applications quickly start connecting directly to different providers. That creates fragmentation: different APIs, different credentials, different logging, different model configurations, and very little central control. Changing models later becomes painful, because provider logic gets built into every application.

The architecture

Apps and agents → AI gateway (LiteLLM: one governed entry point) → Open-source models, Frontier models (when they are needed) and Langfuse (traces every call).

What I built

I engineered an enterprise AI gateway using LiteLLM and Langfuse. Applications talk to one gateway instead of integrating with individual models.

The gateway is the central layer for model routing, governed model access, observability and usage visibility, switching between models, and access to open-source LLMs.

Why it matters

The gateway makes model choice an infrastructure decision instead of an application rewrite.

It also creates one place to add enterprise controls, rather than asking every application team to solve governance on its own.

The result

Governed access to open-source models and less reliance on frontier models, through one entry point every team can use.

Related articles