Microsoft has detailed a three-layer LLM routing architecture for AI agents running on Azure Kubernetes Service (AKS), according to a report from InfoQ. The Microsoft LLM routing architecture speaks to one of the most pressing operational questions in agentic AI today: which model should answer which request, and what part of the stack should make that decision. While the InfoQ headline offers few technical specifics, the disclosure matters because routing is rapidly becoming the control point for cost, latency and quality in production agent systems — and a major cloud vendor formalising it as an architectural pattern is a signal worth reading closely.
Key takeaways
- According to InfoQ, Microsoft has detailed a three-layer LLM routing architecture for AI agents running on Azure Kubernetes Service (AKS).
- LLM routing decides which model handles each request, and has become a key lever for controlling cost, latency and output quality in agent systems.
- Agentic workloads multiply model calls per task, which turns intelligent routing from an optimisation into an economic necessity.
- Building the router on AKS suggests routing is being treated as platform infrastructure rather than application code.
- Beyond the headline, technical specifics remain limited; the wider significance is a large cloud provider elevating model routing to a first-class design pattern.
What InfoQ reports about Microsoft’s three-layer routing design
InfoQ, a publication focused on practitioner-level software architecture, has covered what it describes as a Microsoft three-layer LLM routing architecture for AI agents on AKS. The framing is notable in itself: this is not a consumer product launch but an engineering-oriented account of how agent traffic can be routed across large language models on managed Kubernetes infrastructure. Beyond the headline, the available material does not enumerate the contents of each layer, name specific models involved, or quote performance figures, so any detailed claims about the internals should be treated cautiously until the full write-up is examined. What can be said with confidence is that Microsoft is presenting routing — the logic that sits between an agent and a pool of models — as a structured, layered system in its own right, and that AKS is the substrate it has chosen to describe it on. For infrastructure teams, that combination of routing plus Kubernetes is the story.
Why LLM routing has become critical for AI agents
To understand why this matters, consider how agents actually consume models. A single user request to an agent rarely results in a single model call. The agent may plan, decompose the task, select tools, draft, verify, summarise and retry — each step generating its own inference request. Some of those steps genuinely need a frontier-class model; many do not. Sending every call to the largest, most expensive model inflates bills and latency for no quality gain, while sending everything to a small model degrades results on the hard steps.
LLM routing is the discipline of matching each call to the cheapest model that is adequate for the job. Done well, it can cut spend substantially while holding quality steady, because the distribution of difficulty across agent sub-tasks is heavily skewed towards the easy end. As the number of viable models grows — a landscape you can survey in our AI models database — the routing decision becomes both more valuable and harder to make by hand. That is precisely the gap a formal routing architecture is designed to fill.
Inside a layered routing stack: what three layers typically mean
The InfoQ headline does not spell out what Microsoft’s three layers contain, so the following is general industry context rather than reported fact. In practice, layered routing designs tend to separate three concerns. The first is a gateway or policy layer: authentication, quotas, tenant isolation, safety filtering and observability — the guardrails every request passes through regardless of destination. The second is the decision layer: logic that classifies an incoming request by difficulty, domain or latency sensitivity and selects a model, deployment or region accordingly, sometimes using a small classifier model to make the call. The third is the serving or backend layer: the pools of model endpoints themselves, with load balancing, failover and health checking across them.
Separating these concerns matters because they change at different speeds. Policies are stable; routing heuristics evolve weekly as models and prices shift; backend pools change whenever a new model version ships. A layered architecture lets teams update one tier without destabilising the others — the same reasoning that led web infrastructure to split edge, application and data tiers a generation ago. If Microsoft’s design follows this broad pattern, it would bring a familiar operational discipline to a part of the AI stack that many teams still handle with ad-hoc if-statements.
Why Azure Kubernetes Service as the platform
Placing the router on AKS, as the InfoQ report indicates, is a meaningful choice. Kubernetes gives routing infrastructure properties that application-embedded logic lacks: horizontal autoscaling under bursty agent traffic, network-level isolation between tenants, declarative rollout of new routing rules, and the ability to run GPU-backed model servers in the same cluster as the router itself. It also makes the architecture portable in principle, since Kubernetes primitives translate across environments.
There is a strategic dimension too. A Kubernetes-native router can sit in front of a mixed estate: hosted API models on one side and self-hosted open-weight models running in the cluster on the other, with the routing layer arbitrating between them per request. For organisations weighing that trade-off, our self-hosting vs API calculator quantifies where the crossover point sits for a given workload. Treating routing as cluster infrastructure rather than application code is what makes such hybrid estates manageable at all.
The economics: routing as the new cost-control layer
The commercial logic behind routing is straightforward. Agent workloads are token-hungry, and token prices vary enormously between model tiers. A router that sends even half of an agent’s calls to a model an order of magnitude cheaper — with no perceptible quality loss on those calls — transforms the unit economics of the product built on top. That is why routing has moved from research curiosity to production necessity, and why cloud vendors describing reference architectures for it is significant for every team running agents at scale. You can model the effect of shifting traffic between model tiers with our AI API cost calculator, and compare value across models in our AI price-performance index.
As general context, the main routing approaches compare as follows:
| Approach | How it works | Strengths | Trade-offs |
|---|---|---|---|
| Single model for all calls | Every request goes to one chosen model | Simple; predictable behaviour | Overpays on easy calls or underperforms on hard ones |
| Static rules-based routing | Hand-written rules map request types to models | Transparent; easy to audit | Brittle; needs constant manual updating as models change |
| Layered dynamic routing | Dedicated tiers handle policy, model selection and serving | Adapts per request; scales operationally; supports hybrid estates | More infrastructure to build, monitor and tune |
What it means for teams building AI agents
For developers, the practical takeaway is that routing deserves the same architectural attention as the agent logic itself. Teams shipping agents — from customer-support bots to the developer-facing systems in our guide to AI coding agents — increasingly find that model selection per call, not prompt engineering, is the biggest remaining lever on cost and latency. A vendor-published reference architecture, even one whose details are still emerging, gives platform teams something concrete to benchmark their own designs against.
It also hints at where the ecosystem is heading: routing as a managed capability of the cloud platform rather than something every team rebuilds. If routing logic migrates into cluster-level infrastructure on services like AKS, the differentiation between agent products shifts further up the stack — towards task design, tool integration and evaluation — while the plumbing standardises underneath.
Frequently asked questions
What exactly has Microsoft published? According to InfoQ, Microsoft has detailed a three-layer LLM routing architecture for AI agents running on Azure Kubernetes Service. The specific contents of each layer and any performance data were not spelled out in the material available at the time of writing.
What is LLM routing? It is the practice of directing each inference request to the most appropriate model from a pool of options, balancing cost, latency and expected quality — rather than sending all traffic to a single model.
Why run an LLM router on Kubernetes? Kubernetes provides autoscaling, isolation, declarative configuration and support for GPU workloads, which lets routing operate as shared platform infrastructure serving many agents and teams, instead of logic duplicated inside each application.
Does routing actually reduce AI costs? In general, yes: because most agent sub-tasks are easy, diverting them to cheaper models can cut spend materially while reserving premium models for the calls that need them. The exact saving depends entirely on the workload mix.
What remains unknown about Microsoft’s design? The models involved, the decision logic in the routing layer, benchmark results and general availability details are not confirmed by the headline coverage, and should be verified against the full InfoQ report and Microsoft documentation.
The bottom line
The Microsoft three-layer LLM routing architecture reported by InfoQ is a narrow technical disclosure with broad implications. It confirms that model routing — deciding, per request, which LLM does the work — has matured from an internal optimisation into a named, layered architectural pattern that a major cloud vendor is willing to document on its managed Kubernetes service. For anyone running AI agents in production, the message is clear: the router is becoming the economic and operational heart of the agent stack, and it is worth designing deliberately rather than inheriting by accident. The specifics of Microsoft’s implementation deserve scrutiny as fuller details emerge.
Sources: news.google.com. Reported July 29, 2026.
