Perplexity Publishes Explainer on AI Gateway Mechanics and Limits

Perplexity published a guide on October 20, 2026, explaining how AI gateways route, meter, and govern traffic between applications and large language models and when an organization needs one. Titled “What Is an AI Gateway? A Guide to Routing, Cost Control, and Governance for LLM and Agent Traffic,” the post appeared as the newest entry on the company’s blog.

Definition and Request Handling

The guide defines an AI gateway as a central proxy layer placed between an organization’s applications or AI agents on one side and the large language models and tools they call on the other, with routing, cost control, security enforcement, and usage monitoring handled in one place. Because each provider (the guide names OpenAI, Anthropic, Google, open-source models, and self-hosted endpoints) ships its own SDK, API keys, and request format, the gateway accepts one standard request and translates it for the provider being called. Models are added, swapped, or retired through gateway configuration rather than application code.

Per the guide, every request pointed at the gateway’s base URL moves through the same four steps: the gateway authenticates and authorizes the caller by key and role; applies policy such as token limits, budgets, and guardrails including PII redaction; routes to a model or provider by cost, latency, or availability, failing over to a backup and translating the request into the target provider’s format; and logs the model, token counts, latency, and outcome for spend tracking and debugging.

Capabilities and Stated Limitations

The guide describes six capability areas. Routing and failover choose a model by cost, latency, or capability and redirect requests automatically when a provider is down or rate-limited. Token-based rate limits and budgets cap usage per team, project, or user, rejecting requests once an allocation is reached. Semantic caching matches new prompts to stored answers by meaning and returns the saved response without a model call when prompts are close enough. Guardrails inspect each prompt and response, removing personal or sensitive data before it reaches a provider, reducing prompt-injection risk, and filtering unwanted content. Observability records which model ran, token usage, latency, and failures, while governance features grant role-based model access and keep an audit log of every call.

On limitations, the guide states that a gateway adds latency but describes the delay as relatively small for LLM traffic. It cites a June 2026 independent benchmark by Broadcom’s VMware Cloud Foundation team in which a tuned Envoy AI Gateway added about 2 milliseconds per request at up to 224 concurrent users with routing and rate limiting but no content inspection; the same benchmark, as cited, notes guardrails and semantic caching could add tens to hundreds of milliseconds. The guide also states that gateways do not decide what to ask a model, remember conversations, run an agent’s step-by-step logic, or assess answer quality, and it warns that cost-driven routing can reduce quality when a cheaper model returns a worse answer.

Gateway Types and Agent Traffic

The guide contrasts AI gateways with traditional API gateways across several dimensions: token metering versus request-count limits, semantic versus exact-match caching, prompt inspection for sensitive data and hidden instructions versus key authentication, and routing to models by cost, latency, or capability versus routing to backend services. It also separates two narrower terms: an LLM gateway handles the model call itself, while an MCP gateway governs which tools and data sources an agent can reach through the Model Context Protocol. LiteLLM, Portkey, Azure API Management, and Envoy AI Gateway now add MCP governance, the guide notes, so the terms describe overlapping layers of control.

For agents, which the guide describes as working in loops of model calls and tool use until a task is complete, the gateway controls and monitors all agent-initiated traffic, authorizes which tools and data sources the agent can access, and records all model and tool calls in an audit log.

Cited Adoption Data and Deployment Guidance

To size demand, the guide cites third-party figures. Cisco’s AI Readiness Index 2025 found 83% of 8,039 surveyed senior business leaders at organizations with more than 500 employees plan to develop or deploy AI agents, and Gartner expects task-specific agents in 40% of enterprise applications by the end of 2026, up from under 5% in 2025. On cost, the guide cites the DoiT/Sapio 2026 survey, in which 79% of participating finance leaders said they had overrun their AI budget in the past year, and a 2026 Dataiku and Harris Poll survey in which 93% of enterprise CIOs said different models perform better for different use cases, 81% of enterprises expect to run two or more model providers, and 55% have switched LLM providers at least once.

Independent monitoring cited in the guide put leading LLM provider uptime around 99.85–99.9% in May 2026, a level the guide translates to roughly 9–13 hours of downtime a year; in a Cockroach Labs 2026 survey of 1,125 senior infrastructure leaders, two-thirds estimated an hour of AI downtime would cost their organization over $100,000. The guide also cites OWASP’s 2025 ranking of prompt injection as the top LLM security risk and a Zscaler ThreatLabz finding that enterprise data sent to AI applications rose 93% in a year, with ChatGPT alone triggering 410 million data-loss-prevention violations.

The guide lists signals that an organization needs a gateway: provider keys hardcoded across several codebases, model bills finance cannot attribute, rate-limit or downtime errors tied to dependence on a single provider, uninspected prompts carrying sensitive data, competing provider integrations across teams, MCP tool integrations, and no security or compliance overview. It says a gateway is not necessary for a single prototype, a low-volume side project, or calls to one model endpoint with minimal security requirements. Self-hosted options named include LiteLLM, Portkey, and Apache APISIX on private cloud, Kubernetes, or on-premises infrastructure; managed options named include Cloudflare AI Gateway, Azure API Management, and Amazon Bedrock AgentCore Gateway. The guide frames the choice around data control and privacy, operational burden including state stores such as PostgreSQL, Redis, or etcd, deployment speed, and integrations. Its FAQ states that “generative AI gateway” is interchangeable with “AI gateway” and that there is no single best open-source AI gateway.