Agentic AI Will Hit an Infrastructure Wall Before It Hits an Intelligence Wall

The Models Are Improving Faster Than Our Ability to Serve What They’re Being Asked to Do.

Most of the argument about where AI agents are headed is an argument about capability. Can the model plan reliably? Can it recover from its own mistakes? Can it be trusted with an action that has consequences?

Those are real questions. They’re also not the ones that will decide whether agents get deployed at scale over the next few years, because there’s a nearer constraint and it’s arriving faster. Agents are economically awkward to serve, and the awkwardness grows with exactly the capabilities everyone wants to add.

A Chatbot and an Agent Are Different Workloads

A conversational request is one round trip. Prompt in, tokens out, connection closed. You can batch it, cache the prefix, and reason about throughput with a single number.

An agent request is a session. It calls a tool, waits, reads the result, decides what to do next, calls another tool, and eventually answers. A modest task might be six or eight model invocations with waiting in between.

That difference matters for a reason that has nothing to do with intelligence. Between steps, the agent is holding state — the conversation, the tool outputs, the accumulated reasoning. On the serving side that state lives in GPU memory as KV cache, and it occupies memory for the entire duration of the session, including the seconds the agent spends waiting on an API that isn’t you.

So the unit of capacity stops being “requests per second” and becomes “concurrent sessions resident in memory.” A system that comfortably served a thousand conversational requests per second may serve a few dozen concurrent agent sessions on the same hardware.

The Waiting Is the Expensive Part

Here’s the uncomfortable property. During a tool call, the agent is doing nothing. No tokens are being generated. No compute is being consumed. The GPU memory, however, is still occupied, because evicting and recomputing that context would cost more than holding it.

Your most expensive hardware is reserved for a request that is, at that moment, waiting on a database query.

In conventional distributed systems we solved this a long time ago — you release the resource and pick the work back up when the response arrives. Doing the same for model serving means evicting KV cache to host memory or disk and restoring it, and the transfer cost is high enough that most systems don’t bother. Research on exactly this is active, and it’s active because the problem is expensive.

Sequential Execution Makes It Worse Than It Needs to Be

Nearly every agent framework executes tool calls one after another. The model asks for a weather lookup, gets it, asks for a calendar check, gets it, asks for a flight search, gets it.

Frequently those three calls have no dependency on each other. They could have run at the same time. Executed sequentially, the latency is the sum. Executed concurrently, it’s the longest one.

This isn’t a novel observation about agents — it’s the same structural point that applies to any system running multiple models or services inside one latency budget. Total latency should be the longest path through the dependency graph, not the sum of every stage. The reason it matters more for agents is that the session is resident in GPU memory for the whole duration, so cutting wall-clock time doesn’t just improve the user’s experience, it directly increases how many sessions fit on the hardware.

Halve session duration and you roughly double concurrent capacity, on the same GPUs, with no model change.

Retries Compound in a Way Nobody Designed

Agent frameworks inherit retry defaults from the HTTP libraries underneath them. Usually three attempts.

The model also retries, in its own way. When a tool returns something unhelpful, a reasoning model will often try a different approach — which is a retry, just one that nobody configured.

Stack those and a three-attempt policy wrapped around a model that independently reasons its way into three more approaches is a nine-attempt policy that no one wrote down. Each of those attempts is a full model invocation, with its own tokens and its own GPU seconds.

When a dependency slows down, this amplification is what turns degradation into an outage, and it does so at a cost per attempt that’s orders of magnitude above a retried REST call.

What the Wall Actually Looks Like

Not a capability ceiling. Something more mundane: a unit-economics problem that stops projects at the pilot-to-production boundary.

The pilot works. It’s ten users, latency is acceptable, and the bill is a rounding error. Then someone models it at ten thousand users and the GPU requirement is implausible — not because the model is slow, but because concurrent sessions each hold memory for their full lifetime, and lifetime includes waiting.

This is the stage at which a lot of promising internal demos quietly stop.

What Moves the Wall

Nothing exotic, and all of it is infrastructure rather than modelling:

Execute independent tool calls concurrently. The largest single win, available today, requiring no new hardware.

Treat session duration as the capacity metric. Not tokens per second. Not GPU utilization. How long state stays resident, and how many sessions fit.

Budget retries across the system rather than per call. A global cap — retries as a small percentage of total traffic — prevents the compounding described above. Google’s SRE practice on handling overload and Envoy’s retry budgets both implement this idea outside the AI stack.

Separate reasoning from execution. A large model to plan, a much smaller one for the mechanical steps. Most of what an agent does doesn’t need a frontier model.

Make context eviction cheap. Quantized KV cache, paged attention, and offload paths for idle sessions. This is where the most interesting serving work is happening right now.

The models will keep improving. That part looks reliable. Whether agents become infrastructure or stay demos depends on something less glamorous — whether we can serve a workload that holds expensive memory while it waits, and whether we stop paying the sum of a sequence when we could be paying its longest path.

I published the execution model underlying that last point — concurrent execution of dependent model calls within a single latency budget — in “Asynchronous Inference Graph Execution for Model Routing in Machine Learning Systems”, International Journal of Computer Trends and Technology 72(10), October 2024. On the integration of generative models into latency-bound decision systems specifically, see “LLM-Augmented Trading and Decision Platforms: Bridging Generative Intelligence with Financial Decision Systems”, Journal of Information Systems Engineering and Management 11(1s), 2026.