AI’s New Economy Is Built on Tokens — But We’re Measuring Them All Wrong

There’s a shift happening in AI that most people sense but don’t fully understand yet: we’ve moved from counting requests to counting tokens.

In the web era, we measured systems in requests per second. It was clean, intuitive and mostly accurate. A request came in, a response went out, and you could scale infrastructure around that model.

That abstraction is gone.

In AI, the fundamental unit is no longer the request—it’s the token. Every prompt, every response and every chain of reasoning is broken down into tokens, representing the work the system is doing, the cost incurred and, increasingly, the value created.

Naturally, the industry has rallied around this shift, introducing metrics like tokens per second, cost per token and even revenue per token. It feels like we’ve finally found a way to quantify AI systems. But that framing is incomplete.

The industry shortcut: tokens equal value

There’s a growing narrative that tokens are the new currency of AI—that more tokens equals more intelligence, and by extension, more revenue. It’s a compelling idea, but it oversimplifies what’s actually happening inside these systems.

Not all tokens are equal. Some tokens represent real work: analyzing data, generating insights, automating workflows and supporting decisions that drive business outcomes. Others are far less meaningful—casual content generation, experimentation or use cases that never make it into production.

You can already see this inside enterprises. One team might generate millions of tokens to support developer productivity or customer operations, directly impacting efficiency and revenue. Another might generate the same volume experimenting with tools that never move beyond internal exploration. On paper, the token counts look identical. In reality, the business value is completely different.

Treating all tokens as interchangeable units of value creates a distorted view of what AI is actually doing inside an organization. Businesses aren’t built on token volume; they’re built on what those tokens enable.

To understand that difference, you have to look beneath the surface at how these systems actually operate.

Why your second prompt is faster than your first

If you’ve ever used a tool like ChatGPT, you’ve likely noticed that your second question is often faster than your first. That behavior isn’t about the model getting smarter—it’s about how the system reuses context from previous prompts.

Modern AI systems don’t process each request in isolation. They build context, storing previous prompts and responses in memory, often in what’s called a KV cache. This cache sits close to the GPU so it can be accessed quickly when generating follow-up responses.

The first request is expensive because it initializes that state—allocating memory, processing the input and building the context. Subsequent requests reuse that state, which reduces latency and improves responsiveness.

This dynamic becomes more important as context windows expand from thousands to hundreds of thousands—or even millions—of tokens. The more context a system retains, the more pressure it places on memory and infrastructure, making it critical to decide what gets stored, compressed or discarded.

From a user perspective, it feels like a faster system. From an infrastructure perspective, it’s a complex tradeoff between memory, latency and cost.

This is where the real work is happening: not just in the model itself, but in the system that surrounds it.

The ultimate constraint: energy

Once you move beyond model performance, the conversation shifts quickly.

Teams running AI at scale aren’t primarily asking which model is best. They’re asking how to sustain it.

AI infrastructure is pushing against physical limits: energy availability, cooling capacity and memory bandwidth. Data centers are being redesigned around these constraints, and organizations deploying large-scale AI systems are beginning to operate more like utilities than traditional software companies.

We’re already seeing this in large enterprises building AI infrastructure, where power and cooling—not model capability—are becoming the primary constraint.

AI workloads don’t scale cleanly or predictably. They compound demand across compute, memory and networking at the same time. Generating more tokens isn’t simply a matter of adding more GPUs; it’s a question of whether the underlying infrastructure can sustain the energy and thermal load required to keep those systems running efficiently.

The cost of generating a token therefore extends beyond compute. It includes electricity, cooling, physical infrastructure and the ability to maintain performance under load without degradation.

Most “cost per token” discussions don’t fully reflect this reality. At scale, energy becomes the budget, not just another line item.

The future isn’t bigger models. It’s better systems.

For the past two years, the industry has been focused on model comparisons, looking at benchmarks, rankings and incremental improvements in capability.

That focus is starting to shift.

In production environments, performance is less about which model you choose and more about how you use it. Organizations are moving toward systems of models—combining large and small models, routing tasks intelligently and optimizing for cost, latency and throughput across the entire workflow.

Instead of sending every request to a single large model, systems break workloads into smaller components. Simpler tasks can be handled by more efficient models, while more complex reasoning is reserved for larger ones. Context is reused wherever possible, and caching strategies are applied aggressively.

These decisions often have a greater impact on performance and cost than switching from one model to another. In that sense, tokens remain the unit of work, but the system that generates and manages them becomes the real differentiator.

The most overlooked layer in AI systems

AI is often described in terms of models on one side and applications on the other, but the layer in between is where most of the complexity—and opportunity—lives.

This layer doesn’t just move requests; it shapes them. It determines how traffic flows, how decisions are enforced and how systems behave under real-world conditions.

Delivery and security can’t be treated as separate concerns here. The same layer that routes requests and manages context is also where policies are applied, risks are mitigated and trust is established.

As complexity grows, point solutions break down. What’s needed is a unified platform that can coordinate these functions in real time, rather than stitching them together after the fact.

This is where tradeoffs get made. It’s where cost is controlled, performance is optimized and security decisions are enforced in real time. As AI systems scale, this layer becomes increasingly important. It’s the difference between a system that performs well in a demo and one that operates reliably and efficiently in production. That shift has real implications for how organizations design and manage AI systems.

What this means for organizations

As enterprises move AI into production, the question isn’t just which model to use—it’s how the system around it is designed to operate.

That means thinking beyond token metrics and model benchmarks, and focusing on how requests are routed, how context is managed and how policies are enforced across the entire workflow.

Most organizations are still stitching these pieces together—and that approach doesn’t hold up under real production pressure.

We’re measuring the wrong thing

Tokens provide a useful abstraction. They give the industry a way to quantify something that once felt intangible, but they are not the full picture.

Right now, the industry is gravitating toward what is easiest to measure—token counts, throughput and cost metrics—rather than what matters most. Without context, those numbers can be misleading.

The next phase of AI won’t be defined by who generates the most tokens. It will be defined by who understands what those tokens represent, and who can build systems that turn them into meaningful, efficient and scalable outcomes.

Because in the end, tokens aren’t the product. They’re simply the byproduct of intelligence being created.