Reports
The Inference Economics Scorecard Challenges GPU Utilization as an AI Performance Metric

Key PointsGPU utilization cannot distinguish productive output from expensive context rebuilding.
View more
- GPU utilization cannot distinguish productive output from expensive context rebuilding.
- The scorecard measures useful throughput against power, rack space, and cost rather than chip activity.
- Shared NVMe cache cut context recomputation to 6.8% of prefill cycles in the benchmark.
A busy GPU is not necessarily a productive GPU. That is the argument behind The Inference Economics Scorecard, new research from NAND Research and WEKA that proposes a broader way to judge the infrastructure running production AI.
The framework shifts attention from chip activity to useful output delivered within a service target, measured against electricity, rack space, and cost. Its relevance grows as AI agents maintain long conversations, call tools, and return repeatedly to earlier context. An inference server can spend substantial compute rebuilding that context while its utilization dashboard still looks healthy.
Alongside the announcement, WEKA is highlighting an eight-hour benchmark conducted with Nebius on NVIDIA HGX B300 hardware and releasing workload traces and tooling for others to reproduce the comparison. The scorecard is a proposed evaluation framework, rather than an established industry standard. WEKA, whose storage technology benefits from this framing, sponsored the research.
Why GPU Utilization Can Hide Expensive Repeated Work
Inference is the process of running a trained model to produce an answer. Before generating new tokens, the model processes the input and conversation history in a phase called prefill. During that work, it creates a key-value cache, or KV cache: intermediate attention data that lets it reuse earlier computation when the conversation continues.
The fastest place for that cache is the GPU’s high-bandwidth memory, known as HBM. But HBM must also accommodate model weights and other working data. Long sessions and competing users can force a server to evict cached context to make room.
When the user or agent returns, the missing cache may have to be rebuilt by processing old tokens again. This does not mean the model has permanently lost the conversation. It means the infrastructure has discarded a reusable representation and must pay to reconstruct it.
That creates a measurement trap. Reprocessing yesterday’s context keeps the GPU active, just as generating a useful new answer does. Utilization remains a valuable diagnostic signal, but cannot by itself tell an operator how much productive work customers received.
What the Inference Economics Scorecard Measures
The scorecard calls for continuous tests lasting at least eight hours, long enough to expose behavior that short demonstrations may miss. Comparisons must hold the model, precision, workload, concurrency, cache conditions, quality threshold, and service-level objectives constant. The measurement boundary covers the serving stack, with cooling and facility overhead disclosed.
Only successful output meeting the quality and service targets contributes to useful throughput. Failed requests, retries, timeouts, and discarded output must remain visible rather than disappearing from the calculation.
| Metric | What it helps reveal |
|---|---|
| Tokens per watt | How efficiently power is converted into useful model output. |
| Tokens per rack | How much output the deployment delivers within a physical footprint. |
| Cost per inference | The expense of serving a successful request across the system. |
| GPU goodput | Productive output delivered within service targets, rather than raw chip activity. |
| Time to first token and end-to-end latency | How long users wait for a response to begin and finish. |
| Memory efficiency | How effectively the system uses scarce memory capacity. |
| Operational density | How deployment and operating constraints affect usable capacity. |
| Agent concurrency | How well the system supports multiple active agent workloads. |
| Context retention | How effectively reusable session context survives as workloads grow. |
The distinction between throughput and goodput is especially important. A server that produces more tokens but misses a customer’s response deadline can be less useful than a slower-looking system that consistently meets it. Similarly, a larger cache is only economically attractive if its benefit exceeds the additional storage, networking, and operating expense.
How Shared Storage Extends the KV Cache
WEKA’s technical account of the benchmark describes NeuralMesh with Augmented Memory Grid, which keeps reusable KV data on shared NVMe storage beyond the GPU’s own memory. NVMe is a high-performance interface commonly used with flash drives.
In this design, the cache becomes a resource the serving infrastructure can retrieve across the network, instead of repeatedly reconstructing it on a GPU. RDMA, or remote direct memory access, and NVIDIA GPUDirect Storage help move data along a direct storage-to-GPU path with less CPU involvement.
This is a trade between moving existing data and recomputing it. A sufficiently fast storage path can make retrieval cheaper than rebuilding long histories. The advantage depends on cache reuse, context length, concurrency, and the speed of the entire data path; a short request whose working data already fits in HBM presents a different problem.
What the Eight-Hour Benchmark Actually Found
The joint Nebius and WEKA test used one eight-GPU HGX B300 serving node running DeepSeek-V4-Pro. Its replay workload drew on more than 5,000 real agent traces, maintained a pool of 512 active sessions, and used a scheduler permitting 32 concurrent requests. Active sessions and simultaneous requests are different measures.
Across eight hours, WEKA reports 3.7 times more output tokens per GPU-hour with its shared cache than with HBM-only serving. Among successfully completed requests, 97% reached a first token within five seconds, compared with 44% for the baseline. Recomputing previously encountered context consumed 60% of prefill cycles in the baseline, versus 6.8% with the shared cache.
The additional cache used NVMe drives across an eight-node cluster, even though the reported serving result came from one GPU node. The GPU-normalized gain therefore does not establish an equivalent reduction in total cost or energy: those calculations must include the supporting resources.
The baseline also matters. The technical write-up says a suitable DRAM-offload implementation for this model was unavailable when testing took place. These results compare the tested configuration with HBM-only serving, rather than proving superiority over every alternative cache architecture.
Open Traces Make the Comparison More Useful
WEKA’s open-source kvlab project captures and replays the structure of agent sessions, including timing and cache-reuse patterns, without distributing the original conversation or source-code content. Its loader is documented in NVIDIA’s AIPerf tooling, and the announcement also identifies integration with SemiAnalysis’s AgentX benchmark.
Preserving pauses, growing histories, and related agent activity helps expose cache pressure that a stream of unrelated synthetic prompts would overlook. Such replays measure serving behavior; they do not by themselves demonstrate whether an agent completed a business task correctly.
For enterprises, the central contribution of The Inference Economics Scorecard is a clearer definition of success: useful responses, at the required quality and speed, within the resources available. As agents consume more context and run for longer, keeping a GPU busy becomes a starting observation. What matters commercially is the work that reaches the customer.












