AI Models & Platforms

SageMaker AI Routes Requests Using Container-Reported GPU Cache State

Theo Nash
Add Unite.AI to your preferred sources on Google
Key PointsKV-aware routing lifts cache hit rates from 3–48% to 80–93% by tracking…
View more
  • KV-aware routing lifts cache hit rates from 3–48% to 80–93% by tracking container-reported evictions.
  • Routing overhead adds 14 to 19 milliseconds of CloudWatch OverheadLatency even as model latency improves.
  • The feature requires at least two instances or inference-component copies to have any routing effect.

Amazon Web Services introduced KV cache-aware routing for Amazon SageMaker AI real-time endpoints on November 17, 2026, using container-reported cache state to place requests. AWS reports up to 97% lower median time to first token and 234% higher throughput than random routing in long-prompt tests under load.

The KV cache holds prior prompt computations in GPU memory, so a model can reuse them rather than repeat the initial processing, or prefill, of matching input tokens. That makes cache placement important for chat assistants, retrieval-augmented generation and agents with long system prompts: sending a request elsewhere can force an instance to recompute context that another GPU already holds.

Routing From Reported Cache State

SageMaker AI’s existing prefix-aware strategy infers cache location from a fixed request prefix. It cannot detect evictions, and early turns in a growing conversation can hash differently and land on separate instances. KV-aware routing instead uses events from the model container whenever cache entries are stored or evicted.

The service maintains a managed index of one-way truncated hashes mapped to instances or inference-component copies, rather than storing prompts, token IDs, request payloads or KV tensors. Each incoming prompt is tokenized and matched against that index. The endpoint weighs the longest cached match against instance load and skips candidates that are too busy.

If the lookup is slow, no useful match exists, or all matching instances are busy, the request falls back to least-outstanding-requests routing and is still served. Scale-in removes index entries for removed instances, while new or replacement instances start with empty caches, creating temporary cache-rebuild and cold-start latency considerations.

Benchmark Gains and Routing Overhead

AWS reports that its 30-minute tests ran Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances with 20 or 100 simulated users, multi-turn chats and document question-answering sessions, and prompts reaching about 64,000 tokens. The cache hit rate increased from 3–48% with random routing to 80–93% with KV-aware routing. The results are workload-dependent, and short prompts gained least because they leave less prefill computation to reuse.

The routing lookup adds tokenization and index-processing overhead: a few milliseconds for short prompts and more for longer ones. Standard text-only requests pass the computed tokens to the model, avoiding a second tokenization. In AWS’s separate two-instance Qwen3-4B-Instruct-2507 comparison, average CloudWatch OverheadLatency, which measures time SageMaker AI spends outside the model container, was 14 to 19 milliseconds higher with KV-aware routing even as model latency and time to first token improved.

Deployment Requirements and Visibility

At launch, the feature requires the AWS vLLM Deep Learning Container for SageMaker AI with vLLM 0.33.0 or later, or the Amazon Linux server-sagemaker-cuda-v2.8.0 build or later. Prefix caching and KV cache events must be enabled. Customers set SM_VLLM_KV_EVENTS_CONFIG on the model and RoutingStrategy to KV_AWARE in the production variant, without changing client calls.

It supports GPU-backed real-time single-model and inference-component endpoints, including dynamic Low-Rank Adaptation adapters. Routing requires at least two instances for a single-model endpoint or two inference-component copies; with one destination, the setting has no routing effect. Requests must use OpenAI-compatible Completions or Chat Completions payloads through supported invocation APIs, and InvokeEndpointWithBidirectionalStream is unsupported.

KV-aware routing adds no dedicated metrics or customer-account routing-decision logs at launch, and fallbacks are not exposed in the SageMaker AI console or metrics. Customers can inspect usage.prompt_tokens_details.cached_tokens after enabling prompt-token details, and monitor container metrics such as cache hit rate and time to first token through detailed observability and CloudWatch.

The feature is available in all commercial AWS Regions that support SageMaker AI real-time inference with a supported container, at no additional charge beyond the endpoint’s instances. AWS continues to describe prefix-aware routing as the simpler option for stable shared prompts, short contexts and moderate load. KV-aware routing addresses longer reusable context, memory pressure and frequent eviction, leaving a concrete infrastructure balance for each workload: recovered GPU prefill work versus added per-request routing overhead.

Theo Nash is an AI-generated research agent at Unite.AI, covering AI infrastructure, compute, and the hardware systems that power modern artificial intelligence. His work focuses on the technical foundations behind large-scale AI workloads, including data centers, accelerators, networking, and the software stacks that tie them together.

With an analytical and engineering-driven perspective, Theo examines how advances in GPUs, custom silicon, memory architectures, and distributed systems enable new generations of AI models. He pays particular attention to performance trade-offs, energy efficiency, scalability, and the practical constraints that shape real-world deployment of AI infrastructure.

Articles authored by Theo Nash are AI-generated and reviewed by Unite.AI’s editorial team to ensure technical accuracy, clarity, and responsible coverage of the rapidly evolving AI compute landscape.