AI Models & Platforms
AWS Routes HyperPod Inference Across Regions From One Endpoint

Key PointsThe global gateway adds a cluster-selection tier above the existing pod-level Inference…
View more
- The global gateway adds a cluster-selection tier above the existing pod-level Inference Gateway routing.
- Operators tune four weighted signals including RegionProximity at 0.40 to balance latency against resilience.
- AWS states the gateway does not log or store prompts or completions and routes only over PrivateLink.
AWS introduced Amazon SageMaker HyperPod Global Inference Gateway on November 20, 2026, giving customers one managed endpoint for inference requests across registered HyperPod clusters in a single AWS Region or multiple Regions. The service selects a cluster using capacity, queue, proximity and error signals, with automatic failover when a cluster becomes unavailable.
The launch adds a second routing tier above the existing SageMaker HyperPod Inference Gateway. That cluster-level gateway continues to select individual model pods using signals including key-value (KV) cache utilization and queue depth; the new global tier selects the cluster first, while leaving pod selection unchanged.
AWS is addressing a capacity-utilization problem: requests can queue on a saturated cluster while already-provisioned GPUs sit idle elsewhere. In its example of four models served across five clusters in two Regions, per-model serving requires clients to track 20 endpoints. Cluster-level gateways cut that to five, while the global gateway provides one. The managed tier removes the need to build and maintain load balancers, DNS and cross-Region networking for this routing, and lets peak traffic overflow to clusters with available capacity.
Configurable Cluster Selection
An operator publishes each participating cluster by enabling spec.globalRouting in its existing InferenceGatewayConfig, then calls CreateGlobalInferenceGateway with the clusters to group. Publishing creates a Network Load Balancer, a virtual private cloud (VPC) endpoint service and certificate resources in the customer’s account. An unpublished cluster cannot be added successfully.
Clients send OpenAI-compatible requests signed with AWS Signature Version 4 (SigV4) to the regional endpoint. The gateway authenticates the caller and selects a healthy cluster, AWS PrivateLink carries the request within or across Regions, and the cluster’s Inference Gateway chooses the model pod. SageMaker operates the routing fleet, private endpoints, certificate validation, health checks, authentication and rate limiting; a service-managed publisher on each cluster supplies load metrics.
For each request, the gateway scores healthy clusters using four configurable signals:
| Signal | What it measures | Default weight |
|---|---|---|
KVHeadroom |
Available KV cache and GPU memory on the cluster | 0.20 |
QueueDepth |
How backed up the cluster is | 0.25 |
RegionProximity |
Distance between the cluster and the gateway’s Region | 0.40 |
ErrorRate |
Recent failed requests on the cluster | 0.15 |
Customers overriding the defaults must set all four weights, using values from 0.0 to 1.0 that total 1.0, and can disable a signal with zero. UpdateGlobalInferenceGateway changes the weighting for new requests without reprovisioning, while in-flight requests finish under the old settings. AWS says workloads that prioritize resilience over proximity can raise ErrorRate and lower RegionProximity.
Failover and Privacy
The gateway health-checks every cluster and shifts traffic to the next-best cluster after a failure without requiring client changes, bringing a recovered cluster back gradually. A failed request is retried once on another cluster unless streaming has started; if no healthy cluster remains, the gateway returns HTTP 503. Adding a healthy cluster does not change the endpoint or disrupt other clusters, while removing one stops new requests but lets in-flight work finish.
IAM separately controls standard and streaming invocation, and customer JSON Web Tokens, when used, travel inside the SigV4 signature. Requests reach clusters only over AWS PrivateLink, and AWS states that the gateway does not log or store prompts or completions. Deleting the global gateway stops its endpoint but leaves the per-cluster endpoints serving.
Bagelcode VP of AI Platform Jinhyung Park said in AWS’s announcement that beta tests routed requests across two Regions through the gateway and continued serving them when one cluster could no longer serve the model. That gave the company’s small platform team cross-Region failover without building or maintaining its own routing layer.
Availability and Migration
The service is available in 13 Regions: US East (N. Virginia and Ohio), US West (N. California and Oregon), South America (São Paulo), Europe (Frankfurt, Ireland, London and Stockholm), and Asia Pacific (Tokyo, Mumbai, Singapore and Sydney).
AWS advises single-cluster users to remain on the cluster Inference Gateway endpoint and adopt the global service when adding a second cluster. Existing per-cluster endpoints continue serving during client migration, and a gateway containing only clusters in one Region can keep a workload there. A published cluster left outside any gateway receives no traffic even though its Network Load Balancer continues to run and incur charges.
By putting proximity, capacity and failure signals in one policy, the service makes the use of otherwise idle GPU capacity a configuration decision rather than a separate networking project.












