AWS adds Global Inference Gateway to SageMaker HyperPod for multi-cluster routing

Amazon Web Services has introduced the Global Inference Gateway for Amazon SageMaker HyperPod, a single managed endpoint that sends inference requests to HyperPod clusters within one AWS Region or across several Regions.
What the gateway handles
The Global Inference Gateway gives customers one managed endpoint in front of every HyperPod cluster they register. For each request it selects a cluster and forwards the request over the AWS backbone, with automatic failover and authentication handled through AWS Signature Version 4 (SigV4) and AWS Identity and Access Management (IAM). Within each cluster, the existing Inference Gateway still picks the model pod.
AWS says the gateway is aimed at capacity spread across clusters and Regions. Large models are harder to serve and GPU capacity is harder to secure, so customers often divide it among more clusters and Regions, and the gateway is meant to take over the routing work that this creates.
Why fragmented capacity causes trouble
In AWS's example, a team runs four models across five HyperPod clusters in two Regions. Without the new gateway, clients would have to track 20 endpoints, one for each model on each cluster. The Inference Gateway cuts that to five, one per cluster, but each client must still choose a cluster itself.
According to AWS, this arrangement has consequences. During a traffic peak a request can wait in the queue of a saturated cluster while GPUs on another cluster, sometimes in another Region, sit unused. When a cluster fails, an engineer has to change configuration to move traffic away from it. Spreading traffic across Regions currently means building and maintaining one's own load balancers, DNS and cross-Region networking.
Setup and request path
Setting up the gateway involves two decisions: which clusters to make available for routing, and which of them to group behind one gateway. A cluster can be reached only after it has been published. Publishing creates a Network Load Balancer in front of the cluster's Inference Gateway, a VPC endpoint service on that balancer, and stores the listener's certificate in the customer's TLS bucket. Adding an unpublished cluster to a gateway makes the call fail with a message explaining the fix.
AWS describes the route a request takes: the client signs it with SigV4 and sends it to the gateway's regional endpoint, the gateway selects a cluster, AWS PrivateLink carries the request to that cluster in the same or another Region, and the cluster's Inference Gateway passes it to a model pod. Each cluster reports its load back to the gateway. Customers decide which clusters to publish, which to group and how to weight routing, while AWS runs the routing fleet.
Customer view
AWS quotes Jinhyung Park, VP (AI Platform) at Bagelcode, on the service's benefits. The announcement lists three main advantages: requests overflow during peaks to clusters with spare capacity, clients use a single endpoint, and every request is authenticated with SigV4 and IAM, reaching the clusters only over AWS PrivateLink.
Open questions
- How much extra latency does routing across Regions add compared with a single cluster?
- What charges apply to the gateway beyond the underlying HyperPod clusters and data transfer?