WORLDTECH NEWS Global technology intelligence.Contact

Salesforce uses cross-Region routing to serve AI models from fragmented capacity

Warehouse interior showing workers handling boxes and organized shelves filled with products.
Illustrative photo.Photo by Tiger Lily on PexelsAmazon Web Services logo shown for identification only; no affiliation with or endorsement of WORLDTECH is implied.

Salesforce has published an account, confirmed by the company itself, of how it uses the cross-Region inference router in Amazon SageMaker AI to keep its models available across more than 30 global Regions. The router lets the company draw on capacity that had been split between Regions.

Capacity split across Regions

Salesforce's Agentforce Foundation team serves models for products including Agentforce and the Salesforce Trust Layer, among them models trained in house and open source models. Requests are served from the Region they originate in, across more than 30 global Regions and hundreds of AWS accounts, with capacity provisioned according to each customer's traffic.

Customer peak windows often overlap, and provisioning for the shared peak would cost too much. New business units are also onboarded at short notice, so the fleet has to grow quickly. Adding capacity one Region at a time has left some Regions holding idle instances while others run short, and a shortfall in one place then becomes an availability problem.

Limits of the earlier gateway path

Before adopting the router, Salesforce used its Internal Gateway to forward traffic that local endpoints could not handle to other Regions. The team built and ran that path with basic health checks, retries and Region-switching logic.

The post identifies two weaknesses. The gateway received no capacity signals from either the local or the remote endpoint, so it could not judge when to move traffic or whether a destination had room. Cross-Region traffic also had to leave the virtual private cloud through a NAT gateway and an internet gateway and cross public AWS networking. Design reviews raised the risk of cascading failures, because moving traffic during an incident can overload the next Region, and the team concluded that cross-Region traffic needed a cap. Writing this logic separately into every application would not scale across the company.

How the router directs requests

The inference router is a Regional SageMaker AI resource in the customer's own account. Applications call the router instead of an endpoint, naming a routing target and sending the same request payload and content type they currently send to InvokeEndpoint. Each routing target is a name chosen for one logical model and lists the endpoints and inference components that may serve it, identified by their full Amazon Resource Names.

Each router keeps a separate routing target per model, so trouble with one model's endpoints does not shift traffic for the others. A routing target can combine single-model endpoints and inference components, and because destinations are identified by ARN they can sit in another Region or another AWS account. The owning account grants access through a resource-based endpoint policy, which is how Salesforce runs it. The router favours its own Region and keeps requests there while the local destination responds successfully.

Open questions

  • What results did Salesforce observe after running the router in production?

Key facts

  • Salesforce uses the cross-Region inference router in Amazon SageMaker AI to turn capacity spread across Regions into dependable model serving.

Sources & evidence