WORLDTECHAssemblyAI runs speech-to-text on AWS Spot GPUs with retry queues
AssemblyAI has described how it runs cost-optimized speech-to-text inference at scale on AWS, combining Amazon ECS, Spot GPU instances on Amazon EC2 and a queue-based retry design. The organisation says the setup yields significant savings while keeping service level agreement commitments across European AWS Regions.
What AssemblyAI offers customers
AssemblyAI exposes an application programming interface that customers call for transcription, speaker detection, sentiment analysis, chapter detection and redaction of personally identifiable information. The service can be obtained through AWS Marketplace.
Its models transcribe and analyse speech for high-volume workloads in sectors that include technology, media, healthcare and financial services.
Spot capacity addresses the cost burden
Every audio file moves through a chain of deep learning models, covering acoustic models, language models and post-processing stages, that execute on NVIDIA GPUs in order to meet latency targets. At scale that amounts to thousands of GPU instances running without pause, a load that would be prohibitively expensive on On-Demand Instances alone.
Spot Instances carry substantial savings, but Amazon EC2 can take them back with a two-minute warning when capacity is required elsewhere. The architecture must therefore absorb interruptions smoothly while still honouring SLA commitments.
Architecture built on ECS, Spot and SQS
The platform runs on Amazon ECS, with compute capacity governed through ECS capacity providers. The fleet combines Spot and On-Demand Instances across several NVIDIA GPU instance families, with one inference model running per GPU.
An On-Demand capacity provider acts as an always-available baseline, while a Spot capacity provider spans different GPU instance families and Availability Zones to absorb spikes in demand. When a Spot Instance receives an interruption notice, Amazon EventBridge triggers a handler built on AWS Lambda, which returns the unfinished job to the main Amazon SQS queue so a healthy task can pick it up. Jobs that use up their retry attempts are sent to a dead letter queue for investigation, and Amazon CloudWatch watches fleet health and Spot interruption events.
Data residency in European regions
Customers in regulated industries, including healthcare, finance and legal, must keep audio data inside defined geographic boundaries. AssemblyAI states that it applies strict data residency configurations in European AWS Regions, where the architecture is described as delivering savings while holding to SLA commitments.
Open questions
- Which concrete cost saving applies to the production deployment described?