WORLDTECHAmazon makes SGLang-Neuron publicly available for AWS Trainium
Amazon has announced that SGLang-Neuron is now publicly available through an AWS Deep Learning Container, allowing developers to serve models and generate reinforcement learning rollouts on Trainium chips. RadixArk released the plugin in collaboration with Annapurna Labs.
What the release provides
SGLang-Neuron is an out-of-tree plugin that links the SGLang framework with AWS Trainium, which Amazon describes as a portfolio of AI chips offering strong economics for training and inference at scale. Access runs through an AWS Deep Learning Container that RadixArk released in collaboration with Annapurna Labs.
Once the plugin is installed, it registers a neuron platform through SGLang's plugin entry points. Setting SGLANG_PLATFORM=neuron selects that platform when the server starts.
Split between SGLang core and plugin
SGLang core keeps supplying the API server, the request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin supplies the Trainium-specific models, attention, mixture of experts (MoE) components and the graph runner.
Beneath that layer, SGLang-Neuron runs on native PyTorch. TorchNeuron registers Trainium as a native PyTorch device. Model execution relies on standard PyTorch operations plus kernels written with the Neuron Kernel Interface (NKI) for performance. These operations and kernels are compiled into executable programs through torch.compile, which the graph runner prepares and runs during serving.
Working with the existing SGLang workflow
Running SGLang on Trainium keeps the workflow developers already use. Models are launched through SGLang's server interface, and the endpoint is called with the same generation APIs.
Amazon also describes a disaggregated setup in which a prefill server builds the prompt's KV cache and sends it over NIXL and Elastic Fabric Adapter (EFA) to a decode server, which then produces the remaining tokens.