Google Releases TPU Sync: Open-Source KV Cache Transfer for TPUs

What happened

Google released TPU Sync, an open-source C++ library that moves KV caches and weights across TPU HBM (memory stacked next to a processor to feed it data faster), host DRAM, storage and network. Google is an Internet and software company based in Mountain View.

Google Cloud has released TPU Sync , an open-source C++ library for moving KV caches and model weights on Cloud TPUs. Tested on 235B to 480B MoE models.

Google reports over 90% lower KV transfer latency (the delay between asking for something and getting it) on 235B to 480B MoE models. The Google TPU Sync KV cache transfer library treats TPU HBM, host DRAM, shared storage and the data center (a building full of computers that runs online services) network as one memory hierarchy.

It targets a growing bottleneck: moving model state, not computing it. Google reports over 90% lower KV transfer latency on 235B to 480B parameter MoE models.

TL;DR Library: Native C++ core with Python wheels for PyTorch and JAX. Runs on: Cloud TPU v7x (Ironwood) and v6e per the README .

Benchmarks used TPU v7x (2x2x1), PCIe Gen5 and 200 Gbps NICs. Performance: Large cuts to KV transfer latency and time to first token (TTFT) in Google’s internal lab tests.