Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

What happened

Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes (software that runs and manages applications across many servers). It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark.

Buoyant, the company behind Linkerd, has released Kinference , an open source (published so anyone may read, use and change the code) project for building inference platforms on Kubernetes. It sits in front of a fleet of vLLM workers and decides where each request goes.

It applies routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics in one layer. For dev teams serving many open-weight (a model whose trained parameters anyone may download) (published so anyone may download the trained model) models to many internal clients, it targets the layer above the model server.

The project is Apache 2.0 licensed on GitHub . TL;DR Size: v0.1.0-alpha.7, written in Rust, Apache 2.0, configured through 5 Kubernetes CRDs.

Runs on: any GPU-enabled Kubernetes cluster running vLLM. Buoyant’s benchmark used 3 L40S GPUs per pool.

Performance: large gains on prefix-heavy traffic versus plain Kubernetes Service routing (self-reported). Worst: no median TTFT gain on prompts without shared prefixes (6.92 s vs 6.83 s).