Buoyant, the company behind Linkerd, has released Kinference: an open source control plane for running inference workloads on Kubernetes with vLLM. The project is licensed under Apache 2.0 and is currently at version 0.1.0-alpha.7. It sits in front of vLLM workers to manage routing, token rate limits, and load balancing based on KV-cache state.
Core architecture
Kinference separates the control plane from the data plane. vLLM handles the actual model serving, while Kinference decides where each request goes. The system consists of a traffic gateway and an inference handler. The handler exposes a gRPC interface to return the chosen endpoint and any necessary modifications to the request body. It works alongside Buoyant’s Gateway proxy.
Configuration relies on five Kubernetes Custom Resource Definitions (CRDs): GatewayBackend, InferencePolicy, InferencePool, InferenceProvider, and InferenceWorkerSet. The project currently uses NVIDIA’s dynamo-kv-router crate to process cache events.
Routing policies
Users define rules to dictate how requests are routed or rejected. These policies can match against tenant identity, the requested model, GPU pool status, and semantic tags. The documentation lists five common patterns:
- Simple to cheaper: Route low-complexity prompts to a smaller model unless cache overlap makes a larger model faster.
- Burst to cloud: Skip workers with over two queued prompts and fall back to external providers like Bedrock.
- PII stays local: Remove external providers when a PII tag is detected and fail closed by rejecting the request.
- Batch on idle GPUs: Send batch requests only to workers operating under 70% KV-cache usage.
- Model aliasing: Publish a stable name, such as
standard-model, backed by several different models.
Most rules end with minimize: PredictedTTFT. The system predicts time-to-first-token for each candidate endpoint by analysing cache overlap and queue length. Policies are stored in git and can be validated offline using the kinfctl CLI.
Performance benchmarks
Buoyant published a benchmark comparing Kinference against plain Kubernetes load balancing. The test used three vLLM 0.31.0 pods per arm, each running on an L40S GPU. The model tested was Qwen3.8-27B-FP8, and the load generator was GuideLLM 0.8.1. The baseline used a ClusterIP Service where kube-proxy selects a random pod per connection.
Median time-to-first-token (TTFT) results show a significant divergence based on traffic patterns:
| Workload | Kubernetes Service | Kinference | Change |
|---|---|---|---|
| Control (no shared prefix) | 6.92 s | 6.83 s | No significant difference |
| Shared-prefix | 6.88 s | 0.31 s | about -95% |
| Multiturn | 1.70 s | 0.59 s | about -65% |
Median end-to-end latency fell 45% on shared-prefix traffic and 32% on multiturn requests. For control traffic, the p95 TTFT dropped from 21.46 seconds to 12.33 seconds. Buoyant attributes this improvement to queue-aware balancing.
A capacity test measured goodput: requests per minute meeting a specific TTSL SLO. Peak goodput reached 103.0 requests per minute versus 14.0 on shared-prefix traffic, a roughly 7.4x increase. On multiturn requests, the figure was 82.7 versus 23.8, a 3.5x increase. The Service baseline failed to meet the SLO at any tested concurrency level.
These numbers are self-reported. The baseline relies on random routing rather than another cache-aware router like llm-d or Dynamo. Buoyant notes that its SLOs of 8 seconds and 4 seconds suit offline agents rather than chat interfaces.
Comparison with other tools
Kinference competes with llm-d, NVIDIA Dynamo, and the vLLM Production Stack. Kinference’s distinct angle is governance: policies, aliases, provider fallback, and tenant accounting. llm-d and Dynamo lead on raw serving performance, disaggregation, and engine coverage.
| Feature | Kinference | llm-d | NVIDIA Dynamo | vLLM Production Stack |
|---|---|---|---|---|
| Backer | Buoyant | CNCF sandbox (Red Hat, Google Cloud, IBM, CoreWeave, NVIDIA) | NVIDIA | vLLM project |
| License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Apache 2.0 |
| Maturity | v0.1.0-alpha.7 | v0.7 (May 2026) | 1.x (1.5.1 containers) | Reference stack |
| Engines | vLLM (SGLang, TensorRT-LLM planned) | vLLM, SGLang | vLLM, SGLang, TensorRT-LLM | vLLM |
| KV-cache-aware routing | Yes, predicted TTFT | Yes, prefix-cache and predicted latency | Yes, KV-aware router | Session-based; prefix-aware listed as WIP |
| Declarative routing policy CRD | Yes (InferencePolicy) | Not documented in README | Not documented in README | Not documented in README |
| Burst to external providers | Yes (InferenceProvider) | Not documented in README | Not documented in README | Not documented in README |
| Disaggregated prefill/decode | Not disclosed | Yes | Yes | On roadmap |
| Multi-tenant controls | Per-tenant metrics; rate limits planned | Flow control for multi-tenant serving | Gateway-level rate limiting via GAIE | Not documented in README |
| Main result | -95% median TTFT vs kube-proxy (3x L40S) | 3x throughput, 2x faster TTFT vs round-robin (4x MI300X) | 2x faster TTFT with KV-aware routing (Baseten) | Not disclosed |
Deployment and availability
Installation takes five steps according to the getting started guide. Teams must install Gateway API v1.5.1 CRDs, the Buoyant gateway Helm chart, and then the Kinference chart. Kinference ships Prometheus metrics and Grafana dashboards. Metrics cover TTFT, ITL, end-to-end latency, and cached versus uncached prompt tokens, mostly per tenant.
Rate limiting, classifiers, and Anthropic API support are not yet shipped.
What it means
For teams managing many open-weight models, Kinference adds a layer of policy control above the model server. It allows operators to define simple rules that route low-complexity prompts to smaller models or fail requests containing sensitive data. The system handles the logic of when to batch requests or fall back to external cloud providers, removing that manual orchestration from the workflow.
