Buoyant announced the launch of Kinference (kay-inference), an open source inference control plane for Kubernetes. Building on top of the open source vLLM inference engine, Kinference allows organizations to provide safe, reliable, and cost-effective inference across multiple models, GPUs, and machines, while delivering the governance and policies needed for enterprise use at scale.
Kinference is lightweight, easy to deploy, and can be run without a team of specialists. It is agnostic to LLM model, cloud provider, and GPU type. Kinference’s feature set lets organizations offer a true, internal “inference platform” that provides inference guardrails and controls, lowered cost of inference, intent-based routing, burst to cloud, and per-tenant usage and cost attribution.
In public Kinference benchmarks, when compared to standard Kubernetes load balancing, Kinference decreased median time-to-first-token (TTFT) by 95% and median end-to-end latency by 45% of requests on one workload type.
For more information, go to kinference.ai.