Kubernetes WG Serving: Achievements and Next Steps

The Kubernetes Working Group (WG) Serving was created to support the development of the AI inference stack on Kubernetes. The goal of this working group was to ensure that Kubernetes is an orchestration platform of choice for inference workloads. This goal has been accomplished, and the working group is now being disbanded.

Key Achievements of WG Serving

WG Serving formed workstreams to collect requirements from various model servers, hardware providers, and inference vendors. This work resulted in a common understanding of inference workload specifics and trends and laid the foundation for improvements across many SIGs in Kubernetes.

For example, the working group oversaw several key evolutions related to load balancing and workloads. The inference gateway was adopted as a request scheduler. Multiple groups have worked to standardize AI gateway functionality, and early inference gateway participants went on to seed agent networking work in SIG Network.

Impact on the Kubernetes Ecosystem

The use cases and problem statements gathered by the working group informed the design of AIBrix. Many of the unresolved problems in distributed inference — especially benchmarking and recommended best practices — have been picked up by the llm-d project, which hybridizes the infrastructure and ML ecosystems and is better able to steer model server co-evolution.

Additionally, llm-d and AIBrix represent more appropriate forums for driving requirements to Kubernetes SIGs than this working group. llm-d’s goal is to provide well-lit paths for achieving state-of-the-art inference and aims to provide recommendations that can compose into existing inference user platforms. AIBrix provides a complete platform solution for cost-efficient LLM inference.

Next Steps for WG Serving Initiatives

WG Serving helped with Kubernetes AI Conformance requirements. The llm-d project is leveraging multiple components from the profile and making recommendations to end users consistent with Kubernetes direction (including Kueue, inference gateway, LWS, DRA, and related efforts). Widely adopted patterns and solutions are expected to go into the conformance program.

All efforts currently running inside WG Serving can be migrated to other working groups or directly to SIGs. Requirements will be discussed in SIGs and in the llm-d community. Specifically:

• Autoscaling-related questions — mostly related to fast bootstrap — will be discussed in SIG Node or SIG Scheduling.
• Multi-host, multi-node work can continue as part of SIG Apps (for example, for the LWS project), and DRA requirements will be discussed in WG Device Management.
• Orchestration topics will be covered by SIG Scheduling and SIG Node.
• Requirements for DRA will be discussed in WG Device Management.

The Gateway API Inference Extension project is already sponsored by SIG Network and will remain there. The Serving Catalog work can be moved to the Inference Perf project. Originally it was designed for a larger scope, but it has been used mostly for inference performance.

Conclusion and Future Directions

CNCF thanks all contributors who participated in WG Serving and helped advance Kubernetes as a platform for AI inference workloads. As the working group is disbanded, its initiatives and efforts will continue in other parts of the Kubernetes ecosystem, ensuring that Kubernetes remains a leading platform for AI inference workloads.

For more information on the future of WG Serving initiatives, please visit the Kubernetes website. Additionally, you can explore the following FAQs for more details:

Frequently Asked Questions

1. What is the focus of the llm-d project, and how does it relate to WG Serving? The llm-d project aims to provide well-lit paths for achieving state-of-the-art inference and is a key initiative that has picked up some of the unresolved problems in distributed inference from WG Serving.
2. What happens to the initiatives and efforts of WG Serving now that it is being disbanded? All efforts currently running inside WG Serving can be migrated to other working groups or directly to SIGs, ensuring continuity and progress in the Kubernetes ecosystem.
3. How can I get involved in the future of Kubernetes AI inference workloads? You can participate in SIGs, such as SIG Node, SIG Scheduling, and SIG Apps, or join the llm-d community to contribute to the development of AI inference workloads on Kubernetes.
4. What is the role of AIBrix in the Kubernetes ecosystem, and how does it relate to WG Serving? AIBrix provides a complete platform solution for cost-efficient LLM inference and is one of the initiatives that has been informed by the work of WG Serving.
5. How can I learn more about the Kubernetes AI Conformance requirements and the llm-d project? You can visit the Kubernetes website and explore the resources and documentation available on the llm-d project and Kubernetes AI Conformance requirements.