Running Ray at Scale on AKS

GPU Capacity Limits

GPU scarcity is one of the most significant operational challenges in large-scale ML. High-demand accelerators, such as NVIDIA GPUs, often have quota and availability issues in Azure regions. This can delay cluster setup and job scheduling.

Microsoft’s proposed solution uses a multi-cluster, multi-region setup. Distributing Ray clusters across different AKS instances in various Azure regions allows teams to: Aggregate GPU quota beyond regional limits, automatically reroute workloads during outages or capacity issues and extend the compute pool to on-premises systems or other cloud providers using Azure Arc with AKS.

Scattered ML Storage

A common issue in ML operations is transferring training data, model checkpoints, and artifacts between pipeline stages. This includes moving them from pre-training to fine-tuning and then to inference. The guidance addresses this with Azure BlobFuse2, which mounts Azure Blob Storage into Ray worker pods as a POSIX-compatible filesystem.

From Ray’s perspective, the mount point is just a local directory. Tasks and actors read datasets and write checkpoints using standard file I/O. BlobFuse2 then saves data to Azure Blob Storage. This makes data available across pods and node pools. Local caching prevents GPU stalls during large training runs, and because data is decoupled from compute, Ray clusters can scale up and down without data loss.

Problems with Credential Expiry

Another important topic is the authentication reliability. Anyscale and Azure used to integrate with CLI tokens or API keys that expired every 30 days. This meant manual rotation was needed, which risked service disruption.

The new method uses Microsoft Entra service principals and AKS workload identity. It issues short-lived tokens automatically. The Anyscale Kubernetes Operator pod uses a user-assigned managed identity. This identity requests an access token for the Anyscale service principal from Entra ID. Azure handles token refresh transparently, meaning no long-lived credentials are stored in the cluster and no manual rotation is required.

Conclusion

The Anyscale on AKS integration is currently in private preview. Teams wanting access should reach out to their Microsoft account team. They can also file a request on the AKS GitHub repository. Include details about Ray workloads and target regions.

Check out example setups and workloads for fine-tuning with DeepSpeed and LLaMA-Factory in the Azure-Samples/aks-anyscale repository on GitHub. This also includes LLM inference endpoints.