Introduction to Backend Aggregation

Meta is developing a gigawatt-scale AI cluster called Prometheus, which will deliver 1-gigawatt of capacity to enhance and enable new and existing AI experiences across Meta products. A key component in building this infrastructure is backend aggregation (BAG), a centralized Ethernet-based super spine network layer that interconnects multiple spine layer fabrics across various data centers and regions within large clusters.

What is Backend Aggregation?

BAG is designed to support immense bandwidth needs, with inter-BAG capacities reaching the petabit range (e.g., 16-48 Pbps per region pair). We use BAG to interconnect data center regions to share compute and other resources into large clusters.

How BAG is Helping Us Build Gigawatt-Scale AI Clusters

To address the challenge of interconnecting tens of thousands of GPUs, we’re deploying distributed BAG layers regionally. BAG layers are strategically distributed across regions to serve subsets of L2 fabrics, adhering to distance, buffer, and latency constraints.

How We Interconnect BAG Layers

Inter-BAG connectivity utilizes either a planar (direct match) or spread connection topology, chosen based on site size and fiber availability. Planar topology connects BAG switches one-to-one between regions following the plane, offering simplified management but concentrating potential failure domains.

Meanwhile, spread connection topology distributes links across multiple BAG switches/planes, enhancing path diversity and resilience. For example, we use two main fabric technologies, Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF), to build L2 networks.

Hardware and Routing

Meta’s implementation of BAG uses a modular chassis equipped with Jericho3 (J3) ASIC line cards, each providing up to 432x800G ports for high-capacity, scalable, and resilient interconnect. Routing within BAG uses eBGP with link bandwidth attributes, enabling Unequal Cost Multipath (UCMP) for efficient load balancing and robust failure handling.

Designing the Network for Resilience

The network design meticulously details port striping, IP addressing schemes, and comprehensive failure domain analysis to ensure high availability and minimize the impact of failures. Failure modes are analyzed at the BAG, data hall, and power distribution levels.

Conclusion

In conclusion, BAG is playing an important role in Meta’s next generation of AI infrastructure. By centralizing the interconnection of regional networks, BAG helps enable the gigawatt-scale Prometheus cluster, ensuring seamless, high-capacity networking across tens of thousands of GPUs.

Frequently Asked Questions

• What is Backend Aggregation (BAG)? BAG is a centralized Ethernet-based super spine network layer that interconnects multiple spine layer fabrics across various data centers and regions within large clusters.
• How does BAG help build gigawatt-scale AI clusters? BAG helps by interconnecting data center regions to share compute and other resources into large clusters.
• What is the role of BAG in Meta’s AI infrastructure? BAG plays a crucial role in enabling the gigawatt-scale Prometheus cluster, ensuring seamless, high-capacity networking across tens of thousands of GPUs.
• How does BAG ensure network resilience? BAG ensures network resilience through careful design, including port striping, IP addressing schemes, and comprehensive failure domain analysis.
• What are the benefits of using BAG in AI infrastructure? The benefits of using BAG include enabling gigawatt-scale AI clusters, ensuring seamless, high-capacity networking, and providing a scalable and resilient infrastructure for AI workloads.