Facebook is taking the wraps off of its “6-Pack” open modular switch platform designed for the flexibility, efficiency, and scale required in its massive data centers. The architecture allows Facebook to build different size switches using common line card and fabric card building blocks.
Last year, Facebook disclosed the specification of its top-of-rack network switch (code-named “Wedge”) and a Linux-based operating system for that switch (code-named “FBOSS”). It then described the modular network architecture it will use for scaling operations. The new 6-pack switch will serve as the core of this fabric.
In an engineering blog posting, Facebook said the 6-pack switch uses Wedge as its basic building block: it is a full mesh non-blocking two-stage switch that includes 12 independent switching elements. Each independent element can switch 1.28Tbps.
Facebook is currently building two configurations: One configuration exposes 16x40GE ports to the front and 640G (16x40GE) to the back, and the other is used for aggregation and exposes all 1.28T to the back. Each element runs its own operating system on the local server and is completely independent, from the switching aspects to the low-level board control and cooling system. Facebook said the advantage of this unique dual backplane design is the ability to modify any part of the system with no system-level impact, software or hardware.
In november 2014, Facebook’s Alexey Andreyev outlined a new data center switching architecture deployed at the company’s new facility in Altoona, Iowa.
In a blog posting, Andreyev writes that while its previous data centers have been built with a hierarchically oversubscribed system of clusters, Facebook set out to make its newest data center into a single, high-performance network for the whole building.
The new design is characterized by small, identical “server pods” that only require basic mid-size switches to aggregate the Top-of-rack (TOR) switches. Each pod has 48 server racks, each with a 10G connection. This form factor is always the same for all pods.
Each pod is served by a set of four fabric switches. Each TOR currently has 4 x 40G uplinks, providing 160G total bandwidth capacity for a rack of 10G-connected servers. The smaller port density of the fabric switches makes their internal architecture very simple, modular, and robust. The switches are available from multiple sources. Symmetrical bandwidth is provisioned to/from each pod. To implement building-wide connectivity, Facebook created four independent “planes” of spine switches, each scalable up to 48 independent devices within a plane. The network is all Layer 3 – from TOR uplinks to the edge. Standard BGP4 is the only routing protocol. Facebook is using its own centralized BGP controller that is able to override any routing paths on the fabric by pure software decisions.
The full posting is here: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/
In June 2014, Facebook introduced a new top-of-rack network switch, code-named “Wedge,” and a new Linux-based operating system for that switch, code-named “FBOSS.” Both products are part of Facebook’s efforts to “disaggregate” the network from traditional product categories so as to create more flexible, more scalable and more efficient data centers.
The Wedge switch features a modular design that brings capabilities of a micro-server using a range of processors, including Intel, AMD and ARM. On the software side, the “FBOSS” uses the same libraries that Facebook currently uses to manage its server fleet. This will let Facebook program the switch with the same abstraction layer used for other software services.
In an engineering blog posting, Facebook describes its capabilities as a hybrid of distributed and centralized control models.
Facebook is currently testing Wedge and FBOSS and plans to contribute key elements to the Open Compute Project (OCP).
In
