Bare-Metal Switching

Commodity switches are used in both white-box and bare-metal switching (see Figure 1-5). In this way, users can purchase hardware from one vendor, purchase an operating system from another, and then load features and applications from other vendors or open-source communities.

As a result of the OpenFlow hype, white-box switching was a hot topic since it commoditized hardware and centralized the network control in an OpenFlow controller (now known as an SDN controller). Google announced in 2013 that it built and controlled its switches with OpenFlow! It was a topic of much discussion then, but not every user is Google, so not every user will build their hardware and software.

Meanwhile, a few companies emerged solely focused on providing white-box switching solutions. These companies include Big Switch Networks, Cumulus Networks, and Pica8 (now owned by NVIDIA). They also needed hardware for their software to provide an end-to-end solution. Originally, original design manufacturers (ODMs) supplied white-box hardware platforms like Quanta Networks, Supermicro, Alpha Networks, and Accton Technology Corporation. You probably haven’t heard of those vendors, even if you’ve worked in the network industry.

The industry shifted from calling this trend white-box to bare-metal only after Cumulus and Big Switch announced partnerships with HP and Dell Technologies. Name-brand vendors now support third-party operating systems from Big Switch and Cumulus on their hardware platforms. You create bare-metal switches by combining switches from ODMs with NOSs from third parties, including the ones mentioned above. Many of the same switches from ODMs are now also available from traditional network vendors, as they use merchant silicon ASICs.

Landscape Changes

Some data center vendors offer a “Debian” based operating system for network equipment. Their philosophy is that engineers should manage switches just like they manage servers with the ability to use existing server administration tools. They want networking to work as a server application. For example, Cumulus has created the first full-featured Linux distribution for network hardware. It allows designers to break free from proprietary networking equipment and utilize the advantages of the SDN Data Center.

Issues with Traditional Networking

Cloud computing, distributed storage, and virtualization technologies are changing the operational landscape. Traditional networking concepts do not align with new requirements and continually act as blockers to business enablers. Decoupling hardware/software is required to keep pace with the innovation needed to meet the speeds and agility of cloud deployments and emerging technologies.

Before you proceed, you may find the following helpful:

1. LISP Hybrid Cloud
2. Modular Building Blocks
3. Virtual Switch
4. Overlay Virtual Networks
5. Virtual Data Center Design

Back to basic with Merchant silicon

Merchant silicon is a term used to describe chips. Usually, ASICs (Application-Specific Integrated Circuits) are developed by an entity, not the company selling the switches. Then, we have custom silicon, which is the opposite of Merchant Silicon. Custom silicon is a term used to describe chips, usually ASICs, that are custom-designed and traditionally built by the company selling the switches in which they are used.

Benefits of Merchant Silicon:

1. Cost-Effectiveness: One of merchant silicon’s primary advantages is its cost-effectiveness. Since these chips are mass-produced, they are available at a lower cost than custom chips. This allows networking equipment manufacturers to deliver high-performance solutions at a more affordable price, making networking technology more accessible to a broader audience.

2. Flexibility and Innovation: Merchant silicon allows network equipment manufacturers to choose the best chipset. They can select chips from various vendors, offering different features and capabilities. This enables manufacturers to innovate and differentiate their products, creating a more diverse and competitive networking landscape.

3. Time-to-Market: Developing custom chips can be a time-consuming process. By leveraging merchant silicon, networking equipment manufacturers can significantly reduce their time-to-market, as they can quickly integrate pre-existing, tested chips into their products. This allows them to bring new networking solutions to market faster, meeting the ever-increasing demands of the industry.

Impact on the Networking Industry:

Merchant silicon has profoundly impacted the networking industry, transforming how networks are built and operated. Here are some key areas where merchant silicon has made a difference:

1. Performance and Scalability: With the advancements in merchant silicon, networking equipment manufacturers can now deliver higher performance and scalability in their products. These chips offer greater processing power, faster data rates, and improved packet forwarding capabilities, enabling networks to handle more traffic and meet the growing demands of bandwidth-intensive applications.

2. Openness and Interoperability: Merchant silicon promotes openness and interoperability in networking. Since network equipment manufacturers are not tied to proprietary chipsets, they can build solutions that adhere to industry standards and work seamlessly with equipment from different vendors. This fosters a more open, collaborative networking ecosystem where interoperability and compatibility are prioritized.

3. Innovation and Differentiation: By leveraging merchant silicon, networking equipment manufacturers can focus on developing innovative software solutions and features that differentiate their products in the market. This has led to new technologies, such as software-defined networking (SDN) and network function virtualization (NFV), revolutionizing how networks are designed, managed, and optimized.

Disaggregation Model

Disaggregation is the next logical evolution in data center topologies. Cumulus does not reinvent all the wheels; they believe that routing and bridging work well, with no reason to change them. Instead, they use existing protocols to build on the original networking concept base. The technologies they offer are based on well-designed current feature sets. Their O/S enables dis-aggregation of switching design to the server hardware/software disaggregation model.

Disaggregation decouples hardware/software on individual network elements. Modern networking equipment is proprietary today, making it expensive and complicated to manage. Disaggregation allows designers to break free from vertically integrated networking gear. It also allows you to separate the procurement decisions around hardware and software.

Data center topology types and merchant silicon

Previously, we needed proprietary hardware to provide networking functionality. Now, the hardware allows many of those functions in “merchant silicon.” In the last ten years, we have seen a massive increase in the production of merchant silicon. Merchant silicon is a term used to describe the use of “off-the-shelf” chip components to create a network product enabling open networking. Currently, three major players for 10GbE and 40GbE switch ASIC are Broadcom, Fulcrum, and Fujitsu.

In addition, cumulus supports the Broadcom Trident II ASIC switch silicon, also used in the Cisco Nexus 9000 series. Merchant silicon’s price/performance ratio is far better than proprietary ASIC. 

Routing isn’t broken – Simple building blocks.

To disaggregate networking, we must first simplify it. Networking is complicated. Sometimes, less is more. Building robust ecosystems using simple building blocks with existing layer 2 and layer 3 protocols is possible. Internet Protocol (IP) is the underlying base technology and the basis for every large data center. MPLS is an attractive, helpful alternative, but IP is a mature building block today. IP is based on a standard technique, unlike Multichassis Link Aggregation (MLAG), which is vendor-specific.

Multichassis Link Aggregation (MLAG) implementation

Each vendor has various MLAG variations; some operate with unified and separate control planes. MLAG offers suitable control planes: Juniper with Virtual Chassis, HP with Intelligent Resilient Framework (IRF), Cisco Virtual Switching System, and cross-stack EtherChannel. MLAG, with separate control planes, includes Cisco Virtual Port-Channel (vPC) and Arista MLAG.

With all the vendors out there, we have no standard for MLAG. Where specific VLANs can be isolated to particular ToRs, Layer 3 is a preferred alternative. Cumulus Multichassis Link Aggregation (MLAG) implementation is an MLAG daemon written in python.

The specific implementation of how the MLAG gets translated to the hardware is ASIC independent, so in theory, you could run MLAG between two boxes that are not running the same chipset. Similar to other vendor MLAG implementations, it is limited to two spine switches. If you require anything to scale, move to IP. The beauty of IP is that you can do much stuff without relying on proprietary technologies.

Data center topology types: A design for simple failures

Everyone building networks at scale is building them as a loosely coupled system. People are not trying to over-engineer and build exact systems. High-performance clusters are excellent applications and must be made a certain way. A general-purpose cloud is not built that way. Operators build “generic” applications over “generic” infrastructure. Designing and engineering networks with simple building blocks leads to simpler designs with simple failures. Over-engineering networks experience complex failures that are time-consuming to troubleshoot. When things fail, they should fail.

Building blocks should be constructed with straightforward rules. Designers understand you can build extensive networks with simple rules and building blocks. For example, analyzing Spine Leaf architecture looks complicated. But in terms of the networking fabric, the Cumulus ecosystem is made of a straightforward building block – fixed form-factor switches. It makes failures very simple.

On the other hand, if the chassis base switch fails, you need to troubleshoot many aspects. Did the line card not connect to the backplane? Is the backplane failing? All these troubleshooting steps add complexity. With the disaggregated model, when networks fail, they fail in simple ways. Nobody wants to troubleshoot a network when it is down. Cumulus tries to keep the base infrastructure simple and not complement every tool and technology.

For example, if you use Layer 2, MLAG is your only topology. STP is simply a fail-stop mechanism and is not used as a high convergence mechanism. Rapid Spanning Tree Protocol (RSTP) and Bridge Protocol Data Units (BPDU) are all you need; you can build straightforward networks with these.

Virtual router redundancy

First Hop Redundancy Protocol (FHRP) now becomes trivial. Cumulus uses Anycast Virtual IP/MAC, eliminating complex FHRP protocols. You do not need a protocol in your MLAG topology to keep your network running. They support a variation of the Virtual Router Redundancy Protocol (VRRP) known as Virtual Router Redundancy (VRR). It’s like VRRP without the protocol and supports an active-active setup. It allows hosts to communicate with redundant routers without dynamic or router protocols.

Merchant silicon has emerged as a driving force in the networking industry, offering cost-effectiveness, flexibility, and faster time-to-market. This technology has enabled networking equipment manufacturers to deliver high-performance solutions, promote interoperability, and drive innovation. As the demand for faster, more reliable networks continues to grow, merchant silicon will play a pivotal role in shaping the future of networking technology.