LEGOLAND – Cumulus Networks
Cumulus Networks offer a “debian” based operating system for network equipment. Their philosophy is that engineers should manage switches just like they manage servers with the ability to use existing server administration tools. They want networking to work like a server application. Cumulus has created the first full-featured Linux distribution for network hardware. It allows designers to break free from proprietary networking equipment and utilize the advantages of Software Defined Data Centre. Technologies such as cloud computing, distributed storage, and virtualization are changing the operational landscape. Traditional networking concepts are not in line with new requirements and continually act as blockers to business enablers. Decoupling hardware/software is required to keep pace with innovation needed to meet the speeds and agility for cloud deployments and emerging technologies.
Disaggregation is the next logical evolution is networking. Cumulus do not reinvent all the wheels, they believe that routing and bridging work very well with no reason to change them. They build on the original base of networking concepts and use existing protocols. The technologies they offer are based on well-designed existing feature sets. Their O/S enables dis-aggregation of switching design to the model of server hardware/software disaggregation. Dis-aggregation decouple hardware/software on individual network elements. Today modern networking equipment is proprietary, which makes is expensive and hard to manage. Disaggregation allows designers to break free from vertically integrated networking gear. It also allows you to separate the procurement decisions around hardware and software.
Previously, we needed proprietary hardware to provide networking functionality. Now, hardware exists that allows a lot of those functions to be done in what’s called “merchant silicon”. In the last 10 years, we have seen a massive increase in production of merchant silicon. Merchant silicon is a term used to describe the use of “off the shelf” chip components to create a network product. There are currently three major players for 10GbE and 40GbE switch ASIC: Broadcom, Fulcrum, and Fujitsu. Cumulus support the Broadcom Trident II ASIC switch silicon, which is also, used in the Cisco Nexus 9000 series. The price/performance ratio for merchant silicon is far better than proprietary ASIC.
Routing isn’t broken – Simple Building Blocks
In order to disaggregated networking, we must first simplify it. Networking is complicated. Sometimes less is actually more. It’s possible to build very powerful ecosystems using very simple building blocks with existing layer 2 and layer 3 protocols. Internet Protocol (IP) is the underlying base technology and the basis for every large data centre. MPLS is an interesting useful alternative but today IP is a mature building block. IP is based on a standard technique, not like Multichassis Link Aggregation (MLAG), which is vendor specific. There is a variety of MLAG variations for each vendor and some operate with unified and separate control planes. MLAG offering unified control plane are Juniper with Virtual Chassis, HP with Intelligent Resilient Framework (IRF), Cisco Virtual Switching System and cross-stack EtherChannel. MLAG with separate control planes includes Cisco Virtual Port-Channel (vPC) and Arista MLAG. With all the vendors out there, we have no standard for MLAG. Where specific VLAN’s can be isolated to specific ToR, Layer 3 is a preferred alternative. Cumulus Multichassis Link Aggregation (MLAG) implementation is an MLAG daemon written in python. The specific implementation of how the MLAG gets translated to the hardware is ASIC independent so in theory you could run MLAG between two boxes that are not running the same chipset. Similar to other vendor MLAG implementations, limited to two spine switches. If you require anything to scale, move to IP. The beauty of IP is that you can do a lot of stuff without relying on proprietary technologies.
Design for Simple Failures
Everyone that is building networks at scale is building them as a loosely simple coupled system. People are not trying to over engineer and build very precise systems. High-performance clusters are a nice application and it needs to be built a certain way. General purpose cloud is not built that way. Operators build “generic” applications over “generic” infrastructure. When you design and engineer networks with simple building blocks, it leads to simpler designs with simple failures. Over engineering networks experience complex failures that are time-consuming to troubleshoot. When things fail they should fail simply.
Building blocks should be constructed with very simple rules. Designers understand that with simple rules and building blocks you can build very large networks. For example, analyse leaf and spin topology, it looks complication. But in terms of the networking fabric Cumulus ecosystem is made out of one simple building block – fixed form factor switches. It makes failures very simple. On the other hand if chassis base switch fails, you need to troubleshoot many aspects. Did the line card not connect to backplane? is backplane failing? All these troubleshooting steps add complexity. With the disaggregated model when networks fail they fail in simple ways. Nobody wants to troubleshoot a network when down. Cumulus tries to keep the base infrastructure simple and not to complement every type of tool and technology. For example, if you are using Layer 2, MLAG is your only topology. STP is simply a fail stop mechanism and not used as a high convergence mechanism. Rapid Spanning Tree Protocol (RSTP) and Bridge Protocol Data Units (BPDU) is all you need and with these you can build very simple networks.
Virtual Router Redundancy
First Hop Redundancy Protocol (FHRP) now becomes trivial. Cumulus use Anycast Virtual IP/MAC, eliminating complex FHRP protocols. You do not need a protocol in your MLAG topology to keep your network running. They support a variation of Virtual Router Redundancy Protocol (VRRP) known as Virtual Router Redundancy (VRR). It’s like VRRP without the protocol and supports active – active setup. It allows hosts to communicate with redundant routers without any dynamic router protocols or router redundant protocols.