Hierarchical networks

Organizations have built hierarchical networks in the past decades using hierarchical addressing mechanisms such as the Internet Protocol (IP) or creating and interconnecting multiple network domains. Large bridged domains have always presented a challenge for scaling and fault isolation due to Layer 2 and nonhierarchical address spaces. As endpoint mobility has increased, technologies are needed to build more efficient Layer 2 extensions and reintroduce hierarchies.

The Data Center Interconnect (DCI) technology uses dedicated interconnectivity to restore hierarchy within the data center. Even though DCI can interconnect multiple data centers within a single data center, large fabrics enable borderless endpoint placement and mobility. This trend resulted in an explosion of ARP and MAC entries. VXLAN’s Layer 2 over Layer 3 capabilities were supposed to address this challenge. However, they have only added to it, allowing even larger Layer 2 domains to be built as the location boundary is overcome.

**Spine and Leaf Designs**

The spine and leaf, fat tree, and folded Clos topologies became standard for fabrics. VXLAN, an over-the-top network, flattens out the hierarchy of the new network topology models. With the introduction of the overlay network, the network hierarchy was hidden, even though the underlying topology was predominantly Layer 3, and hierarchies were present. In addition to its benefits, flattening has some drawbacks as well. The simplicity of building a network over the top without touching every switch makes it easy to extend across multiple sites.

As a result, this new overlay networking design presents a risk without failure isolation, especially in large, stretched Layer 2 networks. Whatever is sent through the ingress point to the respective egress point will leave the overlay network. This is done using the “closest to the source” and “closest to the destination” approaches.

With EVPN Multi-Site, overlay networks can maintain hierarchies again. A new version of EVPN Multi-Site for VXLAN BGP EVPN networks introduces external BGP (eBGP), while interior BGP (iBGP) has been the dominant model. The Border Gateways (BGWs) were introduced with autonomous systems (ASs) as a response to eBGP next-hop behavior. Hierarchies are effectively used to classify and connect multiple overlay networks using this approach. Moreover, network extensions within and beyond one data center are controlled and enforced by a control point.

**The Role of Layer 2**

It started as a pure Layer 2 solution and got some Layer 3 functionality pretty early on. Later, it got all the blown IP prefixes, so now you can use EPVN to implement complete Layer 2 and Layer VPNs. EVPN is now considered a mature technology that has been available in multiprotocol label switching (MPLS) networks for some time.

Therefore, many refer to this to it as EVPN over MPLS. When discussing EVPN-MPLS or MPLS EVPN, EPVN still uses Route Distinguisher (RD) and Route Targets (RT).

RD creates separate address spaces, and RT integrates VPN membership. Remember that the precursor to EVPN was Over-the-Top Virtualization (OTV), a proprietary technology invented by Dino Farinacci while working at Cisco. Dino also worked heavily with the LISP protocol.

OTV used Intermediate System–to–Intermediate System (IS-IS) as the control plane and ran over IP networks. IS-IS can build paths for both unicast and multicast routes. 

Data center fabric journey

Spanning Tree and Virtual PortChannel

We have evolved data center networks over the past several years. Spanning Tree Protocol (STP)–-based networks served network requirements for several years. Virtual PortChannel (vPC) was introduced to address some of the drawbacks of STP networks while providing dual-homing abilities. Subsequently, overlay technologies such as FabricPath and TRILL came to the forefront, introducing routed Layer 2 networks with a MAC-in-MAC overlay encapsulation. This evolved into a MAC-in-IP overlay with the invention of VXLAN.

While Layer 2 networks evolved beyond the loop-free topologies with STP, the first-hop gateway functions for Layer 3 also became more sophisticated. The traditional centralized gateways hosted at the distribution or aggregation layers have transitioned to distributed gateway implementations, which has allowed for scaling out and the removal of choke points.

Cisco FabricPath is a MAC-in-MAC

Cisco FabricPath is a MAC-in-MAC encapsulation that eliminates the use of STP in Layer 2 networks. Instead, it uses Layer 2 Intermediate System to Intermediate System (IS-IS) with appropriate extensions to distribute the topology information among the network switches. In this way, switches behave like routers, building switch reachability tables and inheriting all the advantages of Layer 3 strategies such as ECMP. In addition, no unused links exist in this scenario, while optimal forwarding between any pair of switches is promoted.

The rise of VXLAN

While FabricPath has been immensely popular and adopted by thousands of customers, it has faced skepticism because it is associated with a single vendor, Cisco, and lacks multivendor support. In addition, with IP being the de facto standard in the networking industry, an IP-based overlay encapsulation was pushed. As a result, VXLAN was introduced. VXLAN, a MAC-in-IP/UDP encapsulation, is currently the most popular overlay encapsulation.

As an open standard, it has received widespread adoption from networking vendors. Just like FabricPath, VXLAN addresses all the STP limitations previously described. However, with VXLAN, a 24-bit number identifies a virtual network segment, thereby allowing support for up to 16 million broadcast domains as opposed to the traditional 4K limitations imposed by VLANs.

EVPN MPLS: History

Layer 3 VPNs and MPLS

In the late 1990s, we witnessed the introduction of Layer 3 VPNs and Multiprotocol Label Switching (MPLS). Layer 3 VPNs distribute IP prefixes with a control plane, offering any connectivity. So, we have MPLS VPN with PE and CE routers, and EVPN still uses these devices. MPLS also has RD and RT to create different address spaces.

This is also used in EVPN. Layer 3 VPN needed MPLS encapsulation. This signaling was done with LDP; you can use segment routing today. MPLS L3 VPN supports a range of topologies that can be created with Route Targets. Some of which led to complex design scenarios.

Layer 2 VPNs and VPLS

Layer 2 VPNs arrived more humbly with a standard point-to-point connectivity model using Frame Relay, ATM, and Ethernet. Finally, in the early 2000s, pseudowires and layer 2 VPNs arrived. Each of these VPN services operates on different VPN connections, with few working on a Level 3 or MPLS connection. Point-to-point connectivity models no longer satisfied all designs, and services required multipoint Ethernet connectivity.

As a result, Virtual Private LAN Service (VPLS) was introduced. Virtual Private LAN Service (VPLS) is an example of L2VPN and has many drawbacks with using pseudowires to create the topology. A mesh of pseudowires with little control plane leads to much complexity.

VPLS with data plane learning

VPLS offered a data plane learning solution that could emulate a bridge and provide multipoint connectivity for Ethernet stations. It was widely deployed but had many shortcomings, such as support for multi-homing, BUM (BUM = Broadcast, Unknown unicast, and Multicast) optimization, flow-based load balancing, and multipathing. So, EVPN was born to answer this problem.

In the last few years, we have entered a different era of data center architecture with other requirements. For example, we need efficient Layer 2 multipoint connectivity, active-active flows, and better multi-homing capability. Unfortunately, the shortcomings of existing data plane solutions hinder these requirements.

EVPN MPLS: Multi-Homing With Per-Flow Capabilities 

Some data centers require Layer 2 DCI (data center interconnect) and active-active flows between locations. Current L2 VPN technologies do not fully address these DCI requirements. A DCI with better multi-homing capability was needed without compromising network convergence and forwarding. Per-flow redundancy and proper load balancing introduced a BGP MPLS-based Ethernet VPN (EVPN) solution.

**No more pseudowires**

With EVPN, pseudowires are no longer needed. All the hard work is done with BGP. A significant benefit of EVPN operations is that MAC learning between PEs occurs not in the data plane but in the control plane (unlike VPLS). It utilizes a hybrid control/data plane model. First, data plane address learning occurs in the access layer.

This would be the CE to PE link in an SP model using IEEE 802.1x, LLDP, or ARP. Then, we have control-plane address advertisements / learning over the MPLS core. The PEs run MP-BGP to advertise and learn customer MAC addresses. EVPN has many capabilities, and its use case is extended to act as the control plane for open standard VXLAN overlays.

L2 VPN challenges

There are several challenges with traditional Layer 2 VPNs. They do not offer an ALL-active per-flow redundancy model, traffic can loop between PEs, MAC flip-flopping may occur, and there is the duplication of BUM traffic (BUM = Broadcast, Unknown unicast, and Multicast).

In the diagram below, a CE has an Ethernet bundle terminating on two PEs: PE1 and PE2. The problem with the pseudowires VPLS data plane learning approach is that PE1 receives traffic on one of the bundle member links. The traffic is sent over the full mesh of PW and eventually learned by PE2. PE2 cannot know if traffic originated on CE1, and PE2 will return it. CEs also get duplicated BUM traffic.

Another challenge with VPLS and L2 VPN is MAC Flip-flopping over pseudowires. Like the above, you have dual-homed CEs sending traffic from the same MAC but with a different IP address. Now, you have MAC address learning by PE1 and forwarded to the remote PE3. PE3 learns that the MAC address is via PE1, but the same MAC with a different flow can arrive via PE2.

PE3 learns the same MAC over the different links, so it keeps flipping the MAC learning from one link to another. All these problems are forcing us to move to a control plane Layer 2 VPN solution – EVPN.

What Is EVPN

EVPN operates with the same principles and operational experiences as Layer 3 VPNs, such as MP-BGP, route targets (RT), and route distinguishers (RD). EVPN takes BGP, puts a Layer 2 address in it, and advertises as if it were a Layer 3 destination with an MPLS rewrite or MPLS tag as the rewritten header or as the next hop.

It enables the routing of Layer 2 addresses through MP-BGP. Instead of encapsulating an Ethernet frame in IPv4, a MAC address with MPLS tags is sent across the core.

The MPLS core swaps labels as usual and thinks it is another IPv4 packet. This is conceptually similar to IPv6 transportation across an IPv4 LDP core, a feature known as 6PE.

EVPN MPLS: Layer 3 principles apply

All Layer 3 principles apply, allowing you to prepend MAC addresses with RDs to make them unique and permitting overlapping addresses for Layer 2. RTS offers separation, allowing constraints on flooding to interested segments. EVPN gives all your policies with BGP – LP, MED, etc., enabling efficient MAC address flooding control. EVPN is more efficient on your BGP tables; you can control the distribution of the MAC address to the edge of your network.

You control where the MAC addresses are going and where the state is being pushed. It’s a lot simpler than VPLS. You look at the destination MAC address at the network edge and shove a label on it. EVPN has many capabilities. Not only do we use BGP to advertise the reachability of MAC addresses and Ethernet segments, but it may also advertise MAC-to-IP correlation. BGP can provide information that hosts A has this IP and MAC address.

VXLAN & EVPN control plane

Datacenter fabrics started with STP, which is the only thing you can do at Layer 2. Its primary deficiency was that you could only have one active link. We later introduced VPC and VSS, allowing all link forwarding in a non-looped topology. Cisco FabricPath / BGP introduces MAC-in-MAC layer 2 multipathing.

In the gateway area, they added Anycast HSRP, which was limited to 4 gateways. More importantly, they exchanged states.

The industry is moving on, and we now see the introduction of VXLAN as a MAC in IP mechanism. VXLAN allows us to cross a layer 3 boundary and build an overlay over a layer 3 network. Its initial forwarding mechanism was flood and learn, but it had many drawbacks. So now, they have added a control plane to VXLAN—EVPN.

A VXLAN/EVPN solution is an MP-BGP-based control plane using the EVPN NLRI. BGP carries out Layer-2 MAC and Layer-3 IP information distribution. It reduces flooding as forwarding decisions are based on the control plane. The VPN control plane offers VTEP peer discovery and end-host reachability information distribution.