fabricpath design

Data Center Fabric

 

fabricpath design

 

Data Center Fabric

In today’s digital age, where vast amounts of data are generated and processed, data centers play a vital role in ensuring seamless and efficient operations. At the heart of these data centers lies the concept of data center fabric – a sophisticated infrastructure that forms the backbone of modern computing. In this blog post, we will delve into the intricacies of data center fabric, exploring its importance, components, and benefits.

Data center fabric refers to the underlying architecture and interconnectivity of networking resources within a data center. It is designed to efficiently handle data traffic between various components, such as servers, storage devices, and switches while ensuring high performance, scalability, and reliability. Think of it as the circulatory system of a data center, facilitating the flow of data and enabling seamless communication between different entities.

 

  • Example: Data Center Fabric – FabricPath

Network devices are deployed in highly interconnected layers, represented as a fabric. Unlike traditional multitier architectures, a data center fabric effectively flattens the network architecture, reducing the distance between endpoints within the data center. An example of a data center fabric is FabricPath.

Cisco has validated FabricPath as an Intra-DC Layer 2 multipath technology. Design cases are also available where FabricPath is deployed for DCI ( Data Center Interconnect ). Regarding a FabricPath DCI option, design carefully over short distances with reliable interconnects, for example, Dark Fiber or Protected Dense Wavelength Division Multiplexing (DWDM ).

FabricPath designs are suitable for a range of topologies, and unlike hierarchical virtual Port Channel ( vPC ) designs, FabricPath does not need to follow any topology. It can accommodate any design type; full mesh, partial mesh, hub, and spoke topologies.

 

  • Example: Data Center Fabric – Cisco ACI 

ACI Cisco is a software-defined networking (SDN) architecture that brings automation and policy-driven application profiles to data centers. By decoupling network hardware and software, ACI provides a flexible and scalable infrastructure to meet dynamic business requirements. It enables businesses to move from traditional, manual network configurations to a more intuitive and automated approach.

One of the defining features of Cisco ACI is its application-centric approach. It allows IT teams to define policies based on application requirements rather than individual network components. This approach simplifies network management, reduces complexity, and ensures that network resources are aligned with the needs of the applications they support.

 

Before you proceed, you may find the following posts helpful:

  1. What Is FabricPath
  2. Data Center Topologies
  3. ACI Networks
  4. Active Active Data Center Design
  5. Redundant Links

 



Data Center Fabric

Key Data Center Fabric Discussion Points:


  • Introduction to the data center fabric and what is involved.

  • Highlighting the details of FabricPath.

  • Critical points on the possible alternatives to FabricPath.

  • Technical details on the load balancing.

 

Back to basic with a data center fabric.

Key Components of Data Center Fabric:

1. Network Switches: Network switches form the core of the data center fabric, providing connectivity between servers, storage devices, and other networking equipment. These switches are designed to handle massive data traffic, offering high bandwidth and low latency to ensure optimal performance.

2. Cabling Infrastructure: A well-designed cabling infrastructure is crucial for data center fabric. High-speed fiber optic cables are commonly used to connect various components within the data center, ensuring rapid data transmission and minimizing signal loss.

3. Network Virtualization: Network virtualization technologies, such as software-defined networking (SDN), play a significant role in the data center fabric. By decoupling the network control plane from the physical infrastructure, SDN enables centralized management, improved agility, and flexibility in allocating resources within the data center fabric.

 

Flattening the network architecture

In this current data center network design, network devices are deployed in two interconnected layers, representing a fabric. Sometimes, massive data centers are interconnected with three layers. Unlike conventional multitier architectures, a data center fabric flattens the network architecture, reducing the distance between endpoints within the data center. This design results in high efficiency and low latency. Very well suited for east-to-west traffic flows.

Data center fabrics provide a solid layer of connectivity in the physical network, and move the complexity of delivering use cases for network virtualization, segmentation, stretched Ethernet segments, workload mobility, and various other services to an overlay that rides on top of the fabric.

When paired with an overlay, the fabric itself is called the underlay. The overlay could be deployed with, for example, VXLAN. To gain network visibility into user traffic, you would examine the overlay, and the underlay is used to route traffic between the overlay endpoints.

VXLAN, short for Virtual Extensible LAN, is a network virtualization technology that enables the creation of virtual networks over an existing physical network infrastructure. It provides a scalable and flexible approach to address the challenges posed by traditional VLANs, such as limited scalability, spanning domain constraints, and the need for manual configuration.

 

  • A key point: Lab guide on overlay networking with VXLAN

The following example shows VLXAN tunnel endpoints on Leaf A and Leaf B. The bridge domain is mapped to a VNI on G3 on both leaf switches. This enables a Layer 2 overlay for the two hosts to communicate. This VXLAN overlay goes across Spine A and Spine B.

Take note that the Spine layer, which acts as the core network, which would be a WAN network or any other type of Routed Layer 3 network, does not have any VXLAN configuration. We have flatted the network while providing Layer 2 connectivity over a routed core.

VXLAN overlay
Diagram: VXLAN Overlay

 

Fabricpath Design: Problem Statement

Key Features of Cisco Fabric Path:

Transparent Interconnection: Cisco Fabric Path allows for creating a multi-path forwarding infrastructure that provides transparent Layer 2 connectivity between devices within a network. This enables the efficient utilization of available bandwidth and simplifies network design.

Scalability: With Cisco Fabric Path, organizations can quickly scale their network infrastructure to accommodate growing data loads. It supports up to 16 million virtual network segments, enabling seamless expansion of network resources without compromising performance.

Fault Tolerance: Cisco Fabric Path incorporates advanced fault-tolerant mechanisms like loop-free topology and equal-cost multipath routing. These features ensure high availability and resiliency, minimizing the impact of network failures and disruptions.

2.4 Traffic Optimization: Cisco Fabric Path employs intelligent load-balancing techniques to distribute traffic across multiple paths, optimizing network utilization and reducing congestion. This results in improved application performance and enhanced user experience.

The problem with traditional classical Ethernet is the flooding behavior of unknown unicasts and broadcasts and the process of MAC learning. All switches must learn all MAC addresses, leading to inefficient resource use. In addition, Ethernet has no Time-to-Live ( TTL ) value, and if precautions are not in place, it could cause an infinite loop.

data center fabric
Diagram: The need for a data center fabric

 

Deploying Spanning Tree Protocol ( STP ) at Layer 2 blocks loops, but STP has many known limitations. One of its most significant flaws is that it offers a single topology for all traffic with one active forwarding path. Scaling the data center with classical Ethernet and spanning trees is inefficient as it blocks all but one path. With spanning trees’ default behavior, the benefits of adding extra spines do not influence bandwidth or scalability.

Possible alternatives

  • Multichassis EtherChannel 

To overcome these limitations, Cisco introduced Multichassis EtherChannel ( MEC ). MEC comes in two flavors; Virtual Switching System ( VSS ) with Catalyst 6500 series or Virtual Port Channel ( vPC ) with Nexus Series. Both offer active/active forwarding but present scalability challenges when scaling out Spine / Core layers. Additionally, complexity increases when deploying additional spines.

 

  • Multiprotocol Label Switching 

Another option would be to scale out with Multiprotocol Label Switching ( MPLS ). Replace Layer 2 switching with Layer 3 forwarding and MPLS with Layer 2 pseudowires. This type of complexity would lead to an operational nightmare. The prevalent option is to deploy Layer 2 multipath with THRILL or FabricPath. In intra-DC communication, Layer 2 and Layer 3 designs are possible in two forms; Traditional DC design and Switched DC design.

FabricPath VLANs use Conversational Learning, meaning a subset of MAC addresses is learned at the network’s edge. Conversation learning consists of a three-way handshake. Each interface learns the MAC addresses of interested hosts. Compared to classical Ethernet, each switch device learns all MAC addresses for that VLAN.

  1. Traditional DC design replaces hierarchical vPC and STP with FabricPath. Core, distribution, and access elements stay the same. The same layered hierarchical model exists but with FabricPath in the core.
  2. Switched DC design based on Clos Fabrics. Integrate additional Spines for Layer 2 and Layer 3 forwarding.

 

Traditional data center design

what is data center fabric
Diagram: what is data center fabric

 

Fabric Path in the core replaces vPC. It still uses port channels, but the hierarchical vPC technology previously used to provide active/active forwarding is not required. Instead, designs are based on modular units called PODs; within each POD, traditional DC technologies exist, for example, vPC. Active/active ( dual-active paths ) forwarding based on a two-node Spine, Hot Standby Router Protocol ( HSRP ), announces the virtual MAC of the emulated switch from each of the two cores. For this to work, implement vPC+ on the inter-spine peer links.

 

Switched data center design

Switched Fabric Data Center
Diagram: Switched Fabric Data Center

 

Each edge node has equidistant endpoints to each other, offering predictable network characteristics. From FabricPath’s outlook, the entire Spine Layer is one large Fabric-based POD. In the traditional model presented above, port and MAC address capacity are key factors influencing the ability to scale out. The key advantage of Clos-type architecture is that it expands the overall port and bandwidth capacity within each POD.

Implementing load balancing 4 wide spines challenges traditional First Hop Redundancy Protocol ( FHRP ) like HSRP, which works with 2 active pairs by default. Implementing load balancing 4 wide spines with VLANs allowed on certain links is possible but can cause link polarization

For optimized designs, utilize a redundancy protocol to work with a 4-node gateway. Deploy Gateway Load Balancing Protocol ( GLBP ) and Anycast FHRP. GLBP uses a weighting parameter that allows Address Resolution Protocol ( ARP ) requests to be answered by MAC addresses pointing to different routers. Anycast FHRP is the recommended solution for designs with 4 or more spine nodes.

 

FabricPath Key Points:

  • FabricPath removes the requirement for a spanning tree and offers a more flexible and scalable design to its vPC-based Layer 2 alternative. No requirement for a spanning tree, enabling Equal Cost Multipath ( ECMP ).

  • FabricPath no longer forwards using spanning tree. Offering designers bi-sectional bandwidth and up to 16-way ECMP. 16 x 10Gbps links equate to 2.56 terabits per second between switches.

  • Data Centers with FabricPath are easy to extend and scale.

  • Layer 2 troubleshooting tools for FabricPath including FabricPath PING and Traceroute can now test multiple equal paths.

  • Control plane based on Intermediate System-to-Intermediate System ( IS-IS ).

  • Loop prevention is now in the data plane based on the TTL field.

 

Multicast VXLAN

Data Center Network Design

Data Center Network Design

Data centers are crucial in today’s digital landscape, serving as the backbone of numerous businesses and organizations. A well-designed data center network ensures optimal performance, scalability, and reliability. This blog post will explore the critical aspects of data center network design and its significance in modern IT infrastructure.

Efficient data center network design is critical for meeting the growing demands of complex applications, high data traffic, and rapid data processing. It enables seamless connectivity, improves application performance, and enhances user experience. A well-designed network also ensures data security, disaster recovery, and efficient resource utilization.

Table of Contents

Highlights: Data Center Network Design

The goal of data center design and interconnection network is to transport end-user traffic from A to B without any packet drops, yet the metrics we use to achieve this goal can be very different. The data center is evolving and progressing through various topology and technology changes, resulting in various data center network designs.

The new data center control plane we are seeing today, such as Fabric Path, LISP, THRILL, and VXLAN, is being driven by a change in the end user’s requirement; the application has changed.

These new technologies may address new challenges, yet the fundamental question of where to create the Layer 2/Layer 3 boundary and the need for Layer 2 in the access layer remains the same. The question stays the same, yet the technologies available to address this challenge have evolved.

The use of Open Networking

We also have the Open Networking Foundation ( ONF ) with open networking. Open networking describes a network that uses open standards and commodity hardware. So, consider open networking in terms of hardware and software. Unlike a vendor approach like Cisco, this gives you much more choice with what hardware and software you use to make up and design your network.

Related: Before you proceed, you may find the following useful:

  1. ACI Networks
  2. IPv6 Attacks
  3. SDN Data Center
  4. Active Active Data Center Design
  5. Virtual Switch

Data Center Control Plane

Key Data Center Network Design Discussion Points:


  • Introduction to data center network design and what is involved.

  • Highlighting the details of VLANs and virtualization.

  • Technical details on the issues of Layer 2 in data centers. 

  • Scenario: Cisco FabricPath and DFA.

  • Details on overlay networking and Cisco OTV.

The Rise of Overlay Networking

What has the industry introduced to overcome these limitations and address the new challenges? – Network virtualization and overlay networking. In its simplest form, an overlay is a dynamic tunnel between two endpoints that enables Layer 2 frames to be transported between them. In addition, these overlay-based technologies provide a level of indirection that enables switch table sizes to not increase in the order of the number of supported end-hosts.

Today’s overlays are Cisco FabricPath, THRILL, LISP, VXLAN, NVGRE, OTV, PBB, and Shorted Path Bridging. They are essentially virtual networks that sit on top of a physical network, often the physical network not being aware of the virtual layer above it.

Lab Guide: VXLAN

The following lab guide displays a VXLAN network. We are running VXLAN in multicast mode. Multicast VXLAN is a variant of VXLAN that utilizes multicast-based IP multicast for transmitting overlay network traffic. VXLAN is an encapsulation protocol that extends Layer 2 Ethernet networks over Layer 3 IP networks.

Linking multicast enables efficient and scalable communication within the overlay network. Notice the multicast group of 239.0.0.10 and the route of 239.0.0.10 forwarding out the tunnel interface. We have multicast enabled on all Layer 3 interfaces, including the core that consists of Spine A and Spine B.

Multicast VXLAN
Diagram: Multicast VXLAN

Traditional Data Center Network Design

How do routers create a broadcast domain boundary? Firstly, using the traditional core, distribution, and access model, the access layer is layer 2, and servers served to each other in the access layer are in the same IP subnet and VLAN. The same access VLAN will span the access layer switches for east-to-west traffic, and any outbound traffic is via a First Hop Redundancy Protocol ( FHRP ) like Hot Standby Router Protocol ( HSRP ).

Servers in different VLANs are isolated from each other and cannot communicate directly; inter-VLAN communications require a Layer 3 device. Virtualization’s humble beginnings started with VLANs, which were used to segment traffic at Layer 2. It was expected to find single VLANs spanning an entire data center fabric.

 

VLAN and Virtualization

The virtualization side of VLANs comes from two servers physically connected to different switches. Assuming the VLAN spans both switches, the same VLAN can communicate with each server. Each VLAN can be defined as a broadcast domain in a single Ethernet switch or shared among connected switches.

Whenever a switch interface belonging to a VLAN receives a broadcast frame ( destination MAC is ffff.ffff.ffff), the device must forward this frame to all other ports defined in the same VLAN.

This approach is straightforward in design and is almost like a plug-and-play network. The first question is, why not connect everything in the data center into one large Layer 2 broadcast domain? Layer 2 is a plug-and-play network, so why not?

 

The issues of Layer 2

The reason is that there are many scaling issues in large layer 2 networks. Layer 2 networks don’t have controlled / efficient network discovery protocols. Address Resolution Protocol ( ARP ) is used to locate end hosts and uses Broadcasts and Unicast replies. A single host might not generate much traffic, but imagine what would happen if 10,000 hosts were connected to the same broadcast domain. VLANs span an entire data center fabric, which can bring a lot of instability due to loops and broadcast storms.

 

No hierarchy in MAC addresses

There is also no hierarchy in MAC addressing. Unlike Layer 3 networks, where you can have summarization and hierarchy addressing, MAC addresses are flat. Creating several thousand hosts to a single broadcast domain will create large forwarding information tables.

Because end hosts are potentially not static, they are likely to be attached and removed from the network at regular intervals, creating a high rate of change in the control plane. You can, of course, have a large Layer 2 data center with multiple tenants if they don’t need to communicate with each.

The shared services requirements, such as WAAS or load balancing, can be solved by spinning up the service VM in the tenant’s Layer 2 broadcast domain. This design will hit scaling and management issues. There is a consensus to move from a Layer 2 design to a more robust and scalable Layer 3 design.

But why is there still a need for Layer 2 in the data center topologies? One solution is Layer 2 VPN with EVPN. But first, let us have a look at Cisco DFA.

The Requirement for Layer 2 in Data Center Network Design

  • Servers that perform the same function might need to communicate with each other due to a clustering protocol or simply as part of the application’s inner functions. If the communication is clustering protocol heartbeats or some server-to-server application packets that are not routable, then you need this communication layer to be on the same VLAN, i.e., Layer 2 domain, as these types of packets are not routable and don’t understand the IP layer.

  • Stateful devices such as firewalls and load balancers need Layer 2 adjacency as they constantly exchange connection and session state information.

  • Dual-homed servers: Single server with two server NICs and one NIC to each switch will require a layer 2 adjacency if the adapter has a standby interface that uses the same MAC and IP addresses after a failure. In this situation, the active and standby interfaces must be on the same VLAN and use the same default gateway.

  • Suppose your virtualization solutions cannot handle Layer 3 VM mobility. In that case, you may need to stretch VLANs between PODS / Virtual Resource Pools or even data centers so you can move VMs around the data center at Layer 2 ( without changing their IP address ).

 

Data Center Design and Cisco DFA

Cisco went one giant step and recently introduced a data center fabric with Dynamic Fabric Automaton ( DFA ), similar to Juniper QFabric, which offers you both Layer 2 switching and Layer 3 routing at the access layer / ToR. Firstly, they have Fabric Path ( IS-IS for Layer 2 connectivity ) in the core, which gives you optimal Layer 2 forwarding between all the edges.

Then they configure the same Layer 3 address on all the edges, which gives you optimal Layer 3 forwarding across the whole Fabric.

On edge, you can have Layer 3 Leaf switches, for example, the Nexus 6000 series, or integrate with Layer 2-only devices like the Nexus 5500 series or the Nexus 1000v. You can also connect external routers or USC or FEX to the Fabric. In addition to running IS-IS as the data center control plane, DFA uses MP-iBGP, with some Spine nodes being the Route Reflector to exchange IP forwarding information.

Cisco FabricPath

DFA also employs a Cisco FabricPath technique called “Conversational Learning.” The first packet triggers a full RIB lookup, and the subsequent packets are switched in the hardware-implemented switching cache.

This technology provides Layer 2 mobility throughout the data center while providing optimal traffic flow using Layer 3 routing. Cisco commented, “DFA provides a scale-out architecture without congestion points in the network while providing optimized forwarding for all applications.”

Terminating Layer 3 at the access / ToR has clear advantages and disadvantages. Other benefits include reducing the size of the broadcast domain, which comes at the cost of reducing the mobility domain across which VMs can be moved.

Terminating Layer 3 at the accesses can also result in sub-optimal routing because there will be hair pinning or traffic tromboning of across-subnet traffic, taking multiple and unnecessary hops across the data center fabric.

 

The role of the Cisco Fabricpath

Cisco FabricPath is a Layer 2 technology that provides Layer 3 benefits, such as multipathing the classical Layer 2 networks using IS-IS at Layer 2. This eliminates the need for spanning tree protocol, avoiding the pitfalls of having large Layer 2 networks. As a result, Fabric Path enables a massive Layer 2 network that supports multipath ( ECMP ). THRILL is an IEEE standard that, like Fabric Path, is a Layer 2 technology that provides the same Layer 3 benefits as Cisco FabricPath to the Layer 2 networks using IS-IS.

LISP is popular in Active / Active data centers for DCI route optimization/mobility and separates the host’s location and the identifier ( EID ), allowing VMs to move across subnet boundaries while keeping the endpoint identification. LISP is often referred to as an Internet locator. 

That can enable some designs of triangular routing. Popular encapsulation formats include VXLAN ( proposed by Cisco and VMware ) and STT (created by Nicira but will be deprecated over time as VXLAN comes to dominate ).

 

Video: LISP networking

In the following video, we will demonstrate the use of LISP in networking. It’s a hands-on demonstration that goes through the various components of a LISP network and how each component operates.

Hands on Video Series – Enterprise Networking | LISP Configuration Intro
Prev 1 of 1 Next
Prev 1 of 1 Next

The role of OTV

OTV is a data center interconnect ( DCI ) technology enabling Layer 2 extension across data center sites. While Fabric Path can be a DCI technology over short distances with dark fiber, OTV has been explicitly designed for DCI. In contrast, the Fabric Path data center control plane is primarily used for intra-DC communications.

Failure boundary and site independence are preserved in OTV networks because OTV uses a data center control plane protocol to sync MAC addresses between sites and prevent unknown unicast floods. In addition, recent IOS versions can allow unknown unicast floods for certain VLANs, which are unavailable if you use Fabric Path as the DCI technology.

 

The Role of Software-defined Networking (SDN)

Another potential trade-off between data center control plane scaling, Layer 2 VM mobility, and optimal ingress/egress traffic flow would be software-defined networking ( SDN ). At a basic level, SDN can create direct paths through the network fabric to isolate private networks effectively.

An SDN network allows you to choose the correct forwarding information on a per-flow basis. This per-flow optimization eliminates VLAN separation in the data center fabric. Instead of using VLANs to enforce traffic separation, the SDN controller has a set of policies allowing traffic to be forwarded from a particular source to a destination.

The ACI Cisco borrows concepts of SDN to the data center. It operates over a leaf and spine design and traditional routing protocols such as BGP and IS-IS. However, it brings a new way to manage the data center with new constructs such as Endpoint Groups (EPGs). In addition, no more VLANs are needed in the data center as everything is routed over a Layer 3 core, with VXLAN as the overlay protocol.

Summary: Recap on Data Center Design

Data centers are the backbone of modern technology infrastructure, providing the foundation for storing, processing, and transmitting vast amounts of data. A critical aspect of data center design is the network architecture, which ensures efficient and reliable data transmission within and outside the facility.  1. Scalability and Flexibility

One of the primary goals of data center network design is to accommodate the ever-increasing demand for data processing and storage. Scalability ensures the network can grow seamlessly as the data center expands. This involves designing a network supporting many devices, servers, and users without compromising performance or reliability. Additionally, flexibility is essential to adapt to changing business requirements and technological advancements.

  • Redundancy and High Availability

Data centers must ensure uninterrupted access to data and services, making redundancy and high availability critical for network design. Redundancy involves duplicating essential components, such as switches, routers, and links, to eliminate single points of failure. This ensures that if one component fails, there are alternative paths for data transmission, minimizing downtime and maintaining uninterrupted operations. High availability further enhances reliability by providing automatic failover mechanisms and real-time monitoring to detect and address network issues promptly.

  • Traffic Optimization and Load Balancing

Efficient data flow within a data center is vital to prevent network congestion and bottlenecks. Traffic optimization techniques, such as Quality of Service (QoS) and traffic prioritization, can be implemented to ensure that critical applications and services receive the necessary bandwidth and resources. Load balancing is crucial in distributing network traffic evenly across multiple servers or paths, preventing overutilization of specific resources and optimizing performance.

  • Security and Data Protection

Data centers house sensitive information and mission-critical applications, making security a top priority. The network design should incorporate robust security measures, including firewalls, intrusion detection systems, and encryption protocols, to safeguard data from unauthorized access and cyber threats. Data protection mechanisms, such as backups, replication, and disaster recovery plans, should also be integrated into the network design to ensure data integrity and availability.

  • Monitoring and Management

Proactive monitoring and effective management are essential for maintaining optimal network performance and addressing potential issues promptly. The network design should include comprehensive monitoring tools and centralized management systems that provide real-time visibility into network traffic, performance metrics, and security events. This enables administrators to promptly identify and resolve network bottlenecks, security breaches, and performance degradation.

Data center network design is critical in ensuring efficient, reliable, and secure data transmission within and outside the facility. Scalability, redundancy, traffic optimization, security, and monitoring are key considerations for designing a robust, high-performance network. By implementing best practices and staying abreast of emerging technologies, data centers can build networks that meet the growing demands of the digital age while maintaining the highest levels of performance, availability, and security.