opencontrail

OpenContrail

OpenContrail

In today's fast-paced world, where cloud computing and virtualization have become the norm, the need for efficient and flexible networking solutions has never been greater. OpenContrail, an open-source software-defined networking (SDN) solution, has emerged as a powerful tool. This blog post explores the capabilities, benefits, and significance of OpenContrail in revolutionizing network management and delivering enhanced connectivity in the cloud era.

OpenContrail, initially developed by Juniper Networks, is an open-source SDN platform offering comprehensive network capabilities for cloud environments. It provides a scalable and flexible network infrastructure that enables automation, network virtualization, and secure multi-tenancy across distributed cloud deployments.

OpenContrail, an open-source network virtualization platform, is designed to simplify the management and orchestration of virtual networks. Built on well-established technologies such as OpenStack and SDN, it provides a comprehensive set of tools and APIs to create and manage virtualized network services. With OpenContrail, organizations can achieve greater scalability, security, and performance while reducing operational complexities.

1. Virtual Network Overlays: OpenContrail leverages virtual network overlays to create isolated and secure network segments, allowing for seamless multi-tenancy and network segmentation.

2. Network Policy and Security: It offers fine-grained network policies to control traffic flow, implement access control, and enforce security measures at the virtual network level.

3. Analytics and Monitoring: OpenContrail provides advanced analytics and monitoring capabilities, allowing administrators to gain insights into network performance, troubleshoot issues, and optimize resource allocation.

1. Cloud Service Providers: OpenContrail empowers cloud service providers to deliver scalable and secure network services to their customers. It enables seamless provisioning of virtual networks, ensuring high-performance connectivity and efficient resource utilization.

2. Enterprise Networks: Enterprises can leverage OpenContrail to build agile and flexible network infrastructures. It simplifies network management, enables seamless integration with existing infrastructure, and provides enhanced security measures.

3. Internet of Things (IoT): With the proliferation of IoT devices, OpenContrail offers a robust solution for managing and securing large-scale IoT deployments. It enables efficient communication between devices, ensures data privacy, and provides centralized control over IoT network resources.

Conclusion: OpenContrail proves to be a groundbreaking solution in the realm of network virtualization. Its feature-rich architecture, open-source nature, and diverse real-world applications make it an invaluable tool for organizations seeking to optimize network performance, enhance security, and embrace the future of virtualized networks.

Highlights: OpenContrail

OpenContrail is an open-source network virtualization platform that enables the creation of virtual networks overlaying physical infrastructure. It provides a scalable and flexible solution for managing network resources, improving security, and enhancing overall network performance. By decoupling the network control plane from the data plane, OpenContrail brings a new level of agility and efficiency to network operations.

1. Virtual Network Creation: OpenContrail allows the creation of virtual networks, each with its own isolated environment, policies, and routing tables. This enables organizations to achieve multi-tenancy and securely isolate their applications and workloads.

2. Network Automation and Orchestration: With OpenContrail, network provisioning and management become automated and orchestrated. This reduces manual configuration efforts and brings more consistency and reliability to network operations.

3. Enhanced Security: OpenContrail provides advanced security features such as micro-segmentation, distributed firewalling, and traffic isolation. These capabilities ensure that applications and data remain protected and isolated, even in complex and dynamic network environments.

Understanding OpenContrail componentns

Controller Node: At the heart of OpenContrail lies the Controller Node, which acts as the brain of the network. It is responsible for managing and orchestrating all the network services, including configuration, control, and analytics. Through its intuitive and user-friendly interface, network administrators can easily define and enforce policies, monitor network performance, and troubleshoot issues.

vRouter: The vRouter, short for virtual router, is a critical component of OpenContrail that ensures efficient packet forwarding within the network. By combining the power of virtualization and routing, the vRouter enables seamless communication between virtual machines and physical hosts. It provides advanced networking capabilities, such as firewalling, NAT, and VPN, while ensuring high performance and scalability.

Analytics Node: To gain valuable insights into network behavior and performance, OpenContrail incorporates an Analytics Node. This component collects and analyzes network data, generating comprehensive reports and metrics. Network operators can leverage this information to optimize network utilization, identify bottlenecks, and proactively address potential issues. The Analytics Node plays a crucial role in ensuring the reliability and efficiency of the entire network infrastructure.

Web User Interface

OpenContrail offers a user-friendly Web User Interface (UI) that simplifies network management and configuration. With its intuitive design and powerful functionalities, network administrators can easily define network topologies, set up policies, and monitor network performance in real time. The Web UI provides a centralized platform for managing the entire network infrastructure, making deploying, scaling, and maintaining OpenContrail deployments easier.

The traditional network vs. SDN network

In a traditional network, each switch/router must be programmed individually because applications are loaded. These applications could include a load balancer, intrusion detection, monitoring, or Voice over IP (VoIP). Based on local logic, each switch/router decides where to route packets as traffic flows through the network. Changing applications or flows in this network requires systematically programming each switch/router.

A traditional network includes both a control plane and a forwarding plane. There are also applications loaded on each device, which must be configured separately.

In an SDN network, a switch/router is not connected to any applications or intelligence. By centralized control of all devices, the network becomes programmable. A controller interfaces with applications, which are then executed across a network. Traffic flows are now supervised by a centralized controller that distributes and manages a flow table for each switch/router. Several factors can be used to define very flexible flow tables.

The flow table also collects statistics, which are fed up to the controller. This improves both visibility and control of the network because issues are immediately reported to the controller, which, in turn, can make immediate adjustments across the entire network.

The role of The VM

Virtual machines have been around for a long time, but we are beginning to spread our computing workloads in several ways. When you throw in docker containers and bare metal servers, networking becomes more interesting. Network challenges arise when all these components require communication within the same subnet, access to Internet gateways, and Layer 3 MPLS/VPNs.

As a result, data center networks are moving towards IP underlay fabrics and Layer 2 overlays. Layer 3 data plane forwarding utilizes efficient Equal-cost multi-path routing (ECMP), but we lack Layer 2 multipathing by default. Now, similar to an SD WAN overlay approach, we can connect dispersed layer 2 segments and leverage all the good features of the IP underlay. To provide Layer 2 overlays and network virtualization, Juniper has introduced an SDN platform called Junipers OpenContrail in direct competition with

Virtualization

For additional pre-information, you may find the following post of use.

  1. ACI Cisco
  2. Network Traffic Engineering
  3. Spine Leaf Architecture
  4. IP Forwarding
  5. SDN Data Center
  6. Network Overlays
  7. Application Traffic Steering
  8. What is BGP Protocol in Networking



MPLS Overlay

Key OpenContrail Discussion Points:


  • Introduction to the OpenContrail solution and what is involved.

  • Highlighting data center networks and ECMP.

  • Critical points on network virtualization.

  • Technical details on the virtual overlay network.

  • Technical details virtual network implementation.

  • Layer 2 VPN and EVPN.

Back to Basics with OpenContrail

Key Features and Benefits:

Network Virtualization:

OpenContrail leverages network virtualization techniques to provide isolated virtual networks within a shared physical infrastructure. It offers a logical abstraction layer, enabling the creation of virtual networks that operate independently, complete with their own routing, security, and quality of service policies. This approach allows for the efficient utilization of resources, simplified network management, and improved scalability.

Secure Multi-Tenancy:

OpenContrail’s security features ensure tenants’ data and applications remain isolated and protected from unauthorized access. It employs micro-segmentation to enforce strict access control policies at the virtual machine level, reducing the risk of lateral movement within the network. Additionally, OpenContrail integrates with existing security solutions, enabling the implementation of comprehensive security measures.

Intelligent Automation:

OpenContrail automates various network provisioning, configuration, and management tasks, reducing manual intervention and minimizing human errors. Its programmable API and centralized control plane simplify the deployment of complex network topologies, accelerate service delivery, and enhance overall operational efficiency.

Scalability and Flexibility:

OpenContrail’s architecture is designed to scale seamlessly, supporting distributed cloud deployments across multiple locations. It offers a highly flexible solution that can adapt to changing network requirements, allowing administrators to dynamically allocate resources, establish new connectivity, and respond to evolving business needs.

OpenContrail in Practice:

OpenContrail has gained significant traction among cloud providers, service providers, and enterprises seeking to build robust, scalable, and secure networks. Its open-source nature has facilitated its adoption, encouraging collaboration, innovation, and customization. OpenContrail’s community-driven development model ensures continuous improvement and the availability of new features and enhancements.

opencontrail
Diagram: OpenContrail.

Highlighting Junipers OpenContrail

OpenContrail is an open-source network virtualization platform. The commercial controller and open-source product are identical; they share the same checksum on the binary image. Maintenance and support are the only difference. Juniper decided to open source to fit into the open ecosystem, which wouldn’t have worked in a closed environment.

OpenContrail offers features similar to VMware NSX, can apply service chaining and high-level security policies, and provides connections to Layer 3 VPNs for WAN integration. OpenContrail works with any hardware, but integration with Juniper’s product sets offers additional rich analytics for the underlay network.

Underlay and overlay network visibility are essential for troubleshooting. You need to look further than the first header of the packet; you need to look deeper into the tunnel to understand what is happening entirely. 

Network virtualization – Isolated networks

With a cloud architecture, network virtualization gives the illusion that each tenant has a separate isolated network. Virtual networks are independent of physical network location or state, and nodes within the physical underlay can fail without disrupting the overlay tenant. A tenant may be a customer or department, depending if it’s a public or private cloud.

The virtual network sits on top of a physical network, the same way the compute virtual machines sit on top of a physical server. Virtual networks are not created with VLANs; Contrail uses a virtual overlay network system for multi-tenancy and cross-tenant communication. Many problems exist with large-scale VLAN deployments for multi-tenancy in today’s networks.

They introduce a lot of states in the physical network, and the Spanning Tree Protocol (STP) also introduces well-documented problems. There are technologies (THRILL, SPB) to overcome these challenges, but they add complexity to the design of the network.

Service Chaining

Customers require the ability to apply policy at virtual network boundaries. Policies may include ACL and stateless firewalls provided within the virtual switch. However, once you require complicated policy pieces between virtual networks, you need a more sophisticated version of policy control and orchestration called service chaining. Service chaining applies intelligent services between traffic from one tenant to another.

For example, if a customer requires content caching and stateful services, you must introduce additional service appliances and force next-hop traffic through these appliances. Once you deploy a virtual appliance, you need a scale-out architecture.

The ability to Scale-out

Scale-out is the ability to instantiate multiple physical and virtual machine instances and load balance traffic across them. Customers may also require the ability to connect with different tenants in dispersed geographic locations or to workloads in a remote private cloud or public cloud. Usually, people build a private cloud for the norm and then burst into a public cloud. 

Juniper has implemented a virtual networking architecture that meets these requirements. It is based on well-known technology, MPLS/layer 3 VPN. MPLS/layer 3 VPN is the base for Juniper designs.

MPLS Overlay

Junipers OpenContrail: Virtual Network Implementation 

MPLS Overlay

The SDN controller is responsible for the networking aspects of virtualization. When creating virtual networks, initiate the Northbound API and issue an instruction that attaches the VM to the VN. The network responsibilities are delegated from Cloudstack or OpenStack to Contrail. The Contrail SDN controller automatically creates the overlay tunnel between virtual machines. The overlay can be either an MPLS overlay style with MPLS-over-GREMPLS-over-UDP, or VXLAN

  • L3VPN for routed traffic and EVPN for bridged traffic

Juniper’s OpenContrail is still a pure MPLS overlay of MPLS/VPN, using L3VPN for routed traffic and EVPN for bridged traffic. Traffic forwarding between end nodes has one MPLS label (VPN label), but they use various encapsulation methods to carry labeled traffic across the IP fabric. As mentioned above, this includes MPLS-over-GRE – a traditional encapsulation mechanism, MPLS-over-UDP – a variation of MPLS-over-GRE that replaces the GRE headers with UDP headers. MPLS-over-VXLAN uses VXLAN packet format but stores the MPLS label in the Virtual Network Identifier (VNI) field.

The forwarding plane

The forwarding plane takes the packet from the VM and gives it to the “Vrouter,” which does a lookup and determines if the destination is a remote network. If it is, it encapsulates the packet and sends it across the tunnel. The underlay that sites between the workloads forward is based on tunnel source and destination only.

No state belongs to end hosts ‘VMs, MAC addresses, or IPs. This type of architecture gives the Core a cleaner and more precise role. Generally, as a best practice, keeping “state” in the Core is a lousy design principle.

Northbound and southbound interfaces

To implement policy and service chaining, use the Northbound Interface and express your policy at a high level. For example, you may require HTTP or NAT and force traffic via load balancers or virtual firewalls. Contrail does this automatically and issues instructions to the Vrouter, forcing traffic to the correct virtual appliance. In addition, it can create all the suitable routes and tunnels, causing traffic through the proper sequence of virtual machines.

Contrail achieves this automatically with southbound protocols, such as XMPP (Extensible Messaging and Presence Protocol) or BGP. XMPP is a communications protocol based on XML (Extensible Markup Language).

WAN Integration

Junipers OpenContrail can connect virtual networks to external Layer 3 MPLS VPN for WAN integration. In addition, they gave the controller the ability to peer BGP to gateway routers. For the data plane, they support MPLS-over-GRE, and for the control plane, they speak MP-BGP.

Contrail communicates directly with PE routers, exchanging VPNv4 routes with MP-BGP and using MPLS-over-GRE encapsulation to pass IP traffic between hypervisor hosts and PE routers. Using standards-based protocols lets you choose any hardware appliance as the gateway node.

mpls overaly

This data and control plane makes integration to an MPLS/VPN backbone a simple task. First, MP-BGP between the controllers and PE-routers should be established. Inter-AS Option B next hop self-approach should be used to demonstrate some demarcation points.

OpenContrail has emerged as a game-changer in software-defined networking, empowering organizations to build agile, secure, and scalable networks in the cloud era. With its advanced features, such as network virtualization, secure multi-tenancy, intelligent automation, and scalability, OpenContrail offers a comprehensive solution that addresses the complex networking challenges of modern cloud environments. As the demand for efficient and flexible network management continues to rise, OpenContrail provides a compelling option for organizations looking to optimize their network infrastructure and unlock the full potential of the cloud.

Summary: OpenContrail

OpenContrail is a powerful open-source software-defined networking (SDN) solution revolutionizing network management and connectivity. In this blog post, we will explore its key features, benefits, and use cases and showcase how it empowers organizations to build robust and scalable networks.

Understanding OpenContrail

OpenContrail, developed by Juniper Networks, is an open-source SDN controller that provides network virtualization and automation capabilities. It is a single control point for managing and orchestrating network resources, enabling organizations to simplify network operations and enhance flexibility. By decoupling the network control plane from the underlying physical infrastructure, OpenContrail brings agility and scalability to modern networks.

Key Features of OpenContrail

OpenContrail offers a wide range of features, making it a preferred choice for network administrators. Some notable features include:

1. Virtual Network Overlay: OpenContrail creates virtual network overlays, allowing multiple virtual networks to coexist on a shared physical infrastructure. This isolation ensures enhanced security and enables efficient resource utilization.

2. Policy-Driven Automation: With policy-driven automation, network administrators can define and enforce network policies and access controls across the infrastructure. OpenContrail simplifies the management and enforcement of complex policies, reducing operational overhead.

3. Analytics and Monitoring: OpenContrail provides extensive analytics and monitoring capabilities, offering real-time insights into network traffic, performance, and security. These insights help administrators optimize network resources and troubleshoot issues effectively.

Use Cases of OpenContrail

OpenContrail finds applications in various use cases across industries. Some prominent use cases include:

1. Cloud Infrastructure: OpenContrail enables cloud service providers to build and manage scalable and secure cloud infrastructures. It facilitates seamless integration with popular cloud platforms and offers rich networking capabilities.

2. Data Centers: OpenContrail simplifies network management in data center environments. It provides dynamic workload placement, automated provisioning, and seamless connectivity between virtual machines and containers, ensuring efficient resource utilization.

3. Multi-Cloud Networking: OpenContrail supports multi-cloud networking, allowing organizations to connect and manage multiple cloud environments securely. It provides seamless connectivity, consistent policies, and centralized control across cloud providers.

Conclusion:

OpenContrail presents a game-changing solution for organizations seeking to enhance their networking capabilities. With its rich feature set, including virtual network overlays, policy-driven automation, and advanced analytics, OpenContrail empowers organizations to build scalable, secure, and agile networks. Whether it’s cloud infrastructure, data centers, or multi-cloud networking, OpenContrail is a reliable and versatile SDN solution.

rsz_overlay_soltuins

Overlay Virtual Networking | Overlay Virtual Networks

Overlay Virtual Networks

In today's interconnected world, networks enable seamless communication and data transfer. Overlay virtual networking has emerged as a revolutionary approach to network connectivity, offering enhanced flexibility, scalability, and security. This blog post aims to delve into the concept of overlay virtual networking, exploring its benefits, use cases, and potential implications for modern network architectures.

Overlay virtual networking is a network virtualization technique that decouples the logical network from the underlying physical infrastructure. It creates a virtual network on top of the existing physical infrastructure, enabling the coexistence of multiple logical networks on the same physical infrastructure. By abstracting the network functions and services from the physical infrastructure, overlay virtual networking provides a flexible and scalable solution for managing complex network environments.

- Scalability and Flexibility: Overlay virtual networks provide the ability to scale network resources on-demand without disrupting the underlying physical infrastructure. This enables organizations to expand their network capabilities swiftly and efficiently, catering to changing business requirements.

- Enhanced Security: Overlay virtual networks offer heightened security by isolating traffic and providing secure communication channels. By segmenting the network into multiple virtual domains, potential threats can be contained, preventing unauthorized access to sensitive data.

- Cloud Computing: Overlay virtual networks are extensively used in cloud computing environments. They allow multiple tenants to have their own isolated virtual networks, ensuring data privacy and security. Additionally, overlay networks enable seamless migration of virtual machines between physical hosts, enhancing resource utilization.

- Software-Defined Networking (SDN): Overlay virtual networks align perfectly with the principles of Software-Defined Networking. By abstracting the logical network from the physical infrastructure, SDN controllers can dynamically manage and provision network resources, optimizing performance and efficiency.

Conclusion: Overlay virtual networks have emerged as a powerful networking solution, providing scalability, flexibility, and enhanced security. Their applications span across various domains, including cloud computing and software-defined networking. As technology continues to evolve, overlay virtual networks are poised to play a vital role in shaping the future of networking.

Highlights: Overlay Virtual Networks

Overlay Network Architecture

Overlay virtual networks are built on the existing physical network infrastructure, creating a logical network layer that operates independently. This architecture allows organizations to leverage the benefits of virtualization without disrupting their underlying network infrastructure.

The virtual network overlay software is at the heart of an overlay virtual network. This software handles the encapsulation and decapsulation of network packets, enabling communication between virtual machines (VMs) or containers across different physical hosts or data centers. It ensures data flows seamlessly within the overlay network, regardless of the underlying network topology.

Tunneling Protocols

Tunneling protocols play a crucial role in overlay virtual networks by facilitating the encapsulation and transportation of network packets over the underlying physical network. Popular tunneling protocols such as VXLAN (Virtual Extensible LAN), NVGRE (Network Virtualization using Generic Routing Encapsulation), and Geneve provide the necessary mechanisms for creating virtual tunnels and encapsulating traffic.

The Network Virtualization Edge (NVE) acts as the endpoint for the overlay virtual network. It connects the physical network infrastructure to the virtual network, ensuring seamless communication between the two. NVEs perform functions like encapsulation, decapsulation, and mapping virtual network identifiers (VNIs) to the appropriate virtual machines or containers.

Example: GRE and the tunneling protocol

GRE, or Generic Routing Encapsulation, is a tunneling protocol widely used in overlay networks. It encapsulates various network layer protocols within IP packets, enabling virtual point-to-point connections over an existing IP network. GRE provides a mechanism to extend private IP addressing schemes over public networks, facilitating secure and efficient communication between remote locations.

GRE without IPsec

 

Overlay Network Control Plane

The control plane in an overlay virtual network manages and maintains the overall network connectivity. It handles tasks such as route distribution, network mapping, and keeping the overlay network’s forwarding tables. Border Gateway Protocol (BGP) and Virtual Extensible LAN Segment Identifier (VXLAN VNI) provide the necessary control plane mechanisms.

Overlays and a Clos Fabric

The underlay of most modern data centers is a 3-stage or 5-stage Clos fabric, with the physical infrastructure and point-to-point Layer 3 interfaces between the spines and leaves. Network virtualization can be created by elevating the endpoints and applications connected to the network into this overlay, thus logically carving out different services on top of it.

Several factors have contributed to the need for network virtualization in such an architecture:

  • Virtualization of hosts and multitenancy have become commonplace in today’s data centers. Building and maintaining a unique physical infrastructure for each tenant is naturally costly.
  • Furthermore, maintaining a state for 100,000 hosts or more may not be feasible in most environments. The network’s core (the fabric’s spines, in this case) can be a pure IP forwarding device without any knowledge of the endpoints.
  • A 4K VLAN range was often a barrier in multitenant environments, especially in cloud computing.
  • Host mobility forces networks to retain constructs such as network gateways to ensure the mobility function does not break across leaf boundaries.
  • Legacy applications, host mobility, and storage clusters drive the need for a flat Layer 2 network over a routed infrastructure.
  • In network virtualization, packets are encapsulated as they enter the network, creating a network overlay. As a result, a physical infrastructure is covered with a logical network. Overlays enable the construction and connection of logically separated and isolated networks atop the same physical network.

Network overlay

In RFC 7348, a Virtual Extensible LAN (VXLAN) is a data plane encapsulation type capable of supporting Layer 2 and Layer 3 payloads. In addition to logically separating broadcast or bridging domains in a network, virtual LANs (VLANs) are limited in their scalability to 4K VLANs. By contrast, VXLAN provides a 24-bit VXLAN Network Identifier (VNI) in the VXLAN header, allowing the network administrator more flexibility to partition the network logically.

VXLAN is, in essence, a stateless tunnel, originating at one endpoint and terminating at another because of its encapsulating trait. The VXLAN Tunnel Endpoints (VTEPs) are the endpoints that encapsulate and decapsulate the VXLAN tunnel. The first thing you need to understand about VXLAN is that these tunnels can originate and terminate on network devices or servers with the help of a virtual switch such as Open vSwitch, with a VXLAN module that is usually accelerated by hardware so that the CPU doesn’t have to process these packets in software.

Network Overlay

Virtual networking overlays are vital for modern data centers, providing flexibility, scalability, and increased security. A virtual overlay network is a computer network built on top of an existing physical network. This abstraction layer creates a virtual overlay solution that can be configured independently of the underlying physical network.

VXLAN multicast mode
Diagram: VXLAN multicast mode

Increased Security

Virtual networks offer many advantages over traditional physical networks. First, they allow for rapidly deploying virtual machines, applications, and services. This can be done in minutes rather than the days or weeks it would take to reconfigure a physical network. Additionally, virtual networks provide increased security, as traffic between VMs can be isolated and encrypted. This can help prevent data breaches and unauthorized access.

Additional Flexibility

Virtual networks are also much more flexible than physical networks. For example, virtual networks can be segmented to separate different types of traffic. This improves performance, as traffic from other applications can be routed separately. Furthermore, virtual networks can be quickly reconfigured to meet changing demands. This agility allows for quick responses to changing business requirements.

Related: Before you proceed, you may find the following useful:

  1. SD-WAN Overlay
  2. Open Networking
  3. Segment Routing
  4. SDN Data Center
  5. Network Overlays
  6. Virtual Switch
  7. Load Balancing
  8. OpenContrail
  9. What is BGP Protocol in Networking



Overlay Virtual Networking

Key Overlay Virtual Network Discussion Points:


  • Introduction to overlay virtual networking and what is involved.

  • Highlighting the details of the virtual overlay solution and the components used.

  • Critical points on the scalability and security concerns.

  • Technical details on the different types of overlays, such as STT and VXLAN.

  • Closing comments on network overlay controllers.

Back to Basics: Virtual networks and virtualization.

Underlay and Overlay Networks

Overlay networks are virtual networks that run on top of physical networks. You have probably seen this terminology even if you have never heard of it. A GRE tunnel can illustrate an overlay network. Physical underlay networks support the GRE tunnel.VXLAN overlays are layer 2 Ethernet networks. Layer 3 IP networks form the underlay network. Transport networks are also known as underlay networks.

Getting packets from A to B is the only job of the underlay network. Layer 2 is not used here, only layer 3. We can load balance traffic on redundant links using an IGP like OSPF or EIGRP.

In addition, the overlay and underlay networks are independent. Underlay networks are virtual, but any changes made to the overlay network won’t affect the underlay network. A routing protocol can reach the destination regardless of how many links you add or remove in the underlay network.

Virtual Networking 

Main Virtual Overlay Networking Components

Overlay Virtual Networks

  • Overlay networks are virtual networks that run on top of physical networks

  • The most common forms of network virtualization are virtual LANs (VLANs), virtual private networks (VPNs), and Multiprotocol Label Switching (MPLS)

  • Like the ACI network, virtual overlay networks work best with Leaf and Spine fabric architectures

  • STT and VXAN can use 5-tuple load balancing as they use port numbers

Virtual overlay solutions

Virtual overlay solutions must have some simple to complex application stacks. Therefore, public or private cloud environments must support austere, complex environments to enable the virtual overlay network. On the other hand, simple customers that require web-hosting solutions need only a single domain with a few segments. Regarding network connectivity, there is one Virtual Machine ( VM ) with a single public IP.

Complex customers require complex multi-tier application stacks with overlay virtual networking, load-balancing, and firewall services in front and between application tiers. Cloud providers must support all types of application stacks as they are isolated virtual segments, and this is done with virtual overlay networks.

Lab guide on VXLAN.

In the following example, we have a lab guide on VXLAN. Here, we created a Layer 2 overlay across the core. The core layer consists of two spines and is a routed layer. The core does not know the subnets assigned to the desktop devices. It is the role of VXLAN to tunnel this information.

Notice we have a VNI set to 6002. This needs to match at both ends of Leaf A and Leaf B. If you change the VNI, you will break connectivity. This is a Layer 2 overlay, as the VNI is mapped to a bridge domain.

VXLAN
Diagram: Changing the VNI

Concept of network virtualization

It’s worth mentioning that network virtualization is nothing new. The most common forms of network virtualization are virtual LANs (VLANs), virtual private networks (VPNs), and Multiprotocol Label Switching (MPLS). VLAN has been the first to extract the location of Layer 2 connectivity across multiple Layer 2 switches. VPN enables overlay networks across untrusted networks such as the WAN, while MPLS segments traffic based on labels.

These technologies enable the administrators to physically separate endpoints into logical groups, making them behave like they are all on the same local (physical) segment. The ability to do this allows for much greater efficiency in traffic control, security, and network management.

    • Enhanced Connectivity:

One of the primary advantages of network overlay is its ability to enhance connectivity. By creating a virtual network layer, overlay networks enable seamless communication between devices and applications, irrespective of their physical location.

This means organizations can effortlessly connect geographically dispersed branches, data centers, and cloud environments, fostering collaboration and resource sharing. Moreover, network overlays offer greater flexibility by allowing organizations to dynamically adjust and optimize their network configurations to meet evolving business needs.

    • Improved Scalability:

Traditional network infrastructures often struggle to keep up with the increasing demands of modern applications and services. Network overlay addresses this challenge by providing a scalable solution. By decoupling the virtual network from the physical infrastructure, overlay networks allow for more efficient resource utilization and easier scaling.

Organizations can easily add or remove network elements without disrupting the entire network. As a result, network overlays enable organizations to scale their networks rapidly and cost-effectively, ensuring optimal performance even during peak usage periods.

Example of an overlay network: MPLS

MPLS overlay is a technique used to create virtual private networks (VPNs) over existing IP networks, enabling organizations to achieve enhanced network scalability, reliability, and security. Unlike traditional IP routing, MPLS overlay relies on labels to forward packets, making it more efficient and flexible.

Overlay with MPLS 

With MPLS, we can have a free BGP core providing an MPLS overlay. MPLS overlay is a network architecture that allows organizations to build virtual private networks (VPNs) on top of their existing network infrastructure. It leverages the capabilities of MPLS technology to create virtual tunnels, known as MPLS tunnels or MPLS paths, which enable the secure and efficient transfer of data between different network endpoints.

Below, we have BGP running between the PEs and carrying customer prefixes for CE 1 and 2. The P, representing the core layer, does not know customer routes and performs label switching. This brings not only scalability, as the P nodes can focus on label switching, but also an added layer of security. No security devices need to be present in the core layer. Although you would need QoS, they are pushing intelligence to the edges.

MPLS forwarding
Diagram: MPLS Overlay

Benefits of MPLS Overlay:

1. Enhanced Performance: MPLS overlay offers improved network performance by enabling faster data transmission and reduced latency. It achieves this by using label switching, which helps prioritize and route data packets efficiently, reducing congestion and optimizing network utilization.

2. Scalability and Flexibility: With MPLS overlay, organizations can quickly expand their network infrastructure without requiring extensive hardware upgrades. It allows for the creation of virtual networks within a shared physical infrastructure, enabling seamless scalability and flexibility.

3. Quality of Service (QoS): MPLS overlay provides enhanced QoS capabilities, enabling organizations to prioritize critical applications or data traffic. This ensures mission-critical applications receive the bandwidth and low latency, optimizing overall network performance.

4. Improved Security: MPLS overlay enhances network security by providing inherent isolation between different VPNs. It creates separate virtual tunnels for each VPN, ensuring that data remains isolated and protected from unauthorized access.

Lab Guide on MPLS TE

In this lab, we will examine MPLS TE with ISIS configuration. Our MPLS core network consists of routers PE1, P1, P2, P3, and PE2. The CE1 and CE2 routers use regular IP routing. All routers are configured to use IS-IS L2. 

MPLS TE is a mechanism that allows network operators to control and manage traffic flows within a Multiprotocol Label Switching (MPLS) network. It is designed to address the limitations of traditional IP routing by providing a more efficient and flexible approach to data forwarding

Note:

There are four main items we have to configure:

  • Enable MPLS TE support:
    • Globally
    • Interfaces
  • Configure IS-IS to support MPLS TE.
  • Configure RSVP.
  • Configure a tunnel interface.
MPLS TE
Diagram: MPLS TE

Example of an overlay network: DMVPN

With the configuration of DMVPN phase 1, we can have a “hub and spoke” topology, where a single hub site acts as the central point for communication, while the other locations, or “spokes,” connect to the hub through virtual tunnels. This topology provides several benefits, including secure communications between spokes, optimized traffic routing, and reduced overhead for managing the network.

DMVPN also supports dynamic routing protocols, such as Open Shortest Path First (OSPF), allowing for dynamic updates to the network topology. This allows for rapid changes in the network, such as adding or removing spokes, without the need to reconfigure the entire network. Additionally, DMVPN supports multicast traffic, allowing the efficient distribution of data and resources to multiple sites simultaneously.

DMVPN
Diagram: DMVPN. Source is techtarget.

Lab guide with DMVPN

In the following lab, we have DMVPM, which creates an overlay network. The hub, which is R1, created an overlay network over the SP router. The SP router represents the WAN; in reality, the number of nodes in the WAN is irrelevant to DMPVN. The overlay is created between R1, R2, and R3, which act as the spokes.

The protocol used in GRE, specifically point-to-point GRE, as we are running DMVPN Phase 1. The Tunneling protocol of mGRE would have been used if we were running DMVPN Phase 3

DMVPN configuration
Diagram: DMVPN Configuration.

Benefits of DMVPN Overlay:

1. Simplified Network Architecture:

Traditional networking often involves complex and static configurations, making it cumbersome to manage and maintain. DMVPN overlay, on the other hand, simplifies network architecture by providing a dynamic and scalable solution. With DMVPN, organizations can establish secure connections between various branch offices, data centers, and remote users, all while leveraging the existing infrastructure. This simplification leads to reduced administrative overhead and improved network agility.

2. Enhanced Flexibility and Scalability:

DMVPN overlay offers unparalleled flexibility and scalability, making it an ideal choice for organizations with dynamic network requirements. As businesses grow and expand, DMVPN allows for the seamless addition of new sites or remote users without requiring extensive configuration changes. Its ability to establish connections on-demand and dynamically allocate resources ensures that network expansion remains hassle-free and cost-effective.

3. Improved Network Performance:

Network performance is crucial for organizations, directly impacting productivity and user experience. DMVPN overlay utilizes multiple paths and load balancing techniques, allowing for efficient utilization of available bandwidth. By optimizing network traffic, DMVPN ensures that applications and services operate smoothly, even during peak usage periods. Moreover, its ability to prioritize critical traffic and dynamically adjust to network conditions further enhances overall performance.

4. Enhanced Security:

Security remains a top concern for organizations, particularly when transmitting sensitive data across networks. DMVPN overlay addresses these concerns by providing robust encryption and authentication mechanisms. By leveraging IPsec protocols, DMVPN ensures that data confidentiality and integrity are maintained, protecting against unauthorized access and potential threats. The inherent security features of DMVPN make it a reliable choice for organizations looking to maintain a secure network environment.

Types of Overlay Networks

1. Virtual Private Networks (VPNs):

VPNs are one of the most common types of overlay networks. They enable secure communication over public networks by creating an encrypted tunnel between the sender and receiver. Individuals and organizations widely use VPNs to protect sensitive data and maintain privacy. Additionally, they allow users to bypass geographical restrictions and access region-restricted content.

2. Software-Defined Networks (SDNs):

In network architecture, SDNs utilize overlay networks to separate the control plane from the data plane. SDNs provide centralized management, flexibility, and scalability by decoupling network control and forwarding functions. Overlay networks in SDNs enable the creation of virtual networks on top of the physical infrastructure, allowing for more efficient resource allocation and dynamic network provisioning.

3. Peer-to-Peer (P2P) Networks:

P2P overlay networks are decentralized systems that facilitate direct communication and file sharing between nodes without relying on a central server. They leverage overlay networks to establish direct connections between peers and enable efficient data distribution. These networks are widely used for content sharing, real-time streaming, and decentralized applications.

4. Content Delivery Networks (CDNs):

CDNs employ overlay networks to optimize content delivery by strategically distributing content across multiple servers in different geographic regions. By bringing content closer to end-users, CDNs reduce latency and improve performance. Overlay networks in CDNs enable efficient content caching, load balancing, and fault tolerance, resulting in faster and more reliable content delivery.

5. Overlay Multicast Networks:

Overlay multicast networks are designed to distribute data to multiple recipients simultaneously efficiently. These networks use overlay protocols to construct multicast trees and deliver data over these trees. Overlay multicast networks benefit applications such as video streaming, online gaming, and live events broadcasting, where data must be transmitted to many recipients in real-time.

Use Cases of Overlay Virtual Networking:

1. Multi-Tenancy:

Overlay virtual networking provides an ideal solution for organizations to segregate their network resources securely. By creating virtual overlays, multiple tenants can coexist on a single physical network infrastructure without interference. This enables service providers and enterprises to offer distinct network environments to customers or departments while ensuring isolation and security.

2. Data Center Interconnect:

Overlay virtual networking enables efficient and scalable data center interconnect (DCI). With traditional networking, interconnecting multiple data centers across geographies can be complex and costly. However, overlay virtual networking simplifies this process by abstracting the underlying physical infrastructure and providing a unified logical network. It allows organizations to seamlessly extend their networks across multiple data centers, enhancing workload mobility and disaster recovery capabilities.

3. Cloud Computing:

Cloud computing heavily relies on overlay virtual networking to deliver agility and scalability. Cloud providers can dynamically provision and manage network resources by leveraging overlay networks, ensuring optimal customer performance and flexibility. Overlay virtual networking enables the creation of virtual networks that are isolated from each other, allowing for secure and efficient multi-tenant cloud environments.

4. Microservices and Containerization:

The rise of microservices architecture and containerization has presented new networking challenges. Overlay virtual networking provides a solution by enabling seamless communication between microservices and containers, regardless of their physical location. It ensures that applications and services can communicate with each other, even across different hosts or clusters, without complex network configurations.

5. Network Segmentation and Security:

Overlay virtual networking enables granular network segmentation, allowing organizations to implement fine-grained security policies. By creating overlay networks, administrators can isolate different workloads, departments, or applications, ensuring each segment has dedicated network resources and security policies. This enhances security by limiting the lateral movement of threats and reducing the attack surface.

Tailored load balancing

Some customers may not require cloud load balancing services provided by the cloud services if they have optimized web delivery by deploying something like Squid or NGINX. Squid is a caching proxy that improves web request response times by caching frequently requested web pages. NGINX ( open source reverse proxy ) is used to load balance Hypertext Transfer Protocol ( HTTP ) among multiple servers.

Example: Traffic flow and the need for a virtual overlay

Traffic would flow to Web servers and trigger application and database requests. Each tier requires different segments, and in large environments, the limitations of using VLANs to create these segments will bring both scalability and performance problems.

This is why we need virtual overlay solutions. These subnets require Layer 3 and sometimes Layer 2 ( MAC ). Layer 2 connectivity might be for high availability services that rely on gratuitous Address Resolution Protocol ( ARP ) between devices or some other non-routable packet that can not communicate over IP. If the packet is not Layer 3 routable, it needs to communicate via Layer 2 VLANs.

Virtual overlay networking
Diagram: Virtual overlay networking and complex application tiers.

Scalability and Security Concerns

The weakest link in a security paradigm is the lowest application in that segment. Make each application an independent tenant so all other applications are unaffected if a security breach or misuse occurs in one application stack.

Designers should always attempt to design application stacks to minimize beachheading, i.e., an attacker compromising one box and using it to jump to another quickly. Public and private clouds should support multi-tenancy with each application stack.

However, scalability issues arise when you deploy each application as an individual segment. For example, customer X’s cloud application requires four segments; 4000 VLANs soon become 1000 applications. Media Access Control ( MAC ) visibility has an entire reach throughout Layer 2 domains.

Some switches support a low count number of MAC addresses. When a switch reaches its MAC limit, it starts flooding packets, increasing network load and consuming available bandwidth that should be used for production services.

…current broadcast domains can support … around 1,000 end hosts in a single bridged LAN of 100 bridges” (RFC 5556 – TRILL)

NIC in promiscuous mode and failure domains

Server administrators configure server NICs in promiscuous mode to save configuration time. NICs in promiscuous mode look at all frames passing even when the frame is not destined for them. Network cards acting in promiscuous mode are essentially the same as having one VLAN spanning the entire domain. Sniffer products set promiscuous modes to capture all data on a link and usually only act in this mode for troubleshooting purposes.

A well-known issue with Layer 2 networks is that they present a single failure domain with extreme scalability and operational challenges. This is related to Layer 2 Spanning Tree Protocol ( STP ); THRILL is also susceptible to broadcast storms and network meltdowns.

The rise of overlay virtual networks

Previously discussed scalability and operational concerns force vendors to develop new data center technologies. One of the most prevalent new technologies is overlay virtual networks, tunneling over IP. An overlay is a tunnel between two endpoints, allowing frames to be transported. The beauty of overlay architectures is that they enable switch table sizes not to increase as the number of hosts attached increases.

Vendors’ Answer: Virtual Overlay Solutions

Diagram: Virtual overlay solutions.

Virtual Overlay Solution: Keep complexity to the edges.

Ideally, we should run virtual networks over IP like SKYPE runs Voice over IP. The recommended design retains complexity at the network’s edge; the IP transport network provides IP transport. A transport network does not need to be a Layer 2 network and can have as many IP subnets and router hops.

All data ( storage, vMotion, user traffic ) traffic becomes an IP application. The concept resembles how Border Gateway Protocol ( BGP ) applies to TCP. End hosts carry out encapsulation and use the network for transport. Again, complexity is at the edge, similar to the Internet. Keeping complexity to the edge makes Layer 3 fabrics efficient and scalable.

VXLAN, STT, and ( NV ) GRE

Numerous encapsulation methods can tunnel over the IP core. This is known as virtual overlay networking and includes VXLAN, STT, and ( NV ) GRE. The main difference between these technologies is the encapsulation method and minor technological differences with TCP offload and load balancing.

virtual overlay solutions
Diagram: Virtual overlay solution.

The Recommended Design: Leaf and Spine.

Like the ACI network, virtual overlay networks work best with Leaf and Spine fabric architectures. Leaf and Spine designs guarantee any two endpoints get equal bandwidth. VMs on the same Top-of-Rack ( ToR ) switch will have access to more bandwidth than if the VM had to communicate across the Spine layer.

Overlay networks assume that the underlying network has a central endpoint. The transport network should avoid oversubscription as much as possible. If security concerns you, you can always place similar VM appliances on dedicated clusters, one type per physical server.

( NV ) GRE, VXLAN, and STT do not have an built-in security features meaning the transport network MUST be secure.

TCP offload, load balancing & scale-out NAT

TCP can push huge segments down the physical NIC and slice the packet into individual TCP segments, improving TCP performance. For example, you can push 10Gbps from a VM with TCP offload. The problem is that NICs only support VLANs and not VXLANs.

NICIRA added another header in front of TCP segments. TCP is embedded in another TCP. Now, you can use the existing NIC to slice the current TCP segment into smaller TCP segments. It is dramatically improving performance.

STT and VXAN

STT and VXAN can use 5-tuple load balancing as they use port numbers. Therefore, traffic sent between a pair of VMs can use more than one link in the network. Unfortunately, not many switches can load balance based on the GRE payload used by NVGRE.

Scale-out NAT is difficult to implement as an asymmetric path is not guaranteed. Furthermore, the shared state is tied to an outside IP address, which limits scale-out options. To scale out effectively, the state has to be spread across all members of the NAT cluster. The new approach uses floating public IP addresses and one-to-one mapping between floating IP and the private IP address inside—there is no state due to the one-to-one mapping.

Distributed layer 2 & layer 3 forwarding  

They distributed Layer 2 forwarding ( data plane ): Most Overlays offer distributed Layer 2 forwarding. VM can be sent to VM in the same segment. The big question is how they distribute MAC to VTEP – some use multicast and traditional Ethernet flooding, while others use control planes. The big question is how scalable is the control plane.

Distributed Layer 3 forwarding ( data plane ): On the other hand, if you have multiple IP subnets between segments ( not layer 2 ), you need to forward between them. The inter-subnet must not be a choke point. If your data center has lots of intra-traffic ( East to West traffic), avoid centralized inter-subnet forwarding, which will quickly become a traffic choke point.

The router will process ARP if you are doing Layer 3 forwarding. But if you are doing a mix of Layer 2 and 3, make sure you can reduce the flooding by intercepting ARP requests and caching ARP replies, known as distributed ARP Caching.

Scale-out control plane 

Initial overlays used multicast and Ethernet-like learning. Now, some vendors are using controller-based overlays. Keep in mind that the controller can now become a scalability bottleneck. However, many vendors, such as Cisco ACI, can scale the controllers and have a quorum.

Efficient controller scalability is seen when controllers do not participate in the data plane ( do not reply to ARP ). This type of controller scales better than controllers that intercept data plane packets and perform data plane activity. So, the data plane will not be affected if a controller is offline. In the early days of Sofware-Defined Networking, this was not the case. If the controller was down, the network was down.

Scale-out controllers 

Attempt to design scale-out controllers by building a cluster of controllers and having some protocol running between them. You now have clear failure domains. For example, controller A looks after VM segment A and Controller B, and control looks after VM segment B. For cloud deployments in multiple locations, deploy multiple controller clusters in each location.

Availability zones

Design availability zones with hierarchical failure domains by splitting infrastructures into regions. Problems arising in one region do not affect all other regions. You have one or more availability zones within an area for physical and logical isolation.

Availability zones limit the impact of a failure in a failure domain. An example of a failure domain could be a VLAN experiencing a broadcast storm. Attempt to determine the span of VLANs across availability zones—define VLANs to one-ToR switch. Never stretch VLANs as you create a single failure domain by merging two zones.

Do not stretch a VLAN across multiple availability zones. This is why we have network overlays in the first place, so we don’t need to stretch VLAN across the data center. For example, VXLAN uses the VNI to differentiate between Layer 2 and Layer 3 traffic over a routed underlay. We can use VXLAN as the overlay network to span large Layer 2 domains over a routed core.

Availability zones
Diagram: Availability zones. The source is cloudconstruct.

Network Overlay Controllers

As a final note on controllers, controller-based SDN networks participate in data planes and perform activities such as MAC learning and ARP replies. As mentioned, this is not common nowadays but was at the start of the SDN days. If the controller performs activities such as MAC learning and APR replies and the controller fails, then you have network failure.

The more involved the controller is in the forwarding decisions, the worse the outage can be. All overlay networking vendors nowadays have controllers that set up the control plane so the data plane can forward traffic without getting involved in data plane activity. This design also allows the controller to be scaled without affecting the data plane activity.

Overlay virtual networking has significant implications for modern network architectures. It enables the creation of software-defined networks (SDNs), where network policies, routing, and security are managed centrally through software-based controllers. This centralized management simplifies network operations, improves agility, and enables network automation.

Recap: Understanding Overlay Virtual Networking

Overlay virtual networking is a method that allows virtual networks to be created on top of existing physical networks. By decoupling the network’s logical and physical infrastructure, overlay virtual networking provides flexibility, scalability, and enhanced security.

Benefits of Overlay Virtual Networking

Overlay virtual networking brings numerous advantages to organizations. Firstly, it enables seamless network scalability, allowing businesses to quickly expand their network resources without needing physical infrastructure upgrades. Additionally, overlay virtual networking enhances network security by providing isolated virtual networks that can be customized with specific security policies.

Implementation of Overlay Virtual Networking

Implementing overlay virtual networking involves utilizing software-defined networking (SDN) technologies. Organizations can create virtual networks independent of the underlying physical infrastructure through SDN controllers and network virtualization overlays. This implementation approach simplifies network management and enables dynamic configuration.

Applications of Overlay Virtual Networking

Overlay virtual networking finds applications in various industries. In data centers, it facilitates efficient resource allocation and workload mobility across virtual machines. For cloud service providers, overlay virtual networking enables the creation of virtual private clouds, ensuring secure and isolated connectivity for their customers. Moreover, overlay virtual networking can enhance the connectivity and security of IoT devices in smart cities and industrial environments.

Overlay virtual networking empowers organizations to unlock new network flexibility, scalability, and security levels. By abstracting the network’s logical layer from the physical infrastructure, overlay virtual networking enables seamless expansion, simplified management, and tailored security policies. As the digital landscape continues to evolve, overlay virtual networking will play a pivotal role in shaping the future of networking.

Summary: Overlay Virtual Networks

Overlay networking has revolutionized the way we design and manage modern networks. In this blog post, we will delve into the fascinating world of overlay networking, exploring its benefits, applications, and critical components.

Understanding Overlay Networking

Overlay networking is a technique for creating virtual networks on top of an existing physical network infrastructure. By decoupling the network services from the underlying hardware, overlay networks provide flexibility, scalability, and enhanced security.

Benefits of Overlay Networking

One of the primary advantages of overlay networking is its ability to abstract the underlying physical infrastructure, allowing for seamless integration of different network technologies and protocols. This flexibility empowers organizations to adapt to changing network requirements without significant disruptions. Additionally, overlay networks facilitate the implementation of advanced network services, such as virtual private networks (VPNs) and load balancing, while maintaining a simplified management approach.

Applications of Overlay Networking

Overlay networking finds applications in various domains, ranging from data centers to cloud computing. In data center environments, overlay networks enable efficient multi-tenancy, allowing different applications or departments to operate within isolated virtual networks. Moreover, overlay networking facilitates the creation of hybrid cloud architectures, enabling seamless connectivity between on-premises infrastructure and public cloud resources.

Key Components of Overlay Networking

Understanding overlay networking’s key components is crucial to comprehending it. These include overlay protocols, which establish and manage virtual network connections, and software-defined networking (SDN) controllers, which orchestrate the overlay network. Additionally, virtual tunnel endpoints (VTEPs) play a vital role in encapsulating and decapsulating network packets, ensuring efficient communication within the overlay network.

Overlay networking has genuinely transformed the landscape of modern network architectures. By providing flexibility, scalability, and enhanced security, overlay networks have become indispensable in various industries. Whether it is for data centers, cloud environments, or enterprise networks, overlay networking offers a powerful solution to meet the evolving demands of the digital era.

Conclusion:

In conclusion, overlay networking has emerged as a game-changer in the world of networking. Its ability to abstract and virtualize network services brings immense value to organizations, enabling them to adapt quickly, enhance security, and optimize resource utilization. As technology continues to advance, overlay networking will likely play an even more significant role in shaping the future of network architectures.

Green data center with eco friendly electricity usage tiny person concept. Database server technology for file storage hosting with ecological and carbon neutral power source vector illustration.

LISP Hybrid Cloud Use Case

LISP Hybrid Cloud Use Case

In the world of networking, the ability to efficiently manage and scale networks is of paramount importance. This is where LISP networking comes into play. LISP, which stands for Locator/ID Separation Protocol, is a powerful networking technology that offers numerous benefits to network administrators and operators. In this blog post, we will explore the world of LISP networking and its key features and advantages.

LISP networking is a revolutionary approach to IP addressing and routing that separates the identity of a device (ID) from its location (locator). Traditional IP addressing relies on combining these two aspects, making it challenging to scale networks and manage mobility. LISP overcomes these limitations by decoupling the device's identity and location, enabling more flexible and scalable network architectures.

LISP, at its core, is a routing architecture that separates location and identity information for IP addresses. By doing so, it enables scalable and efficient routing across networks. LISP hybrid cloud leverages this architecture to seamlessly integrate multiple cloud environments, including public, private, and on-premises clouds.

Enhanced Scalability: LISP hybrid cloud allows organizations to scale their cloud infrastructure effortlessly. By abstracting location information from IP addresses, it enables efficient traffic routing across cloud environments, ensuring optimal utilization of resources.

Improved Security and Privacy: With LISP hybrid cloud, organizations can establish secure and private connections between different cloud environments. This ensures that sensitive data remains protected while being seamlessly accessed across clouds, bolstering data security and compliance.

Simplified Network Management: By centralizing network policies and control, LISP hybrid cloud simplifies network management for organizations. It provides a unified view of the entire cloud infrastructure, enabling efficient monitoring, troubleshooting, and policy enforcement.

Seamless Data Migration: LISP hybrid cloud enables seamless migration of data between different clouds, eliminating the complexities associated with traditional data migration methods. It allows organizations to transfer large volumes of data quickly and efficiently, minimizing downtime and disruption.

Hybrid Application Deployment: Organizations can leverage LISP hybrid cloud to deploy applications across multiple cloud environments. This enables a flexible and scalable infrastructure, where applications can utilize resources from different clouds based on specific requirements, optimizing performance and cost-efficiency.

Conclusion: In conclusion, the LISP hybrid cloud use case presents a compelling solution for organizations seeking to enhance their cloud infrastructure. With its scalability, security, and simplified network management benefits, LISP hybrid cloud opens up a world of possibilities for seamless integration and optimization of multiple cloud environments. Embracing LISP hybrid cloud can drive efficiency, flexibility, and agility, empowering organizations to stay ahead in today's dynamic digital landscape.

Highlights: LISP Hybrid Cloud Use Case

Use Case: Hybrid Cloud

The hybrid cloud connects the public cloud provider to the private enterprise cloud. It consists of two or more distinct infrastructures in dispersed locations that remain unique. These unique entities are bound together logically via a network to enable data and application portability. LISP networking performs hybrid cloud and can overcome the negative drawback of stretched VLAN. How do you support intra-subnet traffic patterns among two dispersed cloud locations? Without a stretched VLAN spanning locations, instability may arise from broadcast storms and Layer 2 loops.

Triangular routing

End to End-to-end connectivity

Enterprises want the ability to seamlessly insert their application right into the heart of the cloud provider without changing any parameters. Customers want to do this without changing the VM’s IP addresses and MAC addresses. This requires the VLAN to be stretched end-to-end. Unfortunately, IP routing cannot support VLAN extension, which puts pressure on the data center interconnect ( DCI ) link to enable extended VLANs. In reality, and from experience, this is not a good solution.

LISP Architecture on Cisco Platforms

There are various Cisco platforms that support LISP, but the platforms are mainly characterized by the operating system software they run. LISP is supported by Cisco’s IOS/IOS-XE, IOS-XR, and NX-OS operating systems. LISP offers several distinctive features and functions, including xTR/MS/MR, IGP Assist, and ESM/ASM Multi-hop. It is not true that all hardware supports all functions or features. Users need to verify that a platform supports key features before implementing it.

IOS-XR and NX-OS do not have distributed architectures, as does Cisco IOS/IOS-XE.RIB and Cisco Express Forwarding (CEF) provide the forwarding architecture for LISP on IOS/IOS-XE platforms using the LISP control process.

Before you proceed, you may find the following helpful:

  1. LISP Protocol
  2. LISP Hybrid Cloud Implementation
  3. Network Stretch
  4. LISP Control Plane
  5. Internet of Things Access Technologies

LISP Networking

Key LISP Hybrid Cloud Discussion Points:


  • Introduction to LISP Hybrid Cloud and what is involved.

  • Highlighting the details of LISP networking and how it can be implemented.

  • Critical points in a step-by-step format.

  • A final note on LISP stretched VLAN and overlay networking.

Back to basics with a LISP network

The LISP Network

The LISP network comprises a mapping system with a global database of RLOC-EID mapping entries. The mapping system is the control plane of the LISP network decoupled from the data plane. The mapping system is address-family agnostic; the EID can be an IPv4 address mapped to an RLOC IPv6 address and vice versa. Or the EID may be a Virtual Extensible LAN (VXLAN) Layer 2 virtual network identifier (L2VNI) mapped to a VXLAN tunnel endpoint (VTEP) address working as an RLOC IP address.

How Does LISP Networking Work?

At its core, LISP networking introduces a new level of indirection between the device’s IP address and location. LISP relies on two key components: the xTR (eXternal Tunnel Router) and the mapping system. The xTR is responsible for encapsulating and forwarding traffic between different LISP sites, while the mapping system stores the mappings between the device’s identity and its current location.

Benefits of LISP Networking:

Scalability: LISP provides a scalable solution for managing large networks by separating the device’s identity from its location. This allows for efficient routing and reduces the amount of routing table information that needs to be stored and exchanged.

Mobility: LISP networking offers seamless mobility support, enabling devices to change locations without disrupting ongoing communications. This is particularly beneficial in scenarios where mobile devices are constantly moving, such as IoT deployments or mobile networks.

Traffic Engineering: LISP allows network administrators to optimize traffic flow by manipulating the mappings between device IDs and locators. This provides greater control over network traffic and enables efficient load balancing and congestion management.

Security: LISP supports secure communications through the use of cryptographic techniques. It provides authentication and integrity verification mechanisms, ensuring the confidentiality and integrity of data transmitted over the network.

Use Cases for LISP Networking:

Data Centers: LISP can significantly simplify the management of large-scale data center networks by providing efficient traffic engineering and seamless mobility support for virtual machines.

Internet Service Providers (ISPs): LISP can help ISPs improve their network scalability and handle the increasing demand for IP addresses. It enables ISPs to optimize their routing tables and efficiently manage address space.

IoT Deployments: LISP’s mobility support and scalability make it an ideal choice for IoT deployments. It efficiently manages large devices and enables seamless connectivity as devices move across different networks.

LISP Networking and Stretched VLAN

Locator Identity Separation Protocol ( LISP ) can extend subnets without the VLAN. I am creating a LISP Hybrid Cloud. A subnet extension with LISP is far more appealing than a Layer 2 LAN extension. The LISP-enabled hybrid cloud solution allows Intra-subnet communication regardless of where the server is. This means you can have two servers in different locations, one in the public cloud and the other in the Enterprise domain; both servers can communicate as if they were on the same subnet.

LISP acts as an overlay technology

LISP operates like an overlay technology; it encapsulates the source packet with UDP and a header consisting of the source and destination RLOC ( RLOC are used to map EIDS). The result is that you can address the servers in the cloud according to your addressing scheme. There is no need to match your addressing scheme to the cloud addressing scheme.

LISP on the Cloud Service Router ( CRS ) 1000V ( virtual router ) solution provides a Layer-3-based approach to a hybrid cloud. It allows you to stretch subnets from the enterprise to the public cloud without needing a Layer 2 LAN extension.

LISP networking
LISP networking and hybrid cloud

LISP networking deployment key points:

  1. LISP can be deployed with the CRS 1000V in the cloud and either a CRS 1000V or ASR 1000 in the enterprise domain.
  2. The enterprise CRS must have at least two interfaces. One interface is the L3 routed interface to the core. The second interface is a Layer 2 interface to support VLAN connectivity for the servers that require mobility.
  3. The enterprise CRS does not need to be the default gateway, and its interaction with the local infrastructure ( via the Layer 2 interface ) is based on Proxy-ARP. As a result, ARP packets must be allowed on the underlying networks.
  4. The Cloud CRS is also deployed with at least two interfaces. One interface is facing the Internet or MPLS network. The second interface faces the local infrastructure, either by VLANs or Virtual Extensible LAN ( VXLAN ).
  5. The CRS offers machine-level high availability and supports all the VMware high-availability features such as dynamic resource scheduling ( DRS ), vMotion, NIC load balancing, and teaming.
Hybrid Cloud
Hybrid cloud and CRS1000V
  1. LISP is a network-based solution and is independent of the hypervisor. You can have different hypervisors in the Enterprise and the public cloud. No changes to virtual servers or hosts. It’s completely transparent.
  2. The PxTR ( also used to forward to non-LISP sites ) is deployed in the enterprise cloud, and the xTR is deployed in the public cloud.
  3. The CRS1000V deployed in the public cloud is secured by an IPSEC tunnel. Therefore, the LISP tunnel should be encrypted using IPSEC tunnel mode. Tunnel mode is preferred to support NAT.
  4. Each CRS must have one unique outside IP address. This is used to form the IPSEC tunnel between the two endpoints.
  5. Dynamic or static Routing must be enabled over the IPSEC tunnel. This is to announce the RLOC IP address used by the LISP mapping system.
  6. The map-resolver ( MR ) and map server ( MS ) can be enabled on the xTR in the Enterprise or the xTR in the cloud.
  7. Traffic symmetry is still required when you have stateful devices in the path.

 

LISP stretched subnets

The two modes of LISP operation are the LISP “Across” subnet and the LISP “Extended” subnet mode. Neither of these modes is used with the LISP-enabled CRS hybrid cloud deployment scenario. The mode of operation utilized is called the LISP stretched subnet model ( SSM ). The same subnet is used on both sides of the network, and mobility is performed between these two segments on the same subnet. You may think that this is the same as LISP “Extended” subnet mode, but in this case, we are not using a LAN extension between sites. Instead, the extended mode requires a LAN extension such as OTV.

LISP stretched subnets
LISP stretched subnets

 

Summary: LISP Hybrid Cloud Use Case

In the rapidly evolving world of cloud computing, businesses constantly seek innovative solutions to optimize their operations. One such groundbreaking approach is the utilization of LISP (Locator/ID Separation Protocol) in hybrid cloud environments. In this blog post, we explored the fascinating use case of LISP Hybrid Cloud and delved into its benefits, implementation, and potential for revolutionizing the industry.

Understanding LISP Hybrid Cloud

LISP Hybrid Cloud combines the best of two worlds: the scalability and flexibility of public cloud services with the security and control of private cloud infrastructure. By separating the location and identity of network devices, LISP allows for seamless communication between public and private clouds. This breakthrough technology enables businesses to leverage the advantages of both environments and optimize their cloud strategies.

Benefits of LISP Hybrid Cloud

Enhanced Scalability: LISP Hybrid Cloud offers unparalleled scalability by allowing businesses to scale their operations across public and private clouds seamlessly. This ensures that organizations can meet evolving demands without compromising performance or security.

Improved Flexibility: With LISP Hybrid Cloud, businesses can choose the most suitable cloud resources. They can leverage the vast capabilities of public clouds for non-sensitive workloads while keeping critical data and applications secure within their private cloud infrastructure.

Enhanced Security: LISP Hybrid Cloud provides enhanced security by leveraging the inherent advantages of private clouds. Critical data and applications can remain within the organization’s secure network, minimizing the risk of unauthorized access or data breaches.

Implementation of LISP Hybrid Cloud

Implementing LISP Hybrid Cloud involves several key steps. First, organizations must evaluate their cloud requirements and determine the optimal balance between public and private cloud resources. Next, they must deploy the necessary LISP infrastructure, including LISP routers and mapping servers. Finally, businesses must establish secure communication channels between their public and private cloud environments, ensuring seamless data transfer and interconnectivity.

Conclusion:

In conclusion, LISP Hybrid Cloud represents a revolutionary approach to cloud computing. By harnessing the power of LISP, businesses can unlock the potential of hybrid cloud environments, enabling enhanced scalability, improved flexibility, and heightened security. As the cloud landscape continues to evolve, LISP Hybrid Cloud is poised to play a pivotal role in shaping the future of cloud computing.

What is OpenFlow

What is OpenFlow

What is OpenFlow?

In today's rapidly evolving digital landscape, network management and data flow control have become critical for businesses of all sizes. OpenFlow is one technology that has gained significant attention and is transforming how networks are managed. In this blog post, we will delve into the concept of OpenFlow, its advantages, and its implications for network control.

OpenFlow is an open-standard communications protocol that separates the control and data planes in a network architecture. It allows network administrators to have direct control over the behavior of network devices, such as switches and routers, by utilizing a centralized controller.

Traditional network architectures follow a closed model, where network devices make independent decisions on forwarding packets. On the other hand, OpenFlow introduces a centralized control plane that provides a global view of the network and allows administrators to define network policies and rules from a centralized location.

OpenFlow operates by establishing a secure channel between the centralized controller and the network switches. The controller is responsible for managing the flow tables within the switches, defining how traffic should be forwarded based on predefined rules and policies. This separation of control and data planes allows for dynamic network management and facilitates the implementation of innovative network protocols.

One of the key advantages of OpenFlow is its ability to simplify network management. By centralizing control, administrators can easily configure and manage the entire network from a single point of control. This reduces complexity and enhances the scalability of network infrastructure. Additionally, OpenFlow enables network programmability, allowing for the development of custom networking applications and services tailored to specific requirements.

OpenFlow plays a crucial role in network virtualization, as it allows for the creation and management of virtual networks on top of physical infrastructure. By abstracting the underlying network, OpenFlow empowers organizations to optimize resource utilization, improve security, and enhance network performance. It opens doors to dynamic provisioning, isolation, and efficient utilization of network resources.

Highlights: What is OpenFlow?

How does OpenFlow work?

OpenFlow allows network controllers to determine the path of network packets in a network of switches. There is a difference between switches and controllers. With separate control and forwarding, traffic management can be more sophisticated than access control lists (ACLs) and routing protocols. An OpenFlow protocol allows switches from different vendors, often with proprietary interfaces and scripting languages, to be managed remotely. Software-defined networking (SDN) is considered to be enabled by OpenFlow by its inventors.

With OpenFlow, Layer 3 switches can add, modify, and remove packet-matching rules and actions remotely. By doing so, routing decisions can be made periodically or ad hoc by the controller and translated into rules and actions with a configurable lifespan, which are then deployed to the switch’s flow table, where packets are forwarded at wire speed for the duration of the rule. If the switch cannot match packets, they can be sent to the controller. The controller can modify existing flow table rules or deploy new rules to prevent a structural traffic flow. It may even forward the traffic itself if the switch is instructed to forward packets rather than just their headers.

OpenFlow uses Transport Layer Security (TLS) over Transmission Control Protocol (TCP). Switches wishing to connect should listen on TCP port 6653. In earlier versions of OpenFlow, port 6633 was unofficially used. The protocol is mainly used between switches and controllers.

Introducing SDN

Recent changes and requirements have driven networks and network services to become more flexible, virtualization-aware, and API-driven. One major trend affecting the future of networking is software-defined networking ( SDN ). The software-defined architecture aims to extract the entire network into a single switch.

Software-defined networking (SDN) is an evolving technology defined by the Open Networking Foundation ( ONF ). It involves the physical separation of the network control plane from the forwarding plane, where a control plane controls several devices. This differs significantly from traditional IP forwarding that you may have used in the past.

The activities around OpenFlow

Even though OpenFlow has received a lot of industry attention, programmable networks and decoupled control planes (control logic) from data planes have been around for many years. To enhance ATM, Internet, and mobile networks’ openness, extensibility, and programmability, the Open Signaling (OPENING) working group held workshops in 1995. A working group within the Internet Engineering Task Force (IETF) developed GSMP to control label switches based on these ideas. June 2002 marked the official end of this group, and GSMPv3 was published.

What is OpenFlow

Data and control plane

Therefore, SDN separates the data and control plane. The main driving body behind software-defined networking (SDN) is the Open Networking Foundation ( ONF ). Introduced in 2008, the ONF is a non-profit organization that wants to provide an alternative to proprietary solutions that limit flexibility and create vendor lock-in.

The insertion of the ONF allowed its members to run proof of concepts on heterogeneous networking devices without requiring vendors to expose their software’s internal code. This creates a path for an open-source approach to networking and policy-based controllers. 

Building blocks: SDN Environment 

As a fundamental building block of an SDN deployment, the controller, the SDN switch (for example, an OpenFlow switch), and the interfaces are present in the controller to communicate with forwarding devices, generally the southbound interface (OpenFlow) and the northbound interface (the network application interface). In an SDN, switches function as basic forwarding hardware, accessible via an open interface, with the control logic and algorithms offloaded to controllers. Hybrid (OpenFlow-enabled) and pure (OpenFlow-only) OpenFlow switches are available.

OpenFlow switches rely entirely on a controller for forwarding decisions, without legacy features or onboard control. Hybrid switches support OpenFlow as well, in addition to traditional operation and protocols. Today, hybrid switches are the most common type of commercial switch. A flow table performs packet lookup and forwarding in an OpenFlow switch.

You may find the following useful for pre-information:

  1. OpenFlow Protocol
  2. Network Traffic Engineering
  3. What is VXLAN
  4. SDN Adoption Report
  5. Virtual Device Context

Identify the Benefits of OpenFlow

Key What is OpenFlow Discussion Points:


  • Introduction to what is OpenFlow and what is involved with the protocol.

  • Highlighting the details and benefits of OpenFlow.

  • Technical details on the lack of session layers in the TCP/IP model.

  • Scenario: Control and data plane separation with SDN. 

  • A final note on proactive vs reactive flow setup.

Back to basics. What is OpenFlow?

What is OpenFlow?

OpenFlow was the first protocol of the Software-Defined Networking (SDN) trend and is the only protocol that allows decoupling a network device’s control plane from the data plane. In most straightforward terms, the control plane can be thought of as the brains of a network device. On the other hand, the data plane can be considered hardware or application-specific integrated circuits (ASICs) that perform packet forwarding.

Numerous devices also support running OpenFlow in a hybrid mode, meaning OpenFlow can be deployed on a given port, virtual local area network (VLAN), or even within a regular packet-forwarding pipeline such that if there is not a match in the OpenFlow table, then the existing forwarding tables (MAC, Routing, etc.) are used, making it more analogous to Policy Based Routing (PBR).

What is OpenFlow
Diagram: What is OpenFlow? The source is cable solutions.

What is SDN?

Despite various modifications to the underlying architecture and devices (such as switches, routers, and firewalls), traditional network technologies have existed since the inception of networking. Using a similar approach, frames, and packets have been forwarded and routed in a limited manner, resulting in low efficiency and high maintenance costs—consequently, the architecture and operation of networks needed to evolve, resulting in SDN.

By enabling network programmability, SDN promises to simplify network control and management and allow innovation in computer networking. Network engineers configure policies to respond to various network events and application scenarios. They can achieve the desired results by manually converting high-level policies into low-level configuration commands.

Often, minimal tools are available to accomplish these very complex tasks. Controlling network performance and tuning network management are challenging and error-prone tasks.

A modern network architecture consists of a control plane, a data plane, and a management plane; the control and data planes are merged into a machine called Inside the Box. To overcome these limitations, programmable networks have emerged.

How OpenFlow Works:

At the core of OpenFlow is the concept of a flow table, which resides in each OpenFlow-enabled switch. The flow table contains match-action rules defining how incoming packets should be processed and forwarded. The centralized controller determines these rules and communicates with the switches using the OpenFlow protocol.

When a packet arrives at an OpenFlow-enabled switch, it is first matched against the rules in the flow table. If a match is found, the corresponding action is executed, including forwarding the packet, dropping it, or sending it to the controller for further processing. This decoupling of the control and data planes allows for flexible and programmable network management.

What is OpenFlow SDN?

The main goal of SDN is to separate the control and data planes and transfer network intelligence and state to the control plane. These concepts have been exploited by technologies like Routing Control Platform (RCP), Secure Architecture for Network Enterprise (SANE), and, more recently, Ethane.

In addition, there is often a connection between SDN and OpenFlow. The Open Networking Foundation (ONF) is responsible for advancing SDN and standardizing OpenFlow, whose latest version is 1.5.0.

  • An SDN deployment starts with these building blocks.

For communication with forwarding devices, the controller has the SDN switch (for example, an OpenFlow switch), the SDN controller, and the interfaces. An SDN deployment is based on two basic building blocks: a southbound interface (OpenFlow) and a northbound interface (the network application interface).

As the control logic and algorithms are offloaded to a controller, switches in SDNs may be represented as basic forwarding hardware. Switches that support OpenFlow come in two varieties: pure (OpenFlow-only) and hybrid (OpenFlow-enabled).

Pure OpenFlow switches do not have legacy features or onboard control for forwarding decisions. A hybrid switch can operate with both traditional protocols and OpenFlow. Hybrid switches make up the majority of commercial switches available today. A flow table performs packet lookup and forwarding in an OpenFlow switch.

OpenFlow reference switch

The OpenFlow protocol and interface allow OpenFlow switches to be accessed as essential forwarding elements. A flow-based SDN architecture like OpenFlow simplifies switching hardware. Still, it may require additional forwarding tables, buffer space, and statistical counters that are difficult to implement in traditional switches with integrated circuits tailored to specific applications.

There are two types of switches in an OpenFlow network: hybrids (which enable OpenFlow) and pores (which only support OpenFlow). OpenFlow is supported by hybrid switches and traditional protocols (L2/L3). OpenFlow switches rely entirely on a controller for forwarding decisions and do not have legacy features or onboard control.

Hybrid switches are the majority of the switches currently available on the market. This link must remain active and secure because OpenFlow switches are controlled over an open interface (through a TCP-based TLS session). OpenFlow is a messaging protocol that defines communication between OpenFlow switches and controllers, which can be viewed as an implementation of SDN-based controller-switch interactions.

Openflow switch
Diagram: OpenFlow switch. The source is cable solution.

Identify the Benefits of OpenFlow

Application-driven routing. Users can control the network paths.

The networks paths.A way to enhance link utilization.

An open solution for VM mobility. No VLAN reliability.

A means to traffic engineer without MPLS.

A solution to build very large Layer 2 networks.

A way to scale Firewalls and Load Balancers.

A way to configure an entire network as a whole as opposed to individual entities.

A way to build your own encryption solution. Off-the-box encryption.

A way to distribute policies from a central controller.

Customized flow forwarding. Based on a variety of bit patterns.

A solution to get a global view of the network and its state. End-to-end visibility.

A solution to use commodity switches in the network. Massive cost savings.

The following table lists the Software Networking ( SDN ) benefits and the problems encountered with existing control plane architecture:

Identify the benefits of OpenFlow and SDN

Problems with the existing approach

Faster software deployment.

Large scale provisioning and orchestration.

Programmable network elements.

Limited traffic engineering ( MPLS TE is cumbersome )

Faster provisioning.

Synchronized distribution policies.

Centralized intelligence with centralized controllers.

Routing of large elephant flows.

Decisions are based on end-to-end visibility.

Qos and load based forwarding models.

Granular control of flows.

Ability to scale with VLANs.

Decreases the dependence on network appliances like load balancers.

  • A key point: The lack of a session layer in the TCP/IP stack.

Regardless of the hype and benefits of SDN, neither OpenFlow nor other SDN technologies address the real problems of the lack of a session layer in the TCP/IP protocol stack. The problem is that the client’s application ( Layer 7 ) connects to the server’s IP address ( Layer 3 ), and if you want to have persistent sessions, the server’s IP address must remain reachable. 

This session’s persistence and the ability to connect to multiple Layer 3 addresses to reach the same device is the job of the OSI session layer. The session layer provides the services for opening, closing, and managing a session between end-user applications. In addition, it allows information from different sources to be correctly combined and synchronized.

The problem is the TCP/IP reference module does not consider a session layer, and there is none in the TCP/IP protocol stack. SDN does not solve this; it gives you different tools to implement today’s kludges.

what is openflow
What is OpenFlow? Lack of a session layer

Control and data plane

When we identify the benefits of OpenFlow, let us first examine traditional networking operations. Traditional networking devices have a control and forwarding plane, depicted in the diagram below. The control plane is responsible for setting up the necessary protocols and controls so the data plane can forward packets, resulting in end-to-end connectivity. These roles are shared on a single device, and the fast packet forwarding ( data path ) and the high-level routing decisions ( control path ) occur on the same device.

What is OpenFlow | SDN separates the data and control plane

Control plane

The control plane is part of the router architecture and is responsible for drawing the network map in routing. When we mention control planes, you usually think about routing protocols, such as OSPF or BGP. But in reality, the control plane protocols perform numerous other functions, including:

Connectivity management ( BFD, CFM )

Interface state management ( PPP, LACP )

Service provisioning ( RSVP for InServ or MPLS TE)

Topology and reachability information exchange ( IP routing protocols, IS-IS in TRILL/SPB )

Adjacent device discovery via HELLO mechanism

ICMP

Control plane protocols run over data plane interfaces to ensure “shared fate” – if the packet forwarding fails, the control plane protocol fails as well.

Most control plane protocols ( BGP, OSPF, BFD ) are not data-driven. A BGP or BFD packet is never sent as a direct response to a data packet. There is a question mark over the validity of ICMP as a control plane protocol. The debate is whether it should be classed in the control or data plane category.

Some ICMP packets are sent as replies to other ICMP packets, and others are triggered by data plane packets, i.e., data-driven. My view is that ICMP is a control plane protocol that is triggered by data plane activity. After all, the “C” is ICMP does stand for “Control.”

Data plane

The data path is part of the routing architecture that decides what to do when a packet is received on its inbound interface. It is primarily focused on forwarding packets but also includes the following functions:

ACL logging

 Netflow accounting

NAT session creation

NAT table maintenance

The data forwarding is usually performed in dedicated hardware, while the additional functions ( ACL logging, Netflow accounting ) usually happen on the device CPU, commonly known as “punting.” The data plane for an OpenFlow-enabled network can take a few forms.

However, the most common, even in the commercial offering, is the Open vSwitch, often referred to as the OVS. The Open vSwitch is an open-source implementation of a distributed virtual multilayer switch. It enables a switching stack for virtualization environments while supporting multiple protocols and standards.

Identify the benefits of OpenFlow

Software-defined networking changes the control and data plane architecture.

The concept of SDN separates these two planes, i.e., the control and forwarding planes are decoupled. This allows the networking devices in the forwarding path to focus solely on packet forwarding. An out-of-band network uses a separate controller ( orchestration system ) to set up the policies and controls. Hence, the forwarding plane has the correct information to forward packets efficiently.

In addition, it allows the network control plane to be moved to a centralized controller on a server instead of residing on the same box carrying out the forwarding. Moving the intelligence ( control plane ) of the data plane network devices to a controller enables companies to use low-cost, commodity hardware in the forwarding path. A significant benefit is that SDN separates the data and control plane, enabling new use cases.

A centralized computation and management plane makes more sense than a centralized control plane.

The controller maintains a view of the entire network and communicates with Openflow ( or, in some cases, BGP with BGP SDN ) with the different types of OpenFlow-enabled network boxes. The data path portion remains on the switch, such as the OVS bridge, while the high-level decisions are moved to a separate controller. The data path presents a clean flow table abstraction, and each flow table entry contains a set of packet fields to match, resulting in specific actions ( drop, redirect, send-out-port ).

When an OpenFlow switch receives a packet it has never seen before and doesn’t have a matching flow entry, it sends the packet to the controller for processing. The controller then decides what to do with the packet.

Applications could then be developed on top of this controller, performing security scrubbing, load balancing, traffic engineering, or customized packet forwarding. The centralized view of the network simplifies problems that were harder to overcome with traditional control plane protocols.

A single controller could potentially manage all OpenFlow-enabled switches. Instead of individually configuring each switch, the controller can push down policies to multiple switches simultaneously—a compelling example of many-to-one virtualization.

Now that SDN separates the data and control plane, the operator uses the centralized controller to choose the correct forwarding information per-flow basis. This allows better load balancing and traffic separation on the data plane. In addition, there is no need to enforce traffic separation based on VLANs, as the controller would have a set of policies and rules that would only allow traffic from one “VLAN” to be forwarded to other devices within that same “VLAN.”

The advent of VXLAN

With the advent of VXLAN, which allows up to 16 million logical entities, the benefits of SDN should not be purely associated with overcoming VLAN scaling issues. VXLAN already does an excellent job with this. It does make sense to deploy a centralized control plane in smaller independent islands; in my view, it should be at the edge of the network for security and policy enforcement roles. Using Openflow on one or more remote devices is easy to implement and scale.

It also decreases the impact of controller failure. If a controller fails and its sole job is implementing packet filters when a new user connects to the network, the only affecting element is that the new user cannot connect. If the controller is responsible for core changes, you may have interesting results with a failure. New users not being able to connect is bad, but losing your entire fabric is not as bad.

Spanning tree VXLAN
Diagram: Loop prevention. Source is Cisco

What Is OpenFlow? Identify the Benefits of OpenFlow

A traditional networking device runs all the control and data plane functions. The control plane, usually implemented in the central CPU or the supervisor module, downloads the forwarding instructions into the data plane structures. Every vendor needs communications protocols to bind the two planes together to download forward instructions. 

Therefore, all distributed architects need a protocol between control and data plane elements. The protocol to bind this communication path for traditional vendor devices is not open-source, and every vendor uses its proprietary protocol (Cisco uses IPC – InterProcess Communication ).

Openflow tries to define a standard protocol between the control plane and the associated data plane. When you think of Openflow, you should relate it to the communication protocol between the traditional supervisors and the line cards. OpenFlow is just a low-level tool.

OpenFlow is a control plane ( controller ) to data plane ( OpenFlow enabled device ) protocol that allows the control plane to modify forwarding entries in the data plane. It enables SDN to separate the data and control planes.

identify the benefits of openflow

Proactive versus reactive flow setup

OpenFlow operations have two types of flow setups: Proactive and Reactive.

With Proactive, the controller can populate the flow tables ahead of time, similar to a typical routing. However, the packet-in event never occurs by pre-defining your flows and actions ahead of time in the switch’s flow tables. The result is all packets are forwarded at line rate. With Reactive, the network devices react to traffic, consult the OpenFlow controller, and create a rule in the flow table based on the instruction. The problem with this approach is that there can be many CPU hits.

OpenFlow protocol

The following table outlines the critical points for each type of flow setup:

Proactive flow setup

Reactive flow setup

Works well when the controller is emulating BGP or OSPF.

 Used when no one can predict when and where a new MAC address will appear.

The controller must first discover the entire topology.

 Punts unknown packets to the controller. Many CPU hits.

Discover endpoints ( MAC addresses, IP addresses, and IP subnets )

Compute forwarding paths on demand. Not off the box computation.

Compute off the box optimal forwarding.

 Install flow entries based on actual traffic.

Download flow entries to the data plane switches.

Has many scalability concerns such as packet punting rate.

No data plane controller involvement with the exceptions of ARP and MAC learning. Line-rate performance.

 Not a recommended setup.

Hop-by-hop versus path-based forwarding

The following table illustrates the keys point for the two types of forwarding methods used by OpenFlow; hop-by-hop forwarding and path-based forwarding:

Hop-by-hop Forwarding

 Path-based Forwarding

Similar to traditional IP Forwarding.

Similar to MPLS.

Installs identical flows on each switch on the data path.

Map flows to paths on ingress switches and assigns user traffic to paths at the edge node

Scalability concerns relating to flow updates after a change in topology.

Compute paths across the network and installs end-to-end path-forwarding entries.

Significant overhead in large-scale networks.

Works better than hop-by-hop forwarding in large-scale networks.

FIB update challenges. Convergence time.

Core switches don’t have to support the same granular functionality as edge switches.

Identify the benefits of OpenFlow with security.

Obviously, with any controller, the controller is a lucrative target for attack. Anyone who knows you are using a controller-based network will try to attack the controller and its control plane. The attacker may attempt to intercept the controller-to-switch communication and replace it with its commands, essentially attacking the control plane with whatever means they like.

An attacker may also try to insert a malformed packet or some other type of unknown packet into the controller ( fuzzing attack ), exploiting bugs in the controller and causing the controller to crash. 

Fuzzing attacks can be carried out with application scanning software such as Burp Suite. It attempts to manipulate data in a particular way, breaking the application.

The best way to tighten security is to encrypt switch-to-controller communications with SSL and self-signed certificates to authenticate the switch and controller. It would also be best to minimize interaction with the data plane, except for ARP and MAC learning.

To prevent denial-of-service attacks on the controller, you can use Control Plane Policing ( CoPP ) on Ingress to avoid overloading the switch and the controller. Currently, NEC is the only vendor implementing CoPP.

sdn separates the data and control plane

The Hybrid deployment model is helpful from a security perspective. For example, you can group specific ports or VLANs to OpenFlow and other ports or VLANs to traditional forwarding, then use traditional forwarding to communicate with the OpenFlow controller.

Identify the Benefits of OpenFlow

Software-defined networking or traditional routing protocols?

The move to a Software-Defined Networking architecture has clear advantages. It’s agile and can react quickly to business needs, such as new product development. For businesses to achieve success, they must have software that continues to evolve.

Otherwise, your customers and staff may lose interest in your product and service. The following table displays the advantages and disadvantages of the existing routing protocol control architecture.

+Reliable and well known.

-Non-standard Forwarding models. Destination-only and not load-aware metrics**

+Proven with 20 plus years field experience.

 -Loosely coupled.

+Deterministic and predictable.

-Lacks end-to-end transactional consistency and visibility.

+Self-Healing. Traffic can reroute around a failed node or link.

-Limited Topology discovery and extraction. Basic neighbor and topology tables.

+Autonomous.

-Lacks the ability to change existing control plane protocol behavior.

+Scalable.

-Lacks the ability to introduce new control plane protocols.

+Plenty of learning and reading materials.

** Basic EIGRP IETF originally proposed an Energy-Aware Control Plane, but the IETF later removed this.

Software-Defined Networking: Use Cases

Edge Security policy enforcement at the network edge.

Authenticate users or VMs and deploy per-user ACL before connecting a user to the network.

Custom routing and online TE.

The ability to route on a variety of business metrics aka routing for dollars. Allowing you to override the default routing behavior.

Custom traffic processing.

For analytics and encryption.

Programmable SPAN ports

 Use Openflow entries to mirror selected traffic to the SPAN port.

DoS traffic blackholing & distributed DoS prevention.

Block DoS traffic as close to the source as possible with more selective traffic targeting than the original RTBH approach**. The traffic blocking is implemented in OpenFlow switches. Higher performance with significantly lower costs.

Traffic redirection and service insertion.

Redirect a subset of traffic to network appliances and install redirection flow entries wherever needed.

Network Monitoring.

 The controller is the authoritative source of information on network topology and Forwarding paths.

Scale-Out Load Balancing.

Punt new flows to the Openflow controller and install per-session entries throughout the network.

IPS Scale-Out.

OpenFlow is used to distribute the load to multiple IDS appliances.

**Remote-Triggered Black Hole: RTBH refers to installing a host route to a bogus IP address ( RTBH address ) pointing to NULL interfaces on all routers. BGP is used to advertise the host routes to other BGP peers of the attacked hosts, with the next hop pointing to the RTBH address, and it is mainly automated in ISP environments.

SDN deployment models

Guidelines:

  1. Start with small deployments away from the mission-critical production path, i.e., the Core. Ideally, start with device or service provisioning systems.
  2. Start at the Edge and slowly integrate with the Core. Minimize the risk and blast radius. Start with packet filters at the Edge and tasks that can be easily automated ( VLANs ).
  3. Integrate new technology with the existing network.
  4. Gradually increase scale and gain trust. Experience is key.
  5. Have the controller in a protected out-of-band network with SSL connectivity to the switches.

There are 4 different models for OpenFlow deployment, and the following sections list the key points of each model.

Native OpenFlow 

  • They are commonly used for Greenfield deployments.
  • The controller performs all the intelligent functions.
  • The forwarding plane switches have little intelligence and solely perform packet forwarding.
  • The white box switches need IP connectivity to the controller for the OpenFlow control sessions. If you are forced to use an in-band network for this communication path using an isolated VLAN with STP, this should be done with an out-of-band network.
  • Fast convergence techniques such as BFD may be challenging to use with a central controller.
  • Many people believe that this approach does not work for a regular company. Companies implementing native OpenFlow, such as Google, have the time and resources to reinvent the wheel when implementing a new control-plane protocol ( OpenFlow ).

Native OpenFlow with Extensions

  • Some control plane functions are handled from the centralized controller to the forwarding plane switches. For example, the OpenFlow-enabled switches could load balancing across multiple links without the controller’s previous decision. You could also run STP, LACP, or ARP locally on the switch without interaction with the controller. This approach is helpful if you lose connectivity to the controller. If the low-level switches perform certain controller functions, packet forwarding will continue in the event of failure.
  • The local switches should support the specific OpenFlow extensions that let them perform functions on the controller’s behalf.

Hybrid ( Ships in the night )

  • This approach is used where OpenFlow runs in parallel with the production network.
  • The same network box is controlled by existing on-box and off-box control planes ( OpenFlow).
  • Suitable for pilot deployment models as switches still run traditional control plane protocols.
  • The Openflow controller manages only specific VLANs or ports on the network.
  • The big challenge is determining and investigating the conflict-free sharing of forwarding plane resources across multiple control planes.

Integrated OpenFlow

  • OpenFlow classifiers and forwarding entries are integrated with the existing control plane. For example, Juniper’s OpenFlow model follows this mode of operation where OpenFlow static routes can be redistributed into the other routing protocols.
  • No need for a new control plane.
  • No need to replace all forwarding hardware
  • It is the most practical approach as long as the vendor supports it.

Closing Points on OpenFlow

Advantages of OpenFlow:

OpenFlow brings several critical advantages to network management and control:

1. Flexibility and Programmability: With OpenFlow, network administrators can dynamically reconfigure the behavior of network devices, allowing for greater adaptability to changing network requirements.

2. Centralized Control: By centralizing control in a single controller, network administrators gain a holistic view of the network, simplifying management and troubleshooting processes.

3. Innovation and Experimentation: OpenFlow enables researchers and developers to experiment with new network protocols and applications, fostering innovation in the networking industry.

4. Scalability: OpenFlow’s centralized control architecture provides the scalability needed to manage large-scale networks efficiently.

Implications for Network Control:

OpenFlow has significant implications for network control, paving the way for new possibilities in network management:

1. Software-Defined Networking (SDN): OpenFlow is a critical component of the broader concept of SDN, which aims to decouple network control from the underlying hardware, providing a more flexible and programmable infrastructure.

2. Network Virtualization: OpenFlow facilitates network virtualization, allowing multiple virtual networks to coexist on a single physical infrastructure.

3. Traffic Engineering: By controlling the flow of packets at a granular level, OpenFlow enables advanced traffic engineering techniques, optimizing network performance and resource utilization.

Conclusion:

OpenFlow represents a paradigm shift in network control, offering a more flexible, scalable, and programmable approach to managing networks. By separating the control and data planes, OpenFlow empowers network administrators to have fine-grained control over network behavior, improving efficiency, innovation, and adaptability. As the networking industry continues to evolve, OpenFlow and its related technologies will undoubtedly play a crucial role in shaping the future of network management.

Summary: What is OpenFlow?

In the rapidly evolving world of networking, OpenFlow has emerged as a game-changer. This revolutionary technology has transformed the way networks are managed, offering unprecedented flexibility, control, and efficiency. In this blog post, we will delve into the depths of OpenFlow, exploring its definition, key features, and benefits.

What is OpenFlow?

OpenFlow can be best described as an open standard communications protocol that enables the separation of the control plane and the data plane in network devices. It allows centralized control over a network’s forwarding elements, making it possible to program and manage network traffic dynamically. By decoupling the intelligence of the network from the underlying hardware, OpenFlow provides a flexible and programmable infrastructure for network administrators.

Key Features of OpenFlow

a) Centralized Control: One of the core features of OpenFlow is its ability to centralize network control, allowing administrators to define and implement policies from a single point of control. This centralized control improves network visibility and simplifies management tasks.

b) Programmability: OpenFlow’s programmability empowers network administrators to define how network traffic should be handled based on their specific requirements. Through the use of flow tables and match-action rules, administrators can dynamically control the behavior of network switches and routers.

c) Software-Defined Networking (SDN) Integration: OpenFlow plays a crucial role in the broader concept of Software-Defined Networking. It provides a standardized interface for SDN controllers to communicate with network devices, enabling dynamic and automated network provisioning.

Benefits of OpenFlow

a) Enhanced Network Flexibility: With OpenFlow, network administrators can easily adapt and customize their networks to suit evolving business needs. The ability to modify network behavior on the fly allows for efficient resource allocation and improved network performance.

b) Simplified Network Management: By centralizing network control, OpenFlow simplifies the management of complex network architectures. Policies and configurations can be applied uniformly across the network, reducing administrative overhead and minimizing the chances of configuration errors.

c) Innovation and Experimentation: OpenFlow fosters innovation by providing a platform for the development and deployment of new network protocols and applications. Researchers and developers can experiment with novel networking concepts, paving the way for future advancements in the field.

Conclusion:

OpenFlow has ushered in a new era of network management, offering unparalleled flexibility and control. Its ability to separate the control plane from the data plane, coupled with centralized control and programmability, has opened up endless possibilities in network architecture design. As organizations strive for more agile and efficient networks, embracing OpenFlow and its associated technologies will undoubtedly be a wise choice.

What is VXLAN

What is VXLAN

What is VXLAN

In the rapidly evolving networking world, virtualization has become critical for businesses seeking to optimize their IT infrastructure. One key technology that has emerged is VXLAN (Virtual Extensible LAN), which enables the creation of virtual networks independent of physical network infrastructure. In this blog post, we will delve into the concept of VXLAN, its benefits, and its role in network virtualization.

VXLAN is an encapsulation protocol designed to extend Layer 2 (Ethernet) networks over Layer 3 (IP) networks. It provides a scalable and flexible solution for creating virtualized networks, enabling seamless communication between virtual machines (VMs) and physical servers across different data centers or geographic regions.

VXLAN is a technology that creates virtual networks within an existing physical network. A Layer 2 overlay network runs on top of the current Layer 2 network. VXLAN utilizes UDP as the transport protocol, providing a secure, efficient, and reliable way to create a virtual network.

VXLAN encapsulates the original Layer 2 Ethernet frames within UDP packets, using a 24-bit VXLAN Network Identifier (VNI) to distinguish between different virtual networks. The encapsulated packets are then transmitted over the underlying IP network, enabling the creation of virtualized Layer 2 networks across Layer 3 boundaries.

- Scalability: VXLAN solves the limitations of traditional VLANs by providing a much larger network identifier space, accommodating up to 16 million virtual networks. This scalability allows for the efficient isolation and segmentation of network traffic in highly virtualized environments.

VXLAN enables the decoupling of virtual and physical networks, providing the flexibility to move virtual machines across different physical hosts or even data centers without the need for reconfiguration. This flexibility greatly simplifies workload mobility and enhances overall network agility.

- Multitenancy: With VXLAN, multiple tenants can securely share the same physical infrastructure while maintaining isolation between their virtual networks. This is achieved by assigning unique VNIs to each tenant, ensuring their traffic remains separate and secure.

- Underlay Network: VXLAN relies on an IP underlay network, which must provide sufficient bandwidth, low latency, and optimal routing. Careful planning and design of the underlay network are crucial to ensure optimal VXLAN performance.

- Network Virtualization Gateway: To enable communication between VXLAN-based virtual networks and traditional VLAN-based networks, a network virtualization gateway, such as a VXLAN Gateway or an overlay-to-underlay gateway, is required. These gateways bridge the gap between virtual and physical networks, facilitating seamless connectivity.

Highlights: What is VXLAN

Data centers evolution

In recent years, data centers have seen a significant evolution. This evolution has brought popular technologies such as virtualization, cloud computing (private, public, and hybrid), and software-defined networking (SDN). Mobile-first and cloud-native data centers must scale, be agile, secure, consolidate, and integrate with compute/storage orchestrators. As well as visibility, automation, ease of management, operability, troubleshooting, and advanced analytics, today’s data center solutions are expected to include many other features.

A more service-centric approach is replacing device-by-device management. Most requests for proposals (RFPs) specify open application programming interfaces (APIs) and standards-based protocols to prevent vendor lock-in. A Cisco Virtual Extensible LAN (VXLAN)-based fabric using Nexus switches2 and NX-OS controllers form Cisco Virtual Extensible LAN (VXLAN).

what is spine and leaf architecture
Diagram: What is spine and leaf architecture. 2-Tier Spine Leaf Design

Issues with STP

When a switch receives redundant paths, the spanning tree protocol must designate one of those paths as blocked to prevent loops. While this mechanism is necessary, it can lead to suboptimal network performance. Blocked ports limit bandwidth utilization, which can be particularly problematic in environments with heavy data traffic.

One significant concern with the spanning tree protocol is its slow convergence time. When a network topology changes, the protocol takes time to recompute the spanning tree and reestablish connectivity. During this convergence period, network downtime can occur, disrupting critical operations and causing frustration for users.

stp port states

What is VXLAN?

The Internet Engineering Task Force (IETF) developed VXLAN, or Virtual eXtensible Local-Area Network, as a network virtualization technology standard. Multi-tenant networks allow multiple organizations to share a physical network without accessing each other’s traffic.

The VXLAN can be compared to individual apartments in a building: each apartment is a separate, private dwelling within a shared physical structure, just as each VXLAN is a discrete, private network segment within a shared physical infrastructure.

With VXLANs, physical networks can be segmented into 16 million logical networks. To encapsulate Layer 2 Ethernet frames, User Datagram Protocol (UDP) packets with a VXLAN header are used. Combining VXLAN with Ethernet virtual private networks (EVPNs), which transport Ethernet traffic over WAN protocols, allows Layer 2 networks to be extended across Layer 3 IP or MPLS networks.

VXLAN vs. GRE

VXLAN, an overlay network technology, is designed to address the limitations of traditional VLANs. It enables the creation of virtual networks over an existing Layer 3 infrastructure, allowing for more flexible and scalable network deployments. VXLAN operates by encapsulating Layer 2 Ethernet frames within UDP packets, extending Layer 2 domains across Layer 3 boundaries.

GRE, on the other hand, is a simple IP packet encapsulation protocol. It provides a mechanism for encapsulating arbitrary protocols over an IP network and is widely used for creating point-to-point tunnels. GRE encapsulates the payload packets within IP packets, making it a versatile option for connecting remote networks securely.

GRE without IPsec

Key VXLAN advantages

Because VXLANs are encapsulated inside UDP packets, they can run on any network that can send UDP packets. No matter how far a VTEP is from the decapsulating VTEP physically or geographically, it must forward UDP datagrams. 

VXLAN and EVPN enable operators to create virtual networks from physical ports on any Layer 3 network switch supporting the standard. Connecting a port on switch A to two ports on switch B and another port on switch C creates a virtual network that appears to all connected devices as one physical network. Devices in this virtual network cannot see VXLANs or the underlying network fabric.

Problems that VXLAN solves

In the same way, as server virtualization has increased agility and flexibility, decoupling virtual networks from physical infrastructure has done the same. Therefore, network operators can scale their infrastructure rapidly and economically to meet growing demand while securely sharing a single physical network. For privacy and security reasons, networks are segmented to prevent one tenant from seeing or accessing the traffic of another.

In a similar way to traditional virtual LANs (VLANs), VXLANs enable operators to overcome the scaling limitations associated with VLANs.

  • Up to 16 million VXLANs can be created in an administrative domain, compared to 4094 traditional VLANs. Cloud and service providers can segment networks using VXLANs to support many tenants.
  • By using a VXLAN, you can create network segments between different data centers. In traditional VLAN networks, broadcast domains are created by segmenting traffic by VLAN tags, but once a packet containing VLAN tags reaches a router, the VLAN information is removed. There is no limit to the distance VLANs can travel within a Layer 2 network. Layer 3 boundaries, such as virtual machine migration, are generally avoided in certain use cases. Segmenting networks based on VXLAN encapsulates packets as UDP packets, while segmenting networks based on VXLAN encapsulates packets as IP packets. A virtual overlay network can extend as far as the physical Layer 3 routed network can reach when all switches and routers in the path support VXLAN without the applications running on the overlay network having to cross any Layer 3 boundaries. Servers connected to the network are still part of the Layer 2 network, even though UDP packets may have transited one or more routers.
  • Using Layer 2 segmentation on top of an underlying Layer 3 network allows one to segment a Layer 2 network over an underlying Layer 3 network and support many network segments. By providing Layer 2 segmentation on top of an underlying Layer 3 network, Layer 2 networks can remain small even if they are distant. Smaller Layer 2 networks can prevent MAC table overflows on switches.

Primary VXLAN applications

A service provider or cloud provider deploys VXLAN for apparent reasons: they have many tenants or customers, and they must separate the traffic of one customer from another due to legal, privacy, and ethical considerations.

Users, departments, or other groups of network-segmented devices may be tenants in enterprise environments for security reasons. Isolating IoT network traffic from production network applications is a good security practice for Internet of Things (IoT) devices such as data center environmental sensors.

VXLAN has been widely adopted and is now used in many large enterprise networks for virtualization and cloud computing. It provides:

  • A secure and efficient way to create virtual networks.
  • Allowing for the creation of multi-tenant segmentation.
  • Efficient routing.
  • Hardware-agnostic capabilities.

With its widespread adoption, VXLAN has become an essential technology for network virtualization.

Related: Before you proceed, you may find the following posts helpful for pre-information:

  1. Data Center Topologies
  2. Segment Routing
  3. What is OpenFlow
  4. Overlay Virtual Networks
  5. Layer 3 Data Center

What is VXLAN

Key What is VXLAN Discussion Points:


  • Introduction to What is VXLAN and what is involved.

  • Highlighting the details of VXLAN vs VLAN.

  • Technical details on the VXLAN Spanning Tree.

  • Scenario: Why introduce VXLAN? VLXAN benefits. 

  • A final note on the VXLAN enhancements.

Back to Basics: The Need For VXLAN

Traditional layer two networks have issues because of the following reasons:

  • Spanning tree: Restricts links.
  • Limited amount of VLANs: Restricts scalability;
  • Large MAC address tables: Restricts scalability and mobility

Spanning-tree avoids loops by blocking redundant links. By blocking connections, we create a loop-free topology and pay for links we can’t use. Although we could switch to a layer three network, some technologies require layer two networking.

VLAN IDs are 12 bits long, so we can create 4094 VLANs (0 and 4095 are reserved). Data centers may need help with only 4094 available VLANs. Let’s say we have a service provider with 500 customers. There are 4094 available VLANs, so each customer can only have eight.

STP Path distribution

The Role of Server Virtualization

Server virtualization has exponentially increased the number of addresses in our switches’ MAC addresses. Before server virtualization, there was only one MAC address per switch port. We can run many virtual machines (VMs) or containers on a single physical server with server virtualization. Virtual NICs and virtual MAC addresses are assigned to each virtual machine. One switch port must learn many MAC addresses.

A data center could have 24 or 48 physical servers connected to a top-of-rack (ToR) switch. Since there may be many racks in a data center, each switch must store the MAC addresses of all VMs that communicate. Networks without server virtualization require much larger MAC address tables.

1st Lab Guide: VXLAN

In the following lab, I created a Layer 2 overlay with VXLAN over a Layer 3 core. A bridge domain VNI of 6001 must match both sides of the overlay tunnel. What Is a VNI? The VLAN ID field in an Ethernet frame has only 12 bits, so VLAN cannot meet isolation requirements on data center networks. The emergence of VNI specifically solves this problem.

Note: The VNI

A VNI is a user identifier similar to a VLAN ID. A VNI identifies a tenant. VMs with different VNIs cannot communicate at Layer 2. During VXLAN packet encapsulation, a 24-bit VNI is added to a VXLAN packet, enabling VXLAN to isolate many tenants.

In the screenshot below, you will notice that I can ping from desktop 0 to desktop one even though the IP addresses are not in the routing table of the core devices, simulating a Layer 2 overlay. Consider VXLAN to be the overlay and the routing Layer 3 core to be the underlay.

VXLAN overlay
Diagram: VXLAN Overlay

In the following screenshot, notice that the VNI has been changed. The VNI needs to be changed in two places in the configuration, as illustrated below. Once changed, the Peers are down; however, the NVE  interface remains up. The VXLAN layer two overlay is not operational.

Diagram: Changing the VNI

How does VXLAN work?

VXLAN uses tunneling to encapsulate Layer 2 Ethernet frames within IP packets. A unique 24-bit segment ID, the VXLAN Network Identifier (VNI), identifies each VXLAN network. The source VM encapsulates the original Ethernet frame with a VXLAN header, including the VNI. The encapsulated packet is then sent over the physical IP network to the destination VM and decapsulated to retrieve the original Ethernet frame.

Analysis:

Notice below that it is running a ping from desktop 0 to desktop 1. The IP addresses assigned to this host are 10.0.0.1 and 10.0.0.2. First, notice that the ping is booming. When I do a packet capture on the links Gi1 connected to Leaf A, we see the encapsulation of the ICMP echo request and reply.

Everything is encapsulated into UDP port 1024. In my configurations of Leaf A and Leaf B, I explicitly set the VXLAN port to 1024.

VXLAN unicast mode

Benefits of VXLAN:

– Scalability: VXLAN allows the creation of up to 16 million logical networks, providing the scalability required for large-scale virtualized environments.

– Network Segmentation: By leveraging VXLAN, organizations can segment their networks into virtual segments, enhancing security and isolating traffic between applications or user groups.

– Flexibility and Mobility: VXLAN enables the movement of VMs across physical servers and data centers without the need to reconfigure network settings. This flexibility is crucial for workload mobility in dynamic environments.

– Interoperability: VXLAN is an industry-standard protocol supported by various networking vendors, ensuring compatibility across different network devices and platforms.

Data Center

VXLAN

VXLAN Benefits

  • Scalability

  • Network Segmentation

  • Flexibility and Mobility

  • Interopability 

Data Center

VXLAN

VLAN Use Cases

  • Data Center Interconnect (DCI)

  • Multi Tenant Environments

  • Network Virtualization

  • Hybrid Cloud Connectivity

Use Cases for VXLAN:

– Data Center Interconnect (DCI): VXLAN allows organizations to interconnect multiple data centers, enabling seamless workload migration, disaster recovery, and workload balancing across different locations.

– Multi-Tenant Environments: VXLAN enables service providers to offer virtualized network services to multiple tenants securely and isolatedly. This is particularly useful in cloud computing environments.

– Network Virtualization: VXLAN plays a crucial role in network virtualization, allowing organizations to create virtual networks independent of the underlying physical infrastructure. This enables greater flexibility and agility in managing network resources.

Back to Basics: VXLAN and Network Virtualization.

VXLAN and network virtualization

VXLAN is a form of network virtualization. Network virtualization cuts a single physical network into many virtual networks, often called network overlays. Virtualizing a resource allows it to be shared by multiple users. Virtualization provides the illusion that each user is on his or her resources. In the case of virtual networks, each user is under the misconception that there are no other users of the network. To preserve the illusion, virtual networks are separated from one another. Packets cannot leak from one virtual network to another.

Network Virtualization
Diagram: Network Virtualization. Source Parallels

VXLAN Loop Detection and Prevention

So, before we dive into the benefits of VXLAN, let us address the basics of loop detection and prevention, which is a significant driver for using network overlays such as VLXAN. The challenge is that data frames can exist indefinitely when loops occur, disrupting network stability and degrading performance.

In addition, loops introduce broadcast radiation, increasing CPU and network bandwidth utilization, which degrades user application access experience. Finally, in multi-site networks, a loop can span multiple data centers, causing disruptions that are difficult to pinpoint. Overlay networking can solve much of this.

VXLAN vs VLAN

However, first-generation Layer-2 Ethernet networks could not natively detect or mitigate looped topologies, while modern Layer-2 overlays implicitly build loop-free topologies. Therefore, overlays do not need loop detection and mitigation as long as no first-gen Layer-2 network is attached. Essentially, there is no need for a VXLAN spanning tree.

So, one of the differences between VXLAN vs VLAN is that the VLAN has a 12-bit VID while VXLAN has a 24-bit VID network identifier, allowing you to create up to 16 million segments. VXLAN has tremendous scale and stable loop-free networking and is a foundation technology in the ACI Cisco.

Spanning tree VXLAN
Diagram: Loop prevention. Source is Cisco

VXLAN and Data Center Interconnect

VXLAN has revolutionized data center interconnect by providing a scalable, flexible, and efficient solution for extending Layer 2 networks. Its ability to enable network segmentation, multi-tenancy support, and seamless mobility makes it a valuable technology for modern businesses.

However, careful planning, consideration of network infrastructure, and security measures are essential for successful implementation. By harnessing the power of VXLAN, organizations can achieve a more agile, scalable, and interconnected data center environment.

Considerations for Implementing VXLAN:

1. Underlying Network Infrastructure: Before implementing VXLAN, it is essential to assess the underlying network infrastructure. Network devices must support VXLAN encapsulation and decapsulation and have sufficient bandwidth to handle the increased traffic.

2. Network Overhead: While VXLAN provides numerous benefits, it does introduce additional network overhead due to encapsulation and decapsulation processes. It is crucial to consider the impact on network performance and plan accordingly.

3. Security: As VXLAN extends Layer 2 networks over Layer 3 infrastructure, appropriate security measures must be implemented. These include encrypting VXLAN traffic, deploying access control policies, and monitoring network traffic for anomalies.

VXLAN vs VLAN: The VXLAN Benefits Drive Adoption

Introduced by Cisco and VMware and now heavily used in open networking, VXLAN stands for Virtual eXtensible Local Area Network and is perhaps the most popular overlay technology for IP-based SDN data centers. And is used extensively with ACI networks.

VXLAN was explicitly designed for Layer 2 over Layer 3 tunneling. Its early competition from NVGRE and STT are fading away, and VXLAN is becoming the industry standard. VLXAN brings many advantages, especially in loop prevention, as there is no need for a VXLAN spanning tree.

VXLAN Benefits
VXLAN Benefits: Scale and loop-free networks.

Today, with overlays such as with VXLAN, the dependency on loop prevention protocols is almost eliminated. However, even though virtualized overlay networks such as VXLAN are loop-free, having a failsafe loop detection and mitigation method is still desirable because loops can be introduced by topologies connected to the overlay network.

Loop prevention traditionally started with Spanning Tree Protocols (STP) to counteract the loop problem in first-gen Layer-2 Ethernet networks. Over time, other approaches evolved by moving networks from “looped topologies” to “loop-free topologies.

While LAG and MLAG were used, other approaches for building loop-free topologies arose using ECMP at the MAC or IP layers. For example, FabricPath or TRILL is a MAC layer ECMP approach that emerged in the last decade. More recently, network virtualization overlays that build loop-free topologies on top of IP layer ECMP became state-of-the-art.

What is VXLAN
What is VXLAN and the components involved?

VXLAN vs VLAN: Why Introduce VXLAN?

  1. STP issues and scalability constraints: STP is undesirable on a large scale and lacks a proper load-balancing mechanism. A solution was needed to leverage the ECMP capabilities of an IP network while offering extended VLANs across an IP core, i.e., virtual segments across the network core. There is no VXLAN spanning tree.
  2. Multi-tenancy: Layer 2 networks are capped at 4000 VLANs, restricting multi-tenancy design—a big difference in the VXLAN vs VLAN debates.
  3. ToR table scalability: Every ToR switch may need to support several virtual servers, and each virtual server requires several NICs and MAC addresses. This pushes the limits on the table sizes for the ToR switch. In addition, after the ToR tables become full, Layer 2 traffic will be treated as unknown unicast traffic, which will be flooded across the network, causing instability to a previously stable core.
STP Blocking.
Diagram: STP Blocking. Source Cisco Press free chapter.

VXLAN use cases

Use Case 

VXLAN Details

Use Case 1

Multi-tenant IaaS Clouds where you need a large number of segments

Use Case 2

Link Virtual to Physical Servers. This is done via software or hardware VXLAN to VLAN gateway

Use Case 3

HA Clusters across failure domains/availability zones

Use Case 4

VXLAN works well over fabrics that have equidistant endpoints

Use Case 5

VXLAN-encapsulated VLAN traffic across availability zones must be rate-limited to prevent broadcast storm propagation across multiple availability zones

What is VXLAN? The operations

When discussing VXLAN vs VLAN, VXLAN employs a MAC over IP/UDP overlay scheme and extends the traditional VLAN boundary of 4000 VLANs. The 12-bit VLAN identifier in traditional VLANs capped scalability within the SDN data center and proved cumbersome if you wanted a VLAN per application segment model. VXLAN scales the 12-bit to a 24-bit identifier and allows for 16 million logical endpoints, with each endpoint potentially offering another 4,000 VLANs.

While tunneling does provide Layer 2 adjacency between these logical endpoints and allows VMs to move across boundaries, the main driver for its insertion was to overcome the challenge of having only 4000 VLAN.

Typically, an application segment has multiple segments; between each segment, you will have firewalling and load-balancing services, and each segment requires a different VLAN. The Layer 2 VLAN segment transfers non-routable heartbeats or state information that can’t cross an L3 boundary. If you are a cloud provider, you will soon reach the 4000k VLAN limit.

vxlan vs vlan
Multiple segments are required per application stack.

The control plane

The control plane is very similar to the spanning tree control plane. If a switch receives a packet destined for an unknown address, the switch will forward the packet to an IP address that floods the packet to all the other switches.

This IP address is, in turn, mapped to a multicast group across the network. VXLAN doesn’t explicitly have a control plane and requires an IP multicast running in the core for forwarding traffic and host discovery.

Best practices for enabling IP Multicast in the core

IP Multicast

In the Core

  1. Bidirectional PIM or PIM Sparse Mode
  1. Redundant Rendezvous Points (RP)
  1. Shared trees (reduce the amount of IP multicast state)
  1. Always check the IP multicast table sizes on core and ToR switches
  1. Single IP multicast address for multiple VXLAN segments is OK

The requirement for IP multicast in the core made VXLAN undesirable from an operation point of view. For example, creating the tunnel endpoints is simple, but introducing a protocol like IP multicast to a core just for the tunnel control plane was considered undesirable. As a result, some of the more recent versions of VXLAN support IP unicast.

VXLAN uses a MAC over IP/UDP solution to eliminate the need for a spanning tree. There is no VXLAN spanning tree. This enables the core to be IP and not run a spanning tree. Many people ask why VXLAN uses UDP. The reason is that the UDP port numbers cause VXLAN to inherit Layer 3 ECMP features. The entropy that enables load balancing across multiple paths is embedded into the UDP source port of the overlay header.

2nd Lab Guide: Multicast VLXAN

In this lab guide, we will look at a VXLAN multicast mode. The multicat mode requires both unicast and multicast connectivity between sites. Similar to the previous one, this configuration guide uses OSPF to provide unicast connectivity, and now we have an additional bidirectional Protocol Independent Multicast (PIM) to provide multicast connectivity.

This does not mean that you don’t have a multicast-enabled core. It would be best if you still had multicast enabled on the core. 

So we are not, let’s say, tunneling multicast over an IPv4 core without having multicast enabled on the core. I have multicast on all Layer 3 interfaces, and the mroute table is populated on all Layer 3 routers. With the command: Show ip mroute we are tunneling the multicast traffic, and with the command: Show nve vni we have multicast group 239.0.0.10, and we have a state of UP.

Multicast VXLAN
Diagram: Multicast VXLAN

VXLAN benefits and stability

The underlying control plan network impacts the stability of VXLAN and the applications running within it. For example, if the underlying IP network cannot converge quickly enough, VLXAN packets may be dropped, and an application cache timeout may be triggered.

The rate of change in the underlying network has a significant impact on the stability of the tunnels, yet the rate and change of the tunnels do not affect the underlying control plane. This is similar to how the strength of an MPLS / VPN overlay is affected by the core’s IGP.

VXLAN Points

VXLAN benefits

VXLAN drawbacks

Point 1

Runs over IP Transport

 No control plane

Point 2

Offers a large number of logical endpoints 

Needs IP Multicast***

Point 3

Reduced flooding scope

No IGMP snooping ( yet )

Point 4

Eliminates STP

No Pvlan support

Point 5

Easily integrated over existing Core

Requires Jumbo frames in the core ( 50 bytes)

Point 6

Minimal host-to-network integration

No built-in security features **

Point 7

Not a DCI solution ( no arp reduction, first-hop gateway localization, no inbound traffic steering i.e, LISP )

** VXLAN has no built-in security features. Anyone who gains access to the core network can insert traffic into segments. The VXLAN transport network must be secure, as no existing firewall or intrusion prevention system (IPS) equipment can be seen in the VXLAN traffic.

*** Recent versions have Unicast VXLAN. Nexus 1000V release 4.2(1)SV2(2.1)

Updated: VXLAN enhancements

MAC distribution mode is an enhancement to VXLAN that prevents unknown unicast flooding and eliminates data plane MAC address learning. Traditionally, this was done by flooding to locate an unknown end host, but it has now been replaced with a control plane solution.

During VM startup, the VSM ( control plane ) collects the list of MAC addresses and distributes the MAC-to-VTEP mappings to all VEMs participating in a VXLAN segment. This technique makes VXLAN more optimal by unicasting more intelligently, similar to Nicira and VMware NVP.

ARP termination works by giving the VSM controller all the ARP and MAC information. This enables the VSM to proxy and respond locally to ARP requests without sending a broadcast. Because 90% of broadcast traffic is ARP requests ( ARP reply is unicast ), this significantly reduces broadcast traffic on the network.

Final Notes: VXLAN

In recent years, the rapid growth of cloud computing and the increasing demand for scalable and flexible networks have led to the development of various technologies to address these needs. One such technology is VXLAN (Virtual Extensible LAN), an overlay network protocol that has gained significant popularity in networking. In this blog post, we will delve into the intricacies of VXLAN, exploring its key features, benefits, and use cases.

What is VXLAN?

VXLAN is a network overlay technology that enables the creation of virtualized Layer 2 networks over existing Layer 3 infrastructure. It was developed to address the limitations of traditional VLANs, which could not scale beyond a few thousand networks due to the limited number of VLAN IDs available. VXLAN solves this problem using a 24-bit VXLAN Network Identifier (VNI), allowing for an impressive 16 million unique network segments.

Key Features of VXLAN:

1. Scalability: As mentioned earlier, VXLAN’s use of a 24-bit VNI allows for a significantly larger number of network segments than traditional VLANs. This scalability makes VXLAN an ideal solution for large-scale virtualized environments.

2. Network Segmentation: VXLAN enables the creation of logical network segments, allowing for network isolation and improved security. By encapsulating Layer 2 Ethernet frames within Layer 3 UDP packets, VXLAN provides a flexible and scalable approach to network segmentation.

3. Multicast Support: VXLAN leverages IP multicast to efficiently distribute broadcast, unknown unicast, and multicast (BUM) traffic across the network. This feature reduces network congestion and improves overall performance.

4. Mobility: VXLAN supports seamless virtual machines (VMs) movement across physical hosts and data centers. By decoupling the VMs from the underlying physical network, VXLAN enables mobility without requiring any changes to the network infrastructure.

Benefits of VXLAN:

1. Enhanced Network Flexibility: VXLAN enables the creation of virtualized networks decoupled from the underlying physical infrastructure. This flexibility allows for easier network provisioning, scaling, and reconfiguration, making it an ideal choice for cloud environments.

2. Improved Scalability: With its larger network segment capacity, VXLAN offers improved scalability compared to traditional VLANs. This scalability is crucial in modern data centers and cloud environments where virtual machines and network segments are continuously growing.

3. Simplified Network Management: VXLAN simplifies network management tasks by abstracting the network infrastructure. Network administrators can define and manage virtual networks independently of the underlying physical infrastructure, streamlining network operations and reducing complexity.

Use Cases for VXLAN:

1. Data Center Interconnect: VXLAN is widely used for interconnecting geographically dispersed data centers. By extending Layer 2 network connectivity over Layer 3 infrastructure, VXLAN facilitates seamless VM mobility, disaster recovery, and workload balancing across data centers.

2. Multi-tenancy in Cloud Environments: VXLAN allows cloud service providers to create isolated network segments for different tenants, enhancing security and providing dedicated network resources. This feature is vital in multi-tenant cloud environments where data privacy and network isolation are critical.

3. Network Virtualization: VXLAN plays a crucial role in network virtualization, enabling the creation of virtual networks that are independent of the underlying physical infrastructure. This virtualization simplifies network management, enhances flexibility, and enables efficient resource utilization.

VXLAN has emerged as a powerful network virtualization technology with many use cases. VXLAN provides the flexibility, scalability, and efficiency required in modern networking environments, from data center virtualization to multi-tenancy, hybrid cloud connectivity, and disaster recovery. As organizations continue to embrace cloud computing and virtualization, VXLAN will undoubtedly play a pivotal role in shaping the future of networking.

Summary: What is VXLAN

VXLAN, short for Virtual Extensible LAN, is a network virtualization technology that has recently gained significant popularity. In this blog post, we will examine VXLAN’s definition, workings, and benefits. So, let’s dive into the world of VXLAN!

Understanding VXLAN Basics

VXLAN is an encapsulation protocol that enables the creation of virtual networks over existing Layer 3 infrastructures. It extends Layer 2 segments over Layer 3 networks, allowing for greater flexibility and scalability. By encapsulating Layer 2 frames within Layer 3 packets, VXLAN enables efficient communication between virtual machines (VMs) across physical hosts or data centers.

VXLAN Operation and Encapsulation

To understand how VXLAN works, we must look at its operation and encapsulation process. When a VM sends a Layer 2 frame, it is encapsulated into a VXLAN packet by adding a VXLAN header. This header includes information such as the VXLAN network identifier (VNI), which helps identify the virtual network to which the packet belongs. The VXLAN packet is then transported over the underlying Layer 3 network to the destination physical host, encapsulated, and delivered to the appropriate VM.

Benefits and Use Cases of VXLAN

VXLAN offers several benefits that make it an attractive choice for network virtualization. Firstly, it enables the creation of large-scale virtual networks, allowing for seamless VM mobility and workload placement flexibility. VXLAN also helps overcome the limitations of traditional VLANs by providing a much larger address space, accommodating the ever-growing number of virtual machines in modern data centers. Additionally, VXLAN facilitates network virtualization across geographically dispersed data centers, making it ideal for multi-site deployments and disaster recovery scenarios.

VXLAN vs. Other Network Virtualization Technologies

While VXLAN is widely used, it is essential to understand its key differences and advantages compared to other network virtualization technologies. For instance, VXLAN offers better scalability and flexibility than traditional VLANs. It also provides better isolation and segmentation of virtual networks, making it an ideal choice for multi-tenant environments. Additionally, VXLAN is agnostic to the physical network infrastructure, allowing it to be easily deployed in existing networks without requiring significant changes.

Conclusion:

In conclusion, VXLAN is a powerful network virtualization technology that has revolutionized how virtual networks are created and managed. Its ability to extend Layer 2 networks over Layer 3 infrastructures, scalability, flexibility, and ease of deployment make VXLAN a go-to solution for modern data centers. Whether for workload mobility, multi-site implementations, or overcoming VLAN limitations, VXLAN offers a robust and efficient solution. Embracing VXLAN can unlock new possibilities in network virtualization, enabling organizations to build agile, scalable, and resilient virtual networks.

multipath tcp

Data Center Topologies

Data Center Topology

In the world of technology, data centers play a crucial role in storing, managing, and processing vast amounts of digital information. However, behind the scenes, a complex infrastructure known as data center topology enables seamless data flow and optimal performance. In this blog post, we will delve into the intricacies of data center topology, its different types, and how it impacts the efficiency and reliability of data centers.

Data center topology refers to a data center's physical and logical layout. It encompasses the arrangement and interconnection of various components like servers, storage devices, networking equipment, and power sources. A well-designed topology ensures high availability, scalability, and fault tolerance while minimizing latency and downtime. As technology advances, so does the landscape of data center topologies. Here are a few emerging trends worth exploring:

Leaf-Spine Architecture: This modern approach replaces the traditional three-tier architecture with a leaf-spine model. It offers high bandwidth, low latency, and improved scalability, making it ideal for cloud-based applications and data-intensive workloads.

Software-Defined Networking (SDN): SDN introduces a new level of flexibility and programmability to data center topologies. By separating the control plane from the data plane, it enables centralized management, automated provisioning, and dynamic traffic optimization.

The chosen data center topology has a significant impact on the overall performance and reliability of an organization's IT infrastructure. A well-designed topology can optimize data flow, minimize latency, and prevent bottlenecks. By considering factors such as fault tolerance, scalability, and network traffic patterns, organizations can tailor their topology to meet their specific needs.

Highlights: Data Center Topology

Choosing a topology

Data centers are the backbone of many businesses, providing the necessary infrastructure to store and manage data and access applications and services. As such, it is essential to understand the different types of available data center topologies. When choosing a topology for a data center, it is necessary to consider the organization’s specific needs and requirements. Each topology offers its advantages and disadvantages, so it is crucial to understand the pros and cons of each before making a decision.

A data center topology refers to the physical layout and interconnection of network devices within a data center. It determines how servers, switches, routers, and other networking equipment are connected, ensuring efficient and reliable data transmission. Topologies are based on scalability, fault tolerance, performance, and cost.

Scalability of the topology

Additionally, it is essential to consider the topology’s scalability, as a data center may need to accommodate future growth. By understanding the different topologies and their respective strengths and weaknesses, organizations can make the best decision for their data centers. For example, in a spine-and-leaf architecture, traffic traveling from one server to another always crosses the same number of devices (unless both servers are located on the same leaf). Payloads need only hop to a spine switch and another leaf switch to reach their destination, thus reducing latency.

what is spine and leaf architecture

Data Center Topology Types

Centralized Model

Smaller data centers (less than 5,000 square feet) may benefit from the centralized model. It is shown that there are separate local area networks (LANs) and storage area networks (SANs), with home-run cables going to each server cabinet and zone. Each server is effectively connected back to the core switches in the main distribution area. As a result, port switches can be utilized more efficiently, and components can be managed and added more quickly. The centralized topology works well for smaller data centers but does not scale up well, making expansion difficult. Many cable runs in larger data centers cause cable pathways and cabinets congestion and increase costs. Zoned or top-of-rack topologies may be used in large data centers for LAN traffic, but centralized architectures may be used for SAN traffic. In particular, port utilization is essential when SAN switch ports are expensive.

Zoned

Distributed switching resources make up a zoned topology. Typically, chassis-based switches support multiple server cabinets and can be distributed among end-of-row (EoR) and middle-of-row (MoR) locations. It is highly scalable, repeatable, and predictable and is recommended by the ANS/TIA-942 Data Center Standards. A zoned architecture provides the highest switch and port utilization level while minimizing cabling costs. Switching at the end of a row can be advantageous in certain situations. Two servers’ local area network (LAN) ports can be connected to the same end-of-row switch for low-latency port-to-port switching. Having to run cable back to the end-of-row switch is a potential disadvantage of end-of-row switching. It is possible for this cabling to exceed that required for a top-of-rack system if every server is connected to redundant switches.

Top-of-rack (ToR)

Switches are typically placed at the top of a server rack to provide top-of-rack (ToR) switching, as shown below. Using this topology is a good option for dense one-rack-unit (1RU) server environments. For redundancy, both switches are connected to all servers in the rack. There are uplinks to the next layer of switching from the top-of-rack switches. It simplifies cable management and minimizes cable containment requirements when cables are managed at the top of the rack. Using this approach, servers within the rack can quickly switch from port to port, and the uplink oversubscription is predictable. In top-of-rack designs, cabling is more efficiently utilized. In exchange, there is usually an increase in the cost of switches and a high cost for under-utilization of ports. There is also the possibility of overheating local area network (LAN) switch gear in server racks when top-of-rack switching is required.

Data Center Architecture Types

Mesh architecture

Mesh networks, known as “network fabrics” or leaf-spine, consist of meshed connections between leaf-and-spine switches.  They are well suited for supporting universal “cloud services” because the mesh of network links enables any-to-any connectivity with predictable capacity and lower latency. The mesh network has multiple switching resources scattered throughout the data center, making it inherently redundant. Compared to huge, centralized switching platforms, these distributed network designs can be more cost-effective to deploy and scale.

Multi-Tier

Multi-tier architectures are commonly used in enterprise data centers. In this design, mainframes, blade servers, 1RU servers, and mainframes run the web, application, and database server tiers.

Mesh point of delivery

Mesh point of delivery (PoD) architectures have leaf switches interconnected within PoDs and spine switches aggregated in a central main distribution area (MDA). This architecture also enables multiple PoDs to connect efficiently to a super-spine tier. Three-tier topologies that support east-west data flows will be able to support new cloud applications with low latency. Mesh PoD networks can provide a pool of low-latency computing and storage for these applications that can be added without disrupting the existing environment.

Super Spine architectecutre

Hyperscale organizations that deploy large-scale data center infrastructures or campus-style data centers often deploy super spine architecture. This type of architecture handles data passing east to west across data halls.

Related: For pre-information, you may find the following post helpful

  1. ACI Cisco
  2. Virtual Switch
  3. Ansible Architecture
  4. Overlay Virtual Networks



Data Center Network Topology

Key Data Center Topologies Discussion Points:


  • End of Row and Top of Rack designs.

  • The use of Fabric Extenders.

  • Layer 2 or Layer 3 to the Core.

  • The rise of Network Virtualization.

  • VXLAN transports.

  • The Cisco ACI and ACI Network.

Back to Basics: Data Center Network Topology

A data center is a physical facility that houses critical applications and data for an organization. It consists of a network of computing and storage resources that support shared applications and data delivery. The components of a data center are routers, switches, firewalls, storage systems, servers, and application delivery controllers.

Enterprise IT data centers support the following business applications and activities:

  • Email and file sharing
  • Productivity applications
  • Customer relationship management (CRM)
  • Enterprise resource planning (ERP) and databases
  • Big data, artificial intelligence, and machine learning
  • Virtual desktops, communications, and collaboration services

A data center consists of the following core infrastructure components:

  • Network infrastructure: Connects physical and virtual servers, data center services, storage, and external connections to end users.
  • Storage Infrastructure: Modern data centers use storage infrastructure to power their operations. Storage systems hold this valuable commodity.
  • A data center’s computing infrastructure is its applications. The computing infrastructure comprises servers that provide processors, memory, local storage, and application network connectivity. In the last 65 years, computing infrastructure has undergone three major waves:
    • In the first wave of replacements of proprietary mainframes, x86-based servers were installed on-premises and managed by internal IT teams.
    • In the second wave, application infrastructure was widely virtualized. The result was improved resource utilization and workload mobility across physical infrastructure pools.
    • The third wave finds us in the present, where we see the move to the cloud, hybrid cloud, and cloud-native (that is, applications born in the cloud).

Common Types of Data Center Topologies:

a) Bus Topology: In this traditional topology, all devices are connected linearly to a common backbone, resembling a bus. While it is simple and cost-effective, a single point of failure can disrupt the entire network.

b) Star Topology: Each device is connected directly to a central switch or hub in a star topology. This design offers centralized control and easy troubleshooting, but it can be expensive due to the requirement of additional cabling.

c) Mesh Topology: A mesh topology provides redundant connections between devices, forming a network where every device is connected to every other device. This design ensures high fault tolerance and scalability but can be complex and costly.

d) Hybrid Topology: As the name suggests, a hybrid topology combines elements of different topologies to meet specific requirements. It offers flexibility and allows organizations to optimize their infrastructure based on their unique needs.

Considerations in Data Center Topology Design:

a) Redundancy: Redundancy is essential to ensure continuous operation even during component failures. By implementing redundant paths, power sources, and network links, data centers can minimize the risk of downtime and data loss.

b) Scalability: As the data center’s requirements grow, the topology should be able to accommodate additional devices and increased data traffic. Scalability can be achieved through modular designs, virtualization, and flexible network architectures.

c) Performance and Latency: The distance between devices, the quality of network connections, and the efficiency of routing protocols significantly impact data center performance and latency. Optimal topology design considers these factors to minimize delays and ensure smooth data transmission.

Impact of Data Center Topology:

Efficient data center topology directly influences the entire infrastructure’s reliability, availability, and performance. A well-designed topology reduces single points of failure, enables load balancing, enhances fault tolerance, and optimizes data flow. It directly impacts the user experience, especially for cloud-based services, where data centers simultaneously cater to many users.

Data Center Topology

Main Data Center Topology Components

Data Center Topology

  • You need to understanding the different topologies and their respective strengths and weaknesses.

  • Rich connectivity among the ToR switches so that all application and end-user requirements are satisfied

  • A well-designed topology reduces single points of failure.

  • Example: Bus, star, mesh, and hybrid topologies

Knowledge Check: Cisco ACI Building Blocks

Before Cisco ACI 4.1, Cisco ACI fabric supported only a two-tier (leaf-and-spine switch) topology in which leaf switches are connected to spine switches without interconnecting them. The Cisco ACI fabric allows multitier (three-tier) fabrics and two tiers of leaf switches, starting with Cisco ACI 4.1, which allows for vertical expansion. As a result, a traditional three-tier aggregation access architecture can be migrated, which is still required for many enterprise networks.

In some situations, building a full-mesh two-tier fabric is not ideal due to the high cost of fiber cables and the limitations of cable distances. A spine-leaf topology is more efficient in these cases, and Cisco ACI continues to automate and improve visibility.

ACI fabric Details
Diagram: Cisco ACI fabric Details

The Role of Networks

A network lives to serve the connectivity requirements of applications and applications. We build networks by designing and implementing data centers. A common trend is that the data center topology is much bigger than a decade ago, with application requirements considerably different from the traditional client–server applications and with deployment speeds in seconds instead of days. This changes how networks and your chosen data center topology are designed and deployed.

The traditional network design was scaled to support more devices by deploying larger switches (and routers). This is the scale-in model of scaling. However, these large switches are expensive and primarily designed to support only a two-way redundancy.

Today, data center topologies are built to scale out. They must satisfy the three main characteristics of increasing server-to-server traffic, scale ( scale on-demand ), and resilience. The following diagram shows a ToR design we discussed at the start of the blog.

Top of Rack (ToR)
Diagram: Data center network topology. Top of Rack (ToR).

The Role of The ToR

Top of rack (ToR) is a term used to describe the architecture of a data center. It is a server architecture in which servers, switches, and other equipment are mounted on the same rack. This allows for the most efficient use of space since the equipment is all within arm’s reach.

ToR is also the most efficient way to manage power and cooling since the equipment is all in the same area. Since all the equipment is close together, ToR also allows faster access times. This architecture can also be utilized in other areas, such as telecommunications, security, and surveillance.

ToR is a great way to maximize efficiency in any data center and is becoming increasingly popular. In contrast to the ToR data center design, the following diagram shows an EoR switch design.

End of Row (EoR)
Diagram: Data center network topology. End of Row (EoR).

The Role of The EoR

The term end-of-row (EoR) design is derived from a dedicated networking rack or cabinet placed at either end of a row of servers to provide network connectivity to the servers within that row. In EoR network design, each server in the rack has a direct connection with the end-of-row aggregation switch, eliminating the need to connect servers directly with the in-rack switch.

Racks are usually arranged to form a row; a cabinet or rack is positioned at the end of this row. This rack has a row aggregation switch, which provides network connectivity to servers mounted in individual racks. This switch, a modular chassis-based platform, sometimes supports hundreds of server connections. However, a large amount of cabling is required to support this architecture.

Data center topology types
Diagram: ToR and EoR. Source. FS Community.

A ToR configuration requires one switch per rack, resulting in higher power consumption and operational costs. Moreover, unused ports are often more significant in this scenario than with an EoR arrangement.

On the other hand, ToR’s cabling requirements are much lower than those of EoR, and faults are primarily isolated to a particular rack, thus improving the data center’s fault tolerance.

If fault tolerance is the ultimate goal, ToR is the better choice, but EoR configuration is better if an organization wants to save on operational costs. The following table lists the differences between a ToR and an EoR data center design.

data center network topology
Diagram: Data center network topology. The differences. Source FS Community

Data Center Topology Types:

Fabric extenders – FEX

Cisco has introduced the concept of Fabric Extenders, which are not Ethernet switches but remote line cards of a virtualized modular chassis ( parent switch ). This allows scalable topologies previously impossible with traditional Ethernet switches in the access layer.

You should relate an FEX device like a remote line card attached to a parent switch. All the configuration is done on the parent switch, yet physically, the fabric extender could be in a different location. The mapping between the parent switch and the FEX ( fabric extender ) is done via a special VN-Link.

The following diagram shows an example of a FEX in a standard data center network topology. More specifically, we are looking at the Nexus 2000 FEX Series. Cisco Nexus 2000 Series Fabric Extenders (FEX) are based on the standard IEEE 802.1BR. They deliver fabric extensibility with a single point of management.

Cisco FEX
Diagram: Cisco FEX design. Source Cisco.

Different types of Fex solution

FEXs come with various connectivity solutions, including 100 Megabit Ethernet, 1 Gigabit Ethernet, 10 Gigabit Ethernet ( copper and fiber ), and 40 Gigabit Ethernet. They can be synchronized with the following models of parent switches – Nexus 5000, Nexus 6000, Nexus 7000, Nexus 9000, and Cisco UCS Fabric Interconnect.

In addition, because of the simplicity of FEX, they have very low latency ( as low as 500 nanoseconds ) compared to traditional Ethernet switches.

Data Center design
Diagram: Data center fabric extenders.

Some network switches can be connected to others and operate as a single unit. These configurations are called “stacks” and are helpful for quickly increasing the capacity of a network. A stack is a network solution composed of two or more stackable switches. Switches that are part of a stack behave as one single device.

Traditional switches like the 3750s still stand in the data center network topology access layer and can be used with stacking technology, combining two physical switches into one logical switch.

This stacking technology allows you to build a highly resilient switching system, one switch at a time. If you are looking at a standard access layer switch like the 3750s, consider the next-generation Catalyst 3850 series.

The 3850 supports BYOD/mobility and offers a variety of performance and security enhancements to previous models. The drawback of stacking is that you can only stack several switches. So, if you want additional throughout, you should aim for a different design type.

Data Center Design: Layer 2 and Layer 3 Solutions

Traditional views of data center design

Depending on the data center network topology deployed, packet forwarding at the access layer can be either Layer 2 or Layer 3. A Layer 3 approach would involve additional management and configuring IP addresses on hosts in a hierarchical fashion that matches the switch’s assigned IP address.

An alternative approach is to use Layer 2, which has less overhead as Layer 2 MAC addresses do not need specific configuration. However, it has drawbacks with scalability and poor performance.

Generally, access switches focus on communicating servers in the same IP subnet, allowing any type of traffic – unicast, multicast, or broadcast. You can, however, have filtering devices such as a Virtual Security Gateway ( VSG ) to permit traffic between servers, but that is generally reserved for inter-POD ( Platform Optimized Design ) traffic.

Leaf and Spine With Layer 3

We use a leaf and spine data center design with Layer 3 everywhere and overlay networking. This modern, robust architecture provides a high-performance, highly available network. With this architecture, data center networks are composed of leaf switches that connect to one or more spine switches.

The leaf switches are connected to end devices such as servers, storage devices, and other networking equipment. The spine switches, meanwhile, act as the network’s backbone, connecting the multiple leaf switches.

The leaf and spine architecture provides several advantages over traditional data center networks. It allows for greater scalability, as additional leaf switches can be easily added to the network. It also offers better fault tolerance, as the network can operate even if one of the spine switches fails.

Furthermore, it enables faster traffic flows, as the spine switches to route traffic between the leaf switches faster than a traditional flat network.

leaf and spine

Data Center Traffic Flow

Datacenter topologies can have North-South or East-to-West traffic. North-south ( up / down ) corresponds to traffic between the servers and the external world ( outside the data center ). East-to-west corresponds to internal server communication, i.e., traffic does not leave the data center.

Therefore, determining the type of traffic upfront is essential as it influences the type of topology used in the data center.

data center traffic flow
Diagram: Data center traffic flow.

For example, you may have a pair of ISCSI switches, and all traffic is internal between the servers. In this case, you would need high-bandwidth inter-switch links. Usually, an ether channel supports all the cross-server talk; the only north-to-south traffic would be management traffic.

In another part of the data center, you may have data server farm switches with only HSRP heartbeat traffic across the inter-switch links and large bundled uplinks for a high volume of north-to-south traffic. Depending on the type of application, which can be either outward-facing or internal, computation will influence the type of traffic that will be dominant. 

Virtual Machine and Containers.

This drive was from virtualization, virtual machines, and container technologies regarding east-west traffic. Many are moving to a leaf and spine data center design if they have a lot of east-to-west traffic and want better performance.

container based virtualization

Network Virtualization and VXLAN

Network virtualization and the ability of a physical server to host many VMs and move those VMs are also used extensively in data centers, either for workload distribution or business continuity. This will also affect the design you have at the access layer.

For example, in a Layer 3 fabric, migrating a VM across that boundary changes its IP address, resulting in a reset of the TCP sessions because, unlike SCTP, TCP does not support dynamic address configuration. In a Layer 2 fabric, migrating a VM incurs ARP overhead and requires forwarding on millions of flat MAC addresses, which leads to MAC scalability and poor performance problems.

1st Lab Guide: VXLAN

The following lab guide displays a VXLAN network. We are running VXLAN in unicast mode. VXLAN can also be configured to run in multicast mode. In the screenshot below, we have created a Layer 2 overlay across a routed Layer 3 core. The command: Show nve interface nve 1 displays an operational tunnel with the encapsulation set to VXLAN.

The screenshot shows a ping test from the desktops that connect to a Layer 3 port on the Leafs.

VXLAN overlay
Diagram: VXLAN Overlay

VXLAN: stability over Layer 3 core

Network virtualization plays a vital role in the data center. Technologies like VXLAN attempt to move the control plane from the core to the edge and stabilize the core so that it only has a handful of addresses for each ToR switch. The following diagram shows the ACI networks with VXLAN as the overlay that operates over a spine leaf architecture.

Layer 2 and 3 traffic is mapped to VXLAN VNIs that run over a Layer 3 core. The Bridge Domain is for layer 2, and the VRF is for layer 3 traffic. Now, we have the separation of layer 2 and 3 traffic based on the VNI in the VXLAN header.  

One of the first notable differences between VXLAN and VLAN was scale. VLAN has a 12-bit identifier called VID, while VXLAN has a 24-bit identifier called a VID network identifier. This means that with VLAN, you can create only 4094 networks over ethernet, while with VXLAN, you can create up to 16 million.

ACI network
Diagram: ACI network.

Whether you can build layer 2 or layer 3 in the access and use VXLAN or some other overlay to stabilize the core, it would help if you modularized the data center. The first step is to build each POD or rack as a complete unit. Each POD will be able to perform all its functions within that POD.

  • A key point: A POD data center design

POD: It is a design methodology that aims to simplify, speed deployment, optimize utilization of resources, and drive the interoperability of the three or more data center components: server, storage, and networks.

A POD example: Data center modularity

For example, one POD might be a specific human resources system. The second is modularity based on the type of resources offered. For example, a storage pod or bare metal compute may be housed in separate pods.

These two modularization types allow designers to control inter-POD traffic with predefined policies easily. Operators can also upgrade PODs and a specific type of service at once without affecting other PODs.

However, this type of segmentation does not address the scale requirements of the data center. Even when we have adequately modularized the data center into specific portions, the MAC table sizes on each switch still increase exponentially as the data center grows.

Current and Future Design Factors

New technologies with scalable control planes must be introduced for a cloud-enabled data center, and these new control planes should offer the following:

Option

Data Center Feature

Data center feature 1

The ability to scale MAC addresses

Data center feature 2

First-Hop Redundancy Protocol ( FHRP ) multipathing and Anycast HSRP

Data center feature 3

Equal-Cost multipathing

Data center feature 4

MAC learning optimizations

Several design factors need to be taken into account when designing a data center. First, what is the growth rate for servers, switch ports, and data center customers? This prevents part of the network topology from becoming a bottleneck or linking congested.

Application bandwidth demand?

This demand is usually translated into oversubscription. In data center networking, oversubscription refers to how much bandwidth switches are offered to downstream devices at each layer.

Oversubscription is expected in a data center design. By limiting oversubscription to the ToR and edge of the network, you offer a single place to start when performance problems occur.

A data center with no oversubscription ratio will be costly, especially with a low latency network design. So, it’s best to determine what oversubscription ratio your applications support and work best. Optimizing your switch buffers to improve performance is recommended before you decide on a 1:1 oversubscription rate.

Ethernet 6-byte MAC addressing is flat.

Ethernet forms the basis of data center networking in tandem with IP. Since its inception 40 years ago, Ethernet frames have been transmitted over various physical media, even barbed wire. Ethernet 6-byte MAC addressing is flat; the manufacturer typically assigns the address without considering its location.

Ethernet-switched networks do not have explicit routing protocols to ensure readability about the flat addresses of the server’s NICs. Instead, flooding and address learning are used to create forwarding table entries.

IP addressing is a hierarchy.

On the other hand, IP addressing is a hierarchy, meaning that its address is assigned by the network operator based on its location in the network. A hierarchy address space advantage is that forwarding tables can be aggregated. If summarization or other routing techniques are employed, changes in one side of the network will not necessarily affect other areas.

This makes IP-routed networks more scalable than Ethernet-switched networks. IP-routed networks also offer ECMP techniques that enable networks to use parallel links between nodes without spanning tree disabling one of those links. The ECMP method hashes packet headers before selecting a bundled link to avoid out-of-sequence packets within individual flows. 

Equal Cost Load Balancing

Equal-cost load balancing is a method for distributing network traffic among multiple paths of equal cost. It provides redundancy and increases throughput. Sending traffic over multiple paths avoids congestion on any single link. In addition, the load is equally distributed across the paths, meaning that each path carries roughly the same total traffic.

ecmp
Diagam: ECMP 5 Tuple hash. Source: Keysight

This allows for using multiple paths at a lower cost, providing an efficient way to increase throughput.

The idea behind equal cost load balancing is to use multiple paths of equal cost to balance the load on each path. The algorithm considers the number of paths, each path’s weight, and each path’s capacity. It also feels the number of packets that must be sent and the delay allowed for each packet.

Considering these factors, it can calculate the best way to distribute the load among the paths.

Equal-cost load balancing can be implemented using various methods. One method is to use a Link Aggregation Protocol (LACP), which allows the network to use multiple links and distribute the traffic among the links in a balanced way.

ecmp
Diagam: ECMP 5 Tuple hash. Source: Keysight
  • A keynote: Data center topologies. The move to VXLAN.

Given the above considerations, a solution that encompasses the benefits of L2’s plug-and-play flat addressing and the scalability of IP is needed. Location-Identifier Split Protocol ( LISP ) has a set of solutions that use hierarchical addresses as locators in the core and flat addresses as identifiers in the edges. However, not much is seen in its deployment these days.

Equivalent approaches such as THRILL and Cisco FabricPath create massive scalable L2 multipath networks with equidistant endpoints. Tunneling is also being used to extend down to the server and access layer to overcome the 4K limitation with traditional VLANs. What is VXLAN? Tunneling with VXLAN is now the standard design in most data center topologies with leaf-spine designs. The following video provides VXLAN guidance.

Data Center Network Topology

Leaf and spine data center topology types

This is commonly seen in a leaf and spine design. For example, in a leaf-spine fabric, We have a Layer 3 IP fabric that supports equal-cost multi-path (ECMP) routing between any two endpoints in the network. Then, on top of the Layer 3 fabric is an overlay protocol, commonly VXLAN.

A spine-leaf architecture consists of a data center network topology of two switching layers—a spine and a leaf. The leaf layer comprises access switches that aggregate traffic from endpoints such as the servers and connect directly to the spine or network core.

Spine switches interconnect all leaf switches in a full-mesh topology. The leaf switches do not directly connect. The Cisco ACI is a data center topology that utilizes the leaf and spine.

The ACI network’s physical topology is a leaf and spine, while the logical topology is formed with VXLAN. From a protocol side point, VXLAN is the overlay network, and the BGP and IS-IS provide the Layer 3 routing, the underlay network that allows the overlay network to function.

As a result, the nonblocking architecture performs much better than the traditional data center design based on access, distribution, and core designs.

Cisco ACI
Diagram: Data center topology types and the leaf and spine with Cisco ACI

Closing Points: Data Center Topologies

A data center topology refers to the physical layout and interconnection of network devices within a data center. It determines how servers, switches, routers, and other networking equipment are connected, ensuring efficient and reliable data transmission. Topologies are based on scalability, fault tolerance, performance, and cost.

  • Hierarchical Data Center Topology:

The hierarchical or tree topology is one of the most commonly used data center topologies. This design consists of multiple core, distribution, and access layers. The core layer connects all the distribution layers, while the distribution layer connects to the access layer. This structure enables better management, scalability, and fault tolerance by segregating traffic and minimizing network congestion.

  • Mesh Data Center Topology:

Every network device is interlinked in a mesh topology, forming a fully connected network with multiple paths for data transmission. This redundancy ensures high availability and fault tolerance. However, this topology can be cost-prohibitive and complex, especially in large-scale data centers.

  • Leaf-Spine Data Center Topology:

The leaf-spine topology is gaining popularity due to its scalability and simplicity. It consists of interconnected leaf switches at the access layer and spine switches at the core layer. This design allows for non-blocking, low-latency communication between any leaf switch and spine switch, making it suitable for modern data center requirements.

  • Full-Mesh Data Center Topology:

As the name suggests, the full-mesh topology connects every network device to every other device, creating an extensive web of connections. This topology offers maximum redundancy and fault tolerance. However, it can be expensive to implement and maintain, making it more suitable for critical applications with stringent uptime requirements.

Summary: Data Center Topology

Data centers are vital in supporting and enabling our digital infrastructure in today’s interconnected world. Behind the scenes, intricate network topologies ensure seamless data flow, allowing us to access information and services easily. In this blog post, we dived into the world of data center topologies, unraveling their complexities and understanding their significance.

Section 1: Understanding Data Center Topologies

Datacenter topologies refer to a data center’s physical and logical layout of networking components. These topologies determine how data flows between servers, switches, routers, and other network devices. By carefully designing the topology, data center operators can optimize performance, scalability, redundancy, and fault tolerance.

Section 2: Common Data Center Topologies

There are several widely adopted data center topologies, each with its strengths and use cases. Let’s explore some of the most common ones:

2.1. Tree Topology:

Tree topology, or hierarchical topology, is widely used in data centers. It features a hierarchical structure with multiple layers of switches, forming a tree-like network. This topology offers scalability and ease of management, making it suitable for large-scale deployments.

2.2. Mesh Topology:

The mesh topology provides a high level of redundancy and fault tolerance. In this topology, every device is connected to every other device, forming a fully interconnected network. While it offers robustness, it can be complex and costly to implement.

2.3. Spine-Leaf Topology:

The spine-leaf topology, also known as a Clos network, has recently gained popularity. It consists of leaf switches connecting to multiple spine switches, forming a non-blocking fabric. This design allows for efficient east-west traffic flow and simplified scalability.

Section 3: Factors Influencing Topology Selection

Choosing the right data center topology depends on various factors, including:

3.1. Scalability:

It is crucial for a topology to accommodate a data center’s growth. Scalable topologies ensure that additional devices can be seamlessly added without causing bottlenecks or performance degradation.

3.2. Redundancy and Fault Tolerance:

Data centers require high availability to minimize downtime. Topologies that offer redundancy and fault tolerance mechanisms, such as link and device redundancy, are crucial in ensuring uninterrupted operations.

3.3. Traffic Patterns:

Understanding the traffic patterns within a data center is essential for selecting an appropriate topology. Some topologies excel in handling east-west traffic, while others are better suited for north-south traffic flow.

Conclusion:

Datacenter topologies form the backbone of our digital infrastructure, providing the connectivity and reliability needed for our ever-expanding digital needs. By understanding the intricacies of these topologies, we can better appreciate the complexity involved in keeping our data flowing seamlessly. Whether it’s the hierarchical tree, the fully interconnected mesh, or the efficient spine-leaf, each topology has its place in the world of data centers.

Data Center Network Design

Data Center Network Design

Data centers are crucial in today’s digital landscape, serving as the backbone of numerous businesses and organizations. A well-designed data center network ensures optimal performance, scalability, and reliability. This blog post will explore the critical aspects of data center network design and its significance in modern IT infrastructure.

Data center network design involves the architectural planning and implementation of networking infrastructure within a data center environment. It encompasses various components such as switches, routers, cables, and protocols. A well-designed network ensures seamless communication, high availability, and efficient data flow.

The traditional three-tier network architecture is being replaced by more streamlined and flexible designs. Two popular approaches gaining traction are the spine-leaf architecture and the fabric-based architecture. The spine-leaf design offers low latency, high bandwidth, and improved scalability, making it ideal for large-scale data centers. On the other hand, fabric-based architectures provide a unified and simplified network fabric, enabling efficient management and enhanced performance.

Network virtualization, powered by technologies like SDN, is transforming data center network design. By decoupling the network control plane from the underlying hardware, SDN enables centralized network management, automation, and programmability. This results in improved agility, better resource allocation, and faster deployment of applications and services.

With the rising number of cyber threats, ensuring robust security and resilience has become paramount. Data center network design should incorporate advanced security measures such as firewalls, intrusion detection systems, and encryption protocols. Additionally, implementing redundant links, load balancing, and disaster recovery mechanisms enhances network resilience and minimizes downtime.

Highlights: Data Center Network Design

Understanding the Requirements

Before embarking on the design process, it’s crucial to understand the data center’s unique requirements. Factors such as power and cooling, network connectivity, scalability, and security are vital in determining the design approach. By thoroughly assessing these requirements, architects can create a blueprint that aligns with the organization’s current and future needs.

Optimizing Physical Layout

The physical layout of a data center significantly impacts its efficiency and performance. This section will delve into rack placement, aisle design, cable management, and airflow optimization. By adopting best practices in physical layout design, data center operators can minimize energy consumption, reduce maintenance costs, and enhance overall operational efficiency.

Redundancy and Resilience

Data centers demand high levels of redundancy and resilience to ensure uninterrupted operations. This section will explore the concept of redundancy in power and cooling systems, backup generators, redundant network connectivity, and failover mechanisms. Implementing robust redundancy measures helps mitigate the risk of downtime and ensures continuous availability of critical services.

Security and Compliance

Data centers store sensitive and valuable information, making security a top priority. This section will discuss the importance of physical security measures, access controls, surveillance systems, and fire suppression mechanisms. Additionally, we will explore compliance standards and regulations that govern data center operations, such as SOC 2, ISO 27001, and GDPR.

Embracing Green Initiatives

As environmental sustainability gains importance, data centers seek ways to minimize their carbon footprint. This section will focus on energy-efficient design practices, including using renewable energy sources, efficient cooling techniques, and server virtualization. Data centers can contribute to a more sustainable future by adopting green initiatives.

Composition of Data Center Architecture

A data center architecture consists of three main components: the data center network, the data center security, and the data center computing architecture. In addition to these three types of architecture, there are also data center physical architectures and data center information architectures. The following are three typical compositions. Network architecture for data centers: Data center networks (DCNs) are arrangements of network devices interconnecting data center resources. They are a crucial research area for Internet companies and large cloud computing firms. The design of a data center depends on its network architecture.

It is common for routers and switches to be arranged in hierarchies of two or three levels. There are three-tier DCNs: fat tree DCNs, DCells, and others. There has always been a focus on scalability, robustness, and reliability regarding data center network architectures.

Data center security refers to physical practices and virtual technologies for protecting data centers from threats, attacks, and unauthorized access. It can be divided into two components: physical security and software security. A firewall between a data center’s external and internal networks can protect it from attack.

Developing a data center network

A network serves applications’ connectivity requirements, and applications serve their organizations’ business needs. To design or operate a network in a modern data center, you must first understand the needs and topology of the data center. Here we begin our journey. My goal is for you to understand the network design of a modern data center network based on the applications’ needs and the size of the data center.

Compared to a decade ago, data centers now have much larger capacity, vastly different applications, and deployment speeds in seconds rather than days. As a result, network design and deployment change.

Border Gateway Protocol (BGP) is data centers’ most commonly used routing protocol. BGP has been used to connect Internet systems around the world for decades. It can also be used outside of a data center. The BGP protocol is a standard-based open-source software package. Its more common to find BGP peering between data centers over the WAN. However, these days we are seeing BGP often used purely inside the data center. 

forwarding routing protoocols

Data Center Requirments

Google and Amazon, two of the world’s leading web-scale pioneers, developed a modern data center. The third wave of application architectures represents these organizations’ search and cloud applications. Towards the end of the 20th century, client-server architectures and monolithic single-machine applications dominated the landscape. This third wave of applications has three primary characteristics:

Unlike client-server architectures, modern data center applications involve a lot of communication between servers. In client-server architectures, clients communicate with monolithic servers, which either handle the request entirely themselves or communicate with fewer than a handful of other servers, such as database servers. Search (or Hadoop, its more popular variant) employs many mappers and reducers instead of search. In the cloud, virtual machines can reside on different nodes but must communicate seamlessly. In some cases, VMs are deployed on servers with the least load, scaled out, or balanced loads.

A microservices architecture also increases server-to-server communication. This architecture is based on separating a single function into smaller building blocks and interacting with them. Each block can be used in several applications and enhanced, modified, and fixed independently in such an architecture. Since diagrams usually show servers next to each other, East-West traffic is often called server communication. Traffic flows north-south between local networks and external networks.

container based virtualization

Scale and resilience

The sheer size of modern data centers is characterized by rows and rows of dark, humming, blinking machines. As opposed to the few hundred or so servers of the past, a modern data center contains between a few hundred and a hundred thousand servers. To address the connectivity requirements at such scales, as well as the need for increased server-to-server connectivity, network design must be rethought. Unlike older architectures, modern data center applications assume failures as a given. Failures should be limited to the smallest possible footprint. Failures must have a limited “blast radius.” By minimizing the impact of network or server failures on the end-user experience, we aim to provide a stable and reliable experience.

Data Center Goal: Interconnect networks

The goal of data center design and interconnection network is to transport end-user traffic from A to B without any packet drops, yet the metrics we use to achieve this goal can be very different. The data center is evolving and progressing through various topology and technology changes, resulting in multiple network designs.  The new data center control planes we see today, such as Fabric Path, LISP, THRILL, and VXLAN, are driven by a change in the end user’s requirements; the application has changed. These new technologies may address new challenges, yet the fundamental question of where to create the Layer 2/Layer three boundaries and the need for Layer 2 in the access layer remains the same. The question stays the same, yet the technologies available to address this challenge have evolved.

what is spine and leaf architecture

Modern Data Centers

There is a vast difference between modern data centers and what they used to be just a few years ago. Physical servers have evolved into virtual networks that support applications and workloads across pools of physical infrastructure and into a multi-cloud environment. There are multiple data centers, the edge, and public and private clouds where data exists and is connected. Both on-premises and cloud-based data centers must be able to communicate. Data centers are even part of the public cloud. Cloud-hosted applications use the cloud provider’s data center resources.

Unified Fabric

Through Cisco’s fabric-based data center infrastructure, tiered silos and inefficiencies of multiple network domains are eliminated, and a unified, flat fabric is provided instead, which allows local area networks (LANs), storage area networks (SANs), and network-attached storage (NASs) to be consolidated into one high-performance, fault-tolerant network. Creating large pools of virtualized network resources that can be easily moved and rapidly reconfigured with Cisco Unified Fabric provides massive scalability and resiliency to the data center.

This approach automatically deploys virtual machines and applications, thereby reducing complexity. Thanks to deep integration between server and network architecture, secure IT services can be delivered from any device within the data center, between data centers, or beyond. In addition to Cisco Nexus switches, Cisco Unified Fabric uses Cisco NX-OS as its operating system.

leaf and spine design

The use of Open Networking

We also have the Open Networking Foundation ( ONF ), which provides open networking. Open networking describes a network that uses open standards and commodity hardware. So, consider open networking in terms of hardware and software. Unlike a vendor approach like Cisco, this gives you much more choice with what hardware and software you use to make up and design your network.

Related: Before you proceed, you may find the following useful:

  1. ACI Networks
  2. IPv6 Attacks
  3. SDN Data Center
  4. Active Active Data Center Design
  5. Virtual Switch

Data Center Control Plane

Key Data Center Network Design Discussion Points:


  • Introduction to data center network design and what is involved.

  • Highlighting the details of VLANs and virtualization.

  • Technical details on the issues of Layer 2 in data centers. 

  • Scenario: Cisco FabricPath and DFA.

  • Details on overlay networking and Cisco OTV.

The Rise of Overlay Networking

What has the industry introduced to overcome these limitations and address the new challenges? – Network virtualization and overlay networking. In its simplest form, an overlay is a dynamic tunnel between two endpoints that enables Layer 2 frames to be transported between them. In addition, these overlay-based technologies provide a level of indirection that allows switching table sizes to not increase in the order of the number of supported end hosts.

Today’s overlays are Cisco FabricPath, THRILL, LISP, VXLAN, NVGRE, OTV, PBB, and Shorted Path Bridging. They are essentially virtual networks that sit on top of a physical network, and often, the physical network is unaware of the virtual layer above it.

1st Lab Guide: VXLAN

The following lab guide displays a VXLAN network. We are running VXLAN in multicast mode. Multicast VXLAN is a variant of VXLAN that utilizes multicast-based IP multicast for transmitting overlay network traffic. VXLAN is an encapsulation protocol that extends Layer 2 Ethernet networks over Layer 3 IP networks.

Linking multicast enables efficient and scalable communication within the overlay network. Notice the multicast group of 239.0.0.10 and the route of 239.0.0.10 forwarding out the tunnel interface. We have multicast enabled on all Layer 3 interfaces, including the core that consists of Spine A and Spine B.

Multicast VXLAN
Diagram: Multicast VXLAN

Traditional Data Center Network Design

How do routers create a broadcast domain boundary? Firstly, using the traditional core, distribution, and access model, the access layer is layer 2, and servers served to each other in the access layer are in the same IP subnet and VLAN. The same access VLAN will span the access layer switches for east-to-west traffic, and any outbound traffic is via a First Hop Redundancy Protocol ( FHRP ) like Hot Standby Router Protocol ( HSRP ).

Servers in different VLANs are isolated from each other and cannot communicate directly; inter-VLAN communications require a Layer 3 device. Virtualization’s humble beginnings started with VLANs, which were used to segment traffic at Layer 2. It was expected to find single VLANs spanning an entire data center fabric.

Redundant Data Centers 

VLAN and Virtualization

The virtualization side of VLANs comes from two servers physically connected to different switches. Assuming the VLAN spans both switches, the same VLAN can communicate with each server. Each VLAN can be defined as a broadcast domain in a single Ethernet switch or shared among connected switches.

Whenever a switch interface belonging to a VLAN receives a broadcast frame ( destination MAC is ffff.ffff.ffff), the device must forward this frame to all other ports defined in the same VLAN.

This approach is straightforward in design and is almost like a plug-and-play network. The first question is, why not connect everything in the data center into one large Layer 2 broadcast domain? Layer 2 is a plug-and-play network, so why not? STP also blocks links to prevent loops.

stp port states

 The issues of Layer 2

The reason is that there are many scaling issues in large layer 2 networks. Layer 2 networks don’t have controlled / efficient network discovery protocols. Address Resolution Protocol ( ARP ) is used to locate end hosts and uses Broadcasts and Unicast replies. A single host might not generate much traffic, but imagine what would happen if 10,000 hosts were connected to the same broadcast domain. VLANs span an entire data center fabric, which can bring a lot of instability due to loops and broadcast storms.

Address Resolution Protocol

 No hierarchy in MAC addresses

MAC addressing also lacks hierarchy. Unlike Layer 3 networks, which allow summarization and hierarchy addressing, MAC addresses are flat. Adding several thousand hosts to a single broadcast domain will create large forwarding information tables.

Because end hosts are potentially not static, they are likely to be attached and removed from the network at regular intervals, creating a high rate of change in the control plane. Of course, you can have a large Layer 2 data center with multiple tenants if they don’t need to communicate with each other.

The shared services requirements, such as WAAS or load balancing, can be solved by spinning up the service VM in the tenant’s Layer 2 broadcast domain. This design will hit scaling and management issues. There is a consensus to move from a Layer 2 design to a more robust and scalable Layer 3 design.

But why is Layer 2 still needed in data center topologies? One solution is Layer 2 VPN with EVPN. But first, let us look at Cisco DFA.

The Requirement for Layer 2 in Data Center Network Design

  • Servers that perform the same function might need to communicate with each other due to a clustering protocol or simply as part of the application’s inner functions. If the communication is clustering protocol heartbeats or some server-to-server application packets that are not routable, then you need this communication layer to be on the same VLAN, i.e., Layer 2 domain, as these types of packets are not routable and don’t understand the IP layer.

  • Stateful devices such as firewalls and load balancers need Layer 2 adjacency as they constantly exchange connection and session state information.

  • Dual-homed servers: Single server with two server NICs and one NIC to each switch will require a layer 2 adjacency if the adapter has a standby interface that uses the same MAC and IP addresses after a failure. In this situation, the active and standby interfaces must be on the same VLAN and use the same default gateway.

  • Suppose your virtualization solutions cannot handle Layer 3 VM mobility. In that case, you may need to stretch VLANs between PODS / Virtual Resource Pools or even data centers so you can move VMs around the data center at Layer 2 ( without changing their IP address ).

Data Center Design and Cisco DFA

Cisco took a giant step and recently introduced a data center fabric with Dynamic Fabric Automaton ( DFA ), similar to Juniper QFabric. This fabric offers Layer 2 switching and Layer 3 routing at the access layer / ToR. Firstly, it has a Fabric Path ( IS-IS for Layer 2 connectivity ) in the core, which gives optimal Layer 2 forwarding between all the edges.

Then they configure the same Layer 3 address on all the edges, which gives you optimal Layer 3 forwarding across the whole Fabric.

On edge, you can have Layer 3 Leaf switches, for example, the Nexus 6000 series, or integrate with Layer 2-only devices like the Nexus 5500 series or the Nexus 1000v. You can also connect external routers or USC or FEX to the Fabric. In addition to running IS-IS as the data center control plane, DFA uses MP-iBGP, with some Spine nodes being the Route Reflector to exchange IP forwarding information.

Cisco FabricPath

DFA also employs a Cisco FabricPath technique called “Conversational Learning.” The first packet triggers a full RIB lookup, and the subsequent packets are switched in the hardware-implemented switching cache.

This technology provides Layer 2 mobility throughout the data center while providing optimal traffic flow using Layer 3 routing. Cisco commented, “DFA provides a scale-out architecture without congestion points in the network while providing optimized forwarding for all applications.”

Terminating Layer 3 at the access / ToR has clear advantages and disadvantages. Other benefits include reducing the size of the broadcast domain, which comes at the cost of reducing the mobility domain across which VMs can be moved.

Terminating Layer 3 at the accesses can also result in sub-optimal routing because there will be hair pinning or traffic tromboning of across-subnet traffic, taking multiple and unnecessary hops across the data center fabric.

FabricPath

The role of the Cisco Fabricpath

Cisco FabricPath is a Layer 2 technology that provides Layer 3 benefits, such as multipathing the classical Layer 2 networks using IS-IS at Layer 2. This eliminates the need for spanning tree protocol, avoiding the pitfalls of having large Layer 2 networks. As a result, Fabric Path enables a massive Layer 2 network that supports multipath ( ECMP ). THRILL is an IEEE standard that, like Fabric Path, is a Layer 2 technology that provides the same Layer 3 benefits as Cisco FabricPath to the Layer 2 networks using IS-IS.

LISP is popular in Active data centers for DCI route optimization/mobility. It separates the host’s location from the identifier ( EID ), allowing VMs to move across subnet boundaries while keeping the endpoint identification. LISP is often referred to as an Internet locator. 

That can enable some designs of triangular routing. Popular encapsulation formats include VXLAN ( proposed by Cisco and VMware ) and STT (created by Nicira but will be deprecated over time as VXLAN comes to dominate ).

The role of OTV

OTV is a data center interconnect ( DCI ) technology enabling Layer 2 extension across data center sites. While Fabric Path can be a DCI technology with dark fiber over short distances, OTV has been explicitly designed for DCI. In contrast, the Fabric Path data center control plane is primarily used for intra-DC communications.

Failure boundary and site independence are preserved in OTV networks because OTV uses a data center control plane protocol to sync MAC addresses between sites and prevent unknown unicast floods. In addition, recent IOS versions can allow unknown unicast floods for certain VLANs, which are unavailable if you use Fabric Path as the DCI technology.

The Role of Software-defined Networking (SDN)

Another potential trade-off between data center control plane scaling, Layer 2 VM mobility, and optimal ingress/egress traffic flow would be software-defined networking ( SDN ). At a basic level, SDN can create direct paths through the network fabric to isolate private networks effectively.

An SDN network allows you to choose the correct forwarding information per-flow basis. This per-flow optimization eliminates VLAN separation in the data center fabric. Instead of using VLANs to enforce traffic separation, the SDN controller has a set of policies allowing traffic to be forwarded from a particular source to a destination.

The ACI Cisco borrows concepts of SDN to the data center. It operates over a leaf and spine design and traditional routing protocols such as BGP and IS-IS. However, it brings a new way to manage the data center with new constructs such as Endpoint Groups (EPGs). In addition, no more VLANs are needed in the data center as everything is routed over a Layer 3 core, with VXLAN as the overlay protocol.

SDN and OpenFlow

Closing Points: Data Center Design

Data centers are the backbone of modern technology infrastructure, providing the foundation for storing, processing, and transmitting vast amounts of data. A critical aspect of data center design is the network architecture, which ensures efficient and reliable data transmission within and outside the facility.  1. Scalability and Flexibility

One of the primary goals of data center network design is to accommodate the ever-increasing demand for data processing and storage. Scalability ensures the network can grow seamlessly as the data center expands. This involves designing a network that supports many devices, servers, and users without compromising performance or reliability. Additionally, flexibility is essential to adapt to changing business requirements and technological advancements.

Redundancy and High Availability

Data centers must ensure uninterrupted access to data and services, making redundancy and high availability critical for network design. Redundancy involves duplicating essential components, such as switches, routers, and links, to eliminate single points of failure. This ensures that if one component fails, there are alternative paths for data transmission, minimizing downtime and maintaining uninterrupted operations. High availability further enhances reliability by providing automatic failover mechanisms and real-time monitoring to detect and address network issues promptly.

Traffic Optimization and Load Balancing

Efficient data flow within a data center is vital to prevent network congestion and bottlenecks. Traffic optimization techniques, such as Quality of Service (QoS) and traffic prioritization, can be implemented to ensure that critical applications and services receive the necessary bandwidth and resources. Load balancing is crucial in evenly distributing network traffic across multiple servers or paths, preventing overutilization of specific resources, and optimizing performance.

Security and Data Protection

Data centers house sensitive information and mission-critical applications, making security a top priority. The network design should incorporate robust security measures, including firewalls, intrusion detection systems, and encryption protocols, to safeguard data from unauthorized access and cyber threats. Data protection mechanisms, such as backups, replication, and disaster recovery plans, should also be integrated into the network design to ensure data integrity and availability.

Monitoring and Management

Proactive monitoring and effective management are essential for maintaining optimal network performance and addressing potential issues promptly. The network design should include comprehensive monitoring tools and centralized management systems that provide real-time visibility into network traffic, performance metrics, and security events. This enables administrators to promptly identify and resolve network bottlenecks, security breaches, and performance degradation.

Data center network design is critical in ensuring efficient, reliable, and secure data transmission within and outside the facility. Scalability, redundancy, traffic optimization, security, and monitoring are key considerations for designing a robust, high-performance network. By implementing best practices and staying abreast of emerging technologies, data centers can build networks that meet the growing demands of the digital age while maintaining the highest levels of performance, availability, and security.

Summary: Data Center Network Design

In today’s digital age, data centers are the backbone of countless industries, powering the storage, processing, and transmitting massive amounts of information. However, the efficiency and scalability of data center network design have become paramount concerns. In this blog post, we explored the challenges traditional data center network architectures face and delved into innovative solutions that are revolutionizing the field.

The Limitations of Traditional Designs

Traditional data center network designs, such as three-tier architectures, have long been the industry standard. However, these designs come with inherent limitations that hinder performance and flexibility. The oversubscription of network links, the complexity of managing multiple layers, and the lack of agility in scaling are just a few of the challenges that plague traditional designs.

Enter the Spine-and-Leaf Architecture

The spine-and-leaf architecture has emerged as a game-changer in data center network design. This approach replaces the hierarchical three-tier model with a more scalable and efficient structure. The spine-and-leaf design comprises spine switches, acting as the core, and leaf switches, connecting directly to the servers. This non-blocking, high-bandwidth architecture eliminates oversubscription and provides improved performance and scalability.

Embracing Software-Defined Networking (SDN)

Software-defined networking (SDN) is another revolutionary concept transforming data center network design. SDN abstracts the network control plane from the underlying infrastructure, allowing centralized network management and programmability. With SDN, data center administrators can dynamically allocate resources, optimize traffic flows, and respond rapidly to changing demands.

The Rise of Network Function Virtualization (NFV)

Network Function Virtualization (NFV) complements SDN by virtualizing network services traditionally implemented using dedicated hardware appliances. By decoupling network functions, such as firewalls, load balancers, and intrusion detection systems, from specialized hardware, NFV enables greater flexibility, scalability, and cost savings in data center network design.

Conclusion:

The landscape of data center network design is undergoing a significant transformation. Traditional architectures are being replaced by more scalable and efficient models like the spine-and-leaf architecture. Moreover, concepts like SDN and NFV empower administrators with unprecedented control and flexibility. As technology evolves, data center professionals must embrace these innovations and stay at the forefront of this paradigm shift.