In today’s fast-paced world, where cloud computing and virtualization have become the norm, the need for efficient and flexible networking solutions has never been greater. OpenContrail, an open-source software-defined networking (SDN) solution, has emerged as a powerful tool. This blog post explores the capabilities, benefits, and significance of OpenContrail in revolutionizing network management and delivering enhanced connectivity in the cloud era.
OpenContrail, initially developed by Juniper Networks, is an open-source SDN platform offering comprehensive network capabilities for cloud environments. It provides a scalable and flexible network infrastructure that enables automation, network virtualization, and secure multi-tenancy across distributed cloud deployments.
Highlights: OpenContrail
The role of The VM
Virtual machines have been around for a long time, but we are beginning to spread our compute workloads in several different ways. When you throw in docker containers and bare metal servers, networking becomes more interesting. Network challenges arise when all these components require communication within the same subnet, access to Internet gateways, and Layer 3 MPLS/VPNs.
As a result, data center networks are moving towards IP underlay fabrics and Layer 2 overlays. Layer 3 data plane forwarding utilizes efficient Equal-cost multi-path routing (ECMP), but we lack Layer 2 multipathing by default. Now, similar to an SD WAN overlay approach, we can connect dispersed layer 2 segments and leverage all the good features of the IP underlay. To provide Layer 2 overlays and network virtualization, Juniper has introduced an SDN platform called Junipers OpenContrail in direct competition with
For additional pre-information, you may find the following post of use.
Introduction to the OpenContrail solution and what is involved.
Highlighting data center networks and ECMP.
Critical points on network virtualization.
Technical details on the virtual overlay network.
Technical details virtual network implementation.
Layer 2 VPN and EVPN.
Back to Basics with OpenContrail
Key Features and Benefits:
Network Virtualization:
OpenContrail leverages network virtualization techniques to provide isolated virtual networks within a shared physical infrastructure. It offers a logical abstraction layer, enabling the creation of virtual networks that operate independently, complete with their own routing, security, and quality of service policies. This approach allows for the efficient utilization of resources, simplified network management, and improved scalability.
Secure Multi-Tenancy:
OpenContrail’s security features ensure tenants’ data and applications remain isolated and protected from unauthorized access. It employs micro-segmentation to enforce strict access control policies at the virtual machine level, reducing the risk of lateral movement within the network. Additionally, OpenContrail integrates with existing security solutions, enabling the implementation of comprehensive security measures.
Intelligent Automation:
OpenContrail automates various network provisioning, configuration, and management tasks, reducing manual intervention and minimizing human errors. Its programmable API and centralized control plane simplify the deployment of complex network topologies, accelerate service delivery, and enhance overall operational efficiency.
Scalability and Flexibility:
OpenContrail’s architecture is designed to scale seamlessly, supporting distributed cloud deployments across multiple locations. It offers a highly flexible solution that can adapt to changing network requirements, allowing administrators to dynamically allocate resources, establish new connectivity, and respond to evolving business needs.
OpenContrail in Practice:
OpenContrail has gained significant traction among cloud providers, service providers, and enterprises seeking to build robust, scalable, and secure networks. Its open-source nature has facilitated its adoption, which encourages collaboration, innovation, and customization. OpenContrail’s community-driven development model ensures continuous improvement and the availability of new features and enhancements.
Highlighting Junipers OpenContrail
OpenContrail is an open-source network virtualization platform. The commercial controller and open-source product are identical; they share the same checksum on the binary image. Maintenance and support are the only difference. Juniper decided to open source to fit into the open ecosystem, which wouldn’t have worked in a closed environment.
OpenContrail offers similar features to VMware NSX and can apply service chaining and high-level security policies and provide the connection to Layer 3 VPNs for WAN integration. OpenContrail works with any hardware, but integration with Juniper’s product sets offers additional rich analytics for the underlay network.
Underlay and overlay network visibility are essential for troubleshooting. You need to look further than the first header of the packet; you need to look deeper into the tunnel to understand what is happening entirely.
Network virtualization – Isolated networks
With a cloud architecture, network virtualization gives the illusion that each tenant has a separate isolated network. Virtual networks are independent of physical network location or state, and nodes within the physical underlay can fail without disrupting the overlay tenant. A tenant may be a customer or department, depending if it’s a public or private cloud.
The virtual network sits on top of a physical network, the same way the compute virtual machines sits on top of a physical server. Virtual networks are not created with VLANs; Contrail uses a system of the virtual overlay network for multi-tenancy and cross-tenant communication. Many problems exist with large-scale VLAN deployments for multi-tenancy in today’s networks.
They introduce a lot of states in the physical network, and the Spanning Tree Protocol (STP) also introduces well-documented problems. There are technologies (THRILL, SPB) to overcome these challenges, but they add complexity to the design of the network.
Service Chaining
Customers require the ability to apply policy at virtual network boundaries. Policies may include ACL and stateless firewalls provided within the virtual switch. But once you require complicated policy pieces between virtual networks, you need a more sophisticated version of policy control and orchestration called service chaining. Service chaining applies intelligent services between traffic from one tenant to another.
For example, if a customer requires content caching and stateful services, you must introduce additional service appliances and force next-hop traffic through these appliances. Once you deploy a virtual appliance, you need a scale-out architecture.
The ability to Scale-out
Scale-out is the ability to instantiate multiple physical and virtual machine instances and load balance traffic across them. Customers may also require the ability to connect with different tenants in dispersed geographic locations or connect to workloads in a remote private cloud or public cloud. Usually, people build a private cloud for the norm and then burst into a public cloud.
Juniper has implemented a virtual networking architecture that meets these requirements. It is based on well-known technology, MPLS/layer 3 VPN. MPLS/layer 3 VPN is the base for Juniper designs.
The SDN controller is responsible for the networking aspects of virtualization. When creating virtual networks, initiate the Northbound API and issue an instruction that attaches VM to the VN. The network responsibilities are delegated from Cloudstack or OpenStack to Contrail. The Contrail SDN controller automatically creates the overlay tunnel between virtual machines. The overlay can be either an MPLS overlay style with MPLS-over-GRE, MPLS-over-UDP, or VXLAN.
L3VPN for routed traffic and EVPN for bridged traffic
Juniper’s OpenContrail is still a pure MPLS overlay of MPLS/VPN, using L3VPN for routed traffic and EVPN for bridged traffic. Traffic forwarding between end nodes has one MPLS label (VPN label), but they use various encapsulation methods to carry labeled traffic across the IP fabric. As mentioned above, this includes MPLS-over-GRE – a traditional encapsulation mechanism, MPLS-over-UDP – a variation of MPLS-over-GRE that replaces the GRE headers with UDP headers. And MPLS-over-VXLAN uses VXLAN packet format but stores the MPLS label in the Virtual Network Identifier (VNI) field.
The forwarding plane
The forwarding plane takes the packet from the VM and gives it to the “Vrouter,” which does a lookup and determines if the destination is a remote network. It encapsulates the packet and sends it across the tunnel if it is a remote network. The underlay that sites between the workloads forward are based on tunnel source and destination only.
No state belongs to end hosts ‘VMs, MAC addresses, or IPs. This type of architecture gives the Core a cleaner and more precise role. Generally, as a best practice, keeping “state” in the Core is a lousy design principle.
Northbound and southbound interfaces
To implement policy and service chaining, use the Northbound Interface and express your policy at a high level. For example, you may require HTTP or NAT and force traffic via load balancers or virtual firewalls. Contrail does this automatically and issues instructions to the Vrouter, forcing traffic to the correct virtual appliance. In addition, it can create all the right routes and tunnels, causing traffic through the correct sequence of virtual machines.
Contrail achieves this automatically with southbound protocols, such as XMPP (Extensible Messaging and Presence Protocol) or BGP. XMPP is a communications protocol based on XML (Extensible Markup Language).
WAN Integration
Junipers OpenContrail can connect virtual networks to external Layer 3 MPLS VPN for WAN integration. In addition, they gave the controller the ability to peer BGP to gateway routers. For the data plane, they support MPLS-over-GRE, and for the control plane, they speak MP- BGP.
Contrail communicates directly with PE routers, exchanging VPNv4 routes with MP-BGP and using MPLS-over-GRE encapsulation to pass IP traffic between hypervisor hosts and PE routers. Using standards-based protocols lets you choose any hardware appliance as the gateway node.
This data and control plane makes integration to an MPLS/VPN backbone a simple task. First, establish MP-BGP between the controllers to PE-routers. Inter-AS Option B next hop self-approach should be used to establish some demarcation points.
Conclusion:
OpenContrail has emerged as a game-changer in software-defined networking, empowering organizations to build agile, secure, and scalable networks in the cloud era. With its advanced features, such as network virtualization, secure multi-tenancy, intelligent automation, and scalability, OpenContrail offers a comprehensive solution that addresses the complex networking challenges of modern cloud environments. As the demand for efficient and flexible network management continues to rise, OpenContrail provides a compelling option for organizations looking to optimize their network infrastructure and unlock the full potential of the cloud.
In today’s interconnected world, networks enable seamless communication and data transfer. Overlay virtual networking has emerged as a revolutionary approach to network connectivity, offering enhanced flexibility, scalability, and security. This blog post aims to delve into the concept of overlay virtual networking, exploring its benefits, use cases, and potential implications for modern network architectures.
Overlay virtual networking is a network virtualization technique that decouples the logical network from the underlying physical infrastructure. It creates a virtual network on top of the existing physical infrastructure, enabling the coexistence of multiple logical networks on the same physical infrastructure. By abstracting the network functions and services from the physical infrastructure, overlay virtual networking provides a flexible and scalable solution for managing complex network environments.
Highlights: Network Overlay
Network Overlay
Virtual networking overlays are a vital technology for modern data centers, providing flexibility, scalability, and increased security. A virtual overlay network is a computer network built on top of an existing physical network. This abstraction layer creates a virtual overlay solution, which can be configured independently of the underlying physical network.
Increased Security
Virtual networks offer many advantages over traditional physical networks. First, they allow for rapidly deploying virtual machines, applications, and services. This can be done in minutes rather than the days or weeks it would take to reconfigure a physical network. Additionally, virtual networks provide increased security, as traffic between VMs can be isolated and encrypted. This can help prevent data breaches and unauthorized access.
Additional Flexibility
Virtual networks are also much more flexible than physical networks. For example, virtual networks can be segmented to separate different types of traffic. This improves performance, as traffic from other applications can be routed separately. Furthermore, virtual networks can be quickly reconfigured to meet changing demands. This agility allows for quick responses to changing business requirements.
Related: Before you proceed, you may find the following useful:
Introduction to overlay virtual networking and what is involved.
Highlighting the details of the virtual overlay solution and the components used.
Critical points on the scalability and security concerns.
Technical details on the different types of overlays, such as STT and VXLAN.
Closing comments on network overlay controllers.
A key point: Video introducing overlay network
The following video introduces overlay networking. The network overlay is placing Layer 2 or Layer 3 over a Layer 3 core. The Layer 3 core is known as the underlay. This removes many drawbacks and scaling issues with traditional Layer 2 connectivity, which uses VLANs. The multi-tenant nature of overlays is designed to avoid these L2 challenges, allowing you to build networks at a much larger scale. One of the critical differences between VXLAN vs VLAN is scale.
Back to Basics: Virtual networks and virtualization.
Underlay and Overlay Networks
Overlay networks are virtual networks that run on top of physical networks. You have probably seen this terminology even if you have never heard of it. A GRE tunnel can illustrate an overlay network. Physical underlay networks support the GRE tunnel.VXLAN overlays are layer 2 Ethernet networks. Layer 3 IP networks form the underlay network. Transport networks are also known as underlay networks.
Getting packets from A to B is the only job of the underlay network. Layer 2 is not used here, only layer 3. We can load balance traffic on redundant links using an IGP like OSPF or EIGRP.
In addition, the overlay and underlay networks are independent. Underlay networks are virtual, but any changes made to the overlay network won’t affect the underlay network. A routing protocol can reach the destination regardless of how many links you add or remove in the underlay network.
Virtual Networking
Main Virtual Overlay Networking Components
Overlay Virtual Networks
Overlay networks are virtual networks that run on top of physical networks
The most common forms of network virtualization are virtual LANs (VLANs), virtual private networks (VPNs), and Multiprotocol Label Switching (MPLS)
Like the ACI network, virtual overlay networks work best with Leaf and Spine fabric architectures
STT and VXAN can use 5-tuple load balancing as they use port numbers
Virtual overlay solutions
Virtual overlay solutions must have some simple to complex application stacks. Therefore, public or private cloud environments must support austere, complex environments to enable the virtual overlay network. On the other hand, simple customers that require web-hosting solutions need only a single domain with a few segments. In terms of network connectivity is one Virtual Machine ( VM ) with a single public IP.
Complex customers require complex multi-tier application stacks with overlay virtual networking, load-balancing, and firewall services in front and between application tiers. Cloud providers must support all types of application stacks as they are isolated virtual segments, and this is done with virtual overlay networks.
Lab guide on VXLAN.
In the following example, we have a lab guide on VXLAN. Here, we created a Layer 2 overlay across the core. The core layer consists of two spines and is a routed layer. The core does not know the subnets assigned to the desktop devices. It is the role of VXLAN to tunnel this information.
Notice we have a VNI set to 6002. This needs to match at both ends of Leaf A and Leaf B. If you change the VNI, you will break connectivity. This is a Layer 2 overlay, as the VNI is mapped to a bridge domain.
Concept of network virtualization
It’s worth mentioning that network virtualization is nothing new. The most common forms of network virtualization are virtual LANs (VLANs), virtual private networks (VPNs), and Multiprotocol Label Switching (MPLS). VLAN has been the first to extract the location of Layer 2 connectivity across multiple Layer 2 switches. VPN enables overlay networks across untrusted networks such as the WAN, while MPLS segments traffic based on labels.
These technologies enable the administrators to physically separate endpoints into logical groups, making them behave like they are all on the same local (physical) segment. The ability to do this allows for much greater efficiency in traffic control, security, and network management.
Enhanced Connectivity:
One of the primary advantages of network overlay is its ability to enhance connectivity. By creating a virtual network layer, overlay networks enable seamless communication between devices and applications, irrespective of their physical location.
This means organizations can effortlessly connect geographically dispersed branches, data centers, and cloud environments, fostering collaboration and resource sharing. Moreover, network overlays offer greater flexibility by allowing organizations to dynamically adjust and optimize their network configurations to meet evolving business needs.
Improved Scalability:
Traditional network infrastructures often struggle to keep up with the increasing demands of modern applications and services. Network overlay addresses this challenge by providing a scalable solution. By decoupling the virtual network from the physical infrastructure, overlay networks allow for more efficient resource utilization and easier scaling.
Organizations can easily add or remove network elements without disrupting the entire network. As a result, network overlays enable organizations to scale their networks rapidly and cost-effectively, ensuring optimal performance even during peak usage periods.
Example of an overlay network: MPLS
MPLS overlay is a technique used to create virtual private networks (VPNs) over existing IP networks, enabling organizations to achieve enhanced network scalability, reliability, and security. Unlike traditional IP routing, MPLS overlay relies on labels to forward packets, making it more efficient and flexible.
Overlay with MPLS
With MPLS, we can have a free BGP core providing an MPLS overlay. MPLS overlay is a network architecture that allows organizations to build virtual private networks (VPNs) on top of their existing network infrastructure. It leverages the capabilities of MPLS technology to create virtual tunnels, known as MPLS tunnels or MPLS paths, which enable the secure and efficient transfer of data between different network endpoints.
Below, we have BGP running between the PEs and carrying customer prefixes for CE 1 and 2. The P representing the core layer does not know customer routes and performs label switching. This brings not only scalability as the P nodes can focus on label switching and an added layer of security. No security devices need to be present in the core layer. Although you would need QoS, they are pushing intelligence to the edges.
Benefits of MPLS Overlay:
1. Enhanced Performance: MPLS overlay offers improved network performance by enabling faster data transmission and reduced latency. It achieves this by using label switching, which helps prioritize and route data packets efficiently, reducing congestion and optimizing network utilization.
2. Scalability and Flexibility: With MPLS overlay, organizations can quickly expand their network infrastructure without requiring extensive hardware upgrades. It allows for creating of virtual networks within a shared physical infrastructure, enabling seamless scalability and flexibility.
3. Quality of Service (QoS): MPLS overlay provides enhanced QoS capabilities, enabling organizations to prioritize critical applications or data traffic. This ensures mission-critical applications receive the bandwidth and low latency, optimizing overall network performance.
4. Improved Security: MPLS overlay enhances network security by providing inherent isolation between different VPNs. It creates separate virtual tunnels for each VPN, ensuring that data remains isolated and protected from unauthorized access.
Lab Guide on MPLS TE
In this lab, we will look at MPLS TE with ISIS configuration. Routers PE1, P1, P2, P3, and PE2 are our MPLS core network. The CE1 and CE2 routers use regular IP routing. All routers are configured to use IS-IS L2.
MPLS TE is a mechanism that allows network operators to control and manage traffic flows within a Multiprotocol Label Switching (MPLS) network. It is designed to address the limitations of traditional IP routing by providing a more efficient and flexible approach to data forwarding
Note:
There are four main items we have to configure:
Enable MPLS TE support:
Globally
Interfaces
Configure IS-IS to support MPLS TE.
Configure RSVP.
Configure a tunnel interface.
Example of an overlay network: DMVPN
With the configuration of DMVPN phase 1, we can have a “hub and spoke” topology, where a single hub site acts as the central point for communication, while the other locations, or “spokes,” connect to the hub through virtual tunnels. This topology provides several benefits, including secure communications between spokes, optimized traffic routing, and reduced overhead for managing the network.
DMVPN also supports dynamic routing protocols, such as Open Shortest Path First (OSPF), allowing for dynamic updates to the network topology. This allows for rapid changes in the network, such as adding or removing spokes, without the need to reconfigure the entire network. Additionally, DMVPN supports multicast traffic, allowing the efficient distribution of data and resources to multiple sites simultaneously.
A key point: Video on DMVPN. A WAN overlay.
In this technical demo, we will start with the first network topology, with a Hub and Spoke design, and recap DMVPN Phase 1. This was the starting point of the DMVPN design phases. However, today, you will probably see DMVPN phase 3, which allows for spoke-to-spoke tunnels that may be better suited if you don’t need a true hub and spoke.
In the following lab, we have DMVPM, which creates an overlay network. The hub, which is R1, created an overlay network over the SP router. The SP router represents the WAN; in reality, the number of nodes in the WAN is irrelevant to DMPVN. The overlay is created between R1 R2, and R3, which act as the spokes.
The protocol used in GRE, specifically point-to-point GRE, as we are running DMVPN Phase 1. The Tunneling protocol of mGRE would have been used if we were running DMVPN Phase 3
Benefits of DMVPN Overlay:
1. Simplified Network Architecture:
Traditional networking often involves complex and static configurations, making it cumbersome to manage and maintain. DMVPN overlay, on the other hand, simplifies network architecture by providing a dynamic and scalable solution. With DMVPN, organizations can establish secure connections between various branch offices, data centers, and remote users, all while leveraging the existing infrastructure. This simplification leads to reduced administrative overhead and improved network agility.
2. Enhanced Flexibility and Scalability:
DMVPN overlay offers unparalleled flexibility and scalability, making it an ideal choice for organizations with dynamic network requirements. As businesses grow and expand, DMVPN allows for the seamless addition of new sites or remote users without requiring extensive configuration changes. Its ability to establish connections on-demand and dynamically allocate resources ensures that network expansion remains hassle-free and cost-effective.
3. Improved Network Performance:
Network performance is crucial for organizations, directly impacting productivity and user experience. DMVPN overlay utilizes multiple paths and load balancing techniques, allowing for efficient utilization of available bandwidth. By optimizing network traffic, DMVPN ensures that applications and services operate smoothly, even during peak usage periods. Moreover, its ability to prioritize critical traffic and dynamically adjust to network conditions further enhances overall performance.
4. Enhanced Security:
Security remains a top concern for organizations, particularly when transmitting sensitive data across networks. DMVPN overlay addresses these concerns by providing robust encryption and authentication mechanisms. By leveraging IPsec protocols, DMVPN ensures that data confidentiality and integrity are maintained, protecting against unauthorized access and potential threats. The inherent security features of DMVPN make it a reliable choice for organizations looking to maintain a secure network environment.
Types of Overlay Networks
1. Virtual Private Networks (VPNs):
VPNs are one of the most common types of overlay networks. They enable secure communication over public networks by creating an encrypted tunnel between the sender and receiver. Individuals and organizations widely use VPNs to protect sensitive data and maintain privacy. Additionally, they allow users to bypass geographical restrictions and access region-restricted content.
2. Software-Defined Networks (SDNs):
In network architecture, SDNs utilize overlay networks to separate the control plane from the data plane. SDNs provide centralized management, flexibility, and scalability by decoupling network control and forwarding functions. Overlay networks in SDNs enable the creation of virtual networks on top of the physical infrastructure, allowing for more efficient resource allocation and dynamic network provisioning.
3. Peer-to-Peer (P2P) Networks:
P2P overlay networks are decentralized systems facilitating direct communication and file sharing between nodes without relying on a central server. P2P networks leverage overlay networks to establish direct connections between peers and enable efficient data distribution. These networks are widely used for content sharing, real-time streaming, and decentralized applications.
4. Content Delivery Networks (CDNs):
CDNs employ overlay networks to optimize content delivery by strategically distributing content across multiple servers in different geographic regions. By bringing content closer to end-users, CDNs reduce latency and improve performance. Overlay networks in CDNs enable efficient content caching, load balancing, and fault tolerance, resulting in faster and more reliable content delivery.
5. Overlay Multicast Networks:
Overlay multicast networks are designed to distribute data to multiple recipients simultaneously efficiently. These networks use overlay protocols to construct multicast trees and deliver data over these trees. Overlay multicast networks benefit applications such as video streaming, online gaming, and live events broadcasting, where data must be transmitted to many recipients in real time.
Use Cases of Overlay Virtual Networking:
1. Multi-Tenancy:
Overlay virtual networking provides an ideal solution for organizations to segregate their network resources securely. Multiple tenants can coexist on a single physical network infrastructure without interference by creating virtual overlays. This enables service providers and enterprises to offer distinct network environments to customers or departments while ensuring isolation and security.
2. Data Center Interconnect:
Overlay virtual networking enables efficient and scalable data center interconnect (DCI). With traditional networking, interconnecting multiple data centers across geographies can be complex and costly. However, overlay virtual networking simplifies this process by abstracting the underlying physical infrastructure and providing a unified logical network. It allows organizations to seamlessly extend their networks across multiple data centers, enhancing workload mobility and disaster recovery capabilities.
3. Cloud Computing:
Cloud computing heavily relies on overlay virtual networking to deliver agility and scalability. Cloud providers can dynamically provision and manage network resources by leveraging overlay networks, ensuring optimal customer performance and flexibility. Overlay virtual networking enables the creation of virtual networks that are isolated from each other, allowing for secure and efficient multi-tenant cloud environments.
4. Microservices and Containerization:
The rise of microservices architecture and containerization has presented new challenges for networking. Overlay virtual networking provides a solution by enabling seamless communication between microservices and containers, regardless of their physical location. It ensures that applications and services can communicate with each other, even across different hosts or clusters, without complex network configurations.
5. Network Segmentation and Security:
Overlay virtual networking enables granular network segmentation, allowing organizations to implement fine-grained security policies. By creating overlay networks, administrators can isolate different workloads, departments, or applications, ensuring each segment has its dedicated network resources and security policies. This enhances security by limiting the lateral movement of threats and reducing the attack surface.
Tailored load balancing
Some customers may not require cloud load balancing services provided by the cloud services if they have optimized web delivery by deploying something like Squid or NGINX. Squid is a caching proxy that improves web request response times by caching frequently requested web pages. NGINX ( open source reverse proxy ) is used to load balance Hypertext Transfer Protocol ( HTTP ) among multiple servers.
Example: Traffic flow and the need for a virtual overlay
Traffic would flow to Web servers and trigger application and database requests. Each tier requires different segments, and in large environments, the limitations of using VLANs to create these segments will bring both scalability and performance problems.
This is why we need virtual overlay solutions. These subnets require Layer 3 and sometimes Layer 2 ( MAC ). Layer 2 connectivity might be for high availability services that rely on gratuitous Address Resolution Protocol ( ARP ) between devices or some other non-routable packet that can not communicate over IP. If the packet is not Layer 3 routable, it needs to communicate via Layer 2 VLANs.
Scalability and Security Concerns
The weakest link in a security paradigm is the lowest application in that segment. Make each application an independent tenant so all other applications are unaffected if a security breach or misuse occurs in one application stack.
Designers should always attempt to design application stacks to minimize beachheading, i.e., an attacker compromising one box and using it to jump to another quickly. Public and private clouds should support multi-tenancy with each application stack.
However, scalability issues arise when you deploy each application as an individual segment. For example, customer X’s cloud application requires four segments; 4000 VLANs soon become 1000 applications. Media Access Control ( MAC ) visibility has an entire reach throughout Layer 2 domains.
Some switches support a low count number of MAC addresses. When a switch reaches its MAC limit, it starts flooding packets, increasing network load and consuming available bandwidth that should be used for production services.
…current broadcast domains can support … around 1,000 end hosts in a single bridged LAN of 100 bridges” (RFC 5556 – TRILL)
NIC in promiscuous mode and failure domains
To save configuration time, server administrators configure server NICs in promiscuous mode. NICs in promiscuous mode look at all frames passing even when the frame is not destined to it. Network cards acting in promiscuous mode are essentially the same as having one VLAN spanning the entire domain. Sniffer products set promiscuous modes to capture all data on a link and usually only act in this mode for troubleshooting purposes.
A well-known issue with Layer 2 networks is that they present a single failure domain with extreme scalability and operational challenges. This is related to Layer 2 Spanning Tree Protocol ( STP ); THRILL is also susceptible to broadcast storms and network meltdowns.
The rise of overlay virtual networks
Previously discussed scalability and operational concerns force vendors to develop new data center technologies. One of the most prevalent new technologies is overlay virtual networks, tunneling over IP. An overlay is a tunnel between two endpoints, allowing frames to be transported. The beauty of overlay architectures is that they enable switch table sizes not to increase as the number of hosts attached increases.
Vendors’ Answer: Virtual Overlay Solutions
Virtual Overlay Solution: Keep complexity to the edges.
Ideally, we should run virtual networks over IP like SKYPE runs Voice over IP. The recommended design retains complexity at the network’s edge; the IP transport network provides IP transport. A transport network does not need to be a Layer 2 network and can have as many IP subnets and router hops.
All data ( storage, vMotion, user traffic ) traffic becomes an IP application. The concept resembles how Border Gateway Protocol ( BGP ) applies to TCP. End hosts carry out encapsulation and use the network for transport. Again, complexity is at the edge, similar to the Internet. Keeping complexity to the edge makes Layer 3 fabrics efficient and scalable.
VXLAN, STT, and ( NV ) GRE
Numerous encapsulation methods can tunnel over the IP core. This is known as virtual overlay networking and includes VXLAN, STT, and ( NV ) GRE. The main difference between these technologies is the encapsulation method and minor technological differences with TCP offload and load balancing.
The Recommended Design: Leaf and Spine.
Like the ACI network, virtual overlay networks work best with Leaf and Spine fabric architectures. Leaf and Spine designs guarantee any two endpoints get equal bandwidth. VMs on the same Top-of-Rack ( ToR ) switch will have access to more bandwidth than if the VM had to communicate across the Spine layer.
Overlay networks assume that the underlying network has a central endpoint. The transport network should avoid oversubscription as much as possible. If security concerns you, you can always place similar VM appliances on dedicated clusters, one type per physical server.
( NV ) GRE, VXLAN, and STT do not have an built-in security features meaning the transport network MUST be secure.
A key point: Vidoe on Leaf and Spine
This quick education tutorial will examine the leaf and spine data center architecture. We know this design is a considerable step from traditional DC design. As a use case, we will focus on how Cisco has adopted the leaf and spine design with its Cisco ACI product. We will address the components and how they form the Cisco ACI fabric.
TCP can push huge segments down the physical NIC and slice the packet into individual TCP segments, improving TCP performance. For example, you can push 10Gbps from a VM with TCP offload. The problem is that NICs only support VLANs and not VXLANs.
NICIRA added another header in front of TCP segments. TCP is embedded in another TCP. Now, you can use the existing NIC to slice the current TCP segment into smaller TCP segments. It is dramatically improving performance.
STT and VXAN
STT and VXAN can use 5-tuple load balancing as they use port numbers. Therefore, traffic sent between a pair of VMs can use more than one link in the network. Unfortunately, not many switches can load balance based on the GRE payload used by NVGRE.
Scale-out NAT is hard to do as an asymmetric path is not guaranteed. Furthermore, the shared state is tied to an outside IP address, which limits scale-out options. To scale out effectively, the state has to be spread across all members of the NAT cluster. The new approach uses floating public IP addresses and one-to-one mapping between floating IP to the private IP address inside—no state due to the one-to-one mapping.
Distributed layer 2 & layer 3 forwarding
They distributed Layer 2 forwarding ( data plane ):Most Overlays offer distributed Layer 2 forwarding. VM can be sent to VM in the same segment. The big question is how they distribute MAC to VTEP – some use multicast and traditional Ethernet flooding, while others use control planes. The big question is how scalable is the control plane.
Distributed Layer 3 forwarding ( data plane ): On the other hand, if you have multiple IP subnets between segments ( not layer 2 ), you need to forward between them. The inter-subnet must not be a choke point. If your data center has lots of intra-traffic ( East to West traffic), avoid centralized inter-subnet forwarding, which will quickly become a traffic choke point.
The router will do ARP processing if you are doing Layer 3 forwarding. But if you are doing a mix of Layer 2 and 3, then make sure you can reduce the flooding by intercepting ARP requests and caching ARP replies, known as distributed ARP Caching.
Scale-out control plane
Initial overlays used multicast and Ethernet-like learning. Now, some vendors are using controller-based overlays. Keep in mind that the controller can now become a scalability bottleneck. However, many vendors, such as Cisco ACI, can scale the controllers and have a quorum.
Efficient controller scalability is seen when controllers do not participate in the data plane ( do not reply to ARP ). This type of controller scales better than controllers that have to intercept data plane packets and perform data plane activity. So, the data plane will not be affected if a controller is offline. In the early days of Sofware-Defined Networking, this was not the case. If the controller was down, the network was down.
Scale-out controllers
Attempt to design scale-out controllers by building a cluster of controllers and having some protocol running between them. You now have clear failure domains. For example, controller A looks after VM segment A and Controller B, and control looks after VM segment B. For cloud deployments in multiple locations, deploy multiple controller clusters in each location.
Availability zones
Design availability zones with hierarchical failure domains by splitting infrastructures into regions. Problems arising in one region do not affect all other regions. You have one or more availability zones within an area for physical and logical isolation.
Availability zones limit the impact of the failure in a failure domain. An example of a failure domain could be a VLAN experiencing a broadcast storm. Attempt to determine the span of VLANs across availability zone – define VLANs to one-ToR switch. Never stretch VLANs as you create a single failure domain by merging two zones.
Do not stretch a VLAN across multiple availability zones. This is why we have network overlays in the first place, so we don’t need to stretch VLAN across the data center. For example, VXLAN uses the VNI to differentiate between Layer 2 and Layer 3 traffic over a routed underlay. We can use VXLAN as the overlay network to span large Layer 2 domains over a routed core.
Network Overlay Controllers
As a final note on controllers, controller-based SDN networks participate in data plane performing activities such as MAC learning and ARP replies. As mentioned, this is not common nowadays but was at the start of the SDN days. If the controller performs activities such as MAC learning and APR replies and the controller fails, then you have network failure.
The more involved the controller is in the forwarding decisions, the worse the outage can be. All overlay networking vendors nowadays have controllers that set up the control plane so the data plane can forward traffic without getting involved in data plane activity. This design also allows the controller to be scaled without affecting the data plane activity.
Overlay virtual networking has significant implications for modern network architectures. It enables the creation of software-defined networks (SDNs), where network policies, routing, and security are managed centrally through software-based controllers. This centralized management simplifies network operations, improves agility, and enables network automation.
Summary: Understanding Overlay Virtual Networking
Overlay virtual networking is a method that allows virtual networks to be created on top of existing physical networks. By decoupling the network’s logical and physical infrastructure, overlay virtual networking provides flexibility, scalability, and enhanced security.
Benefits of Overlay Virtual Networking
Overlay virtual networking brings numerous advantages to organizations. Firstly, it enables seamless network scalability, allowing businesses to quickly expand their network resources without needing physical infrastructure upgrades. Additionally, overlay virtual networking enhances network security by providing isolated virtual networks that can be customized with specific security policies.
Implementation of Overlay Virtual Networking
Implementing overlay virtual networking involves utilizing software-defined networking (SDN) technologies. Organizations can create virtual networks independent of the underlying physical infrastructure through SDN controllers and network virtualization overlays. This implementation approach simplifies network management and enables dynamic configuration.
Applications of Overlay Virtual Networking
Overlay virtual networking finds applications in various industries. In data centers, it facilitates efficient resource allocation and workload mobility across virtual machines. For cloud service providers, overlay virtual networking enables the creation of virtual private clouds, ensuring secure and isolated connectivity for their customers. Moreover, overlay virtual networking can enhance the connectivity and security of IoT devices in smart cities and industrial environments.
Conclusion: Overlay virtual networking empowers organizations to unlock new network flexibility, scalability, and security levels. By abstracting the network’s logical layer from the physical infrastructure, overlay virtual networking enables seamless expansion, simplified management, and tailored security policies. As the digital landscape continues to evolve, overlay virtual networking will play a pivotal role in shaping the future of networking.
In the world of networking, the ability to efficiently manage and scale networks is of paramount importance. This is where LISP networking comes into play. LISP, which stands for Locator/ID Separation Protocol, is a powerful networking technology that offers numerous benefits to network administrators and operators. In this blog post, we will explore the world of LISP networking, exploring its key features and advantages.
LISP networking is a revolutionary approach to IP addressing and routing that separates the identity of a device (ID) from its location (locator). Traditional IP addressing relies on combining these two aspects, making it challenging to scale networks and manage mobility. LISP overcomes these limitations by decoupling the device’s identity and location, enabling more flexible and scalable network architectures.
Use Case: Hybrid Cloud
The hybrid cloud connects the public cloud provider to the private enterprise cloud. It consists of two or more distinct infrastructures in dispersed locations that remain unique. These unique entities are bound together logically via a network to enable data and application portability. LISP networking performs hybrid cloud and can overcome the negative drawback of stretched VLAN. How do you support intra-subnet traffic patterns among two dispersed cloud locations? Without a stretched VLAN spanning locations, that may bring instability with broadcast storms and Layer 2 loops.
End to End Connectivity
Enterprises want the ability to seamlessly insert their application right into the heart of the cloud provider without changing any parameters. Customers want to do this without changing the VM’s IP addresses and MAC addresses. This requires the VLAN to be stretched end-to-end. Unfortunately, IP routing cannot support VLAN extension, which puts pressure on the data center interconnect ( DCI ) link to enable extended VLANs. In reality, and from experience, this is not a good solution.
Before you proceed, you may find the following helpful:
Introduction to LISP Hybrid Cloud and what is involved.
Highlighting the details of LISP networking and how it can be implemented.
Critical points in a step-by-step format.
A final note on LISP stretched VLAN and overlay networking.
A key point: Video on LISP components and their configuration.
In this video, we will bring you through the stages of LISP configuration and the LISP networking components involved, which will help you on our hybrid cloud journey.
Hands on Video Series - Enterprise Networking | LISP Configuration Intro
The LISP network comprises a mapping system with a global database of RLOC-EID mapping entries. The mapping system is the control plane of the LISP network decoupled from the data plane. The mapping system is address-family agnostic; the EID can be an IPv4 address mapped to an RLOC IPv6 address and vice versa. Or the EID may be a Virtual Extensible LAN (VXLAN) Layer 2 virtual network identifier (L2VNI) mapped to a VXLAN tunnel endpoint (VTEP) address working as an RLOC IP address.
How Does LISP Networking Work?
At its core, LISP networking introduces a new level of indirection between the device’s IP address and location. LISP relies on two key components: the xTR (eXternal Tunnel Router) and the mapping system. The xTR is responsible for encapsulating and forwarding traffic between different LISP sites, while the mapping system stores the mappings between the device’s identity and its current location.
Benefits of LISP Networking:
Scalability: LISP provides a scalable solution for managing large networks by separating the device’s identity from its location. This allows for efficient routing and reduces the amount of routing table information that needs to be stored and exchanged.
Mobility: LISP networking offers seamless mobility support, enabling devices to change locations without disrupting ongoing communications. This is particularly beneficial in scenarios where mobile devices are constantly moving, such as IoT deployments or mobile networks.
Traffic Engineering: LISP allows network administrators to optimize traffic flow by manipulating the mappings between device IDs and locators. This provides greater control over network traffic and enables efficient load balancing and congestion management.
Security: LISP supports secure communications through the use of cryptographic techniques. It provides authentication and integrity verification mechanisms, ensuring the confidentiality and integrity of data transmitted over the network.
Use Cases for LISP Networking:
Data Centers: LISP can significantly simplify the management of large-scale data center networks by providing efficient traffic engineering and seamless mobility support for virtual machines.
Internet Service Providers (ISPs): LISP can help ISPs improve their network scalability and handle the increasing demand for IP addresses. It enables ISPs to optimize their routing tables and efficiently manage address space.
IoT Deployments: LISP’s mobility support and scalability make it an ideal choice for IoT deployments. It efficiently manages large devices and enables seamless connectivity as devices move across different networks.
LISP Networking and Stretched VLAN
Locator Identity Separation Protocol ( LISP ) can extend subnets without the VLAN. I am creating a LISP Hybrid Cloud. A subnet extension with LISP is far more appealing than a Layer 2 LAN extension. The LISP-enabled hybrid cloud solution allows Intra-subnet communication regardless of where the server is. This means you can have two servers in different locations, one in the public cloud and the other in the Enterprise domain; both servers can communicate as if they were on the same subnet.
LISP acts as an overlay technology
LISP operates like an overlay technology; it encapsulates the source packet with UDP and a header consisting of the source and destination RLOC ( RLOC are used to map EIDS). The result is that you can address the servers in the cloud according to your addressing scheme. There is no need to match your addressing scheme to the cloud addressing scheme.
LISP on the Cloud Service Router ( CRS ) 1000V ( virtual router ) solution provides a Layer-3-based approach to a hybrid cloud. It allows you to stretch subnets from the enterprise to the public cloud without needing a Layer 2 LAN extension.
LISP networking deployment key points:
LISP can be deployed with the CRS 1000V in the cloud and either a CRS 1000V or ASR 1000 in the enterprise domain.
The enterprise CRS must have at least two interfaces. One interface is the L3 routed interface to the core. The second interface is a Layer 2 interface to support VLAN connectivity for the servers that require mobility.
The enterprise CRS does not need to be the default gateway, and its interaction with the local infrastructure ( via the Layer 2 interface ) is based on Proxy-ARP. As a result, ARP packets must be allowed on the underlying networks.
The Cloud CRS is also deployed with at least two interfaces. One interface is facing the Internet or MPLS network. The second interface faces the local infrastructure, either by VLANs or Virtual Extensible LAN ( VXLAN ).
The CRS offers machine-level high availability and supports all the VMware high-availability features such as dynamic resource scheduling ( DRS ), vMotion, NIC load balancing, and teaming.
LISP is a network-based solution and is independent of the hypervisor. You can have different hypervisors in the Enterprise and the public cloud. No changes to virtual servers or hosts. It’s completely transparent.
The PxTR ( also used to forward to non-LISP sites ) is deployed in the enterprise cloud, and the xTR is deployed in the public cloud.
The CRS1000V deployed in the public cloud is secured by an IPSEC tunnel. Therefore, the LISP tunnel should be encrypted using IPSEC tunnel mode. Tunnel mode is preferred to support NAT.
Each CRS must have one unique outside IP address. This is used to form the IPSEC tunnel between the two endpoints.
Dynamic or static Routing must be enabled over the IPSEC tunnel. This is to announce the RLOC IP address used by the LISP mapping system.
The map-resolver ( MR ) and map server ( MS ) can be enabled on the xTR in the Enterprise or the xTR in the cloud.
Traffic symmetry is still required when you have stateful devices in the path.
LISP stretched subnets
The two modes of LISP operation are the LISP “Across” subnet and the LISP “Extended” subnet mode. Neither of these modes is used with the LISP-enabled CRS hybrid cloud deployment scenario. The mode of operation utilized is called the LISP stretched subnet model ( SSM ). The same subnet is used on both sides of the network, and mobility is performed between these two segments on the same subnet. You may think that this is the same as LISP “Extended” subnet mode, but in this case, we are not using a LAN extension between sites. Instead, the extended mode requires a LAN extension such as OTV.
In today’s rapidly evolving digital landscape, network management, and data flow control have become critical for businesses of all sizes. OpenFlow is one technology that has gained significant attention and is transforming how networks are managed. In this blog post, we will delve into the concept of OpenFlow, its advantages, and its implications for network control.
OpenFlow is an open-standard communications protocol that separates the control and data planes in a network architecture. It allows network administrators to have direct control over the behavior of network devices, such as switches and routers, by utilizing a centralized controller.
Traditional network architectures follow a closed model, where network devices make independent decisions on forwarding packets. On the other hand, OpenFlow introduces a centralized control plane that provides a global view of the network and allows administrators to define network policies and rules from a centralized location.
Introducing SDN
Recent changes and requirements drive networks and network services to become more flexible, virtualization-aware, and API-driven. One major trend that is affecting the future of networking is software-defined networking ( SDN ). The software-defined architecture aims to extract the entire network into a single switch.
Software-defined networking is an evolving technology defined by the Open Networking Foundation ( ONF ). Software Defined Networking is the physical separation of the network control plane from the forwarding plane, where a control plane controls several devices. This somewhat differs significantly from traditional IP forwarding that you may have used in the past.
Data and control plane
Therefore, SDN separates the data and control plane. The main driving body behind software-defined networking (SDN) is the Open Networking Foundation ( ONF ). Introduced in 2008, the ONF is a non-profit organization that wants to provide an alternative to proprietary solutions that limit flexibility and create vendor lock-in.
The insertion of the ONF allowed its members to run proof of concepts on heterogeneous networking devices without requiring vendors to expose the internal code of their software. This creates a path for an open-source approach to networking and policy-based controllers. Now let us identify the benefits of OpenFlow in the following tables.
You may find the following useful for pre-information:
Introduction to what is OpenFlow and what is involved with the protocol.
Highlighting the details and benefits of OpenFlow.
Technical details on the lack of session layers in the TCP/IP model.
Scenario: Control and data plane separation with SDN.
A final note on proactive vs reactive flow setup.
Back to basics. What is OpenFlow?
What is OpenFlow?
OpenFlow was the first protocol of the Software Defined Networking (SDN) trend and is the only protocol that allows the decoupling a network device’s control plane from the data plane. In most straightforward terms, the control plane can be thought of as the brains of a network device. On the other hand, the data plane can be considered hardware or application-specific integrated circuits (ASICs) that perform packet forwarding.
Numerous devices also support running OpenFlow in a hybrid mode, meaning OpenFlow can be deployed on a given port, virtual local area network (VLAN), or even within a regular packet-forwarding pipeline such that if there is not a match in the OpenFlow table, then the existing forwarding tables (MAC, Routing, etc.) are used, making it more analogous to Policy Based Routing (PBR).
What is SDN?
Despite various modifications to the underlying architecture and devices (such as switches, routers, and firewalls), traditional network technologies have existed since the inception of networking. Using a similar approach, frames, and packets have been forwarded and routed in a limited manner, resulting in low efficiency and high maintenance costs—consequently, the architecture and operation of networks needed to evolve, resulting in SDN.
By enabling network programmability, SDN promises to simplify network control and management and allow innovation in computer networking. Network engineers configure policies to respond to various network events and application scenarios. They can achieve the desired results by manually converting high-level policies into low-level configuration commands.
Often, minimal tools are available to accomplish these very complex tasks. Controlling network performance and tuning network management are challenging and error-prone tasks.
A modern network architecture consists of a control plane, a data plane, and a management plane; the control and data planes are merged into a machine called Inside the Box. To overcome these limitations, programmable networks have emerged.
How OpenFlow Works:
At the core of OpenFlow is the concept of a flow table, which resides in each OpenFlow-enabled switch. The flow table contains match-action rules defining how incoming packets should be processed and forwarded. These rules are determined by the centralized controller, which communicates with the switches using the OpenFlow protocol.
When a packet arrives at an OpenFlow-enabled switch, it is first matched against the rules in the flow table. If a match is found, the corresponding action is executed, including forwarding the packet, dropping it, or sending it to the controller for further processing. This decoupling of the control and data planes allows for flexible and programmable network management.
What is OpenFlow SDN?
The main goal of SDN is to separate the control and data planes and transfer network intelligence and state to the control plane. These concepts have been exploited by technologies like Routing Control Platform (RCP), Secure Architecture for Network Enterprise (SANE), and, more recently, Ethane.
In addition, there is often a connection between SDN and OpenFlow. The Open Networking Foundation (ONF) is responsible for advancing SDN and standardizing OpenFlow, whose latest version is 1.5.0.
An SDN deployment starts with these building blocks.
For communication with forwarding devices, the controller has the SDN switch (for example, an OpenFlow switch), the SDN controller, and the interfaces. An SDN deployment is based on two basic building blocks, a southbound interface (OpenFlow) and a northbound interface (the network application interface).
As the control logic and algorithms are offloaded to a controller, switches in SDNs may be represented as basic forwarding hardware. Switches that support OpenFlow come in two varieties: pure (OpenFlow-only) and hybrid (OpenFlow-enabled).
Pure OpenFlow switches do not have legacy features or onboard control for forwarding decisions. A hybrid switch can operate with both traditional protocols and OpenFlow. Hybrid switches make up the majority of commercial switches available today. A flow table performs packet lookup and forwarding in an OpenFlow switch.
OpenFlow reference switch
The OpenFlow protocol and interface allow OpenFlow switches to be accessed as essential forwarding elements. A flow-based SDN architecture like OpenFlow simplifies switching hardware. Still, it may require additional forwarding tables, buffer space, and statistical counters that are difficult to implement in traditional switches with integrated circuits tailored to specific applications.
There are two types of switches in an OpenFlow network: hybrids (which enable OpenFlow) and pures (which only support OpenFlow). OpenFlow is supported by hybrid switches and traditional protocols (L2/L3). OpenFlow switches rely entirely on a controller for forwarding decisions and do not have legacy features or onboard control.
Hybrid switches are the majority of the switches currently available on the market. This link must remain active and secure because OpenFlow switches are controlled over an open interface (through a TCP-based TLS session). OpenFlow is a messaging protocol that defines communication between OpenFlow switches and controllers, which can be viewed as an implementation of SDN-based controller-switch interactions.
Identify the Benefits of OpenFlow
Application-driven routing. Users can control the network paths.
The networks paths.A way to enhance link utilization.
An open solution for VM mobility. No VLAN reliability.
A means to traffic engineer without MPLS.
A solution to build very large Layer 2 networks.
A way to scale Firewalls and Load Balancers.
A way to configure an entire network as a whole as opposed to individual entities.
A way to build your own encryption solution. Off-the-box encryption.
A way to distribute policies from a central controller.
Customized flow forwarding. Based on a variety of bit patterns.
A solution to get a global view of the network and its state. End-to-end visibility.
A solution to use commodity switches in the network. Massive cost savings.
The following table list the Software Defined Networking ( SDN ) benefits and the problems encountered with existing control plane architecture:
Identify the benefits of OpenFlow and SDN
Problems with the existing approach
Faster software deployment.
Large scale provisioning and orchestration.
Programmable network elements.
Limited traffic engineering ( MPLS TE is cumbersome )
Faster provisioning.
Synchronized distribution policies.
Centralized intelligence with centralized controllers.
Routing of large elephant flows.
Decisions are based on end-to-end visibility.
Qos and load based forwarding models.
Granular control of flows.
Ability to scale with VLANs.
Decreases the dependence on network appliances like load balancers.
A key point: The lack of a session layer in the TCP/IP stack.
Regardless of the hype and benefits of SDN, neither OpenFlow nor other SDN technologies address the real problems of the lack of a session layer in the TCP/IP protocol stack. The problem is that the client’s application ( Layer 7 ) connects to the server’s IP address ( Layer 3 ), and if you want to have persistent sessions, the server’s IP address must remain reachable.
This session’s persistence and the ability to connect to multiple Layer 3 addresses to reach the same device is the job of the OSI session layer. The session layer provides the services for opening, closing, and managing a session between end-user applications. In addition, it allows information from different sources to be correctly combined and synchronized.
The problem is the TCP/IP reference module does not consider a session layer, and there is none in the TCP/IP protocol stack. SDN does not solve this; it gives you different tools to implement today’s kludges.
Control and data plane
When we identify the benefits of OpenFlow, let us first examine traditional networking operations. Traditional networking devices have a control and forwarding plane, depicted in the diagram below. The control plane is responsible for setting up the necessary protocols and controls so the data plane can forward packets, resulting in end-to-end connectivity. These roles are shared on a single device, and the fast packet forwarding ( data path ) and the high-level routing decisions ( control path ) occur on the same device.
What is OpenFlow | SDN separates the data and control plane
Control plane
The control plane is part of the router architecture responsible for drawing the network map in routing. When we mention control planes, you usually think about routing protocols, such as OSPF or BGP. But in reality, the control plane protocols perform numerous other functions, including:
Connectivity management ( BFD, CFM )
Interface state management ( PPP, LACP )
Service provisioning ( RSVP for InServ or MPLS TE)
Topology and reachability information exchange ( IP routing protocols, IS-IS in TRILL/SPB )
Adjacent device discovery via HELLO mechanism
ICMP
Control plane protocols run over data plane interfaces to ensure “shared fate” – if the packet forwarding fails, the control plane protocol fails as well.
Most control plane protocols ( BGP, OSPF, BFD ) are not data-driven. A BGP or BFD packet is never sent as a direct response to a data packet. There is a question mark over the validity of ICMP as a control plane protocol. The debate is whether it should be classed in the control or data plane category.
Some ICMP packets are sent as replies to other ICMP packets, and others are triggered by data plane packets, i.e., data-driven. My view is that ICMP is a control plane protocol that is triggered by data plane activity. After all, the “C” is ICMP does stand for “Control.”
Data plane
The data path is part of the routing architecture that decides what to do when a packet is received on its inbound interface. It is primarily focused on forwarding packets but also includes the following functions:
ACL logging
Netflow accounting
NAT session creation
NAT table maintenance
The data forwarding is usually performed in dedicated hardware, while the additional functions ( ACL logging, Netflow accounting ) usually happen on the device CPU, commonly known as “punting.” The data plane for an OpenFlow-enabled network can take a few forms.
However, the most common, even in the commercial offering, is the Open vSwitch. This is often referred to as the OVS. The Open vSwitch is an open-source implementation of a distributed virtual multilayer switch. It enabled a switching stack for virtualization environments while supporting multiple protocols and standards.
A key point: Identify the benefits of OpenFlow
Software-defined networking changes the control and data plane architecture.
The concept of SDN separates these two planes, i.e., the control and forwarding planes are decoupled. This allows the networking devices in the forwarding path to focus solely on packet forwarding. An out-of-band network uses a separate controller ( orchestration system ) to set up the policies and controls. Hence, the forwarding plane has the correct information to forward packets efficiently.
In addition, it allows the network control plane to be moved to a centralized controller on a server instead of residing on the same box carrying out the forwarding. The movement of the intelligence ( control plane ) of the data plane network devices to a controller enables companies to use low-cost, commodity hardware in the forwarding path. A significant benefit is that SDN separates the data and control plane enabling new use cases.
A key point: Identify the benefits of OpenFlow
A centralized computation and management plane makes more sense than a centralized control plane.
The controller maintains a view of the entire network and communicates with Openflow ( or, in some cases, BGP with BGP SDN ) with the different types of OpenFlow-enabled network boxes. The data path portion remains on the switch, such as the OVS bridge, while the high-level decisions are moved to a separate controller. The data path presents a clean flow table abstraction, and each flow table entry contains a set of packet fields to match, resulting in specific actions ( drop, redirect, send-out-port ).
When an OpenFlow switch receives a packet, it has never seen before and doesn’t have a matching flow entry; it sends the packet to the controller for processing. The controller then decides what to do with the packet.
Applications could then be developed on top of this controller, performing security scrubbing, load balancing, traffic engineering, or customized packet forwarding. The centralized view of the network simplifies problems that were harder to overcome with traditional control plane protocols.
A single controller could potentially manage all OpenFlow-enabled switches. Instead of individually configuring each switch, the controller can push down policies to multiple switches simultaneously—a compelling example of many-to-one virtualization.
Now that SDN separates the data and control plane, the operator uses the centralized controller to choose the correct forwarding information per-flow basis. This allows better load balancing and traffic separation on the data plane. In addition, there is no need to enforce traffic separation based on VLANs, as the controller would have a set of policies and rules that would only allow traffic from one “VLAN” to be forwarded to other devices within that same “VLAN.”
The advent of VXLAN
With the advent of VXLAN, which allows up to 16 million logical entities, the benefits of SDN should not be purely associated with overcoming VLAN scaling issues. VXLAN already does an excellent job with this. It does make sense to deploy a centralized control plane in smaller independent islands; in my view, it should be at the edge of the network for security and policy enforcement roles. Using Openflow on one or more remote devices is easy to implement and scale.
It also decreases the impact of controller failure. If a controller fails and its sole job is implementing packet filters when a new user connects to the network, the only affecting element is that the new user cannot connect. If the controller is responsible for core changes, you may have interesting results with a failure. New users not being able to connect is bad, but losing your entire fabric is not as bad.
What Is OpenFlow? Identify the Benefits of OpenFlow
A traditional networking device runs all the control and data plane functions. The control plane, usually implemented in the central CPU or the supervisor module, downloads the forwarding instructions into the data plane structures. Every vendor needs communications protocols to bind the two planes together to download forward instructions.
Therefore, all distributed architects need a protocol between control and data plane elements. The protocol to bind this communication path for traditional vendor devices is not open-source, and every vendor uses its proprietary protocol (Cisco uses IPC – InterProcess Communication ).
Openflow tries to define a standard protocol between the control plane and the associated data plane. When you think of Openflow, you should relate it to the communication protocol between the traditional supervisors and the line cards. OpenFlow is just a low-level tool.
OpenFlow is a control plane ( controller ) to data plane ( OpenFlow enabled device ) protocol that allows the control plane to modify forwarding entries in the data plane. OpenFlow enables the capability so that SDN separates the data and control plane.
Proactive versus reactive flow setup
OpenFlow operations have two types of flow setups, Proactive and Reactive.
With Proactive, the controller can populate the flow tables ahead of time, similar to a typical routing. However, the packet-in event never occurs by pre-defining your flows and actions ahead of time in the switch’s flow tables. The result is all packets are forwarded at line rate. With Reactive, the network devices react to traffic, consults the OpenFlow controller, and create a rule in the flow table based on the instruction. The problem with this approach is that there can be many CPU hits.
The following table outlines the critical points for each type of flow setup:
Proactive flow setup
Reactive flow setup
Works well when the controller is emulating BGP or OSPF.
Used when no one can predict when and where a new MAC address will appear.
The controller must first discover the entire topology.
Punts unknown packets to the controller. Many CPU hits.
Discover endpoints ( MAC addresses, IP addresses, and IP subnets )
Compute forwarding paths on demand. Not off the box computation.
Compute off the box optimal forwarding.
Install flow entries based on actual traffic.
Download flow entries to the data plane switches.
Has many scalability concerns such as packet punting rate.
No data plane controller involvement with the exceptions of ARP and MAC learning. Line-rate performance.
Not a recommended setup.
Hop-by-hop versus path-based forwarding
The following table illustrates the keys point for the two types of forwarding methods used by OpenFlow; hop-by-hop forwarding and path-based forwarding:
Hop-by-hop Forwarding
Path-based Forwarding
Similar to traditional IP Forwarding.
Similar to MPLS.
Installs identical flows on each switch on the data path.
Map flows to paths on ingress switches and assigns user traffic to paths at the edge node
Scalability concerns relating to flow updates after a change in topology.
Compute paths across the network and installs end-to-end path-forwarding entries.
Significant overhead in large-scale networks.
Works better than hop-by-hop forwarding in large-scale networks.
FIB update challenges. Convergence time.
Core switches don't have to support the same granular functionality as edge switches.
Identify the benefits of OpenFlow with security.
Obviously, with any controller, the controller is a lucrative target for attack. Anyone who knows you are using a controller-based network will try to attack the controller and its control plane. The attacker may attempt to intercept the controller-to-switch communication and replace it with its commands, essentially attacking the control plane with whatever means they like.
An attacker may also try to insert a malformed packet or some other type of unknown packet into the controller ( fuzzing attack ), exploiting bugs in the controller and causing the controller to crash.
Fuzzing attacks can be carried out with application scanning software such as Burp Suite. It attempts to manipulate data in a particular way, breaking the application.
The best way to tighten security would be to encrypt switch-to-controller communications with SSL and self-signed certificates to authenticate the switch and controller. It would be best to minimize interaction with the data plane, except for ARP and MAC learning.
To prevent denial of services attacks on the controller, you can use Control Plane Policing ( CoPP ) on Ingress so you don’t overload the switch and the controller. Currently, NEC is the only vendor implementing CoPP.
The Hybrid deployment model is helpful from a security perspective. For example, you can group specific ports or VLANs to OpenFlow and other ports or VLANs to traditional forwarding, then use traditional forwarding to communicate with the OpenFlow controller.
Identify the Benefits of OpenFlow
Software-defined networking or traditional routing protocols?
The move to a Software Defined Networking architecture has its clear advantages. It’s agile and can react quickly to business needs, such as new product development. And for businesses to achieve success, they must have software that continues to move with the times.
Otherwise, your customers and staff may lose interest in your product and service. The following table displays the advantages and disadvantages of the existing routing protocol control architecture.
+Reliable and well known.
-Non-standard Forwarding models. Destination-only and not load-aware metrics**
+Proven with 20 plus years field experience.
-Loosely coupled.
+Deterministic and predictable.
-Lacks end-to-end transactional consistency and visibility.
+Self-Healing. Traffic can reroute around a failed node or link.
-Limited Topology discovery and extraction. Basic neighbor and topology tables.
+Autonomous.
-Lacks the ability to change existing control plane protocol behavior.
+Scalable.
-Lacks the ability to introduce new control plane protocols.
+Plenty of learning and reading materials.
** Basic EIGRP IETF originally proposed an Energy-Aware Control Plane, but the IETF later removed this.
Software-Defined Networking: Use Cases
Edge Security policy enforcement at the network edge.
Authenticate users or VMs and deploy per-user ACL before connecting a user to the network.
Custom routing and online TE.
The ability to route on a variety of business metrics aka routing for dollars. Allowing you to override the default routing behavior.
Custom traffic processing.
For analytics and encryption.
Programmable SPAN ports
Use Openflow entries to mirror selected traffic to the SPAN port.
DoS traffic blackholing & distributed DoS prevention.
Block DoS traffic as close to the source as possible with more selective traffic targeting than the original RTBH approach**. The traffic blocking is implemented in OpenFlow switches. Higher performance with significantly lower costs.
Traffic redirection and service insertion.
Redirect a subset of traffic to network appliances and install redirection flow entries wherever needed.
Network Monitoring.
The controller is the authoritative source of information on network topology and Forwarding paths.
Scale-Out Load Balancing.
Punt new flows to the Openflow controller and install per-session entries throughout the network.
IPS Scale-Out.
OpenFlow is used to distribute the load to multiple IDS appliances.
**Remote-Triggered Black Hole: RTBH refers to installing a host route to a bogus IP address ( RTBH address ) pointing to NULL interfaces on all routers. BGP is used for advertising the host routes to other BGP peers of the attacked hosts, with the next-hop pointing to the RTBH address and mostly automated in ISP environments.
SDN deployment models
Guidelines:
Start with small deployments away from the mission-critical productions path, i.e., the Core. Ideally, start with device or service provisioning systems.
Start at the Edge and slowly integrate with the Core. Minimize the risk and blast radius. Start with packet filters at the Edge and tasks that can be easily automated ( VLANs ).
Integrate new technology with the existing network.
Gradually increase scale and gain trust. Experience is key.
Have the controller in a protected out-of-band network with SSL connectivity to the switches.
There are 4 different models for OpenFlow deployment, and the following sections list the key points of each model.
Native OpenFlow
They are commonly used for Greenfield deployments.
The controller performs all the intelligent functions.
The forwarding plane switches have little intelligence and solely perform packet forwarding.
The white box switches need IP connectivity to the controller for the OpenFlow control sessions. This should be done with an out-of-band network if you are forced to use an in-band network for this communication path using an isolated VLAN with STP.
Fast convergence techniques such as BFD may be challenging to use with a central controller.
Many people view that this approach does not work for a regular company. Companies implementing native OpenFlow, such as Google, have the time and resources to reinvent all the wheels when implementing a new control-plane protocol ( OpenFlow ).
Native OpenFlow with Extensions
Some control plane functions are handled from the centralized controller to the forwarding plane switches. For example, the OpenFlow-enabled switches could load balancing across multiple links without the controller’s previous decision. You could also run STP, LACP, or ARP locally on the switch without interaction with the controller. This approach is helpful if you lose connectivity to the controller. If the low-level switches perform certain controller functions, packet forwarding will continue in the event of failure.
The local switches should support the specific OpenFlow extensions that let them perform functions on the controller’s behalf.
Hybrid ( Ships in the night )
This approach is used where OpenFlow runs in parallel with the production network.
The same network box is controlled by existing on-box and off-box control planes ( OpenFlow).
Suitable for pilot deployment models as switches still run traditional control plane protocols.
The Openflow controller manages only specific VLANs or ports on the network.
The big challenge is determining and investigating the conflict-free sharing of forwarding plane resources across multiple control planes.
Integrated OpenFlow
OpenFlow classifiers and forwarding entries are integrated with the existing control plane. For example, Juniper’s OpenFlow model follows this mode of operation where OpenFlow static routes can be redistributed into the other routing protocols.
No need for a new control plane.
No need to replace all forwarding hardware
Most practical approach as long as the vendor supports it.
Closing Points on OpenFlow
Advantages of OpenFlow:
OpenFlow brings several critical advantages to network management and control:
1. Flexibility and Programmability: With OpenFlow, network administrators can dynamically reconfigure the behavior of network devices, allowing for greater adaptability to changing network requirements.
2. Centralized Control: By centralizing control in a single controller, network administrators gain a holistic view of the network, simplifying management and troubleshooting processes.
3. Innovation and Experimentation: OpenFlow enables researchers and developers to experiment with new network protocols and applications, fostering innovation in the networking industry.
4. Scalability: OpenFlow’s centralized control architecture provides the scalability needed to manage large-scale networks efficiently.
Implications for Network Control:
OpenFlow has significant implications for network control, paving the way for new possibilities in network management:
1. Software-Defined Networking (SDN): OpenFlow is a critical component of the broader concept of SDN, which aims to decouple network control from the underlying hardware, providing a more flexible and programmable infrastructure.
2. Network Virtualization: OpenFlow facilitates network virtualization, allowing multiple virtual networks to coexist on a single physical infrastructure.
3. Traffic Engineering: By controlling the flow of packets at a granular level, OpenFlow enables advanced traffic engineering techniques, optimizing network performance and resource utilization.
Conclusion:
OpenFlow represents a paradigm shift in network control, offering a more flexible, scalable, and programmable approach to managing networks. By separating the control and data planes, OpenFlow empowers network administrators to have fine-grained control over network behavior, improving efficiency, innovation, and adaptability. As the networking industry continues to evolve, OpenFlow and its related technologies will undoubtedly play a crucial role in shaping the future of network management.
In the rapidly evolving networking world, virtualization has become critical for businesses seeking to optimize their IT infrastructure. One key technology that has emerged is VXLAN (Virtual Extensible LAN), which enables the creation of virtual networks independent of physical network infrastructure. In this blog post, we will delve into the concept of VXLAN, its benefits, and its role in network virtualization.
VXLAN is an encapsulation protocol designed to extend Layer 2 (Ethernet) networks over Layer 3 (IP) networks. It provides a scalable and flexible solution for creating virtualized networks, enabling seamless communication between virtual machines (VMs) and physical servers across different data centers or geographic regions.
VXLAN is a technology that creates virtual networks within an existing physical network. A Layer 2 overlay network runs on top of the current Layer 2 network. VXLAN utilizes UDP as the transport protocol, providing a secure, efficient, and reliable way to create a virtual network.
Table of Contents
Highlights: What is VXLAN
Segmentation: Security and policy control
VXLAN provides several advantages over traditional Layer 2 network technologies. It enables the creation of enormous virtual networks with thousands of endpoints, allowing multi-tenant segmentation for security and policy enforcement. It also takes advantage of existing Layer 3 routing protocols, allowing for efficient routing between virtual networks, and it is hardware agnostic, meaning it can be used with any hardware.
VLXAN offerings
VXLAN has been widely adopted and is now used in many large enterprise networks for virtualization and cloud computing. It provides:
A secure and efficient way to create virtual networks.
Allowing for the creation of multi-tenant segmentation.
Efficient routing.
Hardware-agnostic capabilities.
With its widespread adoption, VXLAN has become an essential technology for network virtualization.
Related: Before you proceed, you may find the following posts helpful for pre-information:
Introduction to What is VXLAN and what is involved.
Highlighting the details of VXLAN vs VLAN.
Technical details on the VXLAN Spanning Tree.
Scenario: Why introduce VXLAN? VLXAN benefits.
A final note on the VXLAN enhancements.
Back to Basics: The Need For VXLAN
Traditional layer two networks have issues because of the following reasons:
Spanning tree: Restricts links.
Limited amount of VLANs: Restricts scalability;
Large MAC address tables: Restricts scalability and mobility
Spanning-tree avoids loops by blocking redundant links. By blocking connections, we create a loop-free topology and pay for links we can’t use. Although we could switch to a layer three network, some technologies require layer two networking.
VLAN IDs are 12 bits long, so we can create 4094 VLANs (0 and 4095 are reserved). Data centers may need help with only 4094 available VLANs. Let’s say we have a service provider with 500 customers. There are 4094 available VLANs, so each customer can only have eight.
The Role of Server Virtualization
Server virtualization has exponentially increased the number of addresses in our switches’ MAC addresses. There was only one MAC address per switch port before server virtualization. With server virtualization, we can run many virtual machines (VMs) or containers on a single physical server. Virtual NICs and virtual MAC addresses are assigned to each virtual machine. One switch port must learn many MAC addresses.
There could be 24 or 48 physical servers connected to a Top of Rack (ToR) switch in a data center. There may be many racks in a data center, so each switch must store the MAC addresses of all VMs that communicate. Networks without server virtualization require much larger MAC address tables.
Lab Guide: VXLAN
In the following lab, I created a Layer 2 overlay with VXLAN over a Layer 3 core. A bridge domain VNI of 6001 must match both sides of the overlay tunnel. What Is a VNI? The VLAN ID field in an Ethernet frame has only 12 bits, so VLAN cannot meet isolation requirements on data center networks. The emergence of VNI is specifically to solve this problem.
Note: The VNI
A VNI is a user identifier similar to a VLAN ID. A VNI identifies a tenant. VMs with different VNIs cannot communicate at Layer 2. During VXLAN packet encapsulation, a 24-bit VNI is added to a VXLAN packet, enabling VXLAN to isolate many tenants.
You will notice in the screenshot below that I can ping from desktop 0 to desktop one even though the IP addresses are not in the routing table of the core devices, simulating a Layer 2 overlay. Consider VXLAN to be the overlay and the routing Layer 3 core to be the underlay.
In the following screenshot, notice that the VNI has been changed. The VNI needs to be changed in two places in the configuration, as illustrated below. Once changed, the Peers are down; however, the NVE interface remains up. The VXLAN layer two overlay is not operational.
How does VXLAN work?
VXLAN uses tunneling to encapsulate Layer 2 Ethernet frames within IP packets. A unique 24-bit segment ID identifies each VXLAN network, the VXLAN Network Identifier (VNI). The source VM encapsulates the original Ethernet frame with a VXLAN header, including the VNI. The encapsulated packet is then sent over the physical IP network to the destination VM and decapsulated to retrieve the original Ethernet frame.
Analysis:
Notice below that it is running a ping from desktop 0 to desktop 1. The IP addresses assigned to this host are in the 10.0.0.1 and 10.0.0.2. First, notice that the ping is booming, and when I do a packet capture on the links Gi1 connected to Leaf A, we see the encapsulation of the ICMP echo request and reply.
Everything is encapsulated into UDP port 1024. In my configurations of Leaf A and Leaf B, I explicitly set the VXLAN port to 1024.
Benefits of VXLAN:
– Scalability: VXLAN allows creating up to 16 million logical networks, providing the scalability required for large-scale virtualized environments.
– Network Segmentation: By leveraging VXLAN, organizations can segment their networks into virtual segments, enhancing security and isolating traffic between applications or user groups.
– Flexibility and Mobility: VXLAN enables the movement of VMs across physical servers and data centers without the need to reconfigure network settings. This flexibility is crucial for workload mobility in dynamic environments.
– Interoperability: VXLAN is an industry-standard protocol supported by various networking vendors, ensuring compatibility across different network devices and platforms.
Data Center
VXLAN
VXLAN Benefits
Scalability
Network Segmentation
Flexibility and Mobility
Interopability
Data Center
VXLAN
VLAN Use Cases
Data Center Interconnect (DCI)
Multi Tenant Environments
Network Virtualization
Hybrid Cloud Connectivity
Use Cases for VXLAN:
– Data Center Interconnect (DCI): VXLAN allows organizations to interconnect multiple data centers, enabling seamless workload migration, disaster recovery, and workload balancing across different locations.
– Multi-Tenant Environments: VXLAN enables service providers to offer virtualized network services to multiple tenants securely and isolatedly. This is particularly useful in cloud computing environments.
– Network Virtualization: VXLAN plays a crucial role in network virtualization, allowing organizations to create virtual networks independent of the underlying physical infrastructure. This enables greater flexibility and agility in managing network resources.
VXLAN is a form of network virtualization. Network virtualization cuts a single physical network into many virtual networks, often called network overlays. Virtualizing a resource allows it to be shared by multiple users. Virtualization provides the illusion that each user is on his or her resources. In the case of virtual networks, each user is under the misconception that there are no other users of the network. To preserve the illusion, virtual networks are separated from one another. Packets cannot leak from one virtual network to another.
VXLAN Loop Detection and Prevention
So, before we dive into the benefits of VXLAN, let us address the basics of loop detection and prevention, which is a significant driver for using network overlays such as VLXAN. The challenge is that data frames can exist indefinitely when loops occur, disrupting network stability and degrading performance.
In addition, loops introduce broadcast radiation, increasing CPU and network bandwidth utilization, which results in a degradation of user application access experience. Finally, in multi-site networks, a loop can span multiple data centers, causing disruptions that are difficult to pinpoint. A lot of this can be solved with overlay networking.
Video: Overlay Networking and VXLAN
In the following video, we will discuss the basics of overlay networking.Overlay/Underlay Essentially, an overlay is placing Layer 2 or Layer 3 over a Layer 3 Core. The Layer 3 Core is known as the underlay. This removes many drawbacks and scaling issues with traditional Layer 2 connectivity, which uses VLANs.
The multi-tenant nature of overlays is designed to avoid these L2 challenges, allowing you to build networks at a much larger scale. We have Layer 2 and Layer 3 overlays. Layer 2 overlays emulate a Layer 2 network and map Layer 2 frames into an IP underlay.
If you are emulating a Layer 2 network, you must emulate the Layer 2 flooding behavior. This is the bread and butter of how Layer 2 networks work, and that doesn’t change just because you decide to create a Layer 2 overlay.
However, first-generation Layer-2 Ethernet networks could not natively detect or mitigate looped topologies, while modern Layer-2 overlays implicitly build loop-free topologies. Therefore, overlays do not need loop detection and mitigation as long as no first-gen Layer-2 network is attached. Essentially, there is no need for a VXLAN spanning tree.
So, one of the differences between VXLAN vs VLAN is that the VLAN has a 12-bit VID while VXLAN has a 24-bit VID network identifier, allowing you to create up to 16 million segments. VXLAN has tremendous scale and stable loop-free networking and is a foundation technology in the ACI Cisco.
VXLAN and Data Center Interconnect
VXLAN has revolutionized data center interconnect by providing a scalable, flexible, and efficient solution for extending Layer 2 networks. Its ability to enable network segmentation, multi-tenancy support, and seamless mobility makes it a valuable technology for modern businesses.
However, careful planning, consideration of network infrastructure, and security measures are essential for successful implementation. By harnessing the power of VXLAN, organizations can achieve a more agile, scalable, and interconnected data center environment.
Considerations for Implementing VXLAN:
1. Underlying Network Infrastructure: Before implementing VXLAN, it is essential to assess the underlying network infrastructure. Network devices must support VXLAN encapsulation and decapsulation and have sufficient bandwidth to handle the increased traffic.
2. Network Overhead: While VXLAN provides numerous benefits, it does introduce additional network overhead due to encapsulation and decapsulation processes. It is crucial to consider the impact on network performance and plan accordingly.
3. Security: As VXLAN extends Layer 2 networks over Layer 3 infrastructure, it is essential to implement appropriate security measures. This includes encrypting VXLAN traffic, deploying access control policies, and monitoring network traffic for anomalies.
VXLAN vs VLAN: The VXLAN Benefits Drive Adoption
Introduced by Cisco and VMware and now heavily used in open networking, VXLAN stands for Virtual eXtensible Local Area Network and is perhaps the most popular overlay technology for IP-based SDN data centers. And is used extensively with ACI networks.
VXLAN was explicitly designed for Layer 2 over Layer 3 tunneling, and its early competitions from NVGRE and STT are fading away, and VXLAN is becoming the industry standard. VLXAN brings many advantages, especially in loop prevention, as there is no need for a VXLAN spanning tree.
Today, with overlays such as with VXLAN, the dependency on loop prevention protocols is almost eliminated. However, even though virtualized overlay networks such as VXLAN are loop-free, having a failsafe loop detection and mitigation method is still desirable because loops can be introduced by topologies connected to the overlay network.
Loop prevention traditionally started with Spanning Tree Protocols (STP) to counteract the loop problem in first-gen Layer-2 Ethernet networks. Over time, other approaches evolved by moving networks from “looped topologies” to “loop-free topologies.
While LAG and MLAG were used, other approaches for building loop-free topologies arose using ECMP at the MAC or IP layers. For example, FabricPath or TRILL is a MAC layer ECMP approach that emerged in the last decade. More recently, network virtualization overlays that build loop-free topologies on top of IP layer ECMP became state-of-the-art.
VXLAN vs VLAN: Why Introduce VXLAN?
STP issues and scalability constraints: STP is undesirable on a large scale and lacks a proper load-balancing mechanism. A solution was needed to leverage the ECMP capabilities of an IP network while offering extended VLANs across an IP core, i.e., virtual segments across the network core. There is no VXLAN spanning tree.
Multi-tenancy: Layer 2 networks are capped at 4000 VLANs, restricting multi-tenancy design—a big difference in the VXLAN vs VLAN debates.
ToR table scalability: Every ToR switch may need to support several virtual servers, and each virtual server requires several NICs and MAC addresses. This pushes the limits on the table sizes for the ToR switch. In addition, after the ToR tables become full, Layer 2 traffic will be treated as unknown unicast traffic, which will be flooded across the network, causing instability to a previously stable core.
VXLAN use cases
Use Case
VXLAN Details
Use Case 1
Multi-tenant IaaS Clouds where you need a large number of segments
Use Case 2
Link Virtual to Physical Servers. This is done via software or hardware VXLAN to VLAN gateway
Use Case 3
HA Clusters across failure domains/availability zones
Use Case 4
VXLAN works well over fabrics that have equidistant endpoints
Use Case 5
VXLAN-encapsulated VLAN traffic across availability zones must be rate-limited to prevent broadcast storm propagation across multiple availability zones
What is VXLAN? The operations
When discussing VXLAN vs VLAN, VXLAN employs a MAC over IP/UDP overlay scheme and extends the traditional VLAN boundary of 4000 VLANs. The 12-bit VLAN identifier in traditional VLANs capped scalability within the SDN data center and proved cumbersome if you wanted a VLAN per application segment model. VXLAN scales the 12-bit to a 24-bit identifier and allows for 16 million logical endpoints, with each endpoint potentially offering another 4,000 VLANs.
While tunneling does provide Layer 2 adjacency between these logical endpoints with the ability to move VMs across boundaries, the main driver for its insertion was to overcome the challenge of having only 4000 VLAN.
Typically, an application segment would have multiple segments; between each segment, you will have firewalling and load-balancing services, and each segment requires a different VLAN. The Layer 2 VLAN segment transfers non-routable heartbeats or state information that can’t cross an L3 boundary. You will soon reach the 4000k VLAN limit if you are a cloud provider.
The control plane
The control plane is very similar to the spanning tree control plane. If a switch receives a packet destined for an unknown address, the switch will forward the packet to an IP address that floods the packet to all the other switches.
This IP address is, in turn, mapped to a multicast group across the network. VXLAN doesn’t explicitly have a control plane and requires an IP multicast running in the core for forwarding traffic and host discovery.
Video: VXLAN operations
VXLAN is all about discovering the destination VTEP; the big decision is how you discover the destination VTEP IP address. The destination VTEP IP address needs to be mapped to the end host destination MAC address. The mechanism used to do this affects the scalability & VXLAN domain functionality. We need some control plane elements.
The control plane element of VXLAN can be deployed as a flood and learn mechanism, which is not an absolute control plane, or you can have an actual control plane (that does not flood and learn) or even use an orchestration tool for VTEP to IP mapping. Many vendors implement this differently.
Best practices for enabling IP Multicast in the core
IP Multicast
In the Core
Bidirectional PIM or PIM Sparse Mode
Redundant Rendezvous Points (RP)
Shared trees (reduce the amount of IP multicast state)
Always check the IP multicast table sizes on core and ToR switches
Single IP multicast address for multiple VXLAN segments is OK
The requirement for IP multicast in the core made VXLAN undesirable from an operation point of view. For example, creating the tunnel endpoints is simple, but introducing a protocol like IP multicast to a core just for the tunnel control plane was considered undesirable. As a result, some of the more recent versions of VXLAN support IP unicast.
VXLAN uses a MAC over IP/UDP solution to eliminate the need for a spanning tree. There is no VXLAN spanning tree. This enables the core to be IP and not run a spanning tree. Many people ask why VXLAN uses UDP. The reason is that the UDP port numbers cause VXLAN to inherit Layer 3 ECMP features. The entropy that enables load balancing across multiple paths is embedded into the UDP source port of the overlay header.
Lab Guide: Multicast VLXAN
In this lab guide, we are going to have a look at a VXLAN multicast mode. The multicat mode requires both unicast and multicast connectivity between sites. Similar to the previous one, this configuration guide uses OSPF to provide unicast connectivity, and now we have an additional bidirectional Protocol Independent Multicast (PIM) to provide multicast connectivity.
This does not mean that you don’t have a multicast-enabled core. It would be best if you still had multicast enabled on the core.
So we are not, let’s say, tunneling multicast over an IPv4 core without having multicast enabled on the core. I have multicast on all Layer 3 interfaces, and the mroute table is populated on all Layer 3 routers. With the command: Show ip mroute we are tunneling the multicast traffic, and with the command: Show nve vni we have multicast group 239.0.0.10, and we have a state of UP.
VXLAN benefits and stability
The underlying control plan network impacts the stability of VXLAN and the applications running within it. For example, if the underlying IP network cannot converge quickly enough, VLXAN packets may be dropped, and an application cache timeout may be triggered.
The rate of change in the underlying network has a significant impact on the stability of the tunnels, yet the rate and change of the tunnels do not affect the underlying control plane. This is similar to how the strength of an MPLS / VPN overlay is affected by the core’s IGP.
VXLAN Points
VXLAN benefits
VXLAN drawbacks
Point 1
Runs over IP Transport
No control plane
Point 2
Offers a large number of logical endpoints
Needs IP Multicast***
Point 3
Reduced flooding scope
No IGMP snooping ( yet )
Point 4
Eliminates STP
No Pvlan support
Point 5
Easily integrated over existing Core
Requires Jumbo frames in the core ( 50 bytes)
Point 6
Minimal host-to-network integration
No built-in security features **
Point 7
Not a DCI solution ( no arp reduction, first-hop gateway localization, no inbound traffic steering i.e, LISP )
** VXLAN has no built-in security features. Anyone who gains access to the core network can insert traffic into segments. The VXLAN transport network must be secure, as no existing Firewall or Intrusion Prevention System (IPS) equipment has visibility into the VXLAN traffic.
*** Recent versions have Unicast VXLAN. Nexus 1000V release 4.2(1)SV2(2.1)
Updated: VXLAN enhancements
MAC distribution mode is an enhancement to VXLAN that prevents unknown unicast flooding. It eliminates the process of data plane MAC address learning. Traditionally, this was done by flooding to locate an unknown end host, but it has now been replaced with a control plane solution.
During VM startup, the VSM ( control plane ) collects the list of MAC addresses and distributes the MAC-to-VTEP mappings to all VEMs participating in a VXLAN segment. This technique makes VXLAN more optimal by unicasting more intelligently, similar to Nicira and VMware NVP.
ARP termination works by giving the VSM controller all the ARP and MAC information. This enables the VSM to proxy and respond locally to ARP requests without sending a broadcast. Because 90% of broadcast traffic is ARP requests ( ARP reply is unicast ), this significantly reduces broadcast traffic on the network.
Video: The VXLAN Phases
In the following video, we will discuss the VXLAN phases. VXLAN went through several steps to get the remote VTEP IP information. It started with a flood-and-learn process and finally used a proper control plane – EVPN.
In recent years, the rapid growth of cloud computing and the increasing demand for scalable and flexible networks have led to the development of various technologies to address these needs. One such technology is VXLAN (Virtual Extensible LAN), an overlay network protocol that has gained significant popularity in networking. In this blog post, we will delve into the intricacies of VXLAN, exploring its key features, benefits, and use cases.
What is VXLAN?
VXLAN is a network overlay technology that enables the creation of virtualized Layer 2 networks over existing Layer 3 infrastructure. It was developed to address the limitations of traditional VLANs, which could not scale beyond a few thousand networks due to the limited number of VLAN IDs available. VXLAN solves this problem using a 24-bit VXLAN Network Identifier (VNI), allowing for an impressive 16 million unique network segments.
Key Features of VXLAN:
1. Scalability: As mentioned earlier, VXLAN’s use of a 24-bit VNI allows for a significantly larger number of network segments than traditional VLANs. This scalability makes VXLAN an ideal solution for large-scale virtualized environments.
2. Network Segmentation: VXLAN enables the creation of logical network segments, allowing for network isolation and improved security. By encapsulating Layer 2 Ethernet frames within Layer 3 UDP packets, VXLAN provides a flexible and scalable approach to network segmentation.
3. Multicast Support: VXLAN leverages IP multicast to efficiently distribute broadcast, unknown unicast, and multicast (BUM) traffic across the network. This feature reduces network congestion and improves overall performance.
4. Mobility: VXLAN supports seamless movement of virtual machines (VMs) across physical hosts and data centers. By decoupling the VMs from the underlying physical network, VXLAN enables mobility without requiring any changes to the network infrastructure.
Benefits of VXLAN:
1. Enhanced Network Flexibility: VXLAN enables the creation of virtualized networks decoupled from the underlying physical infrastructure. This flexibility allows for easier network provisioning, scaling, and reconfiguration, making it an ideal choice for cloud environments.
2. Improved Scalability: With its larger network segment capacity, VXLAN offers improved scalability compared to traditional VLANs. This scalability is crucial in modern data centers and cloud environments where virtual machines and network segments are continuously growing.
3. Simplified Network Management: VXLAN simplifies network management tasks by abstracting the network infrastructure. Network administrators can define and manage virtual networks independently of the underlying physical infrastructure, streamlining network operations and reducing complexity.
Use Cases for VXLAN:
1. Data Center Interconnect: VXLAN is widely used for interconnecting geographically dispersed data centers. By extending Layer 2 network connectivity over Layer 3 infrastructure, VXLAN facilitates seamless VM mobility, disaster recovery, and workload balancing across data centers.
2. Multi-tenancy in Cloud Environments: VXLAN allows cloud service providers to create isolated network segments for different tenants, enhancing security and providing dedicated network resources. This feature is vital in multi-tenant cloud environments where data privacy and network isolation are critical.
3. Network Virtualization: VXLAN plays a crucial role in network virtualization, enabling the creation of virtual networks that are independent of the underlying physical infrastructure. This virtualization simplifies network management, enhances flexibility, and enables efficient resource utilization.
Conclusion: VXLAN has emerged as a powerful network virtualization technology with many use cases. VXLAN provides the flexibility, scalability, and efficiency required in modern networking environments, from data center virtualization to multi-tenancy, hybrid cloud connectivity, and disaster recovery. As organizations continue to embrace cloud computing and virtualization, VXLAN will undoubtedly play a pivotal role in shaping the future of networking.
In the world of technology, data centers play a crucial role in storing, managing, and processing vast amounts of digital information. However, behind the scenes, a complex infrastructure known as data center topology enables seamless data flow and optimal performance. In this blog post, we will delve into the intricacies of data center topology, its different types, and how it impacts the efficiency and reliability of data centers.
Data center topology refers to a data center's physical and logical layout. It encompasses the arrangement and interconnection of various components like servers, storage devices, networking equipment, and power sources. A well-designed topology ensures high availability, scalability, and fault tolerance while minimizing latency and downtime.
Table of Contents
Highlights: Data Center Topoloigy
Choosing a topology
Data centers are the backbone of many businesses, providing the necessary infrastructure to store and manage data and access applications and services. As such, it is essential to understand the different types of available data center topologies. When choosing a topology for a data center, it is necessary to consider the organization’s specific needs and requirements. Each topology offers its advantages and disadvantages, so it is crucial to understand the pros and cons of each before making a decision.
Data Center Topology
A data center topology refers to the physical layout and interconnection of network devices within a data center. It determines how servers, switches, routers, and other networking equipment are connected, ensuring efficient and reliable data transmission. Topologies are based on scalability, fault tolerance, performance, and cost.
Scalability of the topology
Additionally, it is essential to consider the topology’s scalability, as a data center may need to accommodate future growth. By understanding the different topologies and their respective strengths and weaknesses, organizations can make the best decision for their data centers.
Typical data center topologies
Typical data center topologies connect end hosts to the top rack ( ToR ) switches, typically using 1GigE or 10GigE links. These ToR/access switches contain several end-host ports, usually 48GigE, to physically connect the end stations.
Because this layer has many ports, its configuration aims for simplicity and ease of management. The ToR also has several 10GigE or 40GigE uplink ports to connect to an upstream device. Depending on the data center network topology, these ToR switches sometimes connect to one or more end-of-row ( EoR ) switches, resulting in different data center topology types.
The design of the data center topology is to provide rich connectivity among the ToR switches so that all application and end-user requirements are satisfied. The diagrams below display the ToR and EoR server connectivity models commonly seen in the SDN data center.
Related: For pre-information, you may find the following post helpful
A data center is a physical facility that houses critical applications and data for an organization. It consists of a network of computing and storage resources that support shared applications and data delivery. The components of a data center are routers, switches, firewalls, storage systems, servers, and application delivery controllers.
Enterprise IT data centers support the following business applications and activities:
Email and file sharing
Productivity applications
Customer relationship management (CRM)
Enterprise resource planning (ERP) and databases
Big data, artificial intelligence, and machine learning
Virtual desktops, communications, and collaboration services
A data center consists of the following core infrastructure components:
Network infrastructure: Connects physical and virtual servers, data center services, storage, and external connections to end users.
Storage Infrastructure: Modern data centers use storage infrastructure to power their operations. Storage systems hold this valuable commodity.
A data center’s computing infrastructure is its applications. The computing infrastructure comprises servers that provide processors, memory, local storage, and application network connectivity. In the last 65 years, computing infrastructure has undergone three major waves:
In the first wave of replacements of proprietary mainframes, x86-based servers were installed on-premises and managed by internal IT teams.
In the second wave, application infrastructure was widely virtualized. The result was improved resource utilization and workload mobility across physical infrastructure pools.
The third wave finds us in the present, where we see the move to the cloud, hybrid cloud, and cloud-native (that is, applications born in the cloud).
Common Types of Data Center Topologies:
a) Bus Topology: In this traditional topology, all devices are connected linearly to a common backbone, resembling a bus. While it is simple and cost-effective, a single point of failure can disrupt the entire network.
b) Star Topology: In a star topology, each device is connected directly to a central switch or hub. This design offers centralized control and easy troubleshooting, but it can be expensive due to the requirement of additional cabling.
c) Mesh Topology: A mesh topology provides redundant connections between devices, forming a network where every device is connected to every other device. This design ensures high fault tolerance and scalability but can be complex and costly.
d) Hybrid Topology: As the name suggests, a hybrid topology combines elements of different topologies to meet specific requirements. It offers flexibility and allows organizations to optimize their infrastructure based on their unique needs.
Considerations in Data Center Topology Design:
a) Redundancy: Redundancy is essential to ensure continuous operation even during component failures. By implementing redundant paths, power sources, and network links, data centers can minimize the risk of downtime and data loss.
b) Scalability: As the data center’s requirements grow, the topology should be able to accommodate additional devices and increased data traffic. Scalability can be achieved through modular designs, virtualization, and flexible network architectures.
c) Performance and Latency: The distance between devices, the quality of network connections, and the efficiency of routing protocols significantly impact data center performance and latency. Optimal topology design considers these factors to minimize delays and ensure smooth data transmission.
Impact of Data Center Topology:
Efficient data center topology directly influences the entire infrastructure’s reliability, availability, and performance. A well-designed topology reduces single points of failure, enables load balancing, enhances fault tolerance, and optimizes data flow. It directly impacts the user experience, especially for cloud-based services, where data centers simultaneously cater to many users.
Data Center Topology
Main Data Center Topology Components
Data Center Topology
You need to understanding the different topologies and their respective strengths and weaknesses.
Rich connectivity among the ToR switches so that all application and end-user requirements are satisfied
A well-designed topology reduces single points of failure.
Example: Bus, star, mesh, and hybrid topologies
The Role of Networks
A network lives to serve the connectivity requirements of applications and applications. We build networks by designing and implementing data centers. A common trend is that the data center topology is much bigger than a decade ago, with application requirements considerably different from the traditional client–server applications and with deployment speeds in seconds instead of days. This changes how networks and your chosen data center topology are designed and deployed.
The traditional network design was scaled to support more devices by deploying larger switches (and routers). This is thescale-in model of scaling. However, these large switches are expensive and primarily designed to support only a two-way redundancy.
Today, data center topologies are built to scale out. They must satisfy the three main characteristics of increasing server-to-server traffic, scale ( scale on-demand ), and resilience. The following diagram shows a ToR design we discussed at the start of the blog.
The Role of The ToR
Top of rack (ToR) is a term used to describe the architecture of a data center. It is a server architecture in which servers, switches, and other equipment are mounted on the same rack. This allows for the most efficient use of space since the equipment is all within arm’s reach.
ToR is also the most efficient way to manage power and cooling since the equipment is all in the same area. Since all the equipment is close together, ToR also allows faster access times. This architecture can also be utilized in other areas, such as telecommunications, security, and surveillance.
ToR is a great way to maximize efficiency in any data center and is becoming increasingly popular. In contrast to the ToR data center design, the following diagram shows an EoR switch design.
The Role of The EoR
The term end-of-row (EoR) design is derived from a dedicated networking rack or cabinet placed at either end of a row of servers to provide network connectivity to the servers within that row. In EoR network design, each server in the rack has a direct connection with the end-of-row aggregation switch, eliminating the need to connect servers directly with the in-rack switch.
Racks are usually arranged to form a row; a cabinet or rack is positioned at the end of this row. This rack has a row aggregation switch, which provides network connectivity to servers mounted in individual racks. This switch, a modular chassis-based platform, sometimes supports hundreds of server connections. However, a large amount of cabling is required to support this architecture.
A ToR configuration requires one switch per rack, resulting in higher power consumption and operational costs. Moreover, unused ports are often more significant in this scenario than with an EoR arrangement.
On the other hand, ToR’s cabling requirements are much lower than those of EoR, and faults are primarily isolated to a particular rack, thus improving the data center’s fault tolerance.
If fault tolerance is the ultimate goal, ToR is the better choice, but EoR configuration is better if an organization wants to save on operational costs. The following table lists the differences between a ToR and an EoR data center design.
Data Center Topology Types:
Fabric extenders – FEX
Cisco has introduced the concept of Fabric Extenders, which are not Ethernet switches but remote line cards of a virtualized modular chassis ( parent switch ). This allows scalable topologies previously impossible with traditional Ethernet switches in the access layer.
You should relate an FEX device like a remote line card attached to a parent switch. All the configuration is done on the parent switch, yet physically, the fabric extender could be in a different location. The mapping between the parent switch and the FEX ( fabric extender ) is done via a special VN-Link.
The following diagram shows an example of a FEX in a standard data center network topology. More specifically, we are looking at the Nexus 2000 FEX Series. Cisco Nexus 2000 Series Fabric Extenders (FEX) are based on the standard IEEE 802.1BR. They deliver fabric extensibility with a single point of management.
Different types of Fex solution
FEXs come with various connectivity solutions, including 100 Megabit Ethernet, 1 Gigabit Ethernet, 10 Gigabit Ethernet ( copper and fiber ), and 40 Gigabit Ethernet. They can be synchronized with the following models of parent switches – Nexus 5000, Nexus 6000, Nexus 7000, Nexus 9000, and Cisco UCS Fabric Interconnect.
In addition, because of the simplicity of FEX, they have very low latency ( as low as 500 nanoseconds ) compared to traditional Ethernet switches.
Some network switches can be connected to others and operate as a single unit. These configurations are called “stacks” and are helpful for quickly increasing the capacity of a network. A stack is a network solution composed of two or more stackable switches. Switches that are part of a stack behave as one single device.
Traditional switches like the 3750s still stand in the data center network topology access layer and can be used with stacking technology, combining two physical switches into one logical switch.
This stacking technology allows you to build a highly resilient switching system, one switch at a time. If you are looking at a standard access layer switch like the 3750s, consider the next-generation Catalyst 3850 series.
The 3850 supports BYOD/mobility and offers a variety of performance and security enhancements to previous models. The drawback of stacking is that you can only stack several switches. So, if you want additional throughout, you should aim for a different design type.
Data Center Design: Layer 2 and Layer 3 Solutions
Traditional views of data center design
Depending on the data center network topology deployed, packet forwarding at the access layer can be either Layer 2 or Layer 3. A Layer 3 approach would involve additional management and configuring IP addresses on hosts in a hierarchical fashion that matches the switch’s assigned IP address.
An alternative approach is to use Layer 2, which has less overhead as Layer 2 MAC addresses do not need specific configuration. However, it has drawbacks with scalability and poor performance.
Generally, access switches focus on communicating servers in the same IP subnet, allowing any type of traffic – unicast, multicast, or broadcast. You can, however, have filtering devices such as a Virtual Security Gateway ( VSG ) to permit traffic between servers, but that is generally reserved for inter-POD ( Platform Optimized Design ) traffic.
Lab Guide: IGMPv1
In the following example, we have a lab guide on IGMPv1.I GMPv1, or Internet Group Management Protocol Version 1, is a network-layer protocol designed to facilitate host communication and actively manage multicast group memberships.
It enables hosts to join and leave multicast groups, allowing them to receive IP multicast traffic from a specific source. Notice the output from the packet captures below for the Membership Query and the Membership Report.
Leaf and Spine With Layer 3
We use a leaf and spine data center design with Layer 3 everywhere and overlay networking. This modern, robust architecture provides a high-performance, highly available network. With this architecture, data center networks are composed of leaf switches that connect to one or more spine switches.
The leaf switches are connected to end devices such as servers, storage devices, and other networking equipment. The spine switches, meanwhile, act as the network’s backbone, connecting the multiple leaf switches.
The leaf and spine architecture provides several advantages over traditional data center networks. It allows for greater scalability, as additional leaf switches can be easily added to the network. It also offers better fault tolerance, as the network can operate even if one of the spine switches fails.
Furthermore, it enables faster traffic flows, as the spine switches to route traffic between the leaf switches faster than a traditional flat network.
Data Center Traffic Flow
Datacenter topologies can have North-South or East-to-West traffic. North-south ( up / down ) corresponds to traffic between the servers and the external world ( outside the data center ). East-to-west corresponds to internal server communication, i.e., traffic does not leave the data center.
Therefore, determining the type of traffic upfront is essential as it influences the type of topology used in the data center.
For example, you may have a pair of ISCSI switches, and all traffic is internal between the servers. In this case, you would need high-bandwidth inter-switch links. Usually, an ether channel supports all the cross-server talk; the only north-to-south traffic would be management traffic.
In another part of the data center, you may have data server farm switches with only HSRP heartbeat traffic across the inter-switch links and large bundled uplinks for a high volume of north-to-south traffic. Depending on the type of application, which can be either outward-facing or internal, computation will influence the type of traffic that will be dominant.
Virtual Machine and Containers.
This drive was from virtualization, virtual machines, and container technologies regarding east-west traffic. Many are moving to a leaf and spine data center design if they have a lot of east-to-west traffic and want better performance.
Network Virtualization and VXLAN
Network virtualization and the ability of a physical server to host many VMs and move those VMs are also used extensively in data centers, either for workload distribution or business continuity. This will also affect the design you have at the access layer.
For example, in a Layer 3 fabric, migrating a VM across that boundary changes its IP address, resulting in a reset of the TCP sessions because, unlike SCTP, TCP does not support dynamic address configuration. In a Layer 2 fabric, migrating a VM incurs ARP overhead and requires forwarding on millions of flat MAC addresses, which leads to MAC scalability and poor performance problems.
2nd Lab Guide: VXLAN
The following lab guide displays a VXLAN network. We are running VXLAN in unicast mode. VXLAN can also be configured to run in multicast mode. In the screenshot below, we have created a Layer 2 overlay across a routed Layer 3 core. The command: Show nve interface nve 1 displays an operational tunnel with the encapsulation set to VXLAN.
The screenshot shows a ping test from the desktops that connect to a Layer 3 port on the Leafs.
VXLAN: stability over Layer 3 core
Network virtualization plays a vital role in the data center. Technologies like VXLAN attempt to move the control plane from the core to the edge and stabilize the core so that it only has a handful of addresses for each ToR switch. The following diagram shows the ACI networks with VXLAN as the overlay that operates over a spine leaf architecture.
Layer 2 and 3 traffic is mapped to VXLAN VNIs that run over a Layer 3 core. The Bridge Domain is for layer 2, and the VRF is for layer 3 traffic. Now, we have the separation of layer 2 and 3 traffic based on the VNI in the VXLAN header.
One of the first notable differences between VXLAN and VLAN was scale. VLAN has a 12-bit identifier called VID, while VXLAN has a 24-bit identifier called a VID network identifier. This means that with VLAN, you can create only 4094 networks over ethernet, while with VXLAN, you can create up to 16 million.
Whether you can build layer 2 or layer 3 in the access and use VXLAN or some other overlay to stabilize the core, it would help if you modularized the data center. The first step is to build each POD or rack as a complete unit. Each POD will be able to perform all its functions within that POD.
A key point: A POD data center design
POD: It is a design methodology that aims to simplify, speed deployment, optimize utilization of resources, and drive the interoperability of the three or more data center components: server, storage, and networks.
A POD example: Data center modularity
For example, one POD might be a specific human resources system. The second is modularity based on the type of resources offered. For example, a storage pod or bare metal compute may be housed in separate pods.
These two modularization types allow designers to easily control inter-POD traffic with predefined policies. Operators can also upgrade PODs and a specific type of service at once without affecting other PODs.
However, this type of segmentation does not address the scale requirements of the data center. Even when we have adequately modularized the data center into specific portions, the MAC table sizes on each switch still increase exponentially as the data center grows.
Current and Future Design Factors
New technologies with scalable control planes must be introduced for a cloud-enabled data center, and these new control planes should offer the following:
Option
Data Center Feature
Data center feature 1
The ability to scale MAC addresses
Data center feature 2
First-Hop Redundancy Protocol ( FHRP ) multipathing and Anycast HSRP
Data center feature 3
Equal-Cost multipathing
Data center feature 4
MAC learning optimizations
When designing a data center, several design factors need to be taken into account. First, what is the growth rate for servers, switch ports, and data center customers? This prevents part of the network topology from becoming a bottleneck or linking congested.
Application bandwidth demand?
This demand is usually translated into oversubscription. In data center networking, oversubscription refers to how much bandwidth switches are offered to downstream devices at each layer.
Oversubscription is expected in a data center design. By limiting oversubscription to the ToR and edge of the network, you offer a single place to start when you experience performance problems.
A data center with no oversubscription ratio will be costly, especially with a low latency network design. So, it’s best to determine what oversubscription ratio your applications support and work best. Optimizing your switch buffers to improve performance is recommended before you decide on a 1:1 oversubscription rate.
Ethernet 6-byte MAC addressing is flat.
Ethernet forms the basis of data center networking in tandem with IP. Since its inception 40 years ago, Ethernet frames have been transmitted over various physical media, even barbed wire. Ethernet 6-byte MAC addressing is flat; the manufacturer typically assigns the address without considering its location.
Ethernet-switched networks do not have explicit routing protocols to ensure readability about the flat addresses of the server’s NICs. Instead, flooding and address learning are used to create forwarding table entries.
IP addressing is a hierarchy.
On the other hand, IP addressing is a hierarchy, meaning that its address is assigned by the network operator based on its location in the network. A hierarchy address space advantage is that forwarding tables can be aggregated. If summarization or other routing techniques are employed, changes in one side of the network will not necessarily affect other areas.
This makes IP-routed networks more scalable than Ethernet-switched networks. IP-routed networks also offer ECMP techniques that enable networks to use parallel links between nodes without spanning tree disabling one of those links. The ECMP method hashes packet headers before selecting a bundled link to avoid out-of-sequence packets within individual flows.
Equal Cost Load Balancing
Equal-cost load balancing is a method for distributing network traffic among multiple paths of equal cost. It provides redundancy and increases throughput. Sending traffic over multiple paths avoids congestion on any single link. In addition, the load is equally distributed across the paths, meaning that each path carries roughly the same total traffic.
This allows for using multiple paths at a lower cost, providing an efficient way to increase throughput.
The idea behind equal cost load balancing is to use multiple paths of equal cost to balance the load on each path. The algorithm considers the number of paths, each path’s weight, and each path’s capacity. It also feels the number of packets that must be sent and the delay allowed for each packet.
Considering these factors, it can calculate the best way to distribute the load among the paths.
Equal-cost load balancing can be implemented using various methods. One method is to use a Link Aggregation Protocol (LACP), which allows the network to use multiple links and distribute the traffic among the links in a balanced way.
A keynote: Data center topologies. The move to VXLAN.
Given the above considerations, a solution that encompasses the benefits of L2’s plug-and-play flat addressing and the scalability of IP is needed. Location-Identifier Split Protocol ( LISP ) has a set of solutions that use hierarchical addresses as locators in the core and flat addresses as identifiers in the edges. However, not much is seen in its deployment these days.
Equivalent approaches such as THRILL and Cisco FabricPath create massive scalable L2 multipath networks with equidistant endpoints. Tunneling is also being used to extend down to the server and access layer to overcome the 4K limitation with traditional VLANs. What is VXLAN? Tunneling with VXLAN is now the standard design in most data center topologies with leaf-spine designs. The following video provides VXLAN guidance.
Data Center Network Topology
Leaf and spine data center topology types
This is commonly seen in a leaf and spine design. For example, in a leaf-spine fabric, We have a Layer 3 IP fabric that supports equal-cost multi-path (ECMP) routing between any two endpoints in the network. Then, on top of the Layer 3 fabric is an overlay protocol, commonly VXLAN.
A spine-leaf architecture consists of a data center network topology of two switching layers—a spine and a leaf. The leaf layer comprises access switches that aggregate traffic from endpoints such as the servers and connect directly to the spine or network core.
Spine switches interconnect all leaf switches in a full-mesh topology. The leaf switches do not directly connect. The Cisco ACI is a data center topology that utilizes the leaf and spine.
The ACI network’s physical topology is a leaf and spine, while the logical topology is formed with VXLAN. From a protocol side point, VXLAN is the overlay network, and the BGP and IS-IS provide the Layer 3 routing, the underlay network that allows the overlay network to function.
As a result, the nonblocking architecture performs much better than the traditional data center design based on access, distribution, and core designs.
Closing Points: Data Center Topologies
A data center topology refers to the physical layout and interconnection of network devices within a data center. It determines how servers, switches, routers, and other networking equipment are connected, ensuring efficient and reliable data transmission. Topologies are based on scalability, fault tolerance, performance, and cost.
Hierarchical Data Center Topology:
The hierarchical or tree topology is one of the most commonly used data center topologies. This design consists of multiple core, distribution, and access layers. The core layer connects all the distribution layers, while the distribution layer connects to the access layer. This structure enables better management, scalability, and fault tolerance by segregating traffic and minimizing network congestion.
Mesh Data Center Topology:
Every network device is interlinked in a mesh topology, forming a fully connected network with multiple paths for data transmission. This redundancy ensures high availability and fault tolerance. However, this topology can be cost-prohibitive and complex, especially in large-scale data centers.
Leaf-Spine Data Center Topology:
The leaf-spine topology is gaining popularity due to its scalability and simplicity. It consists of interconnected leaf switches at the access layer and spine switches at the core layer. This design allows for non-blocking, low-latency communication between any leaf switch and spine switch, making it suitable for modern data center requirements.
Full-Mesh Data Center Topology:
As the name suggests, the full-mesh topology connects every network device to every other device, creating an extensive web of connections. This topology offers maximum redundancy and fault tolerance. However, it can be expensive to implement and maintain, making it more suitable for critical applications with stringent uptime requirements.
Summary:Data Center Topoloigy
Data centers are vital in supporting and enabling our digital infrastructure in today’s interconnected world. Behind the scenes, intricate network topologies ensure seamless data flow, allowing us to access information and services easily. In this blog post, we dived into the world of data center topologies, unraveling their complexities and understanding their significance.
Section 1: Understanding Data Center Topologies
Datacenter topologies refer to a data center’s physical and logical layout of networking components. These topologies determine how data flows between servers, switches, routers, and other network devices. By carefully designing the topology, data center operators can optimize performance, scalability, redundancy, and fault tolerance.
Section 2: Common Data Center Topologies
There are several widely adopted data center topologies, each with its strengths and use cases. Let’s explore some of the most common ones:
2.1. Tree Topology:
Tree topology, or hierarchical topology, is widely used in data centers. It features a hierarchical structure with multiple layers of switches, forming a tree-like network. This topology offers scalability and ease of management, making it suitable for large-scale deployments.
2.2. Mesh Topology:
The mesh topology provides a high level of redundancy and fault tolerance. In this topology, every device is connected to every other device, forming a fully interconnected network. While it offers robustness, it can be complex and costly to implement.
2.3. Spine-Leaf Topology:
The spine-leaf topology, also known as a Clos network, has recently gained popularity. It consists of leaf switches connecting to multiple spine switches, forming a non-blocking fabric. This design allows for efficient east-west traffic flow and simplified scalability.
Section 3: Factors Influencing Topology Selection
Choosing the right data center topology depends on various factors, including:
3.1. Scalability:
It is crucial for a topology to accommodate a data center’s growth. Scalable topologies ensure that additional devices can be seamlessly added without causing bottlenecks or performance degradation.
3.2. Redundancy and Fault Tolerance:
Data centers require high availability to minimize downtime. Topologies that offer redundancy and fault tolerance mechanisms, such as link and device redundancy, are crucial in ensuring uninterrupted operations.
3.3. Traffic Patterns:
Understanding the traffic patterns within a data center is essential for selecting an appropriate topology. Some topologies excel in handling east-west traffic, while others are better suited for north-south traffic flow.
Conclusion:
Datacenter topologies form the backbone of our digital infrastructure, providing the connectivity and reliability needed for our ever-expanding digital needs. By understanding the intricacies of these topologies, we can better appreciate the complexity involved in keeping our data flowing seamlessly. Whether it’s the hierarchical tree, the fully interconnected mesh, or the efficient spine-leaf, each topology has its place in the world of data centers.
Data centers are crucial in today’s digital landscape, serving as the backbone of numerous businesses and organizations. A well-designed data center network ensures optimal performance, scalability, and reliability. This blog post will explore the critical aspects of data center network design and its significance in modern IT infrastructure.
Efficient data center network design is critical for meeting the growing demands of complex applications, high data traffic, and rapid data processing. It enables seamless connectivity, improves application performance, and enhances user experience. A well-designed network also ensures data security, disaster recovery, and efficient resource utilization.
Table of Contents
Highlights:Data Center Network Design
The goal of data center design and interconnection network is to transport end-user traffic from A to B without any packet drops, yet the metrics we use to achieve this goal can be very different. The data center is evolving and progressing through various topology and technology changes, resulting in various data center network designs.
The new data center control plane we are seeing today, such as Fabric Path, LISP, THRILL, and VXLAN, is being driven by a change in the end user’s requirement; the application has changed.
These new technologies may address new challenges, yet the fundamental question of where to create the Layer 2/Layer 3 boundary and the need for Layer 2 in the access layer remains the same. The question stays the same, yet the technologies available to address this challenge have evolved.
The use of Open Networking
We also have the Open Networking Foundation ( ONF ) with open networking. Open networking describes a network that uses open standards and commodity hardware. So, consider open networking in terms of hardware and software. Unlike a vendor approach like Cisco, this gives you much more choice with what hardware and software you use to make up and design your network.
Related: Before you proceed, you may find the following useful:
Introduction to data center network design and what is involved.
Highlighting the details of VLANs and virtualization.
Technical details on the issues of Layer 2 in data centers.
Scenario: Cisco FabricPath and DFA.
Details on overlay networking and Cisco OTV.
The Rise of Overlay Networking
What has the industry introduced to overcome these limitations and address the new challenges? – Network virtualization and overlay networking. In its simplest form, an overlay is a dynamic tunnel between two endpoints that enables Layer 2 frames to be transported between them. In addition, these overlay-based technologies provide a level of indirection that enables switch table sizes to not increase in the order of the number of supported end-hosts.
Today’s overlays are Cisco FabricPath, THRILL, LISP, VXLAN, NVGRE, OTV, PBB, and Shorted Path Bridging. They are essentially virtual networks that sit on top of a physical network, often the physical network not being aware of the virtual layer above it.
Lab Guide: VXLAN
The following lab guide displays a VXLAN network. We are running VXLAN in multicast mode. Multicast VXLAN is a variant of VXLAN that utilizes multicast-based IP multicast for transmitting overlay network traffic. VXLAN is an encapsulation protocol that extends Layer 2 Ethernet networks over Layer 3 IP networks.
Linking multicast enables efficient and scalable communication within the overlay network. Notice the multicast group of 239.0.0.10 and the route of 239.0.0.10 forwarding out the tunnel interface. We have multicast enabled on all Layer 3 interfaces, including the core that consists of Spine A and Spine B.
Traditional Data Center Network Design
How do routers create a broadcast domain boundary? Firstly, using the traditional core, distribution, and access model, the access layer is layer 2, and servers served to each other in the access layer are in the same IP subnet and VLAN. The same access VLAN will span the access layer switches for east-to-west traffic, and any outbound traffic is via a First Hop Redundancy Protocol ( FHRP ) like Hot Standby Router Protocol ( HSRP ).
Servers in different VLANs are isolated from each other and cannot communicate directly; inter-VLAN communications require a Layer 3 device. Virtualization’s humble beginnings started with VLANs, which were used to segment traffic at Layer 2. It was expected to find single VLANs spanning an entire data center fabric.
VLAN and Virtualization
The virtualization side of VLANs comes from two servers physically connected to different switches. Assuming the VLAN spans both switches, the same VLAN can communicate with each server. Each VLAN can be defined as a broadcast domain in a single Ethernet switch or shared among connected switches.
Whenever a switch interface belonging to a VLAN receives a broadcast frame ( destination MAC is ffff.ffff.ffff), the device must forward this frame to all other ports defined in the same VLAN.
This approach is straightforward in design and is almost like a plug-and-play network. The first question is, why not connect everything in the data center into one large Layer 2 broadcast domain? Layer 2 is a plug-and-play network, so why not?
The issues of Layer 2
The reason is that there are many scaling issues in large layer 2 networks. Layer 2 networks don’t have controlled / efficient network discovery protocols. Address Resolution Protocol ( ARP ) is used to locate end hosts and uses Broadcasts and Unicast replies. A single host might not generate much traffic, but imagine what would happen if 10,000 hosts were connected to the same broadcast domain. VLANs span an entire data center fabric, which can bring a lot of instability due to loops and broadcast storms.
No hierarchy in MAC addresses
There is also no hierarchy in MAC addressing. Unlike Layer 3 networks, where you can have summarization and hierarchy addressing, MAC addresses are flat. Creating several thousand hosts to a single broadcast domain will create large forwarding information tables.
Because end hosts are potentially not static, they are likely to be attached and removed from the network at regular intervals, creating a high rate of change in the control plane. You can, of course, have a large Layer 2 data center with multiple tenants if they don’t need to communicate with each.
The shared services requirements, such as WAAS or load balancing, can be solved by spinning up the service VM in the tenant’s Layer 2 broadcast domain. This design will hit scaling and management issues. There is a consensus to move from a Layer 2 design to a more robust and scalable Layer 3 design.
But why is there still a need for Layer 2 in the data center topologies? One solution is Layer 2 VPN with EVPN. But first, let us have a look at Cisco DFA.
The Requirement for Layer 2 in Data Center Network Design
Servers that perform the same function might need to communicate with each other due to a clustering protocol or simply as part of the application’s inner functions. If the communication is clustering protocol heartbeats or some server-to-server application packets that are not routable, then you need this communication layer to be on the same VLAN, i.e., Layer 2 domain, as these types of packets are not routable and don’t understand the IP layer.
Stateful devices such as firewalls and load balancers need Layer 2 adjacency as they constantly exchange connection and session state information.
Dual-homed servers: Single server with two server NICs and one NIC to each switch will require a layer 2 adjacency if the adapter has a standby interface that uses the same MAC and IP addresses after a failure. In this situation, the active and standby interfaces must be on the same VLAN and use the same default gateway.
Suppose your virtualization solutions cannot handle Layer 3 VM mobility. In that case, you may need to stretch VLANs between PODS / Virtual Resource Pools or even data centers so you can move VMs around the data center at Layer 2 ( without changing their IP address ).
Data Center Design and Cisco DFA
Cisco went one giant step and recently introduced a data center fabric with Dynamic Fabric Automaton ( DFA ), similar to Juniper QFabric, which offers you both Layer 2 switching and Layer 3 routing at the access layer / ToR. Firstly, they have Fabric Path ( IS-IS for Layer 2 connectivity ) in the core, which gives you optimal Layer 2 forwarding between all the edges.
Then they configure the same Layer 3 address on all the edges, which gives you optimal Layer 3 forwarding across the whole Fabric.
On edge, you can have Layer 3 Leaf switches, for example, the Nexus 6000 series, or integrate with Layer 2-only devices like the Nexus 5500 series or the Nexus 1000v. You can also connect external routers or USC or FEX to the Fabric. In addition to running IS-IS as the data center control plane, DFA uses MP-iBGP, with some Spine nodes being the Route Reflector to exchange IP forwarding information.
Cisco FabricPath
DFA also employs a Cisco FabricPath technique called “Conversational Learning.” The first packet triggers a full RIB lookup, and the subsequent packets are switched in the hardware-implemented switching cache.
This technology provides Layer 2 mobility throughout the data center while providing optimal traffic flow using Layer 3 routing. Cisco commented, “DFA provides a scale-out architecture without congestion points in the network while providing optimized forwarding for all applications.”
Terminating Layer 3 at the access / ToR has clear advantages and disadvantages. Other benefits include reducing the size of the broadcast domain, which comes at the cost of reducing the mobility domain across which VMs can be moved.
Terminating Layer 3 at the accesses can also result in sub-optimal routing because there will be hair pinning or traffic tromboning of across-subnet traffic, taking multiple and unnecessary hops across the data center fabric.
The role of the Cisco Fabricpath
Cisco FabricPath is a Layer 2 technology that provides Layer 3 benefits, such as multipathing the classical Layer 2 networks using IS-IS at Layer 2. This eliminates the need for spanning tree protocol, avoiding the pitfalls of having large Layer 2 networks. As a result, Fabric Path enables a massive Layer 2 network that supports multipath ( ECMP ). THRILL is an IEEE standard that, like Fabric Path, is a Layer 2 technology that provides the same Layer 3 benefits as Cisco FabricPath to the Layer 2 networks using IS-IS.
LISP is popular in Active / Active data centers for DCI route optimization/mobility and separates the host’s location and the identifier ( EID ), allowing VMs to move across subnet boundaries while keeping the endpoint identification. LISP is often referred to as an Internet locator.
That can enable some designs of triangular routing. Popular encapsulation formats include VXLAN ( proposed by Cisco and VMware ) and STT (created by Nicira but will be deprecated over time as VXLAN comes to dominate ).
Video: LISP networking
In the following video, we will demonstrate the use of LISP in networking. It’s a hands-on demonstration that goes through the various components of a LISP network and how each component operates.
Hands on Video Series – Enterprise Networking | LISP Configuration Intro
OTV is a data center interconnect ( DCI ) technology enabling Layer 2 extension across data center sites. While Fabric Path can be a DCI technology over short distances with dark fiber, OTV has been explicitly designed for DCI. In contrast, the Fabric Path data center control plane is primarily used for intra-DC communications.
Failure boundary and site independence are preserved in OTV networks because OTV uses a data center control plane protocol to sync MAC addresses between sites and prevent unknown unicast floods. In addition, recent IOS versions can allow unknown unicast floods for certain VLANs, which are unavailable if you use Fabric Path as the DCI technology.
The Role of Software-defined Networking (SDN)
Another potential trade-off between data center control plane scaling, Layer 2 VM mobility, and optimal ingress/egress traffic flow would be software-defined networking ( SDN ). At a basic level, SDN can create direct paths through the network fabric to isolate private networks effectively.
An SDN network allows you to choose the correct forwarding information on a per-flow basis. This per-flow optimization eliminates VLAN separation in the data center fabric. Instead of using VLANs to enforce traffic separation, the SDN controller has a set of policies allowing traffic to be forwarded from a particular source to a destination.
The ACI Cisco borrows concepts of SDN to the data center. It operates over a leaf and spine design and traditional routing protocols such as BGP and IS-IS. However, it brings a new way to manage the data center with new constructs such as Endpoint Groups (EPGs). In addition, no more VLANs are needed in the data center as everything is routed over a Layer 3 core, with VXLAN as the overlay protocol.
Summary: Recap on Data Center Design
Data centers are the backbone of modern technology infrastructure, providing the foundation for storing, processing, and transmitting vast amounts of data. A critical aspect of data center design is the network architecture, which ensures efficient and reliable data transmission within and outside the facility. 1. Scalability and Flexibility
One of the primary goals of data center network design is to accommodate the ever-increasing demand for data processing and storage. Scalability ensures the network can grow seamlessly as the data center expands. This involves designing a network supporting many devices, servers, and users without compromising performance or reliability. Additionally, flexibility is essential to adapt to changing business requirements and technological advancements.
Redundancy and High Availability
Data centers must ensure uninterrupted access to data and services, making redundancy and high availability critical for network design. Redundancy involves duplicating essential components, such as switches, routers, and links, to eliminate single points of failure. This ensures that if one component fails, there are alternative paths for data transmission, minimizing downtime and maintaining uninterrupted operations. High availability further enhances reliability by providing automatic failover mechanisms and real-time monitoring to detect and address network issues promptly.
Traffic Optimization and Load Balancing
Efficient data flow within a data center is vital to prevent network congestion and bottlenecks. Traffic optimization techniques, such as Quality of Service (QoS) and traffic prioritization, can be implemented to ensure that critical applications and services receive the necessary bandwidth and resources. Load balancing is crucial in distributing network traffic evenly across multiple servers or paths, preventing overutilization of specific resources and optimizing performance.
Security and Data Protection
Data centers house sensitive information and mission-critical applications, making security a top priority. The network design should incorporate robust security measures, including firewalls, intrusion detection systems, and encryption protocols, to safeguard data from unauthorized access and cyber threats. Data protection mechanisms, such as backups, replication, and disaster recovery plans, should also be integrated into the network design to ensure data integrity and availability.
Monitoring and Management
Proactive monitoring and effective management are essential for maintaining optimal network performance and addressing potential issues promptly. The network design should include comprehensive monitoring tools and centralized management systems that provide real-time visibility into network traffic, performance metrics, and security events. This enables administrators to promptly identify and resolve network bottlenecks, security breaches, and performance degradation.
Data center network design is critical in ensuring efficient, reliable, and secure data transmission within and outside the facility. Scalability, redundancy, traffic optimization, security, and monitoring are key considerations for designing a robust, high-performance network. By implementing best practices and staying abreast of emerging technologies, data centers can build networks that meet the growing demands of the digital age while maintaining the highest levels of performance, availability, and security.
Shopping Basket
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.