Underlay and Overlay Networks

Overlay networks are virtual networks that run on top of physical networks. You have probably seen this terminology even if you have never heard of it. A GRE tunnel can illustrate an overlay network. Physical underlay networks support the GRE tunnel.VXLAN overlays are layer 2 Ethernet networks. Layer 3 IP networks form the underlay network. Transport networks are also known as underlay networks.

Getting packets from A to B is the only job of the underlay network. Layer 2 is not used here, only layer 3. We can load balance traffic on redundant links using an IGP like OSPF or EIGRP.

In addition, the overlay and underlay networks are independent. Underlay networks are virtual, but any changes made to the overlay network won’t affect the underlay network. A routing protocol can reach the destination regardless of how many links you add or remove in the underlay network.

Virtual overlay solutions

Virtual overlay solutions must have some simple to complex application stacks. Therefore, public or private cloud environments must support austere, complex environments to enable the virtual overlay network. On the other hand, simple customers that require web-hosting solutions need only a single domain with a few segments. Regarding network connectivity, there is one Virtual Machine ( VM ) with a single public IP.

Complex customers require complex multi-tier application stacks with overlay virtual networking, load-balancing, and firewall services in front and between application tiers. Cloud providers must support all types of application stacks as they are isolated virtual segments, and this is done with virtual overlay networks.

Lab guide on VXLAN.

In the following example, we have a lab guide on VXLAN. Here, we created a Layer 2 overlay across the core. The core layer consists of two spines and is a routed layer. The core does not know the subnets assigned to the desktop devices. It is the role of VXLAN to tunnel this information.

Notice we have a VNI set to 6002. This needs to match at both ends of Leaf A and Leaf B. If you change the VNI, you will break connectivity. This is a Layer 2 overlay, as the VNI is mapped to a bridge domain.

Concept of network virtualization

It’s worth mentioning that network virtualization is nothing new. The most common forms of network virtualization are virtual LANs (VLANs), virtual private networks (VPNs), and Multiprotocol Label Switching (MPLS). VLAN has been the first to extract the location of Layer 2 connectivity across multiple Layer 2 switches. VPN enables overlay networks across untrusted networks such as the WAN, while MPLS segments traffic based on labels.

These technologies enable the administrators to physically separate endpoints into logical groups, making them behave like they are all on the same local (physical) segment. The ability to do this allows for much greater efficiency in traffic control, security, and network management.

• • Enhanced Connectivity:

One of the primary advantages of network overlay is its ability to enhance connectivity. By creating a virtual network layer, overlay networks enable seamless communication between devices and applications, irrespective of their physical location.

This means organizations can effortlessly connect geographically dispersed branches, data centers, and cloud environments, fostering collaboration and resource sharing. Moreover, network overlays offer greater flexibility by allowing organizations to dynamically adjust and optimize their network configurations to meet evolving business needs.

• • Improved Scalability:

Traditional network infrastructures often struggle to keep up with the increasing demands of modern applications and services. Network overlay addresses this challenge by providing a scalable solution. By decoupling the virtual network from the physical infrastructure, overlay networks allow for more efficient resource utilization and easier scaling.

Organizations can easily add or remove network elements without disrupting the entire network. As a result, network overlays enable organizations to scale their networks rapidly and cost-effectively, ensuring optimal performance even during peak usage periods.

Example of an overlay network: MPLS

MPLS overlay is a technique used to create virtual private networks (VPNs) over existing IP networks, enabling organizations to achieve enhanced network scalability, reliability, and security. Unlike traditional IP routing, MPLS overlay relies on labels to forward packets, making it more efficient and flexible.

Overlay with MPLS 

With MPLS, we can have a free BGP core providing an MPLS overlay. MPLS overlay is a network architecture that allows organizations to build virtual private networks (VPNs) on top of their existing network infrastructure. It leverages the capabilities of MPLS technology to create virtual tunnels, known as MPLS tunnels or MPLS paths, which enable the secure and efficient transfer of data between different network endpoints.

Below, we have BGP running between the PEs and carrying customer prefixes for CE 1 and 2. The P, representing the core layer, does not know customer routes and performs label switching. This brings not only scalability, as the P nodes can focus on label switching, but also an added layer of security. No security devices need to be present in the core layer. Although you would need QoS, they are pushing intelligence to the edges.

Benefits of MPLS Overlay:

1. Enhanced Performance: MPLS overlay offers improved network performance by enabling faster data transmission and reduced latency. It achieves this by using label switching, which helps prioritize and route data packets efficiently, reducing congestion and optimizing network utilization.

2. Scalability and Flexibility: With MPLS overlay, organizations can quickly expand their network infrastructure without requiring extensive hardware upgrades. It allows for creating virtual networks within a shared physical infrastructure, enabling seamless scalability and flexibility.

3. Quality of Service (QoS): MPLS overlay provides enhanced QoS capabilities, enabling organizations to prioritize critical applications or data traffic. This ensures mission-critical applications receive the bandwidth and low latency, optimizing overall network performance.

4. Improved Security: MPLS overlay enhances network security by providing inherent isolation between different VPNs. It creates separate virtual tunnels for each VPN, ensuring that data remains isolated and protected from unauthorized access.

Guide on MPLS TE

In this lab, we will examine MPLS TE with ISIS configuration. Our MPLS core network consists of routers PE1, P1, P2, P3, and PE2. The CE1 and CE2 routers use regular IP routing. All routers are configured to use IS-IS L2. 

MPLS TE is a mechanism that allows network operators to control and manage traffic flows within a Multiprotocol Label Switching (MPLS) network. It is designed to address the limitations of traditional IP routing by providing a more efficient and flexible approach to data forwarding

Note:

There are four main items we have to configure:

• Enable MPLS TE support:
  • Globally
  • Interfaces
• Configure IS-IS to support MPLS TE.
• Configure RSVP.
• Configure a tunnel interface.

Example of an overlay network: DMVPN

With the configuration of DMVPN phase 1, we can have a “hub and spoke” topology, where a single hub site acts as the central point for communication, while the other locations, or “spokes,” connect to the hub through virtual tunnels. This topology provides several benefits, including secure communications between spokes, optimized traffic routing, and reduced overhead for managing the network.

DMVPN also supports dynamic routing protocols, such as Open Shortest Path First (OSPF), allowing for dynamic updates to the network topology. This allows for rapid changes in the network, such as adding or removing spokes, without the need to reconfigure the entire network. Additionally, DMVPN supports multicast traffic, allowing the efficient distribution of data and resources to multiple sites simultaneously.

Guide with DMVPN

In the following lab, we have DMVPM, which creates an overlay network. The hub, which is R1, created an overlay network over the SP router. The SP router represents the WAN; in reality, the number of nodes in the WAN is irrelevant to DMPVN. The overlay is created between R1, R2, and R3, which act as the spokes.

The protocol used in GRE, specifically point-to-point GRE, as we are running DMVPN Phase 1. The Tunneling protocol of mGRE would have been used if we were running DMVPN Phase 3

Benefits of DMVPN Overlay:

1. Simplified Network Architecture:

Traditional networking often involves complex and static configurations, making it cumbersome to manage and maintain. DMVPN overlay, on the other hand, simplifies network architecture by providing a dynamic and scalable solution. With DMVPN, organizations can establish secure connections between various branch offices, data centers, and remote users, all while leveraging the existing infrastructure. This simplification leads to reduced administrative overhead and improved network agility.

2. Enhanced Flexibility and Scalability:

DMVPN overlay offers unparalleled flexibility and scalability, making it an ideal choice for organizations with dynamic network requirements. As businesses grow and expand, DMVPN allows for the seamless addition of new sites or remote users without requiring extensive configuration changes. Its ability to establish connections on-demand and dynamically allocate resources ensures that network expansion remains hassle-free and cost-effective.

3. Improved Network Performance:

Network performance is crucial for organizations, directly impacting productivity and user experience. DMVPN overlay utilizes multiple paths and load balancing techniques, allowing for efficient utilization of available bandwidth. By optimizing network traffic, DMVPN ensures that applications and services operate smoothly, even during peak usage periods. Moreover, its ability to prioritize critical traffic and dynamically adjust to network conditions further enhances overall performance.

4. Enhanced Security:

Security remains a top concern for organizations, particularly when transmitting sensitive data across networks. DMVPN overlay addresses these concerns by providing robust encryption and authentication mechanisms. By leveraging IPsec protocols, DMVPN ensures that data confidentiality and integrity are maintained, protecting against unauthorized access and potential threats. The inherent security features of DMVPN make it a reliable choice for organizations looking to maintain a secure network environment.

Types of Overlay Networks

1. Virtual Private Networks (VPNs):

VPNs are one of the most common types of overlay networks. They enable secure communication over public networks by creating an encrypted tunnel between the sender and receiver. Individuals and organizations widely use VPNs to protect sensitive data and maintain privacy. Additionally, they allow users to bypass geographical restrictions and access region-restricted content.

2. Software-Defined Networks (SDNs):

In network architecture, SDNs utilize overlay networks to separate the control plane from the data plane. SDNs provide centralized management, flexibility, and scalability by decoupling network control and forwarding functions. Overlay networks in SDNs enable the creation of virtual networks on top of the physical infrastructure, allowing for more efficient resource allocation and dynamic network provisioning.

3. Peer-to-Peer (P2P) Networks:

P2P overlay networks are decentralized systems that facilitate direct communication and file sharing between nodes without relying on a central server. They leverage overlay networks to establish direct connections between peers and enable efficient data distribution. These networks are widely used for content sharing, real-time streaming, and decentralized applications.

4. Content Delivery Networks (CDNs):

CDNs employ overlay networks to optimize content delivery by strategically distributing content across multiple servers in different geographic regions. By bringing content closer to end-users, CDNs reduce latency and improve performance. Overlay networks in CDNs enable efficient content caching, load balancing, and fault tolerance, resulting in faster and more reliable content delivery.

5. Overlay Multicast Networks:

Overlay multicast networks are designed to distribute data to multiple recipients simultaneously efficiently. These networks use overlay protocols to construct multicast trees and deliver data over these trees. Overlay multicast networks benefit applications such as video streaming, online gaming, and live events broadcasting, where data must be transmitted to many recipients in real-time.

Use Cases of Overlay Virtual Networking:

1. Multi-Tenancy:

Overlay virtual networking provides an ideal solution for organizations to segregate their network resources securely. By creating virtual overlays, multiple tenants can coexist on a single physical network infrastructure without interference. This enables service providers and enterprises to offer distinct network environments to customers or departments while ensuring isolation and security.

2. Data Center Interconnect:

Overlay virtual networking enables efficient and scalable data center interconnect (DCI). With traditional networking, interconnecting multiple data centers across geographies can be complex and costly. However, overlay virtual networking simplifies this process by abstracting the underlying physical infrastructure and providing a unified logical network. It allows organizations to seamlessly extend their networks across multiple data centers, enhancing workload mobility and disaster recovery capabilities.

3. Cloud Computing:

Cloud computing heavily relies on overlay virtual networking to deliver agility and scalability. Cloud providers can dynamically provision and manage network resources by leveraging overlay networks, ensuring optimal customer performance and flexibility. Overlay virtual networking enables the creation of virtual networks that are isolated from each other, allowing for secure and efficient multi-tenant cloud environments.

4. Microservices and Containerization:

The rise of microservices architecture and containerization has presented new networking challenges. Overlay virtual networking provides a solution by enabling seamless communication between microservices and containers, regardless of their physical location. It ensures that applications and services can communicate with each other, even across different hosts or clusters, without complex network configurations.

5. Network Segmentation and Security:

Overlay virtual networking enables granular network segmentation, allowing organizations to implement fine-grained security policies. By creating overlay networks, administrators can isolate different workloads, departments, or applications, ensuring each segment has dedicated network resources and security policies. This enhances security by limiting the lateral movement of threats and reducing the attack surface.

Tailored load balancing

Some customers may not require cloud load balancing services provided by the cloud services if they have optimized web delivery by deploying something like Squid or NGINX. Squid is a caching proxy that improves web request response times by caching frequently requested web pages. NGINX ( open source reverse proxy ) is used to load balance Hypertext Transfer Protocol ( HTTP ) among multiple servers.

Example: Traffic flow and the need for a virtual overlay

Traffic would flow to Web servers and trigger application and database requests. Each tier requires different segments, and in large environments, the limitations of using VLANs to create these segments will bring both scalability and performance problems.

This is why we need virtual overlay solutions. These subnets require Layer 3 and sometimes Layer 2 ( MAC ). Layer 2 connectivity might be for high availability services that rely on gratuitous Address Resolution Protocol ( ARP ) between devices or some other non-routable packet that can not communicate over IP. If the packet is not Layer 3 routable, it needs to communicate via Layer 2 VLANs.

Scalability and Security Concerns

The weakest link in a security paradigm is the lowest application in that segment. Make each application an independent tenant so all other applications are unaffected if a security breach or misuse occurs in one application stack.

Designers should always attempt to design application stacks to minimize beachheading, i.e., an attacker compromising one box and using it to jump to another quickly. Public and private clouds should support multi-tenancy with each application stack.

However, scalability issues arise when you deploy each application as an individual segment. For example, customer X’s cloud application requires four segments; 4000 VLANs soon become 1000 applications. Media Access Control ( MAC ) visibility has an entire reach throughout Layer 2 domains.

Some switches support a low count number of MAC addresses. When a switch reaches its MAC limit, it starts flooding packets, increasing network load and consuming available bandwidth that should be used for production services.

…current broadcast domains can support … around 1,000 end hosts in a single bridged LAN of 100 bridges” (RFC 5556 – TRILL)

NIC in promiscuous mode and failure domains

Server administrators configure server NICs in promiscuous mode to save configuration time. NICs in promiscuous mode look at all frames passing even when the frame is not destined for them. Network cards acting in promiscuous mode are essentially the same as having one VLAN spanning the entire domain. Sniffer products set promiscuous modes to capture all data on a link and usually only act in this mode for troubleshooting purposes.

A well-known issue with Layer 2 networks is that they present a single failure domain with extreme scalability and operational challenges. This is related to Layer 2 Spanning Tree Protocol ( STP ); THRILL is also susceptible to broadcast storms and network meltdowns.

The rise of overlay virtual networks

Previously discussed scalability and operational concerns force vendors to develop new data center technologies. One of the most prevalent new technologies is overlay virtual networks, tunneling over IP. An overlay is a tunnel between two endpoints, allowing frames to be transported. The beauty of overlay architectures is that they enable switch table sizes not to increase as the number of hosts attached increases.

Vendors’ Answer: Virtual Overlay Solutions

Virtual Overlay Solution: Keep complexity to the edges.

Ideally, we should run virtual networks over IP like SKYPE runs Voice over IP. The recommended design retains complexity at the network’s edge; the IP transport network provides IP transport. A transport network does not need to be a Layer 2 network and can have as many IP subnets and router hops.

All data ( storage, vMotion, user traffic ) traffic becomes an IP application. The concept resembles how Border Gateway Protocol ( BGP ) applies to TCP. End hosts carry out encapsulation and use the network for transport. Again, complexity is at the edge, similar to the Internet. Keeping complexity to the edge makes Layer 3 fabrics efficient and scalable.

VXLAN, STT, and ( NV ) GRE

Numerous encapsulation methods can tunnel over the IP core. This is known as virtual overlay networking and includes VXLAN, STT, and ( NV ) GRE. The main difference between these technologies is the encapsulation method and minor technological differences with TCP offload and load balancing.

The Recommended Design: Leaf and Spine.

Like the ACI network, virtual overlay networks work best with Leaf and Spine fabric architectures. Leaf and Spine designs guarantee any two endpoints get equal bandwidth. VMs on the same Top-of-Rack ( ToR ) switch will have access to more bandwidth than if the VM had to communicate across the Spine layer.

Overlay networks assume that the underlying network has a central endpoint. The transport network should avoid oversubscription as much as possible. If security concerns you, you can always place similar VM appliances on dedicated clusters, one type per physical server.

( NV ) GRE, VXLAN, and STT do not have an built-in security features meaning the transport network MUST be secure.

TCP offload, load balancing & scale-out NAT

TCP can push huge segments down the physical NIC and slice the packet into individual TCP segments, improving TCP performance. For example, you can push 10Gbps from a VM with TCP offload. The problem is that NICs only support VLANs and not VXLANs.

NICIRA added another header in front of TCP segments. TCP is embedded in another TCP. Now, you can use the existing NIC to slice the current TCP segment into smaller TCP segments. It is dramatically improving performance.

STT and VXAN

STT and VXAN can use 5-tuple load balancing as they use port numbers. Therefore, traffic sent between a pair of VMs can use more than one link in the network. Unfortunately, not many switches can load balance based on the GRE payload used by NVGRE.

Scale-out NAT is difficult to implement as an asymmetric path is not guaranteed. Furthermore, the shared state is tied to an outside IP address, which limits scale-out options. To scale out effectively, the state has to be spread across all members of the NAT cluster. The new approach uses floating public IP addresses and one-to-one mapping between floating IP and the private IP address inside—there is no state due to the one-to-one mapping.

Distributed layer 2 & layer 3 forwarding  

They distributed Layer 2 forwarding ( data plane ): Most Overlays offer distributed Layer 2 forwarding. VM can be sent to VM in the same segment. The big question is how they distribute MAC to VTEP – some use multicast and traditional Ethernet flooding, while others use control planes. The big question is how scalable is the control plane.

Distributed Layer 3 forwarding ( data plane ): On the other hand, if you have multiple IP subnets between segments ( not layer 2 ), you need to forward between them. The inter-subnet must not be a choke point. If your data center has lots of intra-traffic ( East to West traffic), avoid centralized inter-subnet forwarding, which will quickly become a traffic choke point.

The router will process ARP if you are doing Layer 3 forwarding. But if you are doing a mix of Layer 2 and 3, make sure you can reduce the flooding by intercepting ARP requests and caching ARP replies, known as distributed ARP Caching.

Scale-out control plane 

Initial overlays used multicast and Ethernet-like learning. Now, some vendors are using controller-based overlays. Keep in mind that the controller can now become a scalability bottleneck. However, many vendors, such as Cisco ACI, can scale the controllers and have a quorum.

Efficient controller scalability is seen when controllers do not participate in the data plane ( do not reply to ARP ). This type of controller scales better than controllers that intercept data plane packets and perform data plane activity. So, the data plane will not be affected if a controller is offline. In the early days of Sofware-Defined Networking, this was not the case. If the controller was down, the network was down.

Scale-out controllers 

Attempt to design scale-out controllers by building a cluster of controllers and having some protocol running between them. You now have clear failure domains. For example, controller A looks after VM segment A and Controller B, and control looks after VM segment B. For cloud deployments in multiple locations, deploy multiple controller clusters in each location.

Availability zones

Design availability zones with hierarchical failure domains by splitting infrastructures into regions. Problems arising in one region do not affect all other regions. You have one or more availability zones within an area for physical and logical isolation.

Availability zones limit the impact of a failure in a failure domain. An example of a failure domain could be a VLAN experiencing a broadcast storm. Attempt to determine the span of VLANs across availability zones—define VLANs to one-ToR switch. Never stretch VLANs as you create a single failure domain by merging two zones.

Do not stretch a VLAN across multiple availability zones. This is why we have network overlays in the first place, so we don’t need to stretch VLAN across the data center. For example, VXLAN uses the VNI to differentiate between Layer 2 and Layer 3 traffic over a routed underlay. We can use VXLAN as the overlay network to span large Layer 2 domains over a routed core.

Network Overlay Controllers

As a final note on controllers, controller-based SDN networks participate in data planes and perform activities such as MAC learning and ARP replies. As mentioned, this is not common nowadays but was at the start of the SDN days. If the controller performs activities such as MAC learning and APR replies and the controller fails, then you have network failure.

The more involved the controller is in the forwarding decisions, the worse the outage can be. All overlay networking vendors nowadays have controllers that set up the control plane so the data plane can forward traffic without getting involved in data plane activity. This design also allows the controller to be scaled without affecting the data plane activity.

Overlay virtual networking has significant implications for modern network architectures. It enables the creation of software-defined networks (SDNs), where network policies, routing, and security are managed centrally through software-based controllers. This centralized management simplifies network operations, improves agility, and enables network automation.