combination lock and different gadgets on white office table. privacy protection, encrypted connection concept, buying online

VMware NSX – Network and Security Virtualization

 

VMware NSX Security

 

VMware NSX Security

Ensuring robust network security is paramount in today’s rapidly evolving digital landscape. Organizations need to safeguard their data and protect their infrastructure from cyber threats. One powerful solution that has emerged in recent years is VMware NSX Security. In this blog post, we will explore the key features and benefits of VMware NSX Security and how it can help organizations enhance their network security.

 

Highlights: VMware NSX Security

Thank Andreas Gautschi from VMware for giving me a 2-hour demonstration and brain dump about NSX. Initially, even as an immature product, SDN got massive hype in its first year. However, the ratio from slide to production deployments was minimal. It was getting a lot of publicity even though it was mostly an academic and PowerPoint reality.

  • The Role of SDN

Recently, the ratio has changed, and the concepts of SDN apply to different types of network security components meeting various requirements. SDN enables network virtualization with many companies, such as VMware NSX, Midokura, Juniper Contrail, and Nuage, offering network virtualization solutions. The following post generally discusses network virtualization but focuses more on the NSX functionality.  

 

For additional pre-information, you may find the following helpful:

  1. WAN Virtualization
  2. Nexus 1000v
  3. Docker Security Options

 



Network Security Virtualization

Key VMware NSX Security Discussion points:


  • Introduction to VMware NSX Security, and where it can be used.

  • Discussion on Network Security Virtualization.

  • The role of containers and the changing workloads.

  • Distributed Firewalling and attack surface.

  •  Policy classification.

 

  • A key point: Back to basics with the Virtualization

The Virtualization of resources is crucial in fulfilling the required degree of adaptability. Therefore, we can perform Virtualization in many areas, including the Virtualization of servers, applications, storage devices, security appliances, and, not surprisingly, the network infrastructure. Server virtualization was the starting point for most.

Remember that security is a building block behind the virtualized network and a key driver. An essential component of a security policy is the definition of a network perimeter. Communications between the inside and the outside of the perimeter must occur through a checkpoint. With Virtualization, this checkpoint can now be at multiple network parts. Not just the traditional edge.

 

Key VMware NSX Points

1. Network Segmentation:

One of the fundamental aspects of VMware NSX Security is its ability to provide network segmentation. Organizations can create isolated environments for different applications and workloads by dividing the network into multiple virtual segments. This isolation helps prevent lateral movement of threats and limits the impact of a potential security breach.

2. Micro-segmentation:

With VMware NSX Security, organizations can implement micro-segmentation, which allows them to apply granular security policies to individual workloads within a virtualized environment. This level of control enables organizations to establish a zero-trust security model, where each workload is protected independently, reducing the attack surface and minimizing the risk of unauthorized access.

3. Distributed Firewall:

VMware NSX Security incorporates a distributed firewall that operates at the hypervisor level. Unlike traditional perimeter firewalls, which are typically centralized, the distributed firewall provides virtual machine-level security. This approach ensures that security policies are enforced regardless of the virtual machine’s location, providing consistent protection across the entire virtualized infrastructure.

4. Advanced Threat Prevention:

VMware NSX Security leverages advanced threat prevention techniques to detect and mitigate potential security threats. It incorporates intrusion detection and prevention systems (IDPS), malware detection, and network traffic analysis. These capabilities enable organizations to proactively identify and respond to potential security incidents, reducing the risk of data breaches and system compromises.

5. Automation and Orchestration:

Automation and orchestration are integral components of VMware NSX Security. With automation, organizations can streamline security operations, reducing the probability of human errors and speeding up the response to security incidents. Orchestration allows for integrating security policies with other IT processes, enabling consistent and efficient security management.

6. Integration with Existing Security Solutions:

VMware NSX Security can seamlessly integrate with existing security solutions, such as threat intelligence platforms, security information and event management (SIEM) systems, and endpoint protection tools. This integration enhances the overall security posture of an organization by aggregating security data from various sources and providing a holistic view of the network’s security landscape.

 

Network Security Virtualization

The Role of Network Virtualization

Virtualization provides network services independent of the physical infrastructure in its simplest form. Traditional network services were tied to physical boxes, lacking elasticity and flexibility. This results in many technical challenges, including central chokepoints, hair pinning, traffic trombones, and not to mention the underutilization of network devices.

Network virtualization combats this and abstracts network services ( different firewalling such as context firewall, routing, etc.) into software, making it easier to treat the data center fabric as a pool of network services. When a service is put into the software, it gains elasticity and fluidity qualities not present with physical nodes. The physical underlay provides a connectivity model only concerned with endpoint connectivity.

The software layer on top of the physical world provides the abstraction for the workloads, offering excellent application continuity. Now, we can take two data centers and make them feel like one. You can help facilitate this connection by incorporating kubernetes software to help delegate when a service needs to be done correctly, keeping on top of workload traffic.

The Different Traffic Flows

All east and west traffic flows via the tunnels. VMware’s NSX optimizes local egress traffic so that traffic exits the right data center and does not need to flow via the data center interconnect to egress. We used hacks with HSRP localization or complicated protocols such as LISP to overcome outbound TE with traditional designs. 

The application has changed from the traditional client-server model, where you know how many hosts you run on top of. To an application that moves and scales on numerous physical nodes that may change. With network virtualization, we don’t need to know what physical Host the application is on, as all the computing, storage, and networking move with the application.

If application X moves to location X, all its related services move to location X too. The network becomes a software abstract. Apps can have multiple tiers – front end, database, and storage with scale capabilities, automatically reacting to traffic volumes. It’s far more efficient to scale up docker containers with container schedules to meet traffic volumes than to deploy 100 physical servers, leaving them idle for half the year. If performance is not a vital issue, it makes sense to move everything to software.

 

VMware NSX Security: Changing Endpoints

The endpoints the network must support has changed. We now have container based virtualization, VMs, and mobile and physical devices. Networking is evolving, and it’s all about connecting all these heterogeneous endpoints that are becoming very disconnected from the physical infrastructure. Traditionally, a server is connected to the network with an Ethernet port.

Then, virtualization came along, offering the ability to architect new applications. Instead of single servers hosting single applications, we have multiple VMs hosting different applications on a single physical server. More recently, we see the introduction of docker containers, spawning in as little as 350ms.

 

The Challenge with Traditional VM

Traditional VLANs cannot meet this type of fluidity as each endpoint type holds different requirements on the network. The network must now support conventional physical servers, VMs, and Docker containers. All these stacks must cross each other and, more importantly, be properly secured in a multitenant environment.

Can traditional networking meet this? VMware NSX is a reasonably mature product offering virtualized network and security services that can secure various endpoints. 

Network endpoints have different trust levels. Administrators trust hypervisors more now, with only two VMware hypervisor attacks in the last few years. Unfortunately, the Linux kernel has numerous ways to attack it. Security depends on the attack surface, and an element with a large surface has more potential for exploitation. The Linux kernel has a large attack surface, while hypervisors have a small one.

The more options an attacker can exploit, the larger the attack surface. Containers run many workloads, so the attack surface is more extensive and varied. The virtual switch inside the container has a different trust level than a vswitch inside a hypervisor. Therefore, you must operate different security paradigms relating to containers than hypervisors. 

 

  • A key point: VMware NSX Security and Network Security Virtualization.

NSX provides isolation to all these endpoint types with microsegmentation. Microsegmentation allows you to apply security policy at a VM-NIC level. This offers the ability to protect east-west traffic and move policies with the workload.

This doesn’t mean that each VM NIC requires an individual configuration. NSX uses a distributed firewalls kernel module, and the hosts obtain the policy without individual config. Everything is controlled centrally but installed locally on each vSphere host. It scales horizontally, so you get more firewalls if you add more computing capacity.

All the policies are implemented in a distributed fashion, and the firewall is situated right in front of the VM in the hypervisor. So you can apply policy at a VM NIC level without hair pinning or trombone the traffic. Traffic doesn’t need to go across the data center to a central policy engine: offering optimum any to any traffic.

Even though the distributed firewall is a Kernal loadable module (KLM) of the ESXi Host, the policy enforcement is at the vNIC of the VM.

 

Network Security Virtualization: Policy Classification

A central selling point with NSX is that you get an NSX-distributed firewall. VMware operates with three styles of security:

  1. We have traditional network-focused 5-tuple matching.
  2. We then move up a layer with infrastructure-focused rules such as port groups, vCenter objects, etc.
  3. We have application-focused rule sets at a very high level, from the Web tier to the App tier permit TCP port 80.

Traditionally, we have used network-based rules, so the shift to application-based, while more efficient, will present the most barriers. People’s mindset needs to change. But the real benefit of NSX comes from this type of endpoint labeling and security. Sometimes more than a /8 is required!

What happens when you run out of /8? We start to implement kludges with NAT etc. Security labeling has been based on IP addresses in the past, and we should start moving with tagging or other types of labeling.

IP addresses are just a way to get something from point A to point B, but if we can focus on different ways to class traffic, the IP address should be irrelevant to security classification. The less tied we are to IP addresses as a security mechanism, the better we will be.

With NSX, endpoints are managed based on high-level policy language that adequately describes the security function. IP is a terrible way to do this as it imposes hard limits on mobile VMs and reduces flexibility. The policy should be independent of IP address assignment.

Conclusion:

Organizations must adopt robust and versatile security solutions in an era of constant cybersecurity threats. VMware NSX Security offers comprehensive features and capabilities that can significantly enhance network security. Organizations can build a robust security infrastructure that protects their data and infrastructure from evolving cyber threats by implementing network segmentation, micro-segmentation, a distributed firewall, advanced threat prevention, automation, and integration with existing security solutions. VMware NSX Security empowers organizations to take control of their network security and ensure the confidentiality, integrity, and availability of their critical assets.

 

network security virtualization

opencontrail

OpenContrail

 

 

OpenContrail

In today’s fast-paced world, where cloud computing and virtualization have become the norm, the need for efficient and flexible networking solutions has never been greater. OpenContrail, an open-source software-defined networking (SDN) solution, has emerged as a powerful tool. This blog post explores the capabilities, benefits, and significance of OpenContrail in revolutionizing network management and delivering enhanced connectivity in the cloud era.

OpenContrail, initially developed by Juniper Networks, is an open-source SDN platform offering comprehensive network capabilities for cloud environments. It provides a scalable and flexible network infrastructure that enables automation, network virtualization, and secure multi-tenancy across distributed cloud deployments.

 

Highlights: OpenContrail

  • The role of The VM

Virtual machines have been around for a long time, but we are beginning to spread our compute workloads in several different ways. When you throw in docker containers and bare metal servers, networking becomes more interesting. Network challenges arise when all these components require communication within the same subnet, access to Internet gateways, and Layer 3 MPLS/VPNs.

As a result, data center networks are moving towards IP underlay fabrics and Layer 2 overlays. Layer 3 data plane forwarding utilizes efficient Equal-cost multi-path routing (ECMP), but we lack Layer 2 multipathing by default. Now, similar to an SD WAN overlay approach, we can connect dispersed layer 2 segments and leverage all the good features of the IP underlay. To provide Layer 2 overlays and network virtualization, Juniper has introduced an SDN platform called Junipers OpenContrail in direct competition with

 

For additional pre-information, you may find the following post of use.

  1. ACI Cisco
  2. Network Traffic Engineering
  3. Spine Leaf Architecture
  4. IP Forwarding
  5. SDN Data Center
  6. Network Overlays
  7. Application Traffic Steering
  8. What is BGP Protocol in Networking

 



MPLS Overlay

Key OpenContrail Discussion Points:


  • Introduction to the OpenContrail solution and what is involved.

  • Highlighting data center networks and ECMP.

  • Critical points on network virtualization.

  • Technical details on the virtual overlay network.

  • Technical details virtual network implementation.

  • Layer 2 VPN and EVPN.

 

Back to Basics with OpenContrail

Key Features and Benefits:

Network Virtualization:

OpenContrail leverages network virtualization techniques to provide isolated virtual networks within a shared physical infrastructure. It offers a logical abstraction layer, enabling the creation of virtual networks that operate independently, complete with their own routing, security, and quality of service policies. This approach allows for the efficient utilization of resources, simplified network management, and improved scalability.

Secure Multi-Tenancy:

OpenContrail’s security features ensure tenants’ data and applications remain isolated and protected from unauthorized access. It employs micro-segmentation to enforce strict access control policies at the virtual machine level, reducing the risk of lateral movement within the network. Additionally, OpenContrail integrates with existing security solutions, enabling the implementation of comprehensive security measures.

Intelligent Automation:

OpenContrail automates various network provisioning, configuration, and management tasks, reducing manual intervention and minimizing human errors. Its programmable API and centralized control plane simplify the deployment of complex network topologies, accelerate service delivery, and enhance overall operational efficiency.

Scalability and Flexibility:

OpenContrail’s architecture is designed to scale seamlessly, supporting distributed cloud deployments across multiple locations. It offers a highly flexible solution that can adapt to changing network requirements, allowing administrators to dynamically allocate resources, establish new connectivity, and respond to evolving business needs.

OpenContrail in Practice:

OpenContrail has gained significant traction among cloud providers, service providers, and enterprises seeking to build robust, scalable, and secure networks. Its open-source nature has facilitated its adoption, which encourages collaboration, innovation, and customization. OpenContrail’s community-driven development model ensures continuous improvement and the availability of new features and enhancements.

opencontrail
Diagram: OpenContrail.

 

Highlighting Junipers OpenContrail

OpenContrail is an open-source network virtualization platform. The commercial controller and open-source product are identical; they share the same checksum on the binary image. Maintenance and support are the only difference. Juniper decided to open source to fit into the open ecosystem, which wouldn’t have worked in a closed environment.

OpenContrail offers similar features to VMware NSX and can apply service chaining and high-level security policies and provide the connection to Layer 3 VPNs for WAN integration. OpenContrail works with any hardware, but integration with Juniper’s product sets offers additional rich analytics for the underlay network.

Underlay and overlay network visibility are essential for troubleshooting. You need to look further than the first header of the packet; you need to look deeper into the tunnel to understand what is happening entirely. 

 

Network virtualization – Isolated networks

With a cloud architecture, network virtualization gives the illusion that each tenant has a separate isolated network. Virtual networks are independent of physical network location or state, and nodes within the physical underlay can fail without disrupting the overlay tenant. A tenant may be a customer or department, depending if it’s a public or private cloud.

The virtual network sits on top of a physical network, the same way the compute virtual machines sits on top of a physical server. Virtual networks are not created with VLANs; Contrail uses a system of the virtual overlay network for multi-tenancy and cross-tenant communication. Many problems exist with large-scale VLAN deployments for multi-tenancy in today’s networks.

They introduce a lot of states in the physical network, and the Spanning Tree Protocol (STP) also introduces well-documented problems. There are technologies (THRILL, SPB) to overcome these challenges, but they add complexity to the design of the network.

 

Service Chaining

Customers require the ability to apply policy at virtual network boundaries. Policies may include ACL and stateless firewalls provided within the virtual switch. But once you require complicated policy pieces between virtual networks, you need a more sophisticated version of policy control and orchestration called service chaining. Service chaining applies intelligent services between traffic from one tenant to another.

For example, if a customer requires content caching and stateful services, you must introduce additional service appliances and force next-hop traffic through these appliances. Once you deploy a virtual appliance, you need a scale-out architecture.

 

The ability to Scale-out

Scale-out is the ability to instantiate multiple physical and virtual machine instances and load balance traffic across them. Customers may also require the ability to connect with different tenants in dispersed geographic locations or connect to workloads in a remote private cloud or public cloud. Usually, people build a private cloud for the norm and then burst into a public cloud. 

Juniper has implemented a virtual networking architecture that meets these requirements. It is based on well-known technology, MPLS/layer 3 VPN. MPLS/layer 3 VPN is the base for Juniper designs.

MPLS Overlay

 

 

Junipers OpenContrail: Virtual Network Implementation 

MPLS Overlay

The SDN controller is responsible for the networking aspects of virtualization. When creating virtual networks, initiate the Northbound API and issue an instruction that attaches VM to the VN. The network responsibilities are delegated from Cloudstack or OpenStack to Contrail. The Contrail SDN controller automatically creates the overlay tunnel between virtual machines. The overlay can be either an MPLS overlay style with MPLS-over-GREMPLS-over-UDP, or VXLAN

 

  • L3VPN for routed traffic and EVPN for bridged traffic

Juniper’s OpenContrail is still a pure MPLS overlay of MPLS/VPN, using L3VPN for routed traffic and EVPN for bridged traffic. Traffic forwarding between end nodes has one MPLS label (VPN label), but they use various encapsulation methods to carry labeled traffic across the IP fabric. As mentioned above, this includes MPLS-over-GRE – a traditional encapsulation mechanism, MPLS-over-UDP – a variation of MPLS-over-GRE that replaces the GRE headers with UDP headers. And MPLS-over-VXLAN uses VXLAN packet format but stores the MPLS label in the Virtual Network Identifier (VNI) field.

 

  • The forwarding plane

The forwarding plane takes the packet from the VM and gives it to the “Vrouter,” which does a lookup and determines if the destination is a remote network. It encapsulates the packet and sends it across the tunnel if it is a remote network. The underlay that sites between the workloads forward are based on tunnel source and destination only.

No state belongs to end hosts ‘VMs, MAC addresses, or IPs. This type of architecture gives the Core a cleaner and more precise role. Generally, as a best practice, keeping “state” in the Core is a lousy design principle.

 

Northbound and southbound interfaces

To implement policy and service chaining, use the Northbound Interface and express your policy at a high level. For example, you may require HTTP or NAT and force traffic via load balancers or virtual firewalls. Contrail does this automatically and issues instructions to the Vrouter, forcing traffic to the correct virtual appliance. In addition, it can create all the right routes and tunnels, causing traffic through the correct sequence of virtual machines.

Contrail achieves this automatically with southbound protocols, such as XMPP (Extensible Messaging and Presence Protocol) or BGP. XMPP is a communications protocol based on XML (Extensible Markup Language).

 

WAN Integration

Junipers OpenContrail can connect virtual networks to external Layer 3 MPLS VPN for WAN integration. In addition, they gave the controller the ability to peer BGP to gateway routers. For the data plane, they support MPLS-over-GRE, and for the control plane, they speak MP- BGP.

Contrail communicates directly with PE routers, exchanging VPNv4 routes with MP-BGP and using MPLS-over-GRE encapsulation to pass IP traffic between hypervisor hosts and PE routers. Using standards-based protocols lets you choose any hardware appliance as the gateway node.

mpls overaly

 

This data and control plane makes integration to an MPLS/VPN backbone a simple task. First, establish MP-BGP between the controllers to PE-routers. Inter-AS Option B next hop self-approach should be used to establish some demarcation points.

 

Conclusion:

OpenContrail has emerged as a game-changer in software-defined networking, empowering organizations to build agile, secure, and scalable networks in the cloud era. With its advanced features, such as network virtualization, secure multi-tenancy, intelligent automation, and scalability, OpenContrail offers a comprehensive solution that addresses the complex networking challenges of modern cloud environments. As the demand for efficient and flexible network management continues to rise, OpenContrail provides a compelling option for organizations looking to optimize their network infrastructure and unlock the full potential of the cloud.

 

mpls overlay

 

Cisco switch virtualization

Cisco Switch Virtualization Nexus 1000v

 

Cisco switch virtualization

 

Nexus1000v

In today’s digital landscape, virtualization has become integral to modern data centers. With the increasing demand for agility, flexibility, and scalability, organizations are turning to virtual networking solutions to meet their evolving needs. One such solution is the Nexus 1000v, a virtual network switch offering comprehensive features and functionalities. In this blog post, we will delve into the world of the Nexus 1000v, exploring its key features, benefits, and use cases.

The Nexus 1000v is a distributed virtual switch that operates at the hypervisor level, providing advanced networking capabilities for virtual machines (VMs). It is designed to integrate seamlessly with VMware vSphere, offering enhanced network visibility, control, and security.

 

  • Virtual routing and forwarding

Virtual routing and forwarding form the basis of this stack. Firstly, network virtualization comes with two primary methods: 1) One too many and 2) Many to one.  The “one too many” network virtualization method means you segment one physical network into multiple logical segments. Conversely, the “many to one” network virtualization method consolidates multiple physical devices into one logical entity. By definition, they seem to be opposites, but they fall under the same umbrella in network virtualization.

 

Before you proceed, you may find the following posts helpful:

  1. Container Based Virtualization
  2. Virtual Switch
  3. What is VXLAN
  4. Redundant Links
  5. WAN Virtualization
  6. What Is FabricPath

 

Cisco Switch Virtualization.

Key Nexus 1000v Discussion Points:


  • Introduction to Nexus1000v and what is involved.

  • Highlighting the details on Cisco switch virtualization. Logical separation. 

  • Technical details on the additional overhead from virtualization.

  • Scenario: Network virtualization.

  • A final note on software virtual switch designs.

 

  • A key point: Video on network virtualization

In this video, we will address the basics of network virtualization. We have an underlay and an overlay design. Overlay/Underlay Essentially, an overlay is placing Layer 2 or 3 over a Layer 3 Core. The Layer 3 Core is known as the underlay.

This removes many drawbacks and scaling issues with traditional Layer 2 connectivity, which uses VLANs. The multi-tenant nature of overlays is designed to get away from these L2 challenges allowing you to build networks at a much larger scale.

 

Technology Brief : VXLAN - Introducing Overlay Networking
Prev 1 of 1 Next
Prev 1 of 1 Next

 

Back to basics with network virtualization

Before we get stuck in Cisco virtualization, let us address some basics. For example, if you’re going to have multiple virtual endpoints share a physical network, but different virtual endpoints belong to different customers, the communication between these virtual endpoints also needs to be isolated. In other words, the network is a resource, too, and network virtualization is the technology that enables sharing of a standard physical network infrastructure.

Virtualization uses software to simulate traditional hardware platforms and create virtual software-based systems. For example, virtualization allows specialists to construct a single virtual network or partition a physical network into multiple virtual networks.

 

Cisco Switch Virtualization

Logical segmentation: One too many

We have one-to-many network virtualization for the Cisco switch virtualization design; a single physical network is logically segmented into multiple virtual networks. For example, each virtual network could correspond to a user group or a specific security function.

End-to-end path isolation requires the virtualization of networking devices and their interconnecting links. VLANs have been traditionally used, and hosts from one user group are mapped to a single VLAN. To extend the path across multiple switches at Layer 2, VLAN tagging (802.1Q) can carry VLAN information between switches. These VLAN trunks were created to transport multiple VLANs over a single Ethernet interface.

The diagram below displays two independent VLANs, VLAN201 and VLAN101. These VLANs can share one physical wire to provide L2 reachability between hosts connected to Switch B and Switch A via Switch C. VLANs 201, and 101 remain separate entities.

Nexus1000v
Nexus1000v: The operation

 

VLANs are sufficient for small Layer 2 segments. However, today’s networks will likely have a mix of Layer 2 and 3 routed networks. In this case, Layer 2 VLANs alone are insufficient because you must extend the Layer 2 isolation over a Layer 3 device. This can be achieved by using Virtual Routing and Forwarding ( VRF ), the next step in the Cisco switch virtualization. A virtual routing and forwarding instance logically carves a Layer 3 device into several isolated independent L3 devices. The virtual routing and forwarding configured locally cannot communicate directly.

The diagram below displays one physical Layer 3 router with three VRFs, VRF Yellow, VRF Red, and VRF Blue. There is complete separation between these virtual routing and forwarding instances; without explicit configuration, routes in one virtual routing and forwarding cannot be leaked to another virtual routing and forwarding instance.

Virtual Routing and Forwarding

virtual routing and forwarding

The virtualization of the interconnecting links depends on how the virtual routers are connected. If they are physically ( directly ) connected, you could use a technology known as VRF-lite to separate traffic and 802.1Q to label the data plane. This is known as hop-by-hop virtualization. However, it’s possible to run into scalability issues when the number of devices grows. This design is typically used when you connect virtual routing and forwarding back to back, i.e., no more than two devices.

When the virtual routers are connected over multiple hops through an IP cloud, you can use generic routing encapsulation ( GRE ) or Multiprotocol Label Switching ( MPLS ) virtual private networks.

GRE is probably the simpler of the Layer 3 methods, and it can work over any IP core. GRE can encapsulate the contents and transport them over a network with the network unaware of the packet contents. Instead, the core will see the GRE header, virtualizing the network path.

 

 

 

Cisco Switch Virtualization: The additional overhead

When designing Cisco switch virtualization, you need to consider the additional overhead. There are additional 24 bytes overhead for the GRE header, so it may be the case that the forwarding router may break the datagram into two fragments, so the packet may not be larger than the outgoing interface MTU. To resolve the fragmentation issue, you can correctly configure MTU, MSS, and Path MTU parameters on the outgoing and intermediate routers.

The standard for GRE is typically static. You only need to configure tunnel endpoints, and the tunnel will be up as long as you have reachability to those endpoints. However, you can have a dynamic GRE tunnel establishment in recent designs.

MPLS/VPN, on the other hand, is a different beast. It requires signaling to distribute labels and build an end-to-end Label Switched Path ( LSP ). The label distribution can be done with BGP+label, LDP, and RSVP. Unlike GRE tunnels, MPLS VPNs do not have to manage multiple point-to-point tunnels to provide a full mesh of connectivity. Instead, they are used for connectivity, and packets’ labels provide traffic separation.

 

Cisco switch virtualization: Many to one

Many-to-one network consolidation refers to grouping two or more physical devices into one. Examples of this Cisco switch virtualization technology include a Virtual Switching System ( VSS ), Stackable switches, and Nexus VPC. Combining many physicals into one logical entity allows STP to view the logical group as one, allowing all ports to be active. By default, STP will block the redundant path.

Software-Defined Networking takes this concept further; it completely abstracts the entire network into a single virtual switch. On traditional routers, the control and the data plane are on the same device, yet the control and data planes are decoupled with SDN. The control plan is now on a policy-driven controller, and the data plane is local on the OpenFlow-enabled switch.

 

Network Virtualization

Server and network virtualization presented the challenge of multiple VMs sharing a single network physical port, such as a network interface controller ( NIC ). The question then arises, how do I link multiple VMs to the same uplink? How do I provide path separation? Today’s networks need to virtualize the physical port and allow the configuration of policies per port.

Nexus 1000

 

NIC-per-VM design

One way to do this is to have a NIC-per-VM design where each VM is assigned a single physical NIC, and the NIC is not shared with any other VM. The hypervisor, aka virtualization layer, would be bypassed, and the VM would access the I/O device directly. This is known as VMDirectPath. This direct path or pass-through can improve performance for hosts that utilize high-speed I/O devices, such as 10 Gigabit Ethernet. However, the lack of flexibility and the ability to move VMs offset higher performance benefits.  

 

Virtual-NIC-per-VM in Cisco UCS (Adapter FEX)

Another way to do this is to create multiple logical NICs on the same physical NIC, such as Virtual-NIC-per-VM in Cisco UCS (Adapter FEX). These logical NICs are assigned directly to VMs, and traffic gets marked with vNIC specific tag in hardware (VN-Tag/802.1ah). The actual VN-Tag tagging is implemented in the server NICs so you can clone the physical NIC in the server to multiple virtual NICs. This technology provides faster switching and enables you to apply a rich set of management features to local and remote traffic.

 

Software Virtual Switch

The 3rd option is to implement a virtual software switch in the hypervisor. For example, VMware introduced virtual switching compatibility with their vSphere ( ESXi ) hypervisor called vSphere Distributed Switch ( VDS ). Initially, they introduced a local L2 software switch, which was soon phased out due to a lack of distributed architecture.

Data physically move between the servers through the external network, but the control plane abstracts this movement to look like one large distributed switch spanning multiple servers. This approach has a single management and configuration point, similar to stackable switches – one control plane with many physical data forwarding paths. The data does not move through a parent partition but logically connects directly to the network interface through local vNICs associated with each VM.

 

Network virtualization and Nexus 1000v ( Nexus 1000 )

The VDS introduced by VMware lacked any good networking features, which led Cisco to introduce the Nexus 1000V software-based switch. The Nexus 1000v is a multi-cloud, multi-hypervisor, and multi-services distributed virtual switch. Its function is to enable communication between VMs.

Nexus1000v
Nexus1000v: Virtual Distributed Switch.

 

Nexus 1000 components: VEM and VSM

The Nexus 1000v has two essential components:

  1. The Virtual Supervisor Module ( VSM )
  2. The Virtual Ethernet Module ( VEM ).

Compared to a physical switch, you could view the VSM as the supervisor, setting up the control plane functions for the data plane to forward efficiently, and the VEM as the physical line cards that do all the forwarding of packets. The VEM is the software component that runs within the hypervisor kernel. It handles all VM traffic, including inter-VM frames and Ethernet traffic between a VM and external resources.

The VSM runs its NX-OS code and controls the control and management planes, which integrates into a cloud manager, such as a VMware vCenter. You can have two VSMs for redundancy. Both modules remain constantly synchronized with unicast VSM-to-VSM heartbeats to provide stateful failover in the event of an active VSM failure.

 

The two available communication options for VSM to VEM are:

  1. Layer 2 control mode: The VSM control interface shares the same VLAN with the VEM.
  2. Layer 3 control mode: The VEM and the VSM are in different IP subnets.

The VSM also uses heartbeat messages to detect loss of connectivity between the VSM and the VEM. However, the VEM does not depend on connectivity to the VSM to perform its data plane functions and will continue forwarding packets if VSM fails.

 

With Layer 3 control mode, the heartbeat messages are encapsulated in a GRE envelope.

 

Nexus 1000 and VSM best practices

  • L2 control is recommended for new installations.
  • Use MAC pinning instead of LACP.
  • Packet, Control, and Management in the same VLAN.
  • Do not use VLAN 1 for Control and Packet.
  • Use 2 x VSM for redundancy. 

The max latency between VSM and VEM is ten milliseconds. Therefore if you have a high-quality DCI link, a VSM can be placed outside the data center and still control the VEM.

 

Nexus 1000v InterCloud – Cisco switch virtualization

A vital element of the Nexus 1000 is its use case for hybrid cloud deployments and its ability to place workloads in private and public environments via a single pane of glass. In addition, the Nexus 1000v interCloud addresses the main challenges with hybrid cloud deployments, such as security concerns and control/visibility challenges within the public cloud.

The Nexus 1000 interCloud works with Cisco Prime Service Controller to create a secure L2 extension between the private data center and the public cloud.

This L2 extension is based on Datagram Transport Layer Security ( DTLS ) protocol and allows you to securely transfer VMs and Network services over a public IP backbone. DTLS derives the SSL protocol and provides communications privacy for datagram protocols, so all data in motion is cryptographically isolated and encrypted.

Nexus 1000
Nexus 1000 and Hybrid Cloud.

 

Nexus 1000v Hybrid Cloud Components 

Cisco Prime Network Service Controller for InterCloud ** A VM that provides a single pane of glass to manage all functions of the inter clouds
InterCloud VSM Manage port profiles for VMs in the InterCloud infrastructure
InterCloud Extender Provides secure connectivity to the InterCloud Switch in the provider cloud. Install in the private data center.
InterCloud Switch Virtual Machine in the provider data center has secure connectivity to the InterCloud Extender in the enterprise cloud and secure connectivity to the Virtual Machines in the provider cloud.
Cloud Virtual Machines VMs in the public cloud running workloads.

 

Prerequisites

Port 80 HTTP access from PNSC for AWS calls and communicating with InterCloud VMs in the provider cloud
Port 443 HTTPS access from PNSC for AWS calls and communicating with InterCloud VMs in the provider cloud
Port 22 SSH from PNSC to InterCloud VMs in the provider cloud
UDP 6644 DTLS data tunnel
TCP 6644 DTLS control tunnel

 

VXLAN – Virtual Extensible LAN

The requirement for applications on demand has led to an increased number of required VLANs for cloud providers. The standard 12-bit identifier, which provided 4000 VLANs, proved to be a limiting factor in multi-tier, multi-tenant environments, and engineers started to run out of isolation options.

This has introduced a 24-bit VXLAN identifier, offering 16 million logical networks. Now we can cross Layer 3 boundaries. The MAC in UDP encapsulation uses switch hashing to look into UDP packets and distribute all packets in a port channel efficiently.

nexus 1000
VXLAN operations

 

VXLAN works like a layer 2 bridge ( Flood and Learn ); the VEM learn does all the heavy lifting, learns all the VM source MAC and Host VXLAN IP, and encapsulates the traffic according to the port profile to which the VM belongs. Broadcast, Multicast, and unknown unicast traffic are sent as Multicast.

At the same time, unicast traffic is encapsulated and shipped directly to destination hosts VXLAN IP, aka destination VEM. Enhanced VXLAN offers VXLAN MAC distribution and ARP termination making it more optional.

 

  • A key point: Video on VXLAN and Dynamic MAC learning

The following video discusses dynamic MAC learning. Initially, Ethernet started with a Thick Coax Cable – a single cable was used to connect all workstations. It was later replaced by the twisted pair cables – unshielded UTP and shielded twisted pair STP in the late 1990-2000s. On Ethernet networks, each host has a unique MAC address for identification.

 

Technology Brief : VXLAN - Dynamic MAC Learning
Prev 1 of 1 Next
Prev 1 of 1 Next

 

 

VXLAN Mode Packet Functions

Packet VXLAN(multicast mode) Enhanced VXLAN(unicast mode) Enhanced VXLANMAC Distribution Enhanced VXLANARP Termination
Broadcast /Multicast Multicast Encapsulation Replication plus Unicast Encap Replication plus Unicast Encap Replication plus Unicast Encap
Unknown Unicast Multicast Encapsulation Replication plus Unicast Encap Drop Drop
Known Unicast Unicast Encapsulation Unicast Encap Unicast Encap Unicast Encap
ARP Multicast Encapsulation Replication plus Unicast Encap Replication plus Unicast Encap VEM ARP Reply

 

vPath – Service chaining

Intelligent Policy-based traffic steering through multiple network services.

vPath allows you to intelligently traffic steer VM traffic to virtualized devices. It intercepts and redirects the initial traffic to the service node. Once the service node performs its policy function, the result is cached, and the local virtual switch treats the subsequent packets accordingly. In addition, it enables you to tie services together to push the VM through each service as required. Previously, if you wanted to tie services together in a data center, you needed to stitch the VLANs together, which was limited by design and scale.

Cisco virtualization
Nexus and service chaining

 

vPath 3.0 is now submitted to the IETF for standardization, allowing service chaining with vPath and non-vpath network services. It enables you to use vpath service chaining between multiple physical devices and supporting multiple hypervisors.

 

License Options 

Nexus 1000 Essential Edition Nexus 1000 Advanced Edition
Full Layer-2 Feature Set All Features of Essential Edition
Security, QoS Policies VSG firewall
VXLAN virtual overlays VXLAN Gateway
vPath enabled Virtual Services TrustSec SGA
Full monitoring and management capabilities A platform for other Cisco DC Extensions in the Future
Free $695 per CPU MSRP

 

Nexus 1000 features and benefits

Switching L2 Switching, 802.1Q Tagging, VLAN, Rate Limiting (TX), VXLAN
IGMP Snooping, QoS Marking (COS & DSCP), Class-based WFQ
Security Policy Mobility, Private VLANs w/ local PVLAN Enforcement
Access Control Lists, Port Security, Cisco TrustSec Support
Dynamic ARP inspection, IP Source Guard, DHCP Snooping
Network Services Virtual Services Datapath (vPath) support for traffic steering & fast-path off-load[leveraged by Virtual Security Gateway (VSG), vWAAS, ASA1000V]
Provisioning Port Profiles, Integration with vC, vCD, SCVMM*, BMC CLM
Optimized NIC Teaming with Virtual Port Channel – Host Mode
Visibility VM Migration Tracking, VC Plugin, NetFlow v.9 w/ NDE, CDP v.2
VM-Level Interface Statistics, vTrackerSPAN & ERSPAN (policy-based)
Management Virtual Centre VM Provisioning, vCenter Plugin, Cisco LMS, DCNM
Cisco CLI, Radius, TACACs, Syslog, SNMP (v.1, 2, 3)
Hitless upgrade, SW Installer

 

Advantages and disadvantages of the Nexus 1000

Advantages Disadvantages
The Standard edition is FREE; you can upgrade to an enhanced version when needed. VEM and VSM internal communication is very sensitive to latency. Due to their chatty nature, they may not be good for inter-DC deployments.
Easy and Quick to deploy VSM – VEM, VSM (active) – VSM (standby) heartbeat time of 6 seconds makes it sensitive to network failures and congestion.
It offers you many rich network features unavailable on other distributed software switches. VEM over-dependency on VSM reduces resiliency.
Hypervisor agnostic VSM is required for vSphere HA, FT, and VMotion to work.
Hybrid Cloud functionality  

 

Closing Points on Cisco Nexus 1000v

Key Features and Functionalities:

Virtual Ethernet Module (VEM):

The Nexus 1000v employs the concept of the Virtual Ethernet Module (VEM), which runs as a module inside the hypervisor. This allows for efficient and direct communication between VMs, bypassing the traditional reliance on the hypervisor networking stack.

Virtual Supervisor Module (VSM):

The Virtual Supervisor Module (VSM) serves as the control plane for the Nexus 1000v, providing centralized management and configuration. It enables network administrators to define policies, manage virtual ports, and monitor network traffic.

Policy-Based Virtual Network Management:

With the Nexus 1000v, administrators can define policies to manage virtual networks. These policies ensure consistent network configurations across multiple hosts, simplifying network management and reducing the risk of misconfigurations.

Advanced Security and Monitoring Capabilities:

The Nexus 1000v offers granular security controls, including access control lists (ACLs), port security, and dynamic host configuration protocol (DHCP) snooping. Additionally, it provides comprehensive visibility into network traffic, enabling administrators to monitor and troubleshoot network issues effectively.

Benefits of the Nexus 1000v:

Enhanced Network Performance:

By offloading network processing to the VEM, the Nexus 1000v minimizes the impact on the hypervisor, resulting in improved network performance and reduced latency.

Increased Scalability:

The distributed architecture of the Nexus 1000v allows for seamless scalability, ensuring that organizations can meet the growing demands of their virtualized environments.

Simplified Network Management:

With its policy-based approach, the Nexus 1000v simplifies network management tasks, enabling administrators to provision and manage virtual networks more efficiently.

Use Cases:

Data Centers:

The Nexus 1000v is particularly beneficial in data center environments where virtualization is prevalent. It provides a robust and scalable networking solution, ensuring optimal performance and security for virtualized workloads.

Cloud Service Providers:

Cloud service providers can leverage the Nexus 1000v to enhance their network virtualization capabilities, offering customers more flexibility and control over their virtual networks.

The Nexus 1000v is a powerful virtual network switch that provides advanced networking capabilities for virtualized environments. Its rich set of features, policy-based management approach, and seamless integration with VMware vSphere allows organizations to achieve enhanced network performance, scalability, and management efficiency. As virtualization continues to shape the future of data centers, the Nexus 1000v remains a valuable tool for optimizing virtual network infrastructures.

 

Cisco switch virtualization

bus network topology LAN design networking hardware backbone connected vector

Data Center Topologies

Data Center Topology

In the world of technology, data centers play a crucial role in storing, managing, and processing vast amounts of digital information. However, behind the scenes, a complex infrastructure known as data center topology enables seamless data flow and optimal performance. In this blog post, we will delve into the intricacies of data center topology, its different types, and how it impacts the efficiency and reliability of data centers.

Data center topology refers to a data center's physical and logical layout. It encompasses the arrangement and interconnection of various components like servers, storage devices, networking equipment, and power sources. A well-designed topology ensures high availability, scalability, and fault tolerance while minimizing latency and downtime.

Table of Contents

Highlights: Data Center Topoloigy

Choosing a topology

Data centers are the backbone of many businesses, providing the necessary infrastructure to store and manage data and access applications and services. As such, it is essential to understand the different types of available data center topologies. When choosing a topology for a data center, it is necessary to consider the organization’s specific needs and requirements. Each topology offers its advantages and disadvantages, so it is crucial to understand the pros and cons of each before making a decision.

Data Center Topology

A data center topology refers to the physical layout and interconnection of network devices within a data center. It determines how servers, switches, routers, and other networking equipment are connected, ensuring efficient and reliable data transmission. Topologies are based on scalability, fault tolerance, performance, and cost.

Scalability of the topology

Additionally, it is essential to consider the topology’s scalability, as a data center may need to accommodate future growth. By understanding the different topologies and their respective strengths and weaknesses, organizations can make the best decision for their data centers.

Typical data center topologies

Typical data center topologies connect end hosts to the top rack ( ToR ) switches, typically using 1GigE or 10GigE links. These ToR/access switches contain several end-host ports, usually 48GigE, to physically connect the end stations.

Because this layer has many ports, its configuration aims for simplicity and ease of management. The ToR also has several 10GigE or 40GigE uplink ports to connect to an upstream device. Depending on the data center network topology, these ToR switches sometimes connect to one or more end-of-row ( EoR ) switches, resulting in different data center topology types.

The design of the data center topology is to provide rich connectivity among the ToR switches so that all application and end-user requirements are satisfied. The diagrams below display the ToR and EoR server connectivity models commonly seen in the SDN data center.

Related: For pre-information, you may find the following post helpful

  1. ACI Cisco
  2. Virtual Switch
  3. Ansible Architecture
  4. Overlay Virtual Networks



Data Center Network Topology

Key Data Center Topologies Discussion Points:


  • End of Row and Top of Rack designs.

  • The use of Fabric Extenders.

  • Layer 2 or Layer 3 to the Core.

  • The rise of Network Virtualization.

  • VXLAN transports.

  • The Cisco ACI and ACI Network.

Back to Basics: Data Center Network Topology

A data center is a physical facility that houses critical applications and data for an organization. It consists of a network of computing and storage resources that support shared applications and data delivery. The components of a data center are routers, switches, firewalls, storage systems, servers, and application delivery controllers.

Enterprise IT data centers support the following business applications and activities:

  • Email and file sharing
  • Productivity applications
  • Customer relationship management (CRM)
  • Enterprise resource planning (ERP) and databases
  • Big data, artificial intelligence, and machine learning
  • Virtual desktops, communications, and collaboration services

A data center consists of the following core infrastructure components:

  • Network infrastructure: Connects physical and virtual servers, data center services, storage, and external connections to end users.
  • Storage Infrastructure: Modern data centers use storage infrastructure to power their operations. Storage systems hold this valuable commodity.
  • A data center’s computing infrastructure is its applications. The computing infrastructure comprises servers that provide processors, memory, local storage, and application network connectivity. In the last 65 years, computing infrastructure has undergone three major waves:
    • In the first wave of replacements of proprietary mainframes, x86-based servers were installed on-premises and managed by internal IT teams.
    • In the second wave, application infrastructure was widely virtualized. The result was improved resource utilization and workload mobility across physical infrastructure pools.
    • The third wave finds us in the present, where we see the move to the cloud, hybrid cloud, and cloud-native (that is, applications born in the cloud).

Common Types of Data Center Topologies:

a) Bus Topology: In this traditional topology, all devices are connected linearly to a common backbone, resembling a bus. While it is simple and cost-effective, a single point of failure can disrupt the entire network.

b) Star Topology: In a star topology, each device is connected directly to a central switch or hub. This design offers centralized control and easy troubleshooting, but it can be expensive due to the requirement of additional cabling.

c) Mesh Topology: A mesh topology provides redundant connections between devices, forming a network where every device is connected to every other device. This design ensures high fault tolerance and scalability but can be complex and costly.

d) Hybrid Topology: As the name suggests, a hybrid topology combines elements of different topologies to meet specific requirements. It offers flexibility and allows organizations to optimize their infrastructure based on their unique needs.

Considerations in Data Center Topology Design:

a) Redundancy: Redundancy is essential to ensure continuous operation even during component failures. By implementing redundant paths, power sources, and network links, data centers can minimize the risk of downtime and data loss.

b) Scalability: As the data center’s requirements grow, the topology should be able to accommodate additional devices and increased data traffic. Scalability can be achieved through modular designs, virtualization, and flexible network architectures.

c) Performance and Latency: The distance between devices, the quality of network connections, and the efficiency of routing protocols significantly impact data center performance and latency. Optimal topology design considers these factors to minimize delays and ensure smooth data transmission.

Impact of Data Center Topology:

Efficient data center topology directly influences the entire infrastructure’s reliability, availability, and performance. A well-designed topology reduces single points of failure, enables load balancing, enhances fault tolerance, and optimizes data flow. It directly impacts the user experience, especially for cloud-based services, where data centers simultaneously cater to many users.

Data Center Topology

Main Data Center Topology Components

Data Center Topology

  • You need to understanding the different topologies and their respective strengths and weaknesses.

  • Rich connectivity among the ToR switches so that all application and end-user requirements are satisfied

  • A well-designed topology reduces single points of failure.

  • Example: Bus, star, mesh, and hybrid topologies

The Role of Networks

A network lives to serve the connectivity requirements of applications and applications. We build networks by designing and implementing data centers. A common trend is that the data center topology is much bigger than a decade ago, with application requirements considerably different from the traditional client–server applications and with deployment speeds in seconds instead of days. This changes how networks and your chosen data center topology are designed and deployed.

The traditional network design was scaled to support more devices by deploying larger switches (and routers). This is the scale-in model of scaling. However, these large switches are expensive and primarily designed to support only a two-way redundancy.

Today, data center topologies are built to scale out. They must satisfy the three main characteristics of increasing server-to-server traffic, scale ( scale on-demand ), and resilience. The following diagram shows a ToR design we discussed at the start of the blog.

Top of Rack (ToR)
Diagram: Data center network topology. Top of Rack (ToR).

The Role of The ToR

Top of rack (ToR) is a term used to describe the architecture of a data center. It is a server architecture in which servers, switches, and other equipment are mounted on the same rack. This allows for the most efficient use of space since the equipment is all within arm’s reach.

ToR is also the most efficient way to manage power and cooling since the equipment is all in the same area. Since all the equipment is close together, ToR also allows faster access times. This architecture can also be utilized in other areas, such as telecommunications, security, and surveillance.

ToR is a great way to maximize efficiency in any data center and is becoming increasingly popular. In contrast to the ToR data center design, the following diagram shows an EoR switch design.

End of Row (EoR)
Diagram: Data center network topology. End of Row (EoR).

The Role of The EoR

The term end-of-row (EoR) design is derived from a dedicated networking rack or cabinet placed at either end of a row of servers to provide network connectivity to the servers within that row. In EoR network design, each server in the rack has a direct connection with the end-of-row aggregation switch, eliminating the need to connect servers directly with the in-rack switch.

Racks are usually arranged to form a row; a cabinet or rack is positioned at the end of this row. This rack has a row aggregation switch, which provides network connectivity to servers mounted in individual racks. This switch, a modular chassis-based platform, sometimes supports hundreds of server connections. However, a large amount of cabling is required to support this architecture.

Data center topology types
Diagram: ToR and EoR. Source. FS Community.

A ToR configuration requires one switch per rack, resulting in higher power consumption and operational costs. Moreover, unused ports are often more significant in this scenario than with an EoR arrangement.

On the other hand, ToR’s cabling requirements are much lower than those of EoR, and faults are primarily isolated to a particular rack, thus improving the data center’s fault tolerance.

If fault tolerance is the ultimate goal, ToR is the better choice, but EoR configuration is better if an organization wants to save on operational costs. The following table lists the differences between a ToR and an EoR data center design.

data center network topology
Diagram: Data center network topology. The differences. Source FS Community

Data Center Topology Types:

Fabric extenders – FEX

Cisco has introduced the concept of Fabric Extenders, which are not Ethernet switches but remote line cards of a virtualized modular chassis ( parent switch ). This allows scalable topologies previously impossible with traditional Ethernet switches in the access layer.

You should relate an FEX device like a remote line card attached to a parent switch. All the configuration is done on the parent switch, yet physically, the fabric extender could be in a different location. The mapping between the parent switch and the FEX ( fabric extender ) is done via a special VN-Link.

The following diagram shows an example of a FEX in a standard data center network topology. More specifically, we are looking at the Nexus 2000 FEX Series. Cisco Nexus 2000 Series Fabric Extenders (FEX) are based on the standard IEEE 802.1BR. They deliver fabric extensibility with a single point of management.

Cisco FEX
Diagram: Cisco FEX design. Source Cisco.

Different types of Fex solution

FEXs come with various connectivity solutions, including 100 Megabit Ethernet, 1 Gigabit Ethernet, 10 Gigabit Ethernet ( copper and fiber ), and 40 Gigabit Ethernet. They can be synchronized with the following models of parent switches – Nexus 5000, Nexus 6000, Nexus 7000, Nexus 9000, and Cisco UCS Fabric Interconnect.

In addition, because of the simplicity of FEX, they have very low latency ( as low as 500 nanoseconds ) compared to traditional Ethernet switches.

Data Center design
Diagram: Data center fabric extenders.

Some network switches can be connected to others and operate as a single unit. These configurations are called “stacks” and are helpful for quickly increasing the capacity of a network. A stack is a network solution composed of two or more stackable switches. Switches that are part of a stack behave as one single device.

Traditional switches like the 3750s still stand in the data center network topology access layer and can be used with stacking technology, combining two physical switches into one logical switch.

This stacking technology allows you to build a highly resilient switching system, one switch at a time. If you are looking at a standard access layer switch like the 3750s, consider the next-generation Catalyst 3850 series.

The 3850 supports BYOD/mobility and offers a variety of performance and security enhancements to previous models. The drawback of stacking is that you can only stack several switches. So, if you want additional throughout, you should aim for a different design type.

Data Center Design: Layer 2 and Layer 3 Solutions

Traditional views of data center design

Depending on the data center network topology deployed, packet forwarding at the access layer can be either Layer 2 or Layer 3. A Layer 3 approach would involve additional management and configuring IP addresses on hosts in a hierarchical fashion that matches the switch’s assigned IP address.

An alternative approach is to use Layer 2, which has less overhead as Layer 2 MAC addresses do not need specific configuration. However, it has drawbacks with scalability and poor performance.

Generally, access switches focus on communicating servers in the same IP subnet, allowing any type of traffic – unicast, multicast, or broadcast. You can, however, have filtering devices such as a Virtual Security Gateway ( VSG ) to permit traffic between servers, but that is generally reserved for inter-POD ( Platform Optimized Design ) traffic.

Lab Guide: IGMPv1

In the following example, we have a lab guide on IGMPv1.I GMPv1, or Internet Group Management Protocol Version 1, is a network-layer protocol designed to facilitate host communication and actively manage multicast group memberships.

It enables hosts to join and leave multicast groups, allowing them to receive IP multicast traffic from a specific source. Notice the output from the packet captures below for the Membership Query and the Membership Report.

IGMPv1
Diagram: IGMPv1

Leaf and Spine With Layer 3

We use a leaf and spine data center design with Layer 3 everywhere and overlay networking. This modern, robust architecture provides a high-performance, highly available network. With this architecture, data center networks are composed of leaf switches that connect to one or more spine switches.

The leaf switches are connected to end devices such as servers, storage devices, and other networking equipment. The spine switches, meanwhile, act as the network’s backbone, connecting the multiple leaf switches.

The leaf and spine architecture provides several advantages over traditional data center networks. It allows for greater scalability, as additional leaf switches can be easily added to the network. It also offers better fault tolerance, as the network can operate even if one of the spine switches fails.

Furthermore, it enables faster traffic flows, as the spine switches to route traffic between the leaf switches faster than a traditional flat network.

leaf and spine

Data Center Traffic Flow

Datacenter topologies can have North-South or East-to-West traffic. North-south ( up / down ) corresponds to traffic between the servers and the external world ( outside the data center ). East-to-west corresponds to internal server communication, i.e., traffic does not leave the data center.

Therefore, determining the type of traffic upfront is essential as it influences the type of topology used in the data center.

data center traffic flow
Diagram: Data center traffic flow.

For example, you may have a pair of ISCSI switches, and all traffic is internal between the servers. In this case, you would need high-bandwidth inter-switch links. Usually, an ether channel supports all the cross-server talk; the only north-to-south traffic would be management traffic.

In another part of the data center, you may have data server farm switches with only HSRP heartbeat traffic across the inter-switch links and large bundled uplinks for a high volume of north-to-south traffic. Depending on the type of application, which can be either outward-facing or internal, computation will influence the type of traffic that will be dominant. 

Virtual Machine and Containers.

This drive was from virtualization, virtual machines, and container technologies regarding east-west traffic. Many are moving to a leaf and spine data center design if they have a lot of east-to-west traffic and want better performance.

Network Virtualization and VXLAN

Network virtualization and the ability of a physical server to host many VMs and move those VMs are also used extensively in data centers, either for workload distribution or business continuity. This will also affect the design you have at the access layer.

For example, in a Layer 3 fabric, migrating a VM across that boundary changes its IP address, resulting in a reset of the TCP sessions because, unlike SCTP, TCP does not support dynamic address configuration. In a Layer 2 fabric, migrating a VM incurs ARP overhead and requires forwarding on millions of flat MAC addresses, which leads to MAC scalability and poor performance problems.

2nd Lab Guide: VXLAN

The following lab guide displays a VXLAN network. We are running VXLAN in unicast mode. VXLAN can also be configured to run in multicast mode. In the screenshot below, we have created a Layer 2 overlay across a routed Layer 3 core. The command: Show nve interface nve 1 displays an operational tunnel with the encapsulation set to VXLAN.

The screenshot shows a ping test from the desktops that connect to a Layer 3 port on the Leafs.

VXLAN overlay
Diagram: VXLAN Overlay

VXLAN: stability over Layer 3 core

Network virtualization plays a vital role in the data center. Technologies like VXLAN attempt to move the control plane from the core to the edge and stabilize the core so that it only has a handful of addresses for each ToR switch. The following diagram shows the ACI networks with VXLAN as the overlay that operates over a spine leaf architecture.

Layer 2 and 3 traffic is mapped to VXLAN VNIs that run over a Layer 3 core. The Bridge Domain is for layer 2, and the VRF is for layer 3 traffic. Now, we have the separation of layer 2 and 3 traffic based on the VNI in the VXLAN header.  

One of the first notable differences between VXLAN and VLAN was scale. VLAN has a 12-bit identifier called VID, while VXLAN has a 24-bit identifier called a VID network identifier. This means that with VLAN, you can create only 4094 networks over ethernet, while with VXLAN, you can create up to 16 million.

ACI network
Diagram: ACI network.

Whether you can build layer 2 or layer 3 in the access and use VXLAN or some other overlay to stabilize the core, it would help if you modularized the data center. The first step is to build each POD or rack as a complete unit. Each POD will be able to perform all its functions within that POD.

  • A key point: A POD data center design

POD: It is a design methodology that aims to simplify, speed deployment, optimize utilization of resources, and drive the interoperability of the three or more data center components: server, storage, and networks.

A POD example: Data center modularity

For example, one POD might be a specific human resources system. The second is modularity based on the type of resources offered. For example, a storage pod or bare metal compute may be housed in separate pods.

These two modularization types allow designers to easily control inter-POD traffic with predefined policies. Operators can also upgrade PODs and a specific type of service at once without affecting other PODs.

However, this type of segmentation does not address the scale requirements of the data center. Even when we have adequately modularized the data center into specific portions, the MAC table sizes on each switch still increase exponentially as the data center grows.

Current and Future Design Factors

New technologies with scalable control planes must be introduced for a cloud-enabled data center, and these new control planes should offer the following:

Option

Data Center Feature

Data center feature 1

The ability to scale MAC addresses

Data center feature 2

First-Hop Redundancy Protocol ( FHRP ) multipathing and Anycast HSRP

Data center feature 3

Equal-Cost multipathing

Data center feature 4

MAC learning optimizations

When designing a data center, several design factors need to be taken into account. First, what is the growth rate for servers, switch ports, and data center customers? This prevents part of the network topology from becoming a bottleneck or linking congested.

Application bandwidth demand?

This demand is usually translated into oversubscription. In data center networking, oversubscription refers to how much bandwidth switches are offered to downstream devices at each layer.

Oversubscription is expected in a data center design. By limiting oversubscription to the ToR and edge of the network, you offer a single place to start when you experience performance problems.

A data center with no oversubscription ratio will be costly, especially with a low latency network design. So, it’s best to determine what oversubscription ratio your applications support and work best. Optimizing your switch buffers to improve performance is recommended before you decide on a 1:1 oversubscription rate.

Ethernet 6-byte MAC addressing is flat.

Ethernet forms the basis of data center networking in tandem with IP. Since its inception 40 years ago, Ethernet frames have been transmitted over various physical media, even barbed wire. Ethernet 6-byte MAC addressing is flat; the manufacturer typically assigns the address without considering its location.

Ethernet-switched networks do not have explicit routing protocols to ensure readability about the flat addresses of the server’s NICs. Instead, flooding and address learning are used to create forwarding table entries.

IP addressing is a hierarchy.

On the other hand, IP addressing is a hierarchy, meaning that its address is assigned by the network operator based on its location in the network. A hierarchy address space advantage is that forwarding tables can be aggregated. If summarization or other routing techniques are employed, changes in one side of the network will not necessarily affect other areas.

This makes IP-routed networks more scalable than Ethernet-switched networks. IP-routed networks also offer ECMP techniques that enable networks to use parallel links between nodes without spanning tree disabling one of those links. The ECMP method hashes packet headers before selecting a bundled link to avoid out-of-sequence packets within individual flows. 

Equal Cost Load Balancing

Equal-cost load balancing is a method for distributing network traffic among multiple paths of equal cost. It provides redundancy and increases throughput. Sending traffic over multiple paths avoids congestion on any single link. In addition, the load is equally distributed across the paths, meaning that each path carries roughly the same total traffic.

This allows for using multiple paths at a lower cost, providing an efficient way to increase throughput.

The idea behind equal cost load balancing is to use multiple paths of equal cost to balance the load on each path. The algorithm considers the number of paths, each path’s weight, and each path’s capacity. It also feels the number of packets that must be sent and the delay allowed for each packet.

Considering these factors, it can calculate the best way to distribute the load among the paths.

Equal-cost load balancing can be implemented using various methods. One method is to use a Link Aggregation Protocol (LACP), which allows the network to use multiple links and distribute the traffic among the links in a balanced way.

ecmp
Diagam: ECMP 5 Tuple hash. Source: Keysight
  • A keynote: Data center topologies. The move to VXLAN.

Given the above considerations, a solution that encompasses the benefits of L2’s plug-and-play flat addressing and the scalability of IP is needed. Location-Identifier Split Protocol ( LISP ) has a set of solutions that use hierarchical addresses as locators in the core and flat addresses as identifiers in the edges. However, not much is seen in its deployment these days.

Equivalent approaches such as THRILL and Cisco FabricPath create massive scalable L2 multipath networks with equidistant endpoints. Tunneling is also being used to extend down to the server and access layer to overcome the 4K limitation with traditional VLANs. What is VXLAN? Tunneling with VXLAN is now the standard design in most data center topologies with leaf-spine designs. The following video provides VXLAN guidance.

 

Data Center Network Topology

Leaf and spine data center topology types

This is commonly seen in a leaf and spine design. For example, in a leaf-spine fabric, We have a Layer 3 IP fabric that supports equal-cost multi-path (ECMP) routing between any two endpoints in the network. Then, on top of the Layer 3 fabric is an overlay protocol, commonly VXLAN.

A spine-leaf architecture consists of a data center network topology of two switching layers—a spine and a leaf. The leaf layer comprises access switches that aggregate traffic from endpoints such as the servers and connect directly to the spine or network core.

Spine switches interconnect all leaf switches in a full-mesh topology. The leaf switches do not directly connect. The Cisco ACI is a data center topology that utilizes the leaf and spine.

The ACI network’s physical topology is a leaf and spine, while the logical topology is formed with VXLAN. From a protocol side point, VXLAN is the overlay network, and the BGP and IS-IS provide the Layer 3 routing, the underlay network that allows the overlay network to function.

As a result, the nonblocking architecture performs much better than the traditional data center design based on access, distribution, and core designs.

Cisco ACI
Diagram: Data center topology types and the leaf and spine with Cisco ACI

Closing Points: Data Center Topologies

A data center topology refers to the physical layout and interconnection of network devices within a data center. It determines how servers, switches, routers, and other networking equipment are connected, ensuring efficient and reliable data transmission. Topologies are based on scalability, fault tolerance, performance, and cost.

  • Hierarchical Data Center Topology:

The hierarchical or tree topology is one of the most commonly used data center topologies. This design consists of multiple core, distribution, and access layers. The core layer connects all the distribution layers, while the distribution layer connects to the access layer. This structure enables better management, scalability, and fault tolerance by segregating traffic and minimizing network congestion.

  • Mesh Data Center Topology:

Every network device is interlinked in a mesh topology, forming a fully connected network with multiple paths for data transmission. This redundancy ensures high availability and fault tolerance. However, this topology can be cost-prohibitive and complex, especially in large-scale data centers.

  • Leaf-Spine Data Center Topology:

The leaf-spine topology is gaining popularity due to its scalability and simplicity. It consists of interconnected leaf switches at the access layer and spine switches at the core layer. This design allows for non-blocking, low-latency communication between any leaf switch and spine switch, making it suitable for modern data center requirements.

  • Full-Mesh Data Center Topology:

As the name suggests, the full-mesh topology connects every network device to every other device, creating an extensive web of connections. This topology offers maximum redundancy and fault tolerance. However, it can be expensive to implement and maintain, making it more suitable for critical applications with stringent uptime requirements.

Summary: Data Center Topoloigy

Data centers are vital in supporting and enabling our digital infrastructure in today’s interconnected world. Behind the scenes, intricate network topologies ensure seamless data flow, allowing us to access information and services easily. In this blog post, we dived into the world of data center topologies, unraveling their complexities and understanding their significance.

Section 1: Understanding Data Center Topologies

Datacenter topologies refer to a data center’s physical and logical layout of networking components. These topologies determine how data flows between servers, switches, routers, and other network devices. By carefully designing the topology, data center operators can optimize performance, scalability, redundancy, and fault tolerance.

Section 2: Common Data Center Topologies

There are several widely adopted data center topologies, each with its strengths and use cases. Let’s explore some of the most common ones:

2.1. Tree Topology:

Tree topology, or hierarchical topology, is widely used in data centers. It features a hierarchical structure with multiple layers of switches, forming a tree-like network. This topology offers scalability and ease of management, making it suitable for large-scale deployments.

2.2. Mesh Topology:

The mesh topology provides a high level of redundancy and fault tolerance. In this topology, every device is connected to every other device, forming a fully interconnected network. While it offers robustness, it can be complex and costly to implement.

2.3. Spine-Leaf Topology:

The spine-leaf topology, also known as a Clos network, has recently gained popularity. It consists of leaf switches connecting to multiple spine switches, forming a non-blocking fabric. This design allows for efficient east-west traffic flow and simplified scalability.

Section 3: Factors Influencing Topology Selection

Choosing the right data center topology depends on various factors, including:

3.1. Scalability:

It is crucial for a topology to accommodate a data center’s growth. Scalable topologies ensure that additional devices can be seamlessly added without causing bottlenecks or performance degradation.

3.2. Redundancy and Fault Tolerance:

Data centers require high availability to minimize downtime. Topologies that offer redundancy and fault tolerance mechanisms, such as link and device redundancy, are crucial in ensuring uninterrupted operations.

3.3. Traffic Patterns:

Understanding the traffic patterns within a data center is essential for selecting an appropriate topology. Some topologies excel in handling east-west traffic, while others are better suited for north-south traffic flow.

Conclusion:

Datacenter topologies form the backbone of our digital infrastructure, providing the connectivity and reliability needed for our ever-expanding digital needs. By understanding the intricacies of these topologies, we can better appreciate the complexity involved in keeping our data flowing seamlessly. Whether it’s the hierarchical tree, the fully interconnected mesh, or the efficient spine-leaf, each topology has its place in the world of data centers.