Consequences of Network Overlays and Tunnels


There has been a major paradigm shift with data center networks. This evolution has driven network overlays which has brought a number of new requirements to data center designs.

Distributed applications are transforming traffic profiles and there is now a rapid rise in the intra-DC traffic ( East-West ). To support this type of scale we as designers are faced with a number challenges. For large cloud deployment there is no other choice but to implement some kind of network virtualization. If a customer requires a logical segment per application and each application requires load balancing or firewall services between segments its simply impossible to have an all physical network using traditional VLANs. The limitations of 4000 VLANS and the requirement for stretched Layer 2 subnets has pushed designers to virtualize workloads over an underlying network.


Problem Statement

Problem Statement


What are the drawbacks of overlays and how does it affect network stability?


Control Plane Interaction in a Tunneled Virtual Topology

Virtualization adds a level of complexity to the network. Consider the example of a standard tunnel. We are essentially virtualizing workloads over an underlying network. From a control plane perspective, there must be more than one control plane. This results in two separate views of the networks forwarding and reachability information – a view from the tunnel endpoints and a view from the underlining network.

The control plane may be static or dynamic and provides reachability through the virtual topology on top of the control plane that provides reachability to the tunnel endpoints.





Router A has two paths to reach Already we have the complexity of influencing and managing what traffic should and shouldn’t go down the tunnel. Modifying metrics for certain destination will influence path selection but this comes with complexity of additional configuration and the manual management of policies. The incorrect configuration of the interaction between two control planes may cause of a routing loop or suboptimal routing through the tunnel interfaces. The “routers in the middle” and the “routers at tunnel edges” have different views of the network – increasing network complexity.

It may seem that these two control planes act independent from each other but it is most certainly not an independent topology. The control plane of the virtual topology relies heavily on the control plane of the underlying network. These control plane should not be allowed to interplay freely as both can react differently to certain failures. The timing of the convergence process and how quickly each control plane reacts may be the same or different. The underlying network could converge more quickly or slowly that the overlaying control plane and that can affect application performance. Design best practice is to design the overlays control plane so that it detects and reacts to network failures faster than the underlying control plane or have the underlying control plane detects and react faster than the overlays control plane.


Encapsulation overhead

Every VXLAN packet originated from the end host and sent towards the IP core will be stamped with a VXLAN header. This leads to additional 50 bytes added on a per-packet basis from the source all the way to the destination server. If the core cannot accommodate for the greater MTU size or Path MTU is broken, it may have to fragment the packet into smaller pieces. Also, the VXLAN header needs to be encapsulated and de-encapsulation on the virtual switch which takes up computing cycles. Both of these are problematic for network performance.


VXLAN overhead

VXLAN overhead


Security in a Virtualized Virtual Topology

There are many security flaws with tunnels and network overlays. The most notable is that they hide path information. It’s possible for a tunnel to pass one route on one day and take another path a different day. The change of path may be unknown to the network administrator.


Tunnel Path Insecurity

Tunnel Path Insecurity


Traditional routing is hop-by-hop and every router makes an independent decision about where the traffic should be routed. The independent hop-by-hop decisions are not signalled and known by the tunnel endpoints. It’s possible for an attacker to take advantage and direct the tunnel traffic via an unintended path where the rerouted traffic can be monitored and snooped.

Tunneled traffic hides itself from any policies or security checkpoints. Many firewalls have HTTP port 80 open to support web browsing. This can allow an attacker to tunnel traffic in an HTTP envelope bypassing all the security checks. There are also a number of security implications if you are tunneling with GRE. GRE performs no encryption on any part of the data journey or any authentication.  The optional 32-bit tunnel key used for identifying individual traffic flows can easily be brute forced due to the restriction of 2×32 number combinations. It has a weak implementation of the sequence number that is used to provide a method of in-order delivery.  These shortcomings have opened GRE up to a number of MTU based attacks and GRE packet injection attacks.


STP and Layer 2 attacks

VXLAN extends layer 2 domains across layer 3 boundaries resulting in a larger layer 2 flat network. The attack zones in terms of intrusion becomes much larger as we are now connecting up two distant disjointed end points. This increases the attack zones over traditional VLANs where the Layer 2 broadcast domain was much smaller. If you are running STP over VXLAN you are open to various STP attacks. Tools such as BSD brconfig and Linux bridge-utilis allow you to generate STP frames into a Layer 2 network and can be used to insert a rogue root bridge to modify the traffic path.


VXLAN inbuilt security?

The VXLAN standard does not have any inbuilt security so if your core is not secure and becomes compromised so will all your VXLAN tunnelled traffic. Schemes such as 802.1x should be deployed for the admission control of VTEP ( tunnel endpoints ). 802.1x at the edges provides defense so that rogue endpoints may not inject traffic into the VXLAN cloud. The VXLAN payload can also be encrypted with IPSEC.


About Matt Conran

Matt Conran has created 184 entries.

Leave a Reply