What is FabricPath

What is FabricPath?

Introduced by Cisco in Nexus OS software Release 5.1(3), FabricPath allows architects to design highly scalable true Layer 2 fabrics. Similar to spanning tree, it provides an almost plug and play deployment model with the benefits of Layer 3 routing allowing FabricPath networks to scale at an unprecedented level.  With its simplicity, Fabricpath enables faster, simpler and flatter data center networks.



Why introduce Fabricpath?

1) No Multipathing support at Layer 2: Spanning Tree Protocol ( STP ) lacks any good Layer 2 multipathing features that are needed for large data centers. There has been enhancements to the protocol with the introduction of PVST per VLAN load balancing but this feature can only load balance on VLANs.

2) MAC address scalability: Layer 2 end hosts are discovered by their MAC address and this type of host addressing cannot be hierarchical and summarized. For example, you cannot have one MAC address represent a stub of networks. Traditionally, Layer 3 networks can overcome this by introducing ABRs is OSPF or summarization / filtering in EIGRP.  Also, in Layer 2 network all the MAC addresses are populated in ALL switches leading to large requirements in the Layer 2 table sizes.

3) Instability of Layer 2 networks: Layer 3 network have a eight-bit Time to Live ( TTL ) field that prevents datagrams from persisting ( e.g going in circles ) on the internet. In comparison to Layer 3 headers, the Layer 2 packet header does not have a TTL field. Lack of a TTL field will cause Layer 2 packets to infinitely loop causing network meltdown.

4) Incompetent path selection: The shortest path for a layer 2 network depends on the placement of the Root switch.  You can influence root port ( forwarding port ) selection based on costs and port priorities but essentially the placement of the root switch is how the forwarding path is built. In the diagram below, the most optimum traffic for server to server flows would be via the inter switch link but as you can see spanning tree blocks this port and traffic takes the sub-optimal path through the distribution switch.

STP Path distribution

STP Path distribution

Typical Use cases

1) Large flat data centers that need Layer 2 multipathing and equidistant endpoints.

2) DC that require a reduction of Layer 2 table sizes ( done via MAC conversational learning ).

Supporting any new protocol can be done elegantly in IS-IS by adding new extensions without modifying the base infrastructure.


Control Plane

FabricPath is a Layer 2 overlay network that works with an IS-IS control plane.  Using FabricPath IS-IS, the switches build their forwarding tables, which is similar to building the forwarding table in Layer 3 networks. The extensions used in IS-IS to support Fabricpath allow this layer 2 overlay to take advantage of all the scalable and load balancing ( ECMP, up to 16 routes )  benefits of a layer 3 network while retaining the benefits of a plug and play layer 2 network.

The FabricPath header has hop count in one of the fields, which mitigates temporary loops in FabricPath networks.  This header uses locally assigned hierarchical MAC addresses for forwarding frames within the network.  To accomplish this the original Layer 2 frames is encapsulated with a FabricPath header and a new CRC is appended to the existing packet.  One of the main elements of the FabricPath header is the SwitchID and core switches forward Fabricpath traffic by examining this field. The switchID is the field used in the FabricPath domain to forward packets to the right destination switch.

FabricPath Data Plane

FabricPath Data Plane

Why use IS-IS as the FabricPath control plane?

1) IS-IS is flexible and can be extended to support other functions with new type-length-values (TLVs). TLV is also known as tag-length value and is used to encode optional information.

2) IS-IS runs directly over the link layer, thereby obviating the need for any underlying layer 3 protocol like IP to work.


Conversational MAC learning

FabricPath by default performs conversational MAC learning which enables a switch to learn only those MACs involved in active bidirectional communication. Similar to a three-way handshake, this new technique leads to the population of only the interested host’s MAC addresses, rather than all MAC address in the domain. This in turn greatly reduces the need for large table sizes as each switch only learns those MAC addresses that the hosts under its interface are actively communicating with. Edge nodes only know the MAC addresses of local nodes or nodes that want to communicate with local nodes directly.


FabricPath Benefits and Drawbacks

Benefits Drawbacks
Plug-and-play features like Classical Ethernet Cisco proprietary
Single control plane for ALL types of traffic and good troubleshooting features to debug problems at Layer 2 Fabric interfaces carry only FabricPath encapsulated traffic
High performance and high availability using multipathing** Useful as a DCI solution only over short distances
Easy to add new devices to an existing FabricPath domain.
Small Layer 2 table sizes resulting in better performance.

** This enables the MSDC networks to have flat topologies, enabling the nodes to be separated by a single hop.

About Matt Conran

Matt Conran has created 176 entries.