Network Design and Modularization

Network Design and Modularization

Why don’t we rebuild the Internet into one single flat switched domain – the flat earth model? The problem designing one large flat architecture is that you would find no way to reduce the state and control plane in individual devices. In order to forward packets efficiently, every device would have to know how to reach every other device, each device would also have to be interrupted every time there was a state change on any router in the entire domain.

Route Probagation

Route Propagation


The amount of state and the rate at which it changes is not possible to maintain and what you would witness would be a case of information overload at the machine level. Machine overload can be diagnosed into three independent problems below. The general idea behind machine overload is that too much information is bad for network efficiency. There are methods that can reduce these defects but no matter how much you try to optimization your design, you will never get away from the fact that less routes in a small domain is better than many routes in a large domain.


Machine Overload Problems

Machine Overload Problems


CPU and memory utilization

On most Catalyst platforms, routing information is stored in special high-speed memory called TCAM. TCAM is not infinite and is generally expensive. Large routing tables not only require more CPU cycles but also take up more physical memory and TCAM.


Rate of state of change

Every time there is a change in the network topology, the control plane must change and adapt to the new topology. The bigger the domain, the more routers will have to recalculate the best path and propagate changes to its neighbors resulting in an increase in rate of state change. Due to the fact that MAC addresses are not hierarchical, a Layer 2 network has a much higher rate of state change than a Layer 3 network.


Positive feedback loops

Positive feedback loops add the concept of rate of change with the rate of information flow.


Positive Feedback Loops

Positive Feedback Loops


  1. Router A sends Router B a large database update which causes Routers B control plane to fail.
     2. Routers B control plane failure is propagated to Router D and causes Router D’s control plane to fail.
     3. Routers D control plane failure is propagated to Router C and causes Router C’s control plane to fail.
     4. Routers C control plane failure is propagated to Router B and causes Router B’s control plane to fail.


How can we address these challenges? The answer is modularization and information hiding.


Information Hiding

Information hiding reduces routing table sizes and rate of state change by either combining multiple destinations into one summary prefix, known as aggregation or separating destinations into sub topologies aka virtualization. Information hiding can also be carried out by configuring route filters at specific network points.

In the diagram below, Router B is summarizing network and sending the aggregate route to Router C. The aggregation process hides more specific routes that are behind Router A. Router C never receives any specifics or state changes for those specifics – so it doesn’t have to do any recalculations if the reachability of those networks change. Link flaps and topology changes on Router A will not be known to Router C and visa versa.

Information Hiding

Information Hiding


Routers A and B are also in separate failure domains than router C. Routers C view of the network is not the same as Routers A and Routers B.

A failure domain is the set of devices that must recalculate their control plane information in the case of a topology change.

When a link or node fails in one fault domain, it does not affect the other fault domain. There is a true split in the network. You could argue that aggregation does not split the network into “true” fault domains as you can still have backup paths ( specific routes ) with different metrics reachable in the other domain.

If we split the network into fault domains, devices within each fault domain only compute paths within their fault domain and this drags the network closer to the
MTTR/MTBF balance point – a another reason why you should divide complexity from complexity.


The essence of network design and fault domain isolation is based on the principle of modularization. Modularization is the breaking up of the control plane so that you have different pieces of information in different sections of the network. You should engineer the network so it can manage organic growth and change with fixed limits. When the network gets too big you can move to the next module. The concept of repeatable configurations creates a more manageable network. Each topology should be designed and configured using the same tools where possible.


Why Modularize?

The prime reason to introduce modularity is to reduce the amount of data any particular network device must deal with when its describes and calculates paths to a particular destination. The less information the routing process has to process, in conjunction with tight modulation limits, the faster the network will converge.

The essence of modularization can be traced back to the reason why the OSI and TCP/IP model were introduced. Why do we have these models?  They allows network engineers to break big problems into little pieces so we can focus on specific elements and not get clouded by the complexity of the entire problem all at once.

With the practice of modulation, specific areas of the network are assigned specific tasks. The core focuses solely on fast packet forwarding while the edge carries out a variety of functions such as policing, packet filtering, Qos classification etc. Modulization is carried out by assigning specific functionality to different points in the network.

Virtualization techniques such as MPLS and 802.1Q are also ways to perform modularization. It’s just that they are vertical as opposed to horizontal. You can think of virtualization as hiding information along vertical layers within a network.

So why don’t we carry out modularization on every single router and put each router into a single domain? The answer is network stretch and it is discussed in my next blog which can be assessed here.

About Matt Conran

Matt Conran has created 184 entries.

Leave a Reply