Cisco ACI

ACI Cisco

Cisco ACI Components

In today's rapidly evolving digital landscape, businesses constantly seek innovative solutions to streamline their network infrastructure. Enter Cisco ACI (Application Centric Infrastructure), a groundbreaking technology that promises to revolutionize how networks are designed, deployed, and managed.

In this blog post, we will delve into the intricacies of Cisco ACI, its key features, and the benefits it brings to organizations of all sizes.

Cisco ACI is an advanced software-defined networking (SDN) solution that enables organizations to build and manage their networks in a more holistic and application-centric manner. By abstracting network policies and services from the underlying hardware, ACI provides a unified and programmable approach to network management, making it easier to adapt to changing business needs.

Table of Contents

Highlights: Cisco ACI Components

Hardware-based Underlay

In ACI, hardware-based underlay switching offers a significant advantage over software-only solutions due to specialized forwarding chips. Furthermore, thanks to Cisco’s ASIC development, ACI brings many advanced features, including security policy enforcement, microsegmentation, dynamic policy-based redirect (inserting external L4-L7 service devices into the data path), or detailed flow analytics—besides the vast performance and flexibility.

The Legacy data center

The legacy data center topologies have a static infrastructure that specifies the constructs to form the logical topology. We must configure the VLAN, Layer 2/Layer 3 interfaces, and the protocols we need on the individual devices. Also, the process we used to define these constructs was done manually. We may have used Ansible playbooks to backup configuration or check for specific network parameters, but we generally operated with a statically defined process.

  • Poor Resources

The main roadblock to application deployment was the physical bare-metal server. It was chunky and could only host one application due to the lack of process isolation. So, the network has one application per server to support and provide connectivity. This is the opposite of how ACI Cisco, also known as Cisco SDN ACI networks operate.

Related: For pre-information, you may find the following helpful:

  1. Data Center Security 
  2. VMware NSX



Cisco SDN ACI 

Key ACI Cisco Discussion points:


  • Birth of virtualization and SDN.

  • Cisco ACI integrations.

  • ACI design and components.

  • VXLAN networking and ECMP.

  • Focus on ACI and SD-WAN.

Back to Basics: Cisco ACI components

Key Features of Cisco ACI

a) Application-Centric Policy Model: Cisco ACI allows administrators to define and manage network policies based on application requirements rather than traditional network constructs. This approach simplifies policy enforcement and enhances application performance and security.

b) Automation and Orchestration: With Cisco ACI, network provisioning and configuration tasks are automated, reducing the risk of human error and accelerating deployment times. The centralized management framework enables seamless integration with orchestration tools, further streamlining network operations.

c) Scalability and Flexibility: ACI’s scalable architecture ensures that networks can grow and adapt to evolving business demands. Spine-leaf topology and VXLAN overlay technology allow for seamless expansion and simplify the deployment of multi-site and hybrid cloud environments.

Cisco Data Center

Cisco ACI

Key Features

  • Application-Centric Policy Model

  • Automation and Orchestration

  • Scalability and Flexibility

  • Built-in Security 

Cisco Data Center

Cisco ACI 

Key Advantages

  • Enhanced Security

  • Agility and Time-to-Market

  • Simplified Operations

  • Open software flexibility for DevOps teams.

Benefits of Cisco ACI

a) Enhanced Security: By providing granular microsegmentation and policy-based controls, Cisco ACI helps organizations strengthen their security posture. Malicious lateral movement within the network can be mitigated, reducing the attack surface and preventing data breaches.

b) Agility and Time-to-Market: The automation capabilities of Cisco ACI significantly reduce the time and effort required for network provisioning and changes. This agility enables organizations to respond faster to market demands, launch new services, and gain a competitive edge.

c) Simplified Operations: The centralized management and policy-driven approach of Cisco ACI simplify network operations, leading to improved efficiency and reduced operational costs. The intuitive user interface and comprehensive analytics provide administrators with valuable insights, enabling proactive troubleshooting and optimization.

The Cisco ACI SDN Solution

Cisco ACI is a software-defined networking (SDN) solution that integrates with software and hardware. With the ACI, we can create software policies and use hardware for forwarding, an efficient and highly scalable approach offering better performance. The hardware for ACI is based on the Cisco Nexus 9000 platform product line. The APIC centralized policy controller drives the software, which stores all configuration and statistical data.

Nexus Family

To build the ACI underlay, you must exclusively use the Nexus 9000 family of switches. You can choose from modular Nexus 9500 switches or fixed 1U to 2U Nexus 9300 models. Specific models and line cards are dedicated to the spine function in ACI fabric; others can be used as leaves, and some can be used for both purposes. You can combine various leaf switches inside one fabric without any limitations.

Spine and Leaf

For Nexus 9000 switches to be used as an ACI spine or leaf, they must be equipped with powerful Cisco CloudScale ASICs manufactured using 16-nm technology. The following figure shows the Cisco ACI based on the Nexus 9000 series. Cisco Nexus 9300 and 9500 platform switches support Cisco ACI. As a result, organizations can use them as the spine or leaf switches to fully utilize an automated, policy-based systems management approach. 

Cisco ACI Components
Diagram: Cisco ACI Components. Source is Cisco
  • A key point: The birth of virtualization

Server virtualization helped to a degree where we could decouple workloads from the hardware, making the compute platform more scalable and agile. However, the server is not the main interconnection point for network traffic. So, we need to look at how we could virtualize the network infrastructure in a way similar to the agility gained from server virtualization.

This is carried out with software-defined networking and overlays that could map network endpoints and be spun up and down as needed without human intervention. In addition, the SDN architecture includes an SDN controller and an SDN network that enables an entirely new data center topology.

server virtualization
Diagram: The need for virtualization and software-defined networking.

ACI Cisco: Integrations

Routing Control Platform

Then came along Cisco SDN ACI, the ACI Cisco, which operates differently from the traditional data center with an application-centric infrastructure. The Cisco application-centric infrastructure achieves resource elasticity with automation through standard policies for data center operations and consistent policy management across multiple on-premises and cloud instances.

It uses a Software-Defined Networking (SDN) architecture like a routing control platform. The Cisco SDN ACI also provides a secure networking environment for Kubernetes. In addition, it integrates with various other solutions, such as Red Hat OpenShift networking.

Cisco ACI: Integration Options

What makes the Cisco ACI interesting is its several vital integrations. I’m not talking about extending the data center with multi-pod and multi-site, for example, with AlgoSec, Cisco AppDynamics, and SD-WAN. AlgoSec enables secure application delivery and policy across hybrid network estates, while AppDynamic lives in a world of distributed systems Observability. SD-WAN enabled path performance per application with virtual WANs.

Cisco ACI Components: ACI Cisco and Multi-Pod

Cisco ACI Multi-Pod is part of the “Single APIC Cluster / Single Domain” family of solutions, as a single APIC cluster is deployed to manage all the interconnected ACI networks. These separate ACI networks are named “pods,” Each looks like a regular two-tier spine-leaf topology. The same APIC cluster can manage several pods, and to increase the resiliency of the solution, the various controller nodes that make up the cluster can be deployed across different pods.

ACI Multi-Pod
Diagram: Cisco ACI Multi-Pod. Source Cisco.

Cisco ACI Components: ACI Cisco and AlgoSec

With AlgoSec integrated with the Cisco ACI, we can now provide automated security policy change management for multi-vendor devices and risk and compliance analysis. The AlgoSec Security Management Solution for Cisco ACI extends ACI’s policy-driven automation to secure various endpoints connected to the Cisco SDN ACI fabric.

These simplify the network security policy management across on-premises firewalls, SDNs, and cloud environments. It also provides the necessary visibility into the security posture of ACI, even across multi-cloud environments. 

Cisco ACI Components: ACI Cisco and AppDynamics 

Then, with AppDynamics, we are heading into Observability and controllability. Now, we can correlate app health and network for optimal performance, deep monitoring, and fast root-cause analysis across complex distributed systems with numbers of business transactions that need to be tracked. This will give your teams complete visibility of your entire technology stack, from your database servers to cloud-native and hybrid environments. In addition, AppDynamics works with agents that monitor application behavior in several ways. We will examine the types of agents and how they work later in this post.

Cisco ACI Components: ACI Cisco and SD-WAN 

SD-WAN brings a software-defined approach to the WAN. These enable a virtual WAN architecture to leverage transport services such as MPLS, LTE, and broadband internet services. So, SD-WAN is not a new technology; its benefits are well known, including improving application performance, increasing agility, and, in some cases, reducing costs.

The Cisco ACI and SD-WAN integration makes active-active data center design less risky than in the past. The following figures give a high-level overview of the Cisco ACI and SD-WAN integration. For pre-information generic to SD-WAN, go here: SD-WAN Tutorial

SD WAN integration
Diagram: Cisco ACI and SD-WAN integration

The Cisco SDN ACI and SD-WAN Integration

The Cisco SDN ACI with SD-WAN integration helps ensure an excellent application experience by defining application Service-Level Agreement (SLA) parameters. Cisco ACI releases 4.1(1i) and adds support for WAN SLA policies. This feature enables admins to apply pre-configured policies to specify the packet loss, jitter, and latency levels for the tenant traffic over the WAN.

When you apply a WAN SLA policy to the tenant traffic, the Cisco APIC sends the pre-configured policies to a vManage controller. The vManage controller, configured as an external device manager that provides SDWAN capability, chooses the best WAN link that meets the loss, jitter, and latency parameters specified in the SLA policy.

Cisco ACI Components: Openshift and Cisco SDN ACI

OpenShift Container Platform (formerly known as OpenShift Enterprise) or OCP is Red Hat’s offering for the on-premises private platform as a service (PaaS). OpenShift is based on the Origin open-source project and is a Kubernetes distribution, the defacto for container-based virtualization. The foundation of the OpenShift networking SDN is based on Kubernetes and, therefore, shares some of the same networking technology along with some enhancements, such as the OpenShift route construct.

Cisco ACI Components: Other data center integrations

Cisco SDN ACI has another integration with Cisco DNA Center/ISE that maps user identities consistently to endpoints and apps across the network, from campus to the data center. Cisco Software-Defined Access (SD-Access) provides policy-based automation from the edge to the data center and the cloud.

Cisco SD-Access provides automated end-to-end segmentation to separate user, device, and application traffic without redesigning the network. This integration will enable customers to use standard policies across Cisco SD-Access and Cisco ACI, simplifying customer policy management using Cisco technology in different operational domains.

Let us recap before we look at the ACI integrations in more detail.

The Cisco SDN ACI Design  

Introduction to leaf and spine

The Cisco SDN ACI works with a Clos architecture, a fully meshed ACI network. Based on a spine leaf architecture. As a result, every Leaf is physically connected to every Spine, enabling traffic forwarding through non-blocking links. Physically, we have a set of Leaf switches creating a Leaf layer attached to the Spines in a full BIPARTITE graph.

This means that each Leaf is connected to each Spine, and each Spine is connected to each Leaf.  The ACI uses a horizontally elongated Leaf and Spine architecture with one hop to every host in an entirely messed ACI fabric, offering good throughput and convergence needed for today’s applications.

Cisco ACI
Diagram: Cisco ACI: Improving application performance.

The ACI fabric: Aggregate

A key point to note in the spine-and-leaf design is the fabric concept, which is like a stretch network. And one of the core ideas around a fabric is that they do not aggregate traffic. This does increase data center performance along with a non-blocking architecture. With the spine-leaf topology, we are spreading a fabric across multiple devices.

The result of the fabric is that each edge device has the total bandwidth of the fabric available to every other edge device. This is one big difference from traditional data center designs; we aggregate the traffic by either stacking multiple streams onto a single link or carrying the streams serially.

SDN data center
Diagram: Cisco ACI fabric checking.

The issues with oversubscription

With the traditional 3-tier design, we aggregate everything at the core, leading to oversubscription ratios that degrade performance. With the ACI Leaf and Spine design, we spread the load across all devices with equidistant endpoints. Therefore, we can carry the streams parallel.

Horizontal scaling load balancing

Then, we have horizontal scaling load balancing.  Load balancing with this topology uses multipathing to achieve the desired bandwidth between the nodes. Even though this forwarding paradigm can be based on Layer 2 forwarding ( bridging) or Layer 3 forwarding ( routing), the ACI leverages a routed approach to the Leaf and Spine design, and we have Equal Cost Multi-Path (ECMP) for both Layer 2 and Layer 3 traffic. 

Highlighting the overlay and underlay

Mapping Traffic

So you may be asking how we can have Layer 3 routed core and pass Layer 2 traffic. This is done using the overlay, which can map different traffic types to other overlays. So, we can have Layer 2 traffic mapped to an overlay over a routed core. ACI links between the Leaf and the Spine switches are L3 active-active links. Therefore, we can intelligently load balance and traffic steer to avoid issues. And we don’t need to rely on STP to block links or involve STP to fix the topology.

When networks were first developed, there was no such thing as an application moving from one place to another while it was in use. So the original architects of IP, the communication protocol used between computers, used the IP address to mean both the identity of a device connected to the network and its location on the network.  Today, in the modern data center, we need to be able to communicate with an application or application tier, no matter where it is.

Overlay Encapsulation

One day, it may be in location A and the next in location B, but its identity, which we communicate with, is the same on both days. An overlay is when we encapsulate an application’s original message with the location to which it needs to be delivered before sending it through the network.

Once it arrives at its final destination, we unwrap it and deliver the original message as desired. The identities of the devices (applications) communicating are in the original message, and the locations are in the encapsulation, thus separating the place from the identity. This wrapping and unwrapping is done per-packet basis and, therefore, must be done quickly and efficiently.

Overlay and underlay components

The Cisco SDN ACI has a concept of overlay and underlay, forming a virtual overlay solution. The role of the underlay is to glue together devices so the overlay can work and be built on top. So, the overlay, which is VXLAN, runs on top of the underlay, which is IS-IS. In the ACI, the IS-IS protocol provides the routing for the overlay, which is why we can provide ECMP from the Leaf to the Spine nodes. The routed underlay provides an ECMP network where all leaves can access Spine and have the same cost links. 

ACI overlay
Diagram: Overlay. Source Cisco

Example: 

Let’s take a simple example to illustrate how this is done. Imagine that application App-A wants to send a packet to App-B. App-A is located on a server attached to switch S1, and App-B is initially on switch S2. When App-A creates the message, it will put App-B as the destination and send it to the network; when the message is received at the edge of the network, whether a virtual edge in a hypervisor or a physical edge in a switch, the network will look up the location of App-B in a “mapping” database and see that it is attached to switch S2.

It will then put the address of S2 outside of the original message. So, we now have a new message addressed to switch S2. The network will forward this new message to S2 using traditional networking mechanisms. Note that the location of S2 is very static, i.e., it does not move, so using traditional mechanisms works just fine.

Upon receiving the new message, S2 will remove the outer address and thus recover the original message. Since App-B is directly connected to S2, it can easily forward the message to App-B. App-A never had to know where App-B was located, nor did the network’s core. Only the edge of the network, specifically the mapping database, had to know the location of App-B. The rest of the network only had to see the location of switch S2, which does not change.

Let’s now assume App-B moves to a new location switch S3. Now, when App-A sends a message to App-B, it does the same thing it did before, i.e., it addresses the message to App-B and gives the packet to the network. The network then looks up the location of App-B and finds that it is now attached to switch S3. So, it puts S3’s address on the message and forwards it accordingly. At S3, the message is received, the outer address is removed, and the original message is delivered as desired.

The movement of App-B was not tracked by App-A at all. The address of App-B identified App-B, while the address of the switch, S2 or S3, identified App-B’s location. App-A can communicate freely with App-B no matter where App-B is located, allowing the system administrator to place App-B in any location and move it as desired, thus achieving the flexibility needed in the data center.

Multicast Distribution Tree (MDT)

We have a Multicast Distribution Tree MDT tree on top that is used to forward multi-destination traffic without having loops. The Multicast distribution tree is dynamically built to send flood traffic for specific protocols. Again, it does this without creating loops in the overlay network. The tunnels created for the endpoints to communicate will have tunnel endpoints. The tunnel endpoints are known as the VTEP. The VTEP addresses are assigned to each Leaf switch from a pool that you specify in the ACI startup and discovery process.

Normalize the transports

VXLAN tunnels in the ACI fabric are used to normalize the transports in the ACI network. Therefore, traffic between endpoints can be delivered using the VXLAN tunnel, resulting in any transport network regardless of the device connecting to the fabric. 

Building the VXLAN tunnels 

So, using VXLAN in the overlay enables any network, and you don’t need to configure anything special on the endpoints for this to happen. The endpoints that connect to the ACI fabric do not need special software or hardware. The endpoints send regular packets to the leaf nodes they are connected to directly or indirectly. As endpoints come online, they send traffic to reach a destination.

Bridge domain and VRF

Therefore, the Cisco SDN ACI under the hood will automatically start to build the VXLAN overlay network for you. The VXLAN network is based on the Bridge Domain (BD), or VRF ACI constructs deployed to the leaf switches. The Bridge Domain is for Layer 2, and the VRF is for Layer 3. So, as devices come online and send traffic to each other, the overlay will grow in reachability in the Bridge Domain or the VRF. 

Horizontal scaling load balancing
Diagram: Horizontal scaling load balancing.

Routing for endpoints

Routing within each tenant, VRF is based on host routing for endpoints directly connected to the Cisco ACI fabric. For IPv4, the host routing is based on the /32, giving the ACI a very accurate picture of the endpoints. Therefore, we have exact routing in the ACI.

In conjunction, we have a COOP database that runs on the Spines that offers remarkably optimized fabric in terms of knowing where all the endpoints are located. To facilitate this, every node in the fabric has a TEP address, and we have different types of TEPs depending on the role of the device. The Spine and the Leaf will have TEP addresses but will differ from each other.

COOP database
Diagram: COOP database

The VTEP and PTEP

The Leaf’s nodes are the Virtual Tunnel Endpoints (VTEP). In ACI, this is known as PTEP, the physical tunnel endpoints. These PTEP addresses represent the “WHERE” in the ACI fabric that an endpoint lives in.

Cisco ACI uses a dedicated VRF and a subinterface of the uplinks from the Leaf to the Spines as the infrastructure to carry VXLAN traffic. In Cisco ACI terminology, the transport infrastructure for VXLAN traffic is known as Overlay-1, which is part of the tenant “infra.” 

The Spine TEP

The Spines also have a PTEP and an additional proxy TEP. This is used for forwarding lookups into the mapping database. The Spines have a global view of where everything is, which is held in the COOP database synchronized across all Spine nodes. All of this is done automatically for you.

For this to work, the Spines have an Anycast IP address known as the Proxy TEP. The Leaf can use this address if they do not know where an endpoint is, so they ask the Spine for any unknown endpoints, and then the Spine checks the COOP database. This brings many benefits to the ACI solution, especially for traffic optimizations and reducing flooded traffic in the ACI. Now, we have an optimized fabric for better performance.

Cisco ACI
Diagram: Routing control platform.

The ACI optimizations

Mouse and elephant flows

This provides better performance for load balancing different flows. For example, in most data centers, we have latency-sensitive flows, known as mouse flows, and long-lived bandwidth-intensive flows, known as elephant flows. 

The ACI has more precisely load-balanced traffic using algorithms that optimize mouse and elephant flows and distribute traffic based on flow lets: flow let load-balancing. Within a Leaf, Spine latency is low and consistent from port to port. The max latency of a packet from one port to another in the architecture is the same regardless of the network size. So you can scale the network without degrading performance. Scaling is often done on a POD-by-POD basis. For more extensive networks, each POD would be its Leaf and Spine network.

ARP optimizations: Anycast gateways

The ACI comes by default with a lot of traffic optimizations. Firstly, instead of using an ARP and broadcasting across the network, that can hamper performance. The Leaf can assume that the Spine will know where the destination is ( and it does via the COOP database ), so there is no need to broadcast to everyone to find a destination.

If the Spine knows where the endpoint is, it will forward it to the other Leaf. If not, it will drop the traffic.

Fabric anycast addressing

This again adds performance benefits to the ACI solution as the table sizes on the Leaf switches can be kept smaller than they would if they needed to know where all the destinations were, even if they were not or never needed to communicate with them. On the Leaf, we have an Anycast address too.

These fabric anycast addresses are available for Layer 3 interfaces. On the Leaf ToR, we can establish an SVI that uses the same MAC address on every ToR; therefore, when an endpoint needs to route to a ToR. It doesn’t matter which ToR you use. The Anycast Address is spread across all ToR leaf switches. 

Pervasive gateway

Now we have predictable latency to the first hop, and you will use the local route VRF table within that ToR instead of traversing the fabric to a different ToR. This is the Pervasive Gateway feature that is used on all Leaf switches. The Cisco ACI has many advanced networking features, but the pervasive gateway is my favorite. It does take away all the configuration mess we had in the past.

The Cisco SDN ACI Integrations

OpenShift and Cisco ACI

  • OpenSwitch virtual network

OpenShift does this with an SDN layer and enhances Kubernetes networking to have a virtual network across all the nodes. It is created with the Open Switch standard. For OpenShift SDN, this pod network is established and maintained by the OpenShift SDN, configuring an overlay network using a virtual switch called the OVS bridge, forming an OVS network that gets programmed with several OVS rules. The OVS is a popular open-source solution for virtual switching.

Openshift sdn
Diagram: OpenShift SDN.

OpenShift SDN plugin

We mentioned that you could tailor the virtual network topology to suit your networking requirements, which can be determined by the OpenShift SDN plugin and the SDN model you select. With the default OpenShift SDN, there are several modes available. This level of SDN mode you choose is concerned with managing connectivity between applications and providing external access to them. Some modes are more fine-grained than others. The Cisco ACI plugins offer the most granular.

Integrating ACI and OpenShift platform

The Cisco ACI CNI plugin for the OpenShift Container Platform provides a single, programmable network infrastructure, enterprise-grade security, and flexible micro-segmentation possibilities. The APIC can provide all networking needs for the workloads in the cluster. Kubernetes workloads become fabric endpoints, like Virtual Machines or Bare Metal endpoints.

The Cisco ACI CNI plugin extends the ACI fabric capabilities to OpenShift clusters to provide IP Address Management, networking, load balancing, and security functions for OpenShift workloads. In addition, the Cisco ACI CNI plugin connects all OpenShift Pods to the integrated VXLAN overlay provided by Cisco ACI.

The Cisco SDN ACI and AppDynamics

AppDynamis overview

So, you have multiple steps or services for an application to work. These services may include logging in and searching to add something to a shopping cart. These services will invoke various applications, web services, third-party APIs, and databases, known as business transactions.

The user’s critical path

A business transaction is the essential user interaction with the system and is the customer’s critical path. Therefore, business transactions are the things you care about. If they start to go, it will cause your system to degrade. So, you need ways to discover your business transactions and determine if there are any deviations from baselines. This should also be done automated, as learning baseline and business transitions in deep systems is nearly impossible using the manual approach.

So, how do you discover all these business transactions?

AppDynamics automatically discovers business transactions and builds an application topology map of how the traffic flows. A topology map can view usage patterns and hidden flows, acting as a perfect feature for an Observability platform.

Cisco AppDynamics
Diagram: Cisco AppDynamics.

AppDynamic topology

AppDynamics will discover the topology for all of your application components. All of this is done automatically for you. It can then build a performance baseline by capturing metrics and traffic patterns. This allows you to highlight issues when services and components are slower than usual.

AppDynamics uses agents to collect all the information it needs. The agent monitors and records the calls that are made to a service. This is from the entry point and follows executions along its path through the call stack. 

Types of Agents for Infrastructure Visibility

If the agent is installed on all critical parts, you can get information about that specific instance. This can help you build a global picture. So we have an Application Agent, Network Agent, and Machine Agent for Server visibility and Hardware/OS.

  • App Agent: This agent will monitor apps and app servers, and example metrics will be slow transitions, stalled transactions, response times, wait times, block times, and errors.  
  • Network Agent: This agent monitors the network packets, TCP connection, and TCP socket. Example metrics include performance impact Events, Packet loss and retransmissions, RTT for data transfers, TCP window size, and connection setup/teardown.
  • Machine Agent Server Visibility: This agent monitors the number of processes, services, caching, swapping, paging, and querying. Example Metrics include hardware/software interrupts, virtual memory/swapping, process faults, and CPU/DISK/Memory utilization by the process.
  • Machine Agent: Hardware/OS – disks, volumes, partitions, memory, CPU. Example metrics: CPU busy time, MEM utilization, and pieces file.

Automatic establishment of the baseline

A baseline is essential, a critical step in your monitoring strategy. Doing this manual is hard, if not impossible, with complex applications. Having this automatically done for you is much better. You must automatically establish the baseline and alert yourself about deviations from the baseline. This will help you pinpoint the issue faster and resolve issues before the problem can be affected. Platforms such as AppDynamics can help you here. Any malicious activity can be seen from deviations from the security baseline and performance issues from the network baseline.

Summary: Cisco ACI Components

In the ever-evolving world of networking, organizations are constantly seeking ways to enhance their infrastructure’s performance, security, and scalability. Cisco ACI (Application Centric Infrastructure) presents a cutting-edge solution to these challenges. By unifying physical and virtual environments and leveraging network automation, Cisco ACI revolutionizes how networks are built and managed.

Section 1: Understanding Cisco ACI Architecture

At the core of Cisco ACI lies a robust architecture that enables seamless integration between applications and the underlying network infrastructure. The architecture comprises three key components:

1. Application Policy Infrastructure Controller (APIC):

The APIC serves as the centralized management and policy engine of Cisco ACI. It provides a single point of control for configuring and managing the entire network fabric. Through its intuitive graphical user interface (GUI), administrators can define policies, allocate resources, and monitor network performance.

2. Nexus Switches:

Cisco Nexus switches form the backbone of the ACI fabric. These high-performance switches deliver ultra-low latency and high throughput, ensuring optimal data transfer between applications and the network. Nexus switches provide the necessary connectivity and intelligence to enable the automation and programmability features of Cisco ACI.

3. Application Network Profiles:

Application Network Profiles (ANPs) are a fundamental aspect of Cisco ACI. ANPs define the policies and characteristics required for specific applications or application groups. By encapsulating network, security, and quality of service (QoS) policies within ANPs, administrators can streamline the deployment and management of applications.

Section 2: The Power of Network Automation

One of the most compelling aspects of Cisco ACI is its ability to automate network provisioning, configuration, and monitoring. Through the APIC’s powerful automation capabilities, network administrators can eliminate manual tasks, reduce human errors, and accelerate the deployment of applications. With Cisco ACI, organizations can achieve greater agility and operational efficiency, enabling them to rapidly adapt to evolving business needs.

Section 3: Security and Microsegmentation with Cisco ACI

Security is a paramount concern for every organization. Cisco ACI addresses this by providing robust security features and microsegmentation capabilities. With microsegmentation, administrators can create granular security policies at the application level, effectively isolating workloads and preventing lateral movement of threats. Cisco ACI also integrates with leading security solutions, enabling seamless network enforcement and threat intelligence sharing.

Conclusion:

Cisco ACI is a game-changer in the realm of network automation and infrastructure management. Its innovative architecture, coupled with powerful automation capabilities, empowers organizations to build agile, secure, and scalable networks. By leveraging Cisco ACI’s components, businesses can unlock new levels of efficiency, flexibility, and performance, ultimately driving growth and success in today’s digital landscape.

kubernetes security best practice

Kubernetes Security Best Practice

 

kubernetes attack vectors

 

Kubernetes Security Best Practice

Kubernetes has become the go-to container orchestration platform for organizations worldwide due to its flexibility and scalability. However, as with any technology, ensuring robust security measures is crucial for safeguarding sensitive data and maintaining the integrity of Kubernetes clusters. In this blog post, we will discuss essential Kubernetes security best practices that every organization should implement to protect their infrastructure and applications.

 

Highlights: Kubernetes Security Best Practice

  • An Orchestration Tool

Kubernetes has quickly become the de facto orchestration tool for deploying Kubernetes microservices and containers to the cloud. It offers a way of running groups of resources as a cluster and provides an entirely different abstraction level to single container deployments, allowing better management. From a developer’s perspective, it allows the rolling out of new features that align with business demands while not worrying about the underlying infrastructure complexities.

  • Security Pitfalls

But there are pitfalls to consider when considering Kubernetes security best practices and Docker container security. Security teams must maintain the required visibility, compliance, and control, which is difficult because they do not control the underlying infrastructure. To help you on your security journey, you should introduce a chaos engineering project, specifically chaos engineering kubernetes.

This will help you find the breaking points and performance-related issues that may indirectly create security gaps in your Kubernetes cluster, making the cluster acceptable to Kubernetes attack vectors.

 

Before you proceed, you may find the following posts helpful:

  1. OpenShift SDN
  2. Hands On Kubernetes
  3. Kubernetes Network Namespace

 



Kubernetes Attack Vectors

Key Kubernetes Security Best Practice Discussion points:


  • The issues with traditional security constructs.

  • The growing hacker sophistication.

  • Recap on the Kubernetes architecture.

  • Details on the Kubernetes security best practice.

  • Security 101 for containers and Kubernetes.

 

  • A key point: Video on Immutable Infrastructure

The following video discusses Immutable Infrastructure. The birth of virtualization gave rise to virtual servers and the ability to instantiate workloads within a shorter period. Similar to how virtualization brings new ways to improve server infrastructure, Immutable server infrastructure takes us one step further. Firstly, mutable server infrastructure is servers that require additional care once they have been deployed.

This may include an upgrade, downgrade, or a tweak configuration file for a specific optimization. Usually, this is done on a server-by-server basis. Secondly, an Immutable server infrastructure is servers that are not tweaked after deployment.

It’s an entirely new paradigm where if a change needs to occur, a new server is built from a standard image along with the required change included in that image.

 

Technology Brief : Cloud Computing - Introducing Immutable Infrastructure
Prev 1 of 1 Next
Prev 1 of 1 Next

 

Back to basics with Kubernetes security best practice

As workloads move to the cloud, Kubernetes is the most common orchestrator for managing them. The reason Kubernetes is popular is its declarative nature: It abstracts infrastructure details and allows users to specify the workloads they want to run and the desired outcomes.

However, securing, observing, and troubleshooting containerized workloads on Kubernetes are challenges. It needs a range of considerations, from infrastructure choices and cluster configuration to deployment controls, runtime, and network security. Keep in mind that Kubernetes is not secure by default.

 

Regularly Update Kubernetes:

Keeping your Kubernetes environment up to date is one of the fundamental aspects of maintaining a secure cluster. Regularly updating Kubernetes ensures that you have the latest security patches and bug fixes, reducing the risk of potential vulnerabilities.

Enable RBAC (Role-Based Access Control):

Implementing Role-Based Access Control (RBAC) is crucial for controlling and restricting user access within your Kubernetes cluster. By assigning appropriate roles and permissions to users, you can ensure that only authorized personnel can perform specific operations, minimizing the chances of unauthorized access and potential security breaches.

Use Network Policies:

Kubernetes Network Policies allow you to define and enforce rules for network traffic within your cluster. By utilizing network policies effectively, you can isolate workloads, control ingress and egress traffic, and prevent unauthorized communication between pods. This helps in reducing the attack surface and enhancing the overall security posture of your Kubernetes environment.

Employ Pod Security Policies:

Pod Security Policies (PSPs) enable you to define security configurations and restrictions for pods running in your Kubernetes cluster. By defining PSPs, you can enforce policies such as running pods with non-root users, disallowing privileged containers, and restricting host namespace sharing. These policies help prevent malicious activities and reduce the impact of potential security breaches.

Implement Image Security:

Ensuring the security of container images is crucial to protect your Kubernetes environment. Employing image scanning tools and registries that provide vulnerability assessments allows you to identify and address potential security issues within your images. Additionally, only using trusted and verified images from reputable sources reduces the risk of running compromised or malicious containers.

Monitor and Audit Cluster Activity:

Implementing robust monitoring and auditing mechanisms is essential for detecting and responding to security incidents promptly. By leveraging Kubernetes-native monitoring solutions, you can gain visibility into cluster activity, detect anomalies, and generate alerts for suspicious behavior. Furthermore, regularly reviewing audit logs helps in identifying potential security breaches and provides valuable insights for enhancing the security posture of your Kubernetes environment.

 

Kubernetes Attack Vectors

The hot topic for security teams is how to improve Kubernetes security, implement Kubernetes security best practice within the cluster, and make container networking deployments more secure. Kubernetes is a framework that provides infrastructure and features. But it would be best to think outside the box regarding Kubernetes security best practices.

We are not saying that Kubernetes doesn’t have built-in security features to protect your application architecture! It includes logging and monitoring, debugging and introspection, identity, and authorization. But ask yourself, do these defaults cover all aspects of your security requirements? Remember that much of the complexity with Kubernetes can be abstract with OpenShift networking.

 

A key point: Video with OpenShift Networking

In the past, the application stack had very few components, maybe just a cache, web server, and database. The most common network service simply allows a source to reach an application endpoint or load balance to a number of endpoints using a load balancing algorithm. The sole purpose of the network was to provide endpoint reachability.

Changes inside the data center are driving networks and network services towards becoming more integrated with the application. Nowadays, the network function exists no longer solely to satisfy endpoint reachability; it is fully integrated and in the case of Red Hat’s Openshift the network is represented as a Software-Defined Networking (SDN) layer.

 

OpenShift Networking Explained
Prev 1 of 1 Next
Prev 1 of 1 Next

 

Traditional Security Constructs  

Traditional security constructs don’t work with containers. And we have seen this with the introduction of Zero Trust Networking. Containers have a different security model as they are short-lived and immutable. For example, securing containers differs from securing a virtual machine (VM).

For one, the attack surface is smaller as containers usually only live for about a week, as opposed to a VM that may stay online forever. As a result, we need to manage vulnerabilities and compliance through the container scheduler lifecycle – from build tools, image registries, and production.

How do you measure security within a Kubernetes environment? No one can ever be 100% secure, but one should follow certain best practices to harden the gates against bad actors. You can have different levels of security within different parts of the Kubernetes domain. However, a general Kubernetes security best practice would be introducing as many layers of security between your Kubernetes environment and the bad actors to secure valuable company assets.

 

Hacker sophistication

Hackers are growing in sophistication. We started with script kiddies in the 1980s and are now entering a world of artificial intelligence ( AI ) and machine learning (ML) in the hands of bad actors. They have the tools to throw everything and anything at your Kubernetes environment, dynamically changing attack vectors on the fly.

If you haven’t adequately secured the domain, once a bad actor gains access to your environment, they can move laterally throughout the application stack, silently accessing valuable assets.

Kubernetes security best practices

Kubernetes Architecture

Before we delve into some of Kubernetes’ security best practices and the Kubernetes attack vectors, let’s recap Kubernetes networking 101. Within Kubernetes, there will always be four common constructs – pods, services, labels, and replication controllers. All constructs combine to create an entire application stack.

  • Pods
    Pods are the smallest scheduling unit within a Kubernetes environment that holds a set of closely related containers. Containers within a Pod share the same network namespace and must be installed on the same physical host. This enables processing to be performed locally without latency traversing from one physical host to another.
  • Labels
    As I said, containers within a pod share the same network namespaces. As a result, the containers within a pod can reach each other’s ports on localhost. Therefore we need to introduce a tagging mechanism known as labels. Labels offer another level of abstraction by tagging items as a group. They are essentially key-value pairs categorizing constructs. For example, labels distinguish containers as part of an application stack’s web or database tier.
  • Replication Controllers (RC)
    Then we have replication controllers. Their primary purpose is to manage the lifecycle and state of the pods by ensuring the desired state always matches the actual state. The replication controllers ensure the correct numbers are running. Either creating or removing pods at any given time.
  • Services
    Other standard Kubernetes components are services. Services represent groups of pods acting as one, allowing pods to access services in other pods without directing service-destined traffic to the pod IP. Pods are targeted by accessing a service that represents a group of pods. A service can be analogous to a load balancer in front of a pod accepting front-end service-destined traffic.
  • Kubernetes API server
    Finally, the Kubernetes API server validates and configures data for the API objects, which include the construct mentioned above – pods, services, and replication controllers, to name a few.

 

Also, consider the practice of chaos engineering to understand your Kubernetes environment better and test for resilience and breaking points. Chaos engineering Kubernetes breaks your system to help you better understand your system.

Chaos Engineering
Diagram: Chaos Engineering Kubernetes.

 

A key point: Video on Chaos Engineering

This educational tutorial will start with guidance on how the application has changed from the monolithic style to the microservices-based approach along with how this has affected failures. I will introduce to you how this can be solved by knowing exactly how your application and infrastructure perform under stress and what are their breaking points

 

Chaos Engineering: How to Start A Project
Prev 1 of 1 Next
Prev 1 of 1 Next

 

Kubernetes Security Best Practice: Security 101

Now that you understand the most commonly used Kubernetes constructs let us delve into some of the central Kubernetes security best practices and Kubernetes attack vectors.

  • Kubernetes API
    Firstly, a bad actor can attack the Kubernetes API server to attack the cluster further. By design, each pod has a service account associated with it. The service account has full permissions and sufficient privileges to access the Kubernetes API server. A bad actor can get the token mounted in the pod and then use that token to gain access to the API server. Unfortunately, once they can access the Kubernetes API server, they can get information, such as secrets, to further compromise the cluster. For example, a bad actor could access the database server where sensitive and regulatory data is held.
  • Kubernetes API – Mitigate
    Another way to mitigate a Kubernetes API attack is to introduce role-based access control (RBAC). Essentially, RBAC gives rules to service accounts that can be assigned to the entire cluster. RBAC allows you to set permission on individual constructs such as pods and containers and then define rules based on individual use cases. Another Kubernetes best practice would be to introduce an API server firewall. This firewall would work similarly to a standard firewall, where you can allow and block ranges of addresses. Essentially, what this does is that it narrows the attack surface, hardening Kubernetes security.
  • Secure pods with network policy
    Introducing ingress and egress network policies would allow you to block network access to certain pods. Then only certain pods can communicate with each other. For example, network policies can be set up to restrict the web front end from communicating directly to the database tier.

 

There is also a risk of a user implementing unsecured pods. In this case, you can have network policies in place so that if an unsecured pod does appear in the Kubernetes environment, the Kubernetes API will reject them. Remember, left to its defaults, intra-cluster pod-to-pod communication is open. You may think a bad actor could sniff this traffic, but that’s harder to do than in everyday environments. A better strategy to harden Kubernetes security would be to use a network policy only to allow certain pods to communicate with each other.

 

  • Container escape: In the past, an unpatched kernel or a bug in the web front-end allows a bad actor to get out of that container and gain access to other containers or even to the Kubelet process itself. Once the bad actor gets access to the Kubelet client, it can access the API server gaining a further foothold in the Kubernetes environment.
  • Sandboxed Pods: One way to mitigate a container escape would be to introduce sandboxed pods inside a lightweight VM. A sandboxed pod provides additional barriers to security as it lies above the kernel itself. Providing layers of security above the kernel brings many Kubernetes security advantages. As the sandboxed pod runs inside a VM, an additional bug is required to break out of the container if there is a kernel exploit.
  • Scanning containers: For adequate Kubernetes security, you should know if there is a vulnerability in your container image. Running containers with vulnerabilities exposes the entire system to attacks. Scanning all container images to discover and remove known vulnerabilities would be best. The scanning function should be integrated with runtime enforcement and remediation capabilities enabling a secure and compliant SDLC (Software Development Life Cycle).
  • Restrict access to Kubectl: Kubectl is a powerful command-line interface for running commands against Kubernetes clusters. It allows you to create and update resources in a Kubernetes environment. Due to its potential, one should restrict access to this tool with firewall rules.
  • Disabling the Kubernetes dashboard: Disabling the Kubernetes Dashboard is a must, as it has been part of some high-profile compromises. Previously, bad actors accessed a publicly unrestricted Kubernetes dashboard and privileged account credentials to access resources and mine cryptocurrency.

 

Conclusion:

As Kubernetes adoption continues to grow, ensuring the security of your cluster becomes increasingly important. By following these Kubernetes security best practices, you can establish a robust security foundation for your infrastructure and applications.

Regularly updating Kubernetes, enabling RBAC, utilizing network policies, employing pod security policies, implementing image security measures, and monitoring cluster activity will significantly enhance your Kubernetes security posture. Embracing these best practices will not only protect your organization’s sensitive data but also provide peace of mind in today’s ever-evolving threat landscape.

 

kubernetes attack vectors

Kubernetes PetSets

Kubernetes Networking 101

 

Kubernetes Networking 101

Kubernetes, an open-source container orchestration platform, has revolutionized how developers deploy and manage applications. As organizations increasingly adopt Kubernetes, understanding its networking capabilities becomes crucial for efficient container communication. This blog post will delve into Kubernetes networking, exploring its architecture, components, and critical concepts.

At its core, Kubernetes networking enables seamless communication between containers running on different nodes within a cluster. To achieve this, Kubernetes leverages a flexible and extensible networking model. The architecture comprises three main components: Pods, Services, and Ingress.

 

Highlights: Kubernetes Networking

  • The Kubernetes History

Kubernetes is an open-source cluster management tool released by Google in June 2014. Google has said it launches over 2 billion containers per week, and Kubernetes was designed to control and manage the orchestration of all these containers along with container networking. Initially, they built a Borg and Omega system, resulting in Kubernetes.

All lessons learned from Borg to Kubernetes are now passed to the open-source community. Kubernetes went 1.0 in July 2015 and is now at version 1.3.0. The Kubernetes deployment supports GCE, AWS, Azure, vSphere, and bare metal, and there are a variety of Kubernetes networking configuration parameters. Kubernetes forms the base for OpenShift networking

  • Kubernetes information check.

For additional information before you with Kubernetes networking 101, the post on Kubernetes chaos engineering discusses the need to stress and break a system, which is the only way to understand and optimize fully. Chaos Engineering starts with a baseline and introduces several controlled experiments. Then we have Kubernetes security best practice discuss the Kubernetes attack vectors and how to protect against them.

 

Before you proceed, you may find the following posts helpful:

  1. OpenShift SDN
  2. Docker Default Networking 101
  3. OVS Bridge
  4. Hands-On Kubernetes

 



Kubernetes Networking 101.

Key Kubernetes Networking 101 Discussion points:


  • Introduction to Kubernetes Networking.

  • Discussion of Microservices.

  • The challenges around network proximity.

  • Highlighting the Kubernetes Networking concepts.

  • Kubernetes Networking operation and traffic flow.

 

Introducing Kubernetes Networking

At a very high level, Kubernetes Networking 101 enables a group of hosts to be viewed as a single compute instance. The single compute instance, consisting of multiple physical hosts, is used to deploy containers. This offers an entirely different abstraction level to our single-container deployments.

Users start to think about high-level application services and the concept of service design only. They are no longer concerned with individual container deployment as the orchestrator looks after the deployment, scale, and management.

For example, if users specify to the orchestration system they want a specific type of application with defined requirements, now deploy it for me. The orchestrator manages the entire rollout, specifies the targeted hosts, and manages the container lifecycle. The user doesn’t get involved with host selection. This abstraction allows users to focus only on design and workload requirements – the orchestrator takes care of all the low-level deployment and management details.

Kubernetes Networking 101
Diagram: Kubernetes Networking 101

 

1. Pods:

Pods are the fundamental building blocks of Kubernetes, consisting of one or more containers that share a common network namespace. Each pod receives a unique IP address, allowing containers within the pod to communicate via localhost. However, communication between different pods requires additional networking components.

2. Services:

Services provide a stable and abstracted network endpoint to access a set of pods. By grouping pods based on a standard label, services ensure that applications can discover and communicate with each other seamlessly. Kubernetes offers four types of services: ClusterIP, NodePort, LoadBalancer, and ExternalName. Each service type caters to specific use cases, providing varying levels of accessibility.

3. Ingress:

Ingress is a Kubernetes resource that enables inbound connections to reach services within the cluster. It acts as a traffic controller, routing external requests to the appropriate service based on rules defined in the Ingress resource. Additionally, Ingress supports TLS termination, allowing secure communication with services.

Networking Concepts:

To comprehend Kubernetes networking fully, it is essential to grasp critical concepts that govern its behavior.

1. Cluster Networking:

Cluster networking refers to the communication between pods running on different nodes within a Kubernetes cluster. To establish node connectivity, Kubernetes leverages various networking solutions, such as overlay networks and software-defined networking (SDN). Popular SDN solutions include Calico, Flannel, and Weave.

2. DNS Resolution:

Kubernetes provides a built-in DNS service that enables easy discovery of services within the cluster. Each service is assigned a DNS name, which can be resolved to its corresponding IP address. This allows applications to communicate with services using their DNS names, enhancing flexibility and decoupling.

3. Network Policies:

Network policies define rules that dictate how pods communicate with each other. Administrators can enforce fine-grained access control and secure application traffic using network policies. Policies can be based on various criteria, such as IP addresses, ports, and protocols.

 

Microservices

Distributed systems are more fine-grained now, with Kubernetes driving microservices. Microservices is a fast-moving topic involving the breaking down applications into many specific services. All services have their lifecycles, collaborating. Splitting the Monolith with microservices is not a new idea ( the term is ), but the emergence of new technologies has a profound effect.

The specific domains/containers require constant communication and access to each other’s services. Therefore, a strategy needs to be maintained to manage container interaction. For example, how do we scale containers? What’s the process for container failure? How do we react to container resource limits? 

Although Docker does help with container management, Kubernetes orchestration works on a different scale and looks at the entire application stack allowing management at a service/application level.

We need a management and orchestration system to take full advantage of the portability of containers and microservices. Containers can’t just be thrown into a sea of computing and expect to tie themselves together and work efficiently.

A management tool is required to govern and manage the life of containers, where they are placed, and to whom they can talk. Containers have a complicated existence, and many pieces are used to patch up their communication flow and management. We have updates, high availability, service discovery, patching, security, and networking. 

The most important aspect to remember with Kubernetes or any content management system is that they are not concerned with individual container placement. Instead, the focus is on workload placement. Users enter high-level requirements, and the scheduler does the rest – where, when, and how many?

 

Kubernetes networking 101 and network proximity.

Looking at workloads to analyze placement optimizes application deployment. For example, some processes that are part of the same service will benefit from network proximity. Front-end tiers sending large chunks of data to a backend database tier should be close to each other, not trombone across the network Kubernetes host for processing.

Likewise, when common data needs to be accessed and processed, it makes sense to put containers “close” to each other in a cluster. The following diagram displays the core Kubernetes architecture.

 

Kubernetes Networking 101: The Constructs

Kubernetes uses four primary constructs to build the application stack – Pods, Services, Labels, and Replication Controllers. All constructs are configured and combined, resulting in a complete application stack with all management components—pods group similar containers on the same hosts.

Labels tag objects; replication controllers manage the desired state at a POD level, not container level and services enable Pod-to-Pod communication. These constructs enable the management of your entire application lifecycle instead of individual application components. Construct definition is through configuration files either in YAML or JSON format.

 

The Kubernetes Pod

Pods are the smallest scheduling unit in Kubernetes and hold a set of closely related containers, all sharing fate and resources. Containers in a Pod share the same Kubernetes network namespace and must be installed on the same host. The main idea of keeping similar or related containers together is that processing is performed locally and does not incur any latency traversing from one physical host to another. As a result, local processing is always faster than remote processing. 

Pods essentially hold containers with related pieces of the application stack. The critical point is that they are ephemeral and follow a specific lifecycle. They should come and go without service interruption as any service-destined traffic directed should be towards the “service” endpoint IP address, not the Pod IP address.

Even Though Pods have a pod-wide-IP address, service reachability is carried out with service endpoints. Services are not as ephemeral ( although they can be deleted ) and don’t go away as much as Pods. They act as the front-end VIP to back-end Pods ( more on later ). This type of architecture hammers home the level of abstraction Kubernetes seeks.

 

Pod definition file

The following example displays a Pod definition file. We have basic configuration parameters, such as the Pod’s name and ID. Also, notice that the object type is set to “Pod.” This will be set according to the object we are defining. Later we will see this set as “service” for defining a service endpoint.

In this example, we define two containers – “testpod80” and “testpod8080”. We also have the option to specify the container image and Label. As Kubernetes assigns the same IP to the Pod where both containers live, we should be able to browse to the same IP but different port numbers, 80 or 8080. Traffic gets redirected to the respective container.

 

Kubernetes Pod

Kubernetes labels

Containers within a Pod share their network namespaces. All containers within can reach each other’s ports on localhost. This reduces the isolation between containers, but any more isolation would go against why we have Pods in the first place. They are meant to group “similar” containers sharing the same resource volumes, RAM, and CPU. For Pod segmentation, we have labels – a Kubernetes tagging system.

Labels offer another level of abstraction by tagging items as a group. They are essentially key-value pairs categorizing constructs. When we create Kubernetes constructs, we can set a label, which acts as a tag for that construct.

This means you can access a group of objects by specifying the label assigned to those objects. For example, labels distinguish containers as part of a web or database tier. The “selector” field tells Kubernetes which labels to use in finding Pods to forward traffic to.

 

Replication Controller

Container Scheduler

The replication controller ( RC ) manages the lifecycle and state of Pods. It ensures the desired state always matches the actual state. When you create an RC, you define how many copies ( aka replicas) of the Pod you want in the cluster.

The RC maintains that the correct numbers are running by creating or removing Pods at any time. Kubernetes doesn’t care about the number of containers running in a Pod; its only concern is the number of Pods. Therefore, it works at a Pod level.

The following is an example of an RC service definition file. Here you can see that the desired state of replicas is “2”. The replica of 2 means that the number of pods each controller should maintain is 2. Changing the number up or down will either increase or decrease the number of Pods the replication controller manages.

For example, if the RC notices too many, it will stop some from returning the replication controller to the desired state. The RC keeps track of the desired state and returns it to the state specified in the service definition file. We may also assign a label for grouping replication controllers.

 

Kubernetes replication controller

 

Kubernetes Services

Service endpoints enable the abstraction of services and the ability to scale horizontally. Essentially, it is an abstraction defining a logical set of Pods. Services represent groups of Pods acting as one, allowing Pods to access services in other Pods without directing service-destined traffic to the Pod IP. Remember, Pods are short-lived!

The service endpoint’s IP address is from the “Portal Net” range defined on the API service. The address is local to the host, so make sure it doesn’t clash with the docker0 bridge IP address.

Pods are targeted by accessing a service that represents a group of Pods. A service can be viewed with a similar analogy to a load balancer, sitting in front of Pods accepting front-end service-destined traffic. Services act as the main hooking point for service / Pod interactions. They offer high-level abstraction to Pods and the containers within.

All traffic gets redirected to the service IP endpoint, which performs the redirection to the correct backend. Traffic hits the service IP address ( Portal Net ), and a Netfilter IPtable rules forward to a local host high port number.

The -proxy service creates the high port number forming the basis for load balancing. The load balancing object then listens to that port. The Kub proxy acts as a full proxy, maintaining two different TCP connections.

One separates the connection from the container to the proxy and another from the proxy to the load-balanced destination. The following is an example of a service definition file. The service listens on port 80 and sends traffic to the backend container port on 8080. Notice how the object kind is set to “service” and not “Pods” like in the previous definition file.

 

Kubernetes Services

Kubernetes Networking 101 Model

The Kubernetes networking model details that each Pod should have a routable IP address. This makes communication between Pods easier by not requiring any NAT or port mappings we had with earlier versions of Docker networking.

With Kubernetes, for example, we can have a web server and database server placed in the same Pod and use the local interface for cross-communication. Furthermore, as there is no additional translation, performance is better than a NAT’d approach.

 

Kubernetes network proxy

Kubernetes fulfills service -> Pods integration by enabling a network proxy called the Kube-proxy on every node in a cluster. The network proxy is always there even if Pods are not running. Its main task is to route traffic to the correct Pod and can do TCP, UDP stream forwarding or round robin TCP, UDP forwarding.

The Kube-proxy captures service-destination traffic and proxies requests from the service endpoint back to the application’s Pod. The traffic is forwarded to the Pods on the target port defined in the definition file. The target port is a random port assigned during service creation.

To make all this work, Kubernetes uses IPtables and Virtual IP addresses.

When using Kubernetes alongside, for example, Opencontrail, the Kube-proxy is disabled on all hosts, and the OpenContrail router module implements connectivity via overlays ( MPLS over UDP encapsulation ). Another vendor on the forefront is Midokura, the co-founder behind OpenStack Project Kuryr. This project aims to bring any SDN plugin (MidoNet, Dragon flow, OVS, etc.) to Containers—more on these another time.

 

Kubernetes Pod-IP approach

The Pods IP address is reachable by all other Pods and hosts in the Kubernetes cluster. The address is not usually routable outside of the cluster. This should not be too much of a concern as most traffic stays within application tiers inside the cluster. Mapping external load-balancers achieve any inbound external traffic to services in the cluster.

The Pod-IP approach assumes that all Pods can reach each other without creating specific links. They can access each other by IP rather than through a port mapping on the physical host. Port mappings hide the original address by performing a masquerade – Source NAT.

Similar to how your home residential router hides local PC and laptop IP addresses from the public Internet. Cross-node communication is much simpler as every Pod has an IP address. There isn’t any port mapping or NAT like there is with default docker networking. If the Kube-proxy receives traffic for a Pod, not on its host, it simply forwards the traffic to the correct Pod-IP for that service.

The IP per POD offers a simplified approach to K8 networking. A unique IP per host would potentially need port mappings on the host IP as the number of containers increases. Managing port assignment would become an operational and management burden, similar to earlier versions of Docker. Conversely, a unique IP per container would surely hit scalability limits.

 

Kubernetes PAUSE container

Kubernetes has what’s known as a PAUSE container, also referred to as a Pod infrastructure container. It handles the networking by holding the networking namespace and IP address for the containers on that Pod. Some refer to the PAUSE container as an implementation detail you can safely ignore.

Each container uses a Pod’s “mapped container” mode to connect to the pause container. The mapped container mode is implemented with a source and target container grouping. The source container is the user-created container, and the target container is the infrastructure pause container.

Destination Pod IP traffic first lands on the pause container and gets translated to the backend containers. The pause container and the user-built containers all share the same network stack. Remember we created a service destination file with two containers – port 80 and port 8080? It is the pause container that listens on these port numbers.

In summary, the Kubernetes model introduces three methods of communication.

  • a) Pod-to-Pod communication directly by IP address. Kubernetes has a Pod-IP-wide metric simplifying communication.
  • b) Pod-to-Service Communication – Clients’ traffic is directed to the virtual service IP, which is then intercepted by the kub-proxy process ( running on all hosts) and directed to the correct Pod.
  • c) External-to-Internal Communication – external access is captured by an external load balancer that targets nodes in a cluster. The Kub proxy determines the correct Pod to send traffic to—more on this in a separate post.

 

Docker & Kubernetes networking comparison

Docker uses host-private networking. The Docker engine creates a default bridge, and every container gets a virtual ethernet to that bridge. The veth acts like a pipe – one end is mapped to the docker0 bridge namespace and the other to the container’s Linux namespace. This provides connectivity between containers on the same Docker bridge.

All containers are assigned an address from the 172.17.42.0 range and 172.17.42.1 to the default bridge acting as the container gateway. Any off-host traffic requires port mappings and NAT for communication. Therefore, the container’s IP address is hidden, and the network would see the container traffic coming from the docker nodes’ physical IP address. 

The effect is that containers can only talk to each other by IP address on the same virtual bridge. Any off-host container communication requires messy port allocations. Recently, there have been enhancements to docker networking and multi-host native connectivity without translations. Although there are enhancements to the Docker network, the NAT / Port mapping design is not a clean solution.

The K8 model offers a different approach, and the docker0 bridge gets a routable IP address. Any outside host can access that Pod by IP address rather than through a port mapping on the physical host. Kubernetes has no NAT for container-to-container or container-to-node traffic.

 

Conclusion:

Understanding Kubernetes networking is crucial for building scalable and resilient applications within a containerized environment. By leveraging its flexible architecture and components like Pods, Services, and Ingress, developers can enable seamless container communication and ensure efficient network management. Moreover, comprehending network concepts like cluster networking, DNS resolution, and network policies empowers administrators to establish robust and secure communication channels within the Kubernetes ecosystem. Embracing Kubernetes networking capabilities unlocks the full potential of this powerful container orchestration platform.

 

Container Networking

 

docker network types

 

Container Networking

Containerization has revolutionized the way we develop, deploy, and manage applications. Organizations have gained newfound flexibility and scalability by encapsulating applications in lightweight, isolated containers. However, as the number of containers increases, so does the networking complexity among them. This blog post will explore container networking, its challenges, solutions, and best practices.

Container networking refers to the communication and connectivity between containers within a distributed system. Unlike traditional monolithic applications, containers are designed to be ephemeral and can be dynamically created, scaled, and destroyed. This dynamic nature necessitates a flexible and efficient networking infrastructure to facilitate seamless communication between containers, regardless of their physical location.

 

Highlights: Container Networking

  • Network Services

The most common network service is to allow a source to reach an application endpoint. Nowadays, the network function no longer solely satisfies endpoint reachability; it is fully integrated into the application. And in the case of OpenShift networking, the Route and Sevice construct provides both reachability and an abstraction layer for application access.

In the past, applications had three standard components; cache, web server, and database. Applications look very different now. Several services are interacting, completely decoupled into units, and packaged in containers; all are mobile and may move around.

 

Before you proceed, you may find the following helpful:

  1. Container Based Virtualization
  2. Neutron Network

 



What is Container Networking?

Key Container Networking Discussion points:


  • Introduction to Container networking and its operations.

  • Discussion Docker default networking and different Docker network types.

  • Namespaces, port mapping and traffic flow.

  • Kubernetes networking and the Pod concept.

  • A final note on container services.

 

Back to basics with Container Networking

Docker Networking

The Docker networking model uses a virtual bridge network by default, defined per host, and a private network where containers attach. The container’s IP address is allocated a private IP address, which indicates containers operating on different machines cannot communicate with each other.

In this case, you will have to map host ports to container ports and then proxy the traffic to reach across nodes with Docker. Therefore, it is up t the administrator to avoid port clashes between containers. Kubernetes networking handles this differently.

 

Challenges in Container Networking:

Container networking presents several challenges that must be addressed to ensure optimal performance and reliability. Some of the key challenges include:

1. Network Isolation: Containers should be isolated from each other to prevent unauthorized access and potential security breaches.

2. IP Address Management: Containers are assigned unique IP addresses, which can quickly become challenging to manage as the number of containers grows.

3. Scalability: As the container ecosystem expands, the networking infrastructure must scale effortlessly to accommodate the increasing number of containers.

4. Service Discovery: Containers need a reliable mechanism to discover and communicate with other services within the network, especially in a microservices architecture.

Solutions and Best Practices:

To overcome these challenges, several solutions and best practices have emerged in the realm of container networking:

1. Container Network Interface (CNI): CNI is a specification that defines how container runtimes interact with networking plugins. It enables easy integration of various networking solutions into container orchestration platforms like Kubernetes and Docker.

2. Overlay Networking: Overlay networks create a virtual network that spans multiple hosts, allowing containers to communicate seamlessly, regardless of physical location. Technologies like VXLAN, GRE, and WireGuard are commonly used for overlay networking.

3. Network Policies: Network policies define the rules and restrictions for incoming and outgoing traffic between containers. Organizations can enforce security and control network traffic flow within their containerized environments by implementing network policies.

4. Service Mesh: Service mesh technologies, such as Istio and Linkerd, provide advanced networking capabilities, including traffic management, load balancing, and observability. They enhance the resilience and reliability of containerized applications by offloading complex networking tasks from individual services.

 

Container Networking: A Different Application Philosophy

Computing is distributed over multiple elements, and they all interact arbitrarily. Network integration allows the application to be divided into several microservice components. Microservices allow the application to be packaged into pieces and deployed on different hosts or even different cloud providers.

The application stack no longer belongs to a single server. Small composable units enhance application replication and fault tolerance services. All this is made possible by using containers and the ability to interconnect them.

Containers offer a single-purpose environment. They are a bunch of lightweight namespaces and processes sharing a common kernel. Typically, you don’t run a full stack in a single container.

Ideally, only one process per container, which makes them very lightweight. VMs with guest O/S is resource-heavy; containers are a far better option if the application can be containerized.

However, containers offer an utterly different endpoint type for the network. With virtual machine spinning, they arrive and disappear quickly, measured in milliseconds, not seconds or minutes. The speed is down to their light properties. Some containerized application transactions only live for the length of transaction time. The infrastructure and network must be pre-built to support this type of endpoint.

Despite the advantages of containerization, keep in mind that Docker container security and Docker security options are enabled at each point in the defense layer.

 

Introducing Docker Network Types

Docker Default Networking 101

Docker networking comes with several Docker network types and setups. The latest release is Docker version 1.10 and has some enhancements, including linking with user-defined networks. They are other solutions available to enhance Docker networking functionality.

Docker is pluggable and allows ecosystem partners to plug into Docker networking. Project Calico offers a pure IP-based solution that utilizes the same principles of the Internet. Every host is an IP router. Calico uses a Felix agent and a BGP BIRD demon. This would be a clean option if the application only needs Layer 3 connectivity. 

The weave is another solution that operates an overlay function and aims to fit the multi-data center requirements. Each host in a Weave network thinks it belongs to one large switched fabric. The physical locations are abstracted, and they all have reachability. A multi-datacenter solution must concern itself with metrics other than endpoint reachability.

 

Container Networking with Linux Kernel and User Namespaces

Inside each container, a bunch of resources, such as network interfaces and file systems, are unique. These resources look as if they are isolated even though the containers share the Linux kernel. Global resources are abstracted to appear as if they were unique per container. An abstraction that is made available by the use of Linux namespaces.

Namespaces initially provided resource isolation for the first Linux containers project, offering a process virtualization solution. They do not create additional operating system instances on the host but instead use a single system with resource isolation.

Similarly, FreeBSD, where Jails provides resource isolation while running one kernel instance. In 2002, mount namespaces were the first type of Linux namespace with kernel 2.4.19. User namespaces emerged with kernel 3.8. 

  • The Different Namespaces

Containers have namespaces for each type of resource. We have six namespaces. 

    • Mount namespace makes the container feel like it has its filesystem. 
    • UTS namespace offers individual hostnames and domain names. 
    • User namespace provides isolation between the user and group IDs. 
    • IPC namespace isolates message queue systems. 
    • PID namespace offers different PIDs inside the container.

Finally, the network namespace gives the container a separate network stack. When you issue the docker ps command, you will see what ports are bound; these ports are on the namespace network interface.

 

Docker Networking and Docker Network Types

Install docker creates three network types – bridge, host, and none. You cannot delete these networks; you only interact with the default bridge network. There is the option to create user-defined networks and customized plugins. 

Network plugins (LibNetwork project) extend the docker network to support additional networking features such as IP VLAN or macvlan. User-defined networks can take the form of bridge or overlay networks.

Bridge networks have a single-host local scope, and overlay networks have multi-host global scope. The diagram below displays the default bridge and the corresponding attached containers. The driver is “default,” meaning it has local scope.

 

Container Networking

 

The user-defined bridge is similar to the default bridge0. Containers from the same host are added and can cross-communicate. External access is not prohibited, but you can expose network sections with port mappings.

The user-defined overlay networking feature enables multi-host networking using the VXLAN networking driver called libnetwork and Docker’s libkv library. One of the requirements for the overlay function to work is for a valid key-value store.

The Docker libkv library supports Consul, Etcd, and ZooKeeper. With Docker default networking, a veth pair is created – one veth pair is put inside the container and the other outside in the namespaces. All are connected via the docker bridge. The veth is mapped to appear as eth0 in the container, using Linux namespaces.

Container Networking

 

Container networking, port mapping, and traffic flow.

Docker container networking cross-communicate if they are on the same machine and thus connect to the same virtual bridge. Containers can also connect to multiple networks at the same time. By default, containers on different machines can not reach other. Cross-communication on different nodes must be allocated ports on the machine’s IP address, which are then proxied to the containers.

Port mapping is used to provide access to the container from the outside. Docker allocates a DNAT port in the range of 49153 – 65535. This additional functionality continues to use the default Docker0 bridge but adds IPtables rules for the DNAT.

When you spin up a container and do a port mapping, you can see it inside the docker ps command that you have a port mapping from, for example, the host 8080 to container 80. IPtables is setting a port mapping between 8080 to the IP addresses assigned to the container.

The problem with Docker is that you may have to coordinate ports and plenty of NAT. NAT was designed to address the shortage of IPv4 addresses and was only meant to be used for a short period. It is so ingrained in people’s minds we still see it come out in fresh designs.

Ports and NAT are problematic at scale and expose users to cluster-level issues outside their control. It brings the risk of port conflicts and many complexities to scheduling. 

 

  • A key point: Kubernetes

Kubernetes networking does not use any NAT. Instead, it applies IP addresses at the Pod scope level. Remember that containers within a Pod share network namespaces, including their IP address. This means containers within a Pod can all reach each other’s ports on localhost. Kubernetes makes finding and configuring Kubernetes services much easier due to the unique IP addresses per Pod model.

 

Kubernetes Networking 101: IP-per-pod-model

Kubernetes network namespace has two fundamental abstractions – Pods and Services. Pods are essentially scheduling ATOMs in Kubernetes. They represent a group of containers, tightly integrated, sharing resources and fate. An example of an application container grouping in a Pod might be a file puller and a web server.

Frontend / Backend tiers usually fall outside this category as they can be scaled separately. Pods share a network namespace and talk to each other as local hosts.

Pods are assigned a private IP that is routable within the internal fabric. Docker doesn’t give you an IP; you must do weird things like going through a host and exposing a port. Not a great idea, as you may have issues and operational complexities with the deployment of ports.

With Kubernetes, all containers talk to each other, even across nodes, without NAT. The entire solution is NAT-less, flat address space. Pods can talk to Pods without any translations. Communications on ports can be done but with well-known port numbers avoiding service discovery systems like DNS-SD, Consul, or Etcd.

 

Container network and services

The second abstraction is services. Services is a similar analogy to that of a load balancer. They are groups of Pods that act as one. It may be better to reference the service with an IP address, not a Pod. This is because Pods can go away, but services are more dedicated.

A typical flow would be something like this – a client on a cluster looks for IP for a particular service. The Kubernetes nodes that it is running on do an iptables DNAT. Instead of going to the service, it reroutes to the Kube proxy. The Kube proxy is a proxy running on every Kubernetes node.

It programs iptables rules to trap access to service IPs and redirect them to the backends using round-robin load balancing. It places a watch on the API server to know what pods are active and ready to serve requests.

Several implementations include Google Compute Engine, Flannel, Calico, and OVS with GRE/VxLAN to support the IP-per-pod model. OpenVSwitch connects Pods on different hosts with GRE or VxLAN. The Linux bridge replaces the docker0 bridge, encapsulating traffic to and from Pods. Flannel may also be used with Kubernetes.

It creates an overlay network and gives a subnet to each host. Flannel can be used on cloud providers that cannot offer an entire /24 to each host. Flannels’ flannel agent runs on each host and controls the IP assignment. Calico, already mentioned, is also an IP-based solution that relies on traditional BGP.

Conclusion:

Container networking is a critical aspect of modern application development and deployment. As organizations continue to embrace containerization, understanding the challenges and implementing appropriate solutions and best practices is crucial for building a robust and efficient networking infrastructure. By leveraging container networking technologies, organizations can unlock the full potential of containerized applications, enabling seamless communication, scalability, and security in their distributed systems.

 

docker network types

container

Container Scheduler

 

 

Container Scheduler

In modern application development and deployment, containerization has gained immense popularity. Containers allow developers to package their applications and dependencies into portable and isolated environments, making them easily deployable across different systems. However, as the number of containers grows, managing and orchestrating them becomes complex. This is where container schedulers come into play.

A container scheduler is a crucial component of container orchestration platforms. Its primary role is to manage the allocation and execution of containers across a cluster of machines or nodes. By efficiently distributing workloads, container schedulers ensure optimal resource utilization, high availability, and scalability.

 

Highlights: Container Scheduler

  • Traditional Application

Applications started with single server deployments and no need for a container scheduler. However, this was an inefficient deployment model, yet it was widely adopted. Applications mapped to specific hardware do not scale. The landscape changed, and the application stack was divided into several tiers. Decoupling the application to a loosely coupled system is a more efficient solution. Nowadays, the application gets divided into different components, spread across the network with various systems, dependencies, and physical servers.

Example: OpenShift Networking

An example of this is with OpenShift networking. OpenShift is based on Kubernetes and borrows many of the Kubernetes constructs. For pre-information, you may find this post informative on Kubernetes and Kubernetes Security Best Practice

  • The Process of Decoupling

The world of application containerization drives the ability to decouple the application. As a result, there has been a massive increase in containerized application deployments and the need for a container scheduler. With all these changes, remember the need for new security concerns to be addressed with Docker container security.

The Kubernetes team conducts regular surveys on container usage, and their recent figures show an increase in all areas of development, testing, pre-production, and production. Currently, Google initiates about 2 billion containers per week. Most of Google’s apps/services, such as its search engine, Docs, and Gmail, are packaged as Linux containers.

 



Container Scheduler.

Key Container Scheduler Discussion points:


  • Introduction to containerized technologies.

  • The role of the scheduler.

  • Discussion on the Kubernetes Orchestrator.

  • Kubernetes POD and Labels.

 

For pre-information, you may find the following helpful

  1. Kubernetes Network Namespace
  2. Docker Default Networking 101

 

 Back to basics with the container scheduler

With a container orchestration layer, we are marrying the container scheduler’s decisions on where to place a container with the primitives provided by lower layers. The container scheduler knows where containers “live,” and we can consider it the absolute source of truth concerning a container’s location.

So, the primary task of a container scheduler is to start containers on the most suitable host and connect them. It has to manage failures by doing automatic fail-overs, and it needs to be able to scalable containers when there is too much data to process/compute for a single instance.

 

Key Features of Container Schedulers:

1. Resource Management: Container schedulers allocate appropriate resources to each container, considering factors such as CPU, memory, and storage requirements. This ensures that containers operate without resource contention, preventing performance degradation.

2. Scheduling Policies: Schedulers implement various scheduling policies to allocate containers based on priorities, constraints, and dependencies. They ensure containers are placed on suitable nodes that meet the required criteria, such as hardware capabilities or network proximity.

3. Scalability and Load Balancing: Container schedulers enable horizontal scalability by automatically scaling up or down the number of containers based on demand. They also distribute the workload evenly across nodes, preventing any single node from becoming overloaded.

4. High Availability: Schedulers monitor the health of containers and nodes, automatically rescheduling failed containers to healthy nodes. This ensures that applications remain available even in node failures or container crashes.

Popular Container Schedulers:

1. Kubernetes: Kubernetes is an open-source container orchestration platform with a powerful scheduler. It provides extensive features for managing and orchestrating containers, making it widely adopted in the industry.

2. Docker Swarm: Docker Swarm is another popular container scheduler provided by Docker. It simplifies container orchestration by leveraging Docker’s ease of use and integrates well with existing workflows.

3. Apache Mesos: Mesos is a distributed systems kernel that provides a framework for managing and scheduling containers and other types of workloads. It offers high scalability and fault tolerance, making it suitable for large-scale deployments.

Benefits of Container Schedulers:

1. Efficient Resource Utilization: Container schedulers optimize resource allocation, allowing organizations to maximize their infrastructure investments. By eliminating resource wastage, they reduce operational costs.

2. Improved Application Performance: Schedulers ensure containers have the necessary resources to operate at their best, preventing resource contention and bottlenecks.

3. Simplified Management: Container schedulers automate the deployment and management of containers, reducing manual effort and enabling faster application delivery.

4. Flexibility and Portability: With container schedulers, applications can be easily moved and deployed across different environments, whether on-premises, in the cloud, or in hybrid setups. This flexibility allows organizations to adapt to changing business needs.

 

Containers – Raising the Abstraction Level

Container networking raises the abstraction level. The abstraction level was at a VM level, but with containers, the abstraction is moved up one layer. So instead of virtual hardware, you have an idealized O/S stack.

Containers change the way applications are packaged. They allow application tiers to be packaged and isolated, so all dependencies are confined to individual islands, not conflicting with other stacks. It provides a simple way to package all application pieces into an easily deployable unit. The ability to create different units radically simplifies deployment.

It creates a predictable isolated stack with ALL userland dependencies. Each application is isolated from others, and dependencies are sealed in. Dependencies are the natural killer as they can slow down deployment lifecycles. Containers combat this and fundamentally change the operational landscape. Docker and Rocket are the main Linux application container stacks in production.

Containers don’t magically appear. They need assistance with where to go; this is the container scheduler’s role. The main job of the scheduler is to start the container on the correct host and connect them. In addition, the scheduler needs to monitor the containers and deal with container/host failures.

The schedulers are Docker Swarm, Google Kubernetes, and Apache Mesos. Docker Swarm is probably the easiest to start with, and it’s not attached to any cloud provider. The container sends to the cluster scheduler several requirements. For example, I have this amount of resources and want to run five copies of this software with this amount of CPU and disk space – now find me a place.

 

Kubernetes – Container scheduler

Hand on Kubernetes. Kubernetes is an open-source cluster solution for containerized environments. It aims to make deploying microservice-based applications easy by using the concepts of PODS and LABELS to group containers into logical units. All containers run inside a POD.

PODS are the main difference between Kubernetes and other scheduling solutions. Initially, Kubernetes focused on continuously running stateless and “cloud native” stateful applications. In the coming future, it is said to support other workload types.

 

container scheduler

 

Kubernetes Networking 101

Kubernetes is not just interested in the deployment phase and works across the entire operational model – scheduling, updating, maintenance, and scaling. Unlike orchestration systems, it actively ensures the state matches the user’s requirements. Kubernetes is involved in monitoring and healing if something goes wrong.

The team at Google refers to this as a flight control mechanism. It provides the cluster and the decoupling between. The application containers view the world as a sea of computing, an entirely homogenous (similar kind) cluster. Every machine you create in your fleet looks the same. There is the complete decoupling of the application from low-level computing.

The user does not need to care about physical placement anymore. The unit of work has changed and becomes a service. The administrator only needs to care about services, such as the amount of CPU, RAM, and disk space. The unit of work presented is now at a service level. The physical location is abstracted, all taken care of by the Kubernetes components.

This does not mean that the application components can be spread randomly. For example, some application components require the same host. But it is no longer the user’s job to select the hosts. Kubernetes provides an abstracted layer over the infrastructure, allowing this type of management.

The scheduling of containers is on a homogenous pool of resources. The VM disappears, and you think about resources such as CPU and RAM. Everything else, like location, disappears.

 

Kubernetes pod and label

The main building block for Kubernetes clusters is PODS and LABELS. So the first step is to create a cluster, and once complete, you can proceed to PODS and other services. The diagram below shows the creation of a Kubernetes cluster. It consists of a 3-node instance created in us-east1-b.

 

containers

 

A POD is a collection of applications running within a shared context. Containers within a POD share fate and some resources, for example, volumes and IP addresses. They are usually installed on the same host. When you create a POD, you should also create a kubernetes replication controller.

It monitors POD health and starts new PODS as required. Most PODS should be built with a replication controller, but it may not be needed if your POD is short-lived and is writing non-persistent data that won’t survive a restart. There are two types of PODS a) single container and b) Multi-container.

The following diagram displays the full details of a POD named example-tglxm. It has a label run=example located in the default network (namespace).

 

Container POD

 

 

A POD may contain either a single container with a private volume or a group with a shared volume. If a container fails within a POD, it is automatically restarted by the Kubelet. However, if an entire POD or host fails, it needs to be restarted by the replication controller.

The replication to another host must be specifically configured. By default, it is not an automatic process. The Kubernetes replication controller dynamically resizes things and ensures that the required number of PODS and containers are running. If there are too many, it will kill some; if not enough, it will start some more.

Kubernetes operates with the concept of LABELS – a key-value pair attached to objects, such as a POD. A label is a tag that can be used to query against. Kubernetes is an abstraction, and you can query whatever item you want using a label in an entire cluster.

For example, you can select all frontend containers with a label called “frontend”; it then selects all front ends. The cluster can be anywhere. Labels can also be building blocks for other services, such as port mappings. For example, a POD whose labels match a specific service selector is accessible through the defined service’s port.