The move to the cloud

Cloud workloads are typically managed with Kubernetes as they move to the cloud. Kubernetes’ popularity can be attributed to its declarative nature: It abstracts infrastructure details and allows users to specify which workloads they want to run. App teams only need to set up configurations in Kubernetes to deploy their applications; they don’t need to worry about how workloads are deployed, where workloads run, or other details like networking.

To achieve this abstraction, Kubernetes manages the creation, shutdown, and restart of workloads. As a typical implementation, workloads can be scheduled on any available network resource (physical host or virtual machine) based on their requirements. Kubernetes clusters consist of resources that run workloads.

As needed, Kubernetes restarts unresponsive nodes based on the status of workloads (which are deployed as pods in Kubernetes). It also manages the network between pods and hosts and pod communication. Network plug-ins allow you to select the type of networking technology. Although the network plug-in has some configuration options, you cannot directly control networking behavior (either IP address assignment or scheduling).

An Orchestration Tool

Kubernetes has quickly become the de facto orchestration tool for deploying Kubernetes microservices and containers to the cloud. It offers a way of running groups of resources as a cluster and provides an entirely different abstraction level to single container deployments, allowing better management. From a developer’s perspective, it will enable the rolling out of new features that align with business demands while not worrying about the underlying infrastructure complexities.

Security Pitfalls

But there are pitfalls to consider when considering Kubernetes security best practices and Docker container security. Security teams must maintain the required visibility, compliance, and control, which is difficult because they do not control the underlying infrastructure. To help you on your security journey, you should introduce a chaos engineering project, specifically chaos engineering kubernetes.

This will help you find the breaking points and performance-related issues that may indirectly create security gaps in your Kubernetes cluster, making the cluster acceptable to Kubernetes attack vectors.

Before you proceed, you may find the following posts helpful:

1. OpenShift SDN
2. Hands On Kubernetes
3. Kubernetes Network Namespace

Back to basics with Kubernetes security best practice

As workloads move to the cloud, Kubernetes is the most common orchestrator for managing them. The reason Kubernetes is popular is its declarative nature: It abstracts infrastructure details and allows users to specify the workloads they want to run and the desired outcomes.

However, securing, observing, and troubleshooting containerized workloads on Kubernetes are challenges. It needs a range of considerations, from infrastructure choices and cluster configuration to deployment controls, runtime, and network security. Keep in mind that Kubernetes is not secure by default.

Regularly Update Kubernetes:

Keeping your Kubernetes environment up to date is one of the fundamental aspects of maintaining a secure cluster. Regularly updating Kubernetes ensures that you have the latest security patches and bug fixes, reducing the risk of potential vulnerabilities.

Enable RBAC (Role-Based Access Control):

Implementing Role-Based Access Control (RBAC) is crucial for controlling and restricting user access within your Kubernetes cluster. By assigning appropriate roles and permissions to users, you can ensure that only authorized personnel can perform specific operations, minimizing the chances of unauthorized access and potential security breaches.

Use Network Policies:

Kubernetes Network Policies allow you to define and enforce rules for network traffic within your cluster. Utilizing network policies effectively can isolate workloads, control ingress and egress traffic, and prevent unauthorized communication between pods. This helps reduce the attack surface and enhance the overall security posture of your Kubernetes environment.

Employ Pod Security Policies:

Pod Security Policies (PSPs) enable you to define security configurations and restrictions for pods running in your Kubernetes cluster. By defining PSPs, you can enforce policies like running pods with non-root users, disallowing privileged containers, and restricting host namespace sharing. These policies help prevent malicious activities and reduce the impact of potential security breaches.

Implement Image Security:

Ensuring the security of container images is crucial to protect your Kubernetes environment. Employing image scanning tools and registries that provide vulnerability assessments allows you to identify and address potential security issues within your images. Additionally, only using trusted and verified images from reputable sources reduces the risk of running compromised or malicious containers.

Monitor and Audit Cluster Activity:

Implementing robust monitoring and auditing mechanisms is essential for promptly detecting and responding to security incidents. By leveraging Kubernetes-native monitoring solutions, you can gain visibility into cluster activity, detect anomalies, and generate alerts for suspicious behavior. Furthermore, regularly reviewing audit logs helps identify potential security breaches and provides valuable insights for enhancing the security posture of your Kubernetes environment.

Kubernetes Attack Vectors

The hot topic for security teams is how to improve Kubernetes security, implement Kubernetes security best practices within the cluster, and make container networking deployments more secure. Kubernetes is a framework that provides infrastructure and features. However, thinking outside the box regarding Kubernetes’ best security practices would be best.

We are not saying that Kubernetes doesn’t have built-in security features to protect your application architecture! These include logging and monitoring, debugging and introspection, identity, and authorization. But ask yourself: Do these defaults cover all aspects of your security requirements? Remember that much of the complexity with Kubernetes can be abstract with OpenShift networking.

Traditional Security Constructs  

Traditional security constructs don’t work with containers, as we have seen with the introduction of Zero Trust Networking. Containers have a different security model as they are short-lived and immutable. For example, securing containers differs from securing a virtual machine (VM).

For one, the attack surface is smaller as containers usually only live for about a week, as opposed to a VM that may stay online forever. As a result, we need to manage vulnerabilities and compliance through the container scheduler lifecycle – from build tools, image registries, and production.

How do you measure security within a Kubernetes environment? No one can ever be 100% secure, but one should follow certain best practices to harden the gates against bad actors. You can have different levels of security within other parts of the Kubernetes domain. However, a general Kubernetes security best practice would be introducing as many layers of protection between your Kubernetes environment as possible and the bad actors to secure valuable company assets.

Hacker sophistication

Hackers are growing in sophistication. We started with script kiddies in the 1980s and are now entering a world of artificial intelligence ( AI ) and machine learning (ML) in the hands of bad actors. They have the tools to throw everything and anything at your Kubernetes environment, dynamically changing attack vectors on the fly.

If you haven’t adequately secured the domain, once a bad actor gains access to your environment, they can move laterally throughout the application stack, silently accessing valuable assets.

Kubernetes Architecture

Before we delve into some of Kubernetes’ security best practices and the Kubernetes attack vectors, let’s recap Kubernetes networking 101. Within Kubernetes, there will always be four common constructs – pods, services, labels, and replication controllers. All constructs combine to create an entire application stack.

Pods
Pods are the smallest scheduling unit within a Kubernetes environment. They hold a set of closely related containers. Containers within a Pod share the same network namespace and must be installed on the same physical host. This enables processing locally without latency traversing from one physical host to another.

Labels
As I said, containers within a pod share the same network namespaces. As a result, the containers within a pod can reach each other’s ports on localhost. Therefore, we need to introduce a tagging mechanism known as labels. Labels offer another level of abstraction by tagging items as a group. They are essentially key-value pairs categorizing constructs. For example, labels distinguish containers as part of an application stack’s web or database tier.

Replication Controllers (RC)
Then, we have replication controllers. Their primary purpose is to manage the pods’ lifecycle and state by ensuring the desired state always matches the actual state. The replication controllers ensure the correct numbers are running by creating or removing pods at any given time.

Services
Other standard Kubernetes components are services. Services represent groups of pods acting as one, allowing pods to access services in other pods without directing service-destined traffic to the pod IP. Pods are targeted by accessing a service that represents a group of pods. A service can be analogous to a load balancer in front of a pod accepting front-end service-destined traffic.

Kubernetes API server
Finally, the Kubernetes API server validates and configures data for the API objects, which include the constructs mentioned above—pods, services, and replication controllers, to name a few. Also, consider the practice of chaos engineering to better understand your Kubernetes environment and test for resilience and breaking points. Chaos engineering Kubernetes breaks your system to help you better understand it.

Kubernetes Security Best Practice: Security 101

Now that you understand the most commonly used Kubernetes constructs, let’s explore some of the central Kubernetes security best practices and attack vectors.

Kubernetes API
Firstly, a bad actor can attack the Kubernetes API server to attack the cluster further. By design, each pod has a service account associated with it. The service account has full permissions and sufficient privileges to access the Kubernetes API server. A bad actor can get the token mounted in the pod and then use that token to gain access to the API server. Unfortunately, once they can access the Kubernetes API server, they can get information, such as secrets, to compromise the cluster further. For example, a bad actor could access the database server where sensitive and regulatory data is held.

Kubernetes API – Mitigate
Another way to mitigate a Kubernetes API attack is to introduce role-based access control (RBAC). Essentially, RBAC gives rules to service accounts that can be assigned to the entire cluster. RBAC allows you to set permission on individual constructs such as pods and containers and then define rules based on personal use cases. Another Kubernetes best practice would be to introduce an API server firewall. This firewall would work similarly to a standard firewall, where you can allow and block ranges of addresses. Essentially, this narrows the attack surface, hardening Kubernetes security.

Secure pods with network policy
Introducing ingress and egress network policies would allow you to block network access to certain pods. Then, only certain pods can communicate with each other. For example, network policies can be set up to restrict the web front end from communicating directly to the database tier.

There is also a risk of a user implementing unsecured pods. In this case, you can have network policies in place so that if an unsecured pod does appear in the Kubernetes environment, the Kubernetes API will reject it. Remember, left to its defaults, intra-cluster pod-to-pod communication is open. You may think a bad actor could sniff this traffic, but that’s harder to do than in everyday environments. A better strategy to harden Kubernetes security would be to use a network policy that only allows certain pods to communicate with each other.

Container escape: In the past, an unpatched kernel or a bug in the web front-end allowed a bad actor to get out of a container and gain access to other containers or even to the Kubelet process itself. Once the bad actor gets access to the Kubelet client, it can access the API server, gaining a further foothold in the Kubernetes environment.

Sandboxed Pods: One way to mitigate a container escape would be to introduce sandboxed pods inside a lightweight VM. A sandboxed pod provides additional barriers to security as it lies above the kernel itself. Providing layers of protection above the kernel brings many Kubernetes security advantages. As the sandboxed pod runs inside a VM, an additional bug is required to break out of the container if there is a kernel exploit.

Scanning containers: For adequate Kubernetes security, you should know if there is a vulnerability in your container image. Running containers with vulnerabilities exposes the entire system to attacks. Scanning all container images to discover and remove known vulnerabilities would be best. The scanning function should integrate runtime enforcement and remediation capabilities, enabling a secure and compliant SDLC (Software Development Life Cycle).

Restrict access to Kubectl: Kubectl is a powerful command-line interface for running commands against Kubernetes clusters. It allows you to create and update resources in a Kubernetes environment. Due to its potential, one should restrict access to this tool with firewall rules.

Disabling the Kubernetes dashboard is a must, as it has been part of some high-profile compromises. Previously, bad actors accessed a publicly unrestricted Kubernetes dashboard and privileged account credentials to access resources and mine cryptocurrency.

As Kubernetes adoption continues to grow, ensuring the security of your cluster becomes increasingly essential. By following these Kubernetes security best practices, you can establish a robust security foundation for your infrastructure and applications.

Regularly updating Kubernetes, enabling RBAC, utilizing network policies, employing pod security policies, implementing image security measures, and monitoring cluster activity will significantly enhance your Kubernetes security posture. Embracing these best practices will protect your organization’s sensitive data and provide peace of mind in today’s ever-evolving threat landscape.