Kubernetes security best practice

Introduction
Kubernetes has quickly become the de facto orchestration tool for deploying microservices and containers to the cloud. It offers a way of running groups of resources as a cluster. 

Kubernetes provides a completely different abstraction level to single container deployments, allowing better management. From a developer’s perspective, it allows the rolling out of new features that are in line with business demands, while not worrying about the underlying infrastructure complexities. But there are pitfalls to keep in mind when thinking about kubernetes security. Security teams must maintain the required visibility, compliance, and control, which is difficult to do as they do not control the underlying infrastructure.

The hot topic for security teams is how to improve kubernetes security within the cluster and make container deployments more secure? Kubernetes is a framework that provides infrastructure and features. But you need to think outside the box when it comes to kubernetes security best practice. We are not saying that Kubernetes doesn’t have built-in security features to protect your application architecture!. It does: such as logging and monitoring, debugging and introspection, identity, and authorization. But ask yourself, do these defaults cover all aspects of your security requirements?

Traditional security constructs don’t work
Traditional security constructs don’t work with containers. Containers have a different security model as they are short-lived and immutable. For example, securing containers is very different from securing a virtual machine (VM). For one the attack surface is smaller as containers usually only live for about a week as opposed to a VM that may stay online forever. As a result, we need to manage vulnerabilities and compliance through the container lifecycle – from build tools, image registries and in production.

How do you measure security within a Kubernetes environment? No one can ever be 100% secure but one should and must follow certain best practice to harden the gates against bad actors. Essentially, you can have different levels of security within different parts of the Kubernetes domain. 

However, an overall kubernetes security best practice would be to introduce as many layers of security between your Kubernetes environment and the bad actors to secure valuable company assets. 

Hacker sophistication
Hackers are growing in sophistication. We started with script kiddies in the 1980s and now we are entering a world of artificial intelligence ( AI ) and machine learning (ML) in the hands of bad actors. They have the tools to throw everything and anything at your kubernetes environment, dynamically changing attack vectors on the fly.  If you haven’t properly secured the domain, once a bad actor gains access to your environment, they can move laterally throughout the application stack, silently accessing valuable assets.

Kubernetes Architecture

Before we delve into some of kubernetes security best practices, lets recap on its architecture. Within kubernetes, there will always be 4 common constructs – pods, services, labels, and replication controllers. All constructs combine together to create a full application stack.

Pods
Pods are the smallest scheduling unit within a Kubernetes environment that holds a set of closely related containers. Containers within a Pod share the same network namespace and have to be installed on the same physical host. This enables processing to be performed locally and does not incur any latency traversing from one physical host to another.

Labels
As I said, containers within a pod share the same network namespaces. As a result, the containers within a pod can reach each other’s ports on localhost. Therefore we need to introduce a tagging mechanism: this is known as labels.

Labels offer another level of abstraction by tagging items as part of a group. They are essentially key-value pairs categorizing constructs. For example, labels distinguish containers as being part of a web or database tier of an application stack.

Replication Controllers (RC)
Then we have replication controllers. Their main purpose is to manage the lifecycle and state of the pods by ensuring the desired state always matches the actual state. The replication controllers ensure the correct numbers are running. Either creating or removing pods at any given time.

Services 
Other common kubernetes components are services. Services represent groups of pods acting as one and allow pods to access services in other pods without directing service-destined traffic to the pod IP. Pods are targeted by accessing a service that represents a group of pods. A service can be viewed as analogous to a load balancer that sits in front of a pod accepting front end service-destined traffic.

Kubernetes API server
Finally, the kubernetes API server validates and configures data for the API objects which include the construct mentioned above – pods, services, replication controllers to name a few.

Security 101

Now that you have a basic understanding of the most commonly used kubernetes constructs let us delve into some kubernetes security best practices.

Kubernetes API
Firstly, a bad actor can attack the Kubernetes API server to further attack the cluster.

By design, each pod has a service account associated with it. The service account has strong permissions and sufficient privileges to access the Kubernetes API server. A bad actor can get the token that is mounted in the pod and then use that token to gain access to the API server. 

Unfortunately, once they have access to the Kubernetes API server they can get information such as secrets, to further compromise the cluster. For example, a bad actor could gain access to the database server where sensitive and regulatory data is held.  

Kubernetes API – Mitigate
Another way to mitigate a Kubernetes API attack is to introduce role-based access control (RBAC). Essentially what RBAC does is that it gives rules to service accounts that can be assigned to the entire cluster. RBAC allows you to set permission on individual constructs such as pods and containers and then define rules based on individual use cases.

Another kubernetes best practice would be to introduce an API server firewall. This firewall would work similarly to a standard firewall, where you can allow and block ranges of addresses. Essentially, what this does is that it narrows the attack surface hardening kubernetes security.

Secure pods with network policy
Introducing both ingress and egress network policy would allow you to block network access to certain pods. Then only certain pods can communicate with each other. For example, network policies can be set up to restrict the web front end from communicating directly to the database tier.

There is also a risk of a user implementing unsecured pods. In this case, you can have network policies in place so that if an unsecured pod does appear in the kubernetes environment,  the kubernetes API will reject them.

Keep in mind, left to its defaults, intra-cluster pod to pod communication is open. You may think that a bad actor could sniff this traffic but that’s harder to do than in normal environments. Anyways, a better strategy to harden kubernetes security would be to use network policy only to allow certain pods to communicate with each other. 

Container escape
In the past, there have been instances of an unpatched kernel or a bug in the web front-end that allows a bad actor to get out of that container and gain access to other containers, or even to the Kubelet process itself. Once the bad actor gets access to the Kubelet client it can access the API server gaining a further foothold in the kubernetes environment.

Sandboxed Pods
One way to mitigate a container escape would be to introduce sandboxed pods that run inside a lightweight VM. A sandboxed pod provides additional barriers to security as it lies above the kernel itself. Providing layers of security above the kernel bring with it many kubernetes security advantages. As the sandboxed pod runs inside a VM, if there is a kernel exploit an additional bug is required to break out of the container.

Scanning containers
For effective kubernetes security, you should know if there is a vulnerability in your container image. Running containers with vulnerabilities expose the entire system to attacks. You need the ability to scan all container images in order to discover and remove known vulnerabilities. The scanning function should be integrated with runtime enforcement and remediation capabilities enabling a secure and compliant SDLC (Software Development Life Cycle).

Restrict access to Kubectl
Kubectl is a powerful command-line interface for running commands against kubernetes clusters. It allows you to create and update resources in a kubernetes environment. Due to its potential, one should restrict access to this tool with firewall rules.

Disabling the kubernetes dashboard
Disabling the kubernetes Dashboard is a must as it has been part of some high profile compromises. Previously, bad actors gained access to a publicly unrestricted kubernetes dashboard, gained privileged account credentials that further allowed them to access resources and mine cryptocurrency. Certainly, something you don’t want to happen on a regular basis.

About Matt Conran

Matt Conran has created 184 entries.

Leave a Reply