Kubernetes Security: Best Practices for 2021

October 11, 2021

Kubernetes makes it easy to scale your infrastructure, but it also introduces new security challenges. Learn the best practices for k8s security here.

Sean Bangalore
Software Engineer

Kubernetes has changed DevOps since its release in 2014. Not only do you have access to hardware and resource allocation management as in a virtualized deployment, but now, you also have access to managed runtimes, updates, and portability. Kubernetes makes it easy to launch and scale without the headaches of orchestration.

Most organizations realize these benefits. In one survey, over 50% of participants reported that they used Kubernetes for container management, and the popularity of Kubernetes and managed Kubernetes services continues to grow at a rapid pace.

However, there’s a darker side to Kubernetes’ adoption. Over 60% of organizations in a different survey said that they had suffered a container security incident. It’s important to consider this in context and to harden your Kubernetes deployment to preempt security issues like these.

In this article, I will share a brief overview of Kubernetes and its core components. Then, I’ll dive into the 4C framework for Kubernetes security and introduce some tactics for proactively addressing security vulnerabilities.

Kubernetes and its Components

To understand security in Kubernetes, you need to understand how Kubernetes works at a high level. Each Kubernetes environment is loosely coupled and distributed so that if one environment is compromised, the others are relatively isolated from impact. however, there are still significant risks that need to be managed.

The two parts of a Kubernetes environment are the control plane and the node components. The control plane hosts the API server and Kube-controller managers, the key-value store, and the scheduler.

Kubernetes control plane and node worker

Within the control plane, there are four components connected through a hub and spoke pattern.

The node components each include Kubelets and Kube-proxies:

  • Kubelet ensures that containers run and are healthy;
  • Kube-proxy maintains network rules on nodes, allowing communication to Pods from network sessions inside and outside the cluster.

With this architecture in mind, let’s explore some of the security best practices for Kubernetes environments.

Security Architecture, Concerns and Best Practices

Although each Kubernetes component is separated, you still have to consider the communication between the control plane and the node worker and the security of the control plane itself. This is heavily influenced by your configurations and practices.

You can consider security in the context of the 4Cs of Kubernetes security: Cloud, Clusters, Containers, and Code.

You can think of these like a Matryoshka (or Russian Nesting Doll). Each layer (“C”) builds on and is vulnerable to the security issues of the layer that it is encompassed by. So, the code layer is vulnerable to security issues in the container, the clusters, and the cloud.

The 4C's of Kubernetes Security

As a result, to secure the code and application data, all four layers - cloud, clusters, container, and code need to be secured.

Best Practices in Each of the 4Cs of Kubernetes Security

There are two main areas to secure in these four layers: the configurable cluster components and the applications which run in the cluster. In the following sections, I’ll cover some best practices in both of these areas for all four layers.

Code

I won’t cover all of the best practices for writing secure software code as this is a huge topic. But Kubernetes administrators primarily need to worry about network communication and attack prevention measures.

Network

The network describes how your code connects to outside services. You can reduce the vectors in which attackers can gain access to your systems by securing communication into and in between your services.

Here are some recommendations:

  1. Network communications on TCP should be set up to perform a TLS handshake and have encryption by default. Third-party service meshes, which are run on each node, could be used for this; consider Istio, Linkerd, Consul), and Citrix.
  2. You should also encrypt communications between services.
  3. You should only expose ports that are essential for communication and network gathering.

Attack Prevention

Alongside securing your communications, routine operation practices can be implemented which scan your code or internal configurations for vulnerabilities. A few tactics you might consider include:

  1. Schedule pen testing to preemptively defend against SQL injection, CSRF, and XSS attacks – a popular tool to address these in the open-source community is Project ZAP.
  2. Analyze code for vulnerabilities such as updates and changes to 3rd party dependencies and unsafe coding practices through at least one static analysis tool, such as kube-score.

Changes to libraries and bad code can introduce bugs and backdoors that could impact the security of your codebase. Automated monitoring can help with ensuring that the libraries are secure and common errors are detected prior to deployment.

Container

Your code, bundled with config files, libraries, and dependencies, is located in containers. There are a few important Kubernetes security considerations for containers:

  1. When choosing the container image/runtime, select runtime classes that have stronger isolation. Kubernetes does not protect against attacks in the runtime and doesn’t detect issues by default in the image.
  2. As part of the build step, scan for known vulnerabilities, and sign container images.
  3. Allow only essential communication between containers (in pods/clusters) and limit privileged access to specific requirements.

Using known containers from a private registry is also a good idea, but remember that images are often built on external base layers that might have vulnerabilities too.

Clusters

Clusters are Containers within groups of pods. For clusters, it’s important to maintain security in the control plane, specifically:

  1. Restrict cluster access to the etcd backend (and only access etcd manually if you need to escalate permissions). A lot of sensitive data and permissions reside on the etcd, so you don’t want the rest of your cluster to be able to access it unless absolutely necessary.
  2. Enable audit logging and archive the audits on a secure server for monitoring and analysis during debugging.
  3. Encrypt secrets at rest so that adversaries can not access critical information if they penetrate your clusters’ defenses.

Cloud

Kubernetes clusters are typically run on cloud providers like Google, Amazon, and Microsoft. Each provider has documentation for their best practices which you should implement:

  1. Amazon Web Services EKS Best Practices;
  2. Microsoft Azure;
  3. Google Cloud Services.

While your provider will take care of most high-level security issues in the cloud, you still have to pay attention to permissions and access controls. A few considerations that apply to all cloud hosting services include:

  1. Provide least privileged access to cloud resources.
  2. Make the auth role-specific and audit access to the resource through sources such as kubectl-who-can: https://github.com/aquasecurity/kubectl-who-can
  3. Never nest Docker in Docker or run processes as root.
  4. Regularly run kube-bench to comply with CIS Kubernetes benchmarks.

If you’re not using the cloud but instead relying on in-house servers, security at this layer gets more complicated as you’ll also need to consider things like physical security and failover.

Secure Kubernetes Base Configuration

When getting started with Kubernetes, base configurations are suggested for your environment. However, these are not intrinsically secure. There are several things you can do to further secure your Kubernetes base configuration, including:

  1. Monitor the source of your images and be diligent about third-party image registries. DockerHub is a good place to find official images.
  2. Define network policies carefully. They allow you to specify which entities internal and external services can connect with, but ingress and egress rules are not defined by default.
  3. Keep user security privileges on an as-needed basis with tight control over access and well-defined roles.
  4. Label both critical issues and identified security issues with unknown fixes in your security scanner of choice; once a fix is released, refer back to your identified security issues.

The Role of Monitoring and Logging in Security

It’s tough to implement all of these recommendations quickly. Keeping a critical eye on the defaults in Kubernetes and the 4C levels requires an ever-growing list of considerations. Even if you follow all of these best practices, new vulnerabilities are introduced by internal and external changes constantly, so the challenge never really ends.

This leads to my final recommendation, which is to implement robust logging and monitoring. With access to your Kubernetes events, logs, and alerts, you can mitigate issues before they spiral out of control.

Tools like ContainIQ can help maintain security and ensure timely responses to exposures. ContainIQ makes it easy for you to track and manage Kubernetes events, monitor the health of your Kubernetes clusters, and more through their pre-built dashboards, metrics, and data feeds.

As you continue to integrate Kubernetes into your growing organization, do your best to not be part of the 60% of organizations that experienced security breaches in their deployments. Consider the security best practices above, and keep an eye out for changes as Kubernetes is an actively evolving ecosystem.

Article by

Sean Bangalore

Software Engineer

Sean is a Software Development Engineer at Amazon and interested in software, artificial intelligence, and quantitative finance.

Read More