Using eBPF To Enhance Kubernetes Monitoring: Powerful Insights

November 30, 2021

eBPF is a piece of advanced Linux functionality that has been gradually put to use in Kubernetes over the last few years. In this guide, you will walk the reader through some use-cases for eBPF in Kubernetes, such as kernel-level network monitoring.

Sanni Michael
Backend Engineer

The Linux operating system divides its memory into two different spaces: the kernel and the userspace. The kernel space contains the core of the operating system and has access to device drivers and system hardware. Due to its structure, the kernel space is only allowed to run secure and trusted code. On the other hand, the userspace contains regular applications and usually has restricted access to device drivers and system hardware. The userspace usually relies on the kernel space to perform operations like network I/O via device drivers and using system calls.

Sometimes, developers need the flexibility to implement new file systems and execute a secure custom system call without having to add code directly to the kernel space. That’s where the extended Berkeley Packet Filter (eBPF) comes in.

In this article, you’ll learn what eBPF is, why you need it, its various use cases, and how you can use it with Kubernetes for monitoring.

Why Do You Need eBPF?

If you have more than a passing familiarity with Kubernetes, you know that it relies heavily on Linux. eBPF is a Linux functionality that’s gaining traction in workflows with auditing, tracking, and monitoring.

eBPF is a kernel technology (starting from Linux 4.x) for Linux applications to securely execute code in Linux kernel space. It can be used to safely extend the capabilities of the kernel without changing the kernel source code.

Compared to strace, a tool for monitoring Linux processes and tracing system calls, eBPF provides system call tracing without too much overhead. Running the strace command on a server with tons of workloads can have catastrophic effects.

Another reason you might consider using eBPF is that a lot of big players are gradually adopting it for monitoring and observability of cloud-native applications. For example, Cilium is used for observing and monitoring connectivity between container workloads and is built with eBPF, and Cloudflare open-sourced a Prometheus exporter based on eBPF.

eBPF is widely used in a lot of open-source software in areas such as networking, monitoring, and creating security tools. Let’s take a closer look at some of its more common use cases when it comes to Kubernetes monitoring in particular:

  • Advanced user-defined observability: With eBPF, you can run custom and user-defined programs in the kernel space to keep a record of network communications between services.
  • Deep monitoring previously not possible in Kubernetes: eBPF can keep track of the latencies of a microservice as well the API endpoints in each service.
  • Networking: The programmability of eBPF enables adding extra protocol parsers and easily programming any forwarding logic to meet changing requirements without ever leaving the packet processing context of the Linux kernel.
  • Tracking TCP connections in Weave Scope: Weave Scope is a tool for monitoring, visualizing, and interacting with container-based systems.
  • Pod and container-level network statistics: Control groups are usually known for grouping processes together hierarchically. In Kubernetes, this grouping happens at the container level. Since Linux 4.10, it’s now possible to attach eBPF programs to each control group. This helps with filtering network traffic from or to a process in the control group.

How Does eBPF Help with Kubernetes Monitoring?

Services are usually deployed as self-contained entities called containers in a cluster. Since these containers are nothing but processes that run as long as the application in them is alive, eBPF provides a consistent tool for tracking the services running in a Kubernetes cluster.

You can also use eBPF for auditing commands executed in the Kubernetes cluster. eBPF can provide detailed visibility into each command, answering questions such as what happened, when did it happen, who initiated it, where did it happen, and so on. For example, it might be possible to see that a user executed kubectl exec to log in into a pod container, but difficult to see the exact parameters supplied like the pod name, namespace, and other relevant information. With eBPF, you can easily locate more information about such commands.

Since eBPF provides a unified framework for tracking processes, it offers more visibility into your system. It’s also quite secure. Unlike Linux kernel modules, eBPF allows you to execute secure code in the operating system kernel. Before these programs are loaded into the kernel space, they usually go through several checks to validate the code and ensure that it’s secure.

When Does It Make Sense to Use eBPF for Monitoring?

There are a few scenarios where using eBPF for monitoring is a particularly solid choice.

  • Monitoring from the kernel layer can mitigate the challenges relating to observability in the cloud. There are several issues that can result from using cloud observability, like request latencies. Some of these can be addressed by implementing eBPF.
  • When there is a need for kernel tracing. eBPF allows you to execute custom code in the kernel. As everything is done in the kernel, tracing the actions and steps of the program becomes faster and more accurate. Also, the programs are executed when an event is triggered, so most actions are tracked.
  • When traditional monitoring methods might not work. It’s possible for a monitoring tool to malfunction due to a bug. eBPF can offer visibility into requests and HTTP traffic even when traditional monitoring tools don’t work.
  • When you need to optimize packet traffic control. In a standard Linux networking path, packets usually travel from source to destination. However, they’re not aware of complex paths. With eBPF and enough context, you can implement programs in the kernel that ensures the packet will avoid complex paths and follow the best and fastest route to the destination.
Looking for an out-of-the-box monitoring solution?

With a simple one-line install, ContainIQ allows you to monitor the health of your cluster with pre-built dashboards and easy-to-set alerts.

What Are Some Limitations of eBPF?

The biggest limitation: eBPF is restricted to Linux and a recent kernel. It’s only available in newer versions of the Linux kernel.

eBPF has limited program functionality. The size of an eBPF program is limited to a maximum of 4096 bytes, which ensures the program terminates without an unbounded loop. This also limits the resources a program can have access to, and the functionality that can be implemented with the programs. Therefore, eBPF programs tend to be very small.

eBPF is not portable like other tracers, as it was solely developed for the Linux kernel and is completely dependent on it. eBPF programs are usually loaded at runtime by the kernel and the program is verified for possible threats. In the case of an unbounded loop, the verifier rejects the code, thus ensuring that all programs will terminate at some point. This limited support for loops may impair the capacity to create powerful networking programs.

Programs are also restricted to calling only an arbitrary set of kernel functions. They can only call the functions defined in the programs loaded into memory. That means the available helper functions that can be accessed by the programs at runtime are limited. While it’s possible to also read from kernel memory when tracing the path of a process using helpers like <terminal inline>bpf_probe_read()<terminal inline> and <terminal inline>bpf_probe_read_str()<terminal inline>, networking programs are not allowed to read arbitrary kernel memory since they don’t have access to that.

eBPF Tools

There are several open-source tools you can use to build custom programs that get loaded into the kernel at runtime, in case you want to get your hands dirty. The list includes:

  • BPF Compiler Collection (BCC) provides a toolkit for building efficient kernel manipulation programs.
  • bpftrace is a high-level tracing language for eBPF programs.
  • There are also language-specific tools like gobpf for Golang, libbpf for C/C++, and redbpf for Rust.

Final Thoughts

eBPF is still an emerging technology, but companies are gradually adopting it in cloud-native applications running in production due to how it can improve monitoring and observation of those applications.

To monitor and manage Kubernetes metrics and events, check out ContainIQ, a monitoring platform for Kubernetes workloads that uses eBPF to power many of its dashboards.

Article by

Sanni Michael

Backend Engineer

Sanni is a Backend Engineer at Salamantex with years of experience working with Cloud Native technologies. Outside of work, Sanni enjoys writing about DevOps topics and has a Kubernetes focused blog called DailyK8s. He has a degree in Computer Science from Yaba College of Technology.

Read More