Start your free 14-day ContainIQ trial

Kubernetes Audit Logs | Use Cases & Best Practices

Kubernetes audit logs are powerful, but only if enabled and correctly configured. This article will help you get started using audit logs, and show you how to get the most out of them.

July 13, 2022
Daniel Olaogun
Software Engineer

Kubernetes is a container orchestration tool that reduces the complexity of deploying and managing containerized applications. It’s fast becoming the industry standard for the deployment and management of large-scale applications and microservices, and is widely used by a diverse group of organizations to manage their applications, both in the cloud and on-premises.

As a Kubernetes administrator, it’s important you have a record of events that happen in your cluster. These records will serve as a source of truth for debugging issues and improving your cluster’s security. Kubernetes auditing documents actions that were executed—or actions that someone attempted to execute—in your cluster.

In this article, you’re going to learn what Kubernetes audit logs are, why they’re important, and how to enable audit logs in your Kubernetes cluster.

Understanding Kubernetes Audit Logs

Logging Architecture

Audit logs are a set of records that contain a chronological list of all requests made to the Kubernetes API. Kubernetes stores the actions generated by each user, as well as by the control plane. These logs are in JSON format, and contain information such as the HTTP method, information of the user who initiated the request, the requests that were initiated, the Kubernetes component handling the request, and much more.

The audit logs can tell you what happened and when, as well as who initiated it, where the request was observed, where it was initiated, and where it was going.

Why Kubernetes Audit Logs are Important

Audit logs provide a comprehensive overview of everything happening in your cluster, allowing you to make informed decisions. It’s important to note that improperly configured audit logs can contain so much information that they’re impossible to use, so it’s important to make sure that your configuration gives you enough information to get the logs you need, but not so much that it’s overwhelming.

Audit logs can also provide visibility into what an attacker is trying to do—or has already done—in your cluster. When a user initiates a request to the API server, the API server checks that the user has sufficient permissions before it processes the request; if the user doesn’t have authorization, the API server rejects or forbids the request. In either case, Kubernetes saves both the request and the subsequent execution or rejection in the audit logs. In a scenario where an attacker hijacks a user or service account, you can view all the actions they performed or tried to perform.

What Are the Use Cases for Audit Logs?

The following is an example of a Kubernetes audit log. Each key in the JSON structure contains information that’s vital to understanding what’s happening in your cluster.



{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "fbc474df-2466-4612-ae36-69af2c927f9d",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/pods/nginx",
  "verb": "get",
  "user": {
    "username": "system:node:minikube",
    "groups": [
      "system:nodes",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.49.2"
  ],
  "userAgent": "kubelet/v1.21.2 (linux/amd64) kubernetes/092fbfb",
  "objectRef": {
    "resource": "pods",
    "namespace": "default",
    "name": "nginx",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "responseObject": {
    "kind": "Pod",
    "apiVersion": "v1",
    "metadata": {...},
    "spec": {...},
    "status": {...}
  },
  "requestReceivedTimestamp": "2022-01-18T06:57:18.944663Z",
  "stageTimestamp": "2022-01-18T06:57:18.968543Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}

Now you’ll take a closer look at the purpose of some of the keys and how they apply to your Kubernetes cluster.

User Information and Originating IP

The <terminal inline>user<terminal inline> key tells you what user or service account initiated the request through the Kubernetes API server, and the <terminal inline>sourceIP<terminal inline> key provides the IP of that account. An IP address can tell you the location (such as city, ZIP code, or area code), and the name of the user’s ISP. IP addresses aren’t always reliable, because they can be altered or hidden by the user making the request, but this information can be useful when trying to block malicious activities originating from a range of IP addresses.

Information About Requests That Were Initiated and Executed

The <terminal inline>verb<terminal inline>, <terminal inline>requestURI<terminal inline>, and <terminal inline>objectRef<terminal inline> all provide information about the requests users initiated in the cluster. The <terminal inline>verb<terminal inline> key indicates the Kubernetes HTTP request method that was performed: <terminal inline>get<terminal inline>, <terminal inline>post<terminal inline>, <terminal inline>list<terminal inline>, <terminal inline>watch<terminal inline>, <terminal inline>patch<terminal inline>, or <terminal inline>delete<terminal inline>. The <terminal inline>requestURI<terminal inline> provides information about the API requests you initiated in the cluster—for example, a request to get all pods, or create new deployments. The <terminal inline>objectRef<terminal inline> contains information (such as the name, namespace, apiVersion) about the Kubernetes object associated with the request. The <terminal inline>objectRef<terminal inline>, <terminal inline>verb<terminal inline>, and <terminal inline>requestURI<terminal inline> provide full information about the request initiated by the user.

Response Status of the Actions Performed

The <terminal inline>responseStatus<terminal inline> and <terminal inline>responseObject<terminal inline>, in combination with the <terminal inline>annotations<terminal inline> key, provide insight into the response to the requests initiated by users or service accounts. The <terminal inline>annotations.authorization.k8s.io/decision<terminal inline> provides either an<terminal inline> allow<terminal inline> or <terminal inline>reject<terminal inline> value. These keys are very useful when performing audits on your Kubernetes cluster to detect anomalous behaviors.

Automatically Collect Anything Logged Inside Your Cluster
Monitor the health of your cluster and troubleshoot issues faster with pre-built dashboards that just work.
Learn More Book a Demo
logging dashboard

Enabling and Accessing Audit Logs

Now that you know what audit logs look like and why you need them, you’ll learn about how you can enable and access them on your Kubernetes cluster.

Self-Hosted Kubernetes Cluster

Self-hosted Kubernetes clusters don’t have audit logs enabled by default. The following steps explain how to enable and view audit logs in your Kubernetes cluster. In this section, you’ll use Minikube to set up a Kubernetes cluster on your local machine so that you can follow along. Minikube is a free tool that allows you to run a single-node Kubernetes cluster on your local machine. It’s a platform to practice and learn Kubernetes, or to use for daily development work. It’s important to note that you can’t use Minikube to host production-based applications.

  • Follow this guide to install <terminal inline>minikube<terminal inline> on your local machine.
  • Once installed, run the command below to start your cluster

minikube start

Minikube creates a cluster consisting of the Kubernetes primary and worker components on your local machine.

To enable audit logs, you’ll have to perform some configuration, such as updating the kube-apiserver config file and creating the directory path for the log file. The kube-apiserver is part of the control plane components that are hosted on the primary node, which means that its config file is located in the primary node. Because of this, you have to SSH into the primary node to update its config file, as you can’t do it imperatively using the kubectl command.

  • To access the <terminal inline>kube-apiserver<terminal inline> config file in your minikube Kubernetes cluster, run the command below.

minikube ssh
  • Create an <terminal inline>audit-policy.yaml<terminal inline> file.

sudo touch /etc/kubernetes/audit-policy.yaml
  • Open the file using the Vim editor.

sudo vi /etc/kubernetes/audit-policy.yaml
  • Paste the following example rules into the file. This is a basic policy file modeled on the example in the Kubernetes documentation, and can be modified as indicated in the comments to suit your logging needs. Kubernetes uses this policy file to identify if events should be logged or excluded.

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy

# Prevent requests in the RequestReceived stage from generating audit events.
omitStages:
  - "RequestReceived"

rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]
  # Exclude logging requests to a configmap called "controller-config"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-config"]
  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]
  # Log deployment changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["deployments"]
  # Log service changes at metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["services"]
  # Log the request body of configmap changes in the kube-system namespace.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # You can use an empty string [""] to select resources not associated with a namespace.
    namespaces: ["kube-system"]
  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]
  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.
  # A wild-card rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"
  • Create <terminal inline>audit.log<terminal inline> in the following directory. This is where Kubernetes will save your audit logs.

sudo mkdir /var/log/kubernetes/ && cd /var/log/kubernetes/
sudo mkdir audit/ && sudo touch audit.log
  • Run the command below to edit the <terminal inline>kube-apiserver<terminal inline> config file.

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
  • Update the volume mount section of the config file.

# ...

volumeMounts:
- mountPath: /etc/kubernetes/audit-policy.yaml
     name: audit
     readOnly: true
- mountPath: /var/log/kubernetes/audit/audit.log
     name: audit-log
     readOnly: false
 # ...
  • Update the volumes section.

 # ...
 volumes:

 - hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
   name: audit
 - hostPath:
      path: /var/log/kubernetes/audit/audit.log
      type: FileOrCreate
   name: audit-log
 # ... 
  • Finally, update the command section to enable auditing in your cluster.

 --audit-policy-file=/etc/kubernetes/audit-policy.yaml
 --audit-log-path=/var/log/kubernetes/audit/audit.log

Test the Auditing Feature

Give Kubernetes a few seconds to apply the changes you made to the <terminal inline>kube-apiserver<terminal inline> config file, and you can test your auditing to make sure it works.

  • Open a new terminal and run the command below to check that your API server is up and running.

kubectl get pods -n kube-system -w
Checking Stats of Kube-apiserver
Checking Stats of Kube-apiserver
  • Create a new pod to see if Kubernetes will log the request in the log file

kubectl run nginx --image=nginx
  • Return to the <terminal inline>minikube ssh<terminal inline> terminal and open the log file to see if it has logs in it by running the command below.

sudo cat /var/log/kubernetes/audit/audit.log | grep -i "nginx"
  • If you see content in the log file that’s similar to the image below, auditing is now enabled and configured on your Kubernetes cluster
Audit Log File
Audit Log File

Managed Kubernetes Cluster by Cloud Providers

Cloud providers offering managed Kubernetes as a service provide advanced auditing for your Kubernetes cluster—that is, they’ve integrated Kubernetes auditing with their auditing and logging tools. Google Kubernetes Engine (GKE) enables audit logging out of the box, while Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) require that you enable logging before you can start viewing logs generated on your cluster.

To enable or disable audit logging on a managed Kubernetes service, follow the guides in the official documentation, linked below.

Using ContainIQ To Monitor Kubernetes Audit Logs

There are several tools you can install in your Kubernetes cluster to provide a richer experience when collecting and viewing logs. One such tool is ContainIQ, a K8s monitoring platform. ContainIQ makes it easy for you and your team to monitor Kubernetes metrics, logs, and events within your cluster. It offers improved viewing and filtering options to help you see what’s really important in your logs.

How to install ContainIQ in your cluster

  • Install the metrics server, if you haven’t already.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
  • Download the ContainIQ deployment file.

curl -L -o deployment.yaml https://raw.githubusercontent.com/containiq/containiq-deployment/master/deployment.yaml
  • Get your ContainIQ API key by creating an account, then open the deployment file, scroll to the bottom, and add your API key to the secret object.
  • Once done, apply the deployment.

kubectl apply -f deployment.yaml

Implementing ContainIQ on your cluster takes your logging to the next level. While Kubernetes audit logs provide valuable information, they only offer access in the form of raw data logs, which are difficult to read and time-consuming to sift through. However, with ContainIQ, you have a user interface that allows you to easily filter logs on the basis of the pod, timestamp, and search query, making it much easier to extract the information you are looking for.

ContainIQ Logging Dashboard

Final Thoughts

Auditing is very important to the security of your Kubernetes cluster and, in this article, you’ve learned more about Kubernetes audit logs, the importance of enabling audit logs in your Kubernetes cluster, and how to enable and configure audit logs on both self-hosted and managed Kubernetes clusters.

You also learned how to supplement Kubernetes audit logs with third-party tools such as ContainIQ, which enables you to get a better understanding of your audit logs with easily readable, user friendly dashboards. ContainIQ also provides insights into cluster latency, resource management, and more to help users fine-tune their multi-tenancy deployments.

Start your free 14-day ContainIQ trial
Start Free TrialBook a Demo
No card required
Daniel Olaogun
Software Engineer

Daniel is a Software Engineer with 5 years of experience building software solutions in a variety of fields. Today, he works as the Lead Software Engineer at Whitesmith. Daniel has a Bachelor’s Degree in Computer Science from the University of Lagos.

READ MORE