Kubernetes Audit Logs | Use Cases & Best Practices

Kubernetes is a container orchestration tool that reduces the complexity of deploying and managing containerized applications. It’s fast becoming the industry standard for the deployment and management of large-scale applications and microservices, and is widely used by a diverse group of organizations to manage their applications, both in the cloud and on-premises.

As a Kubernetes administrator, it’s important you have a record of events that happen in your cluster. These records will serve as a source of truth for debugging issues and improving your cluster’s security. Kubernetes auditing documents actions that were executed—or actions that someone attempted to execute—in your cluster.

In this article, you’re going to learn what Kubernetes audit logs are, why they’re important, and how to enable audit logs in your Kubernetes cluster.

Understanding Kubernetes Audit Logs

Audit logs are a set of records that contain a chronological list of all requests made to the Kubernetes API. Kubernetes stores the actions generated by each user, as well as by the control plane. These logs are in JSON format, and contain information such as the HTTP method, information of the user who initiated the request, the requests that were initiated, the Kubernetes component handling the request, and much more.

The audit logs can tell you what happened and when, as well as who initiated it, where the request was observed, where it was initiated, and where it was going.

Why Kubernetes Audit Logs are Important

Audit logs provide a comprehensive overview of everything happening in your cluster, allowing you to make informed decisions. It’s important to note that improperly configured audit logs can contain so much information that they’re impossible to use, so it’s important to make sure that your configuration gives you enough information to get the logs you need, but not so much that it’s overwhelming.

Audit logs can also provide visibility into what an attacker is trying to do—or has already done—in your cluster. When a user initiates a request to the API server, the API server checks that the user has sufficient permissions before it processes the request; if the user doesn’t have authorization, the API server rejects or forbids the request. In either case, Kubernetes saves both the request and the subsequent execution or rejection in the audit logs. In a scenario where an attacker hijacks a user or service account, you can view all the actions they performed or tried to perform.

What Are the Use Cases for Audit Logs?

The following is an example of a Kubernetes audit log. Each key in the JSON structure contains information that’s vital to understanding what’s happening in your cluster.

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "fbc474df-2466-4612-ae36-69af2c927f9d",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/pods/nginx",
  "verb": "get",
  "user": {
    "username": "system:node:minikube",
    "groups": [
      "system:nodes",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.49.2"
  ],
  "userAgent": "kubelet/v1.21.2 (linux/amd64) kubernetes/092fbfb",
  "objectRef": {
    "resource": "pods",
    "namespace": "default",
    "name": "nginx",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "responseObject": {
    "kind": "Pod",
    "apiVersion": "v1",
    "metadata": {...},
    "spec": {...},
    "status": {...}
  },
  "requestReceivedTimestamp": "2022-01-18T06:57:18.944663Z",
  "stageTimestamp": "2022-01-18T06:57:18.968543Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}

Now you’ll take a closer look at the purpose of some of the keys and how they apply to your Kubernetes cluster.

User Information and Originating IP

The <terminal inline>user<terminal inline> key tells you what user or service account initiated the request through the Kubernetes API server, and the <terminal inline>sourceIP<terminal inline> key provides the IP of that account. An IP address can tell you the location (such as city, ZIP code, or area code), and the name of the user’s ISP. IP addresses aren’t always reliable, because they can be altered or hidden by the user making the request, but this information can be useful when trying to block malicious activities originating from a range of IP addresses.

Information About Requests That Were Initiated and Executed

The <terminal inline>verb<terminal inline>, <terminal inline>requestURI<terminal inline>, and <terminal inline>objectRef<terminal inline> all provide information about the requests users initiated in the cluster. The <terminal inline>verb<terminal inline> key indicates the Kubernetes HTTP request method that was performed: <terminal inline>get<terminal inline>, <terminal inline>post<terminal inline>, <terminal inline>list<terminal inline>, <terminal inline>watch<terminal inline>, <terminal inline>patch<terminal inline>, or <terminal inline>delete<terminal inline>. The <terminal inline>requestURI<terminal inline> provides information about the API requests you initiated in the cluster—for example, a request to get all pods, or create new deployments. The <terminal inline>objectRef<terminal inline> contains information (such as the name, namespace, apiVersion) about the Kubernetes object associated with the request. The <terminal inline>objectRef<terminal inline>, <terminal inline>verb<terminal inline>, and <terminal inline>requestURI<terminal inline> provide full information about the request initiated by the user.

Response Status of the Actions Performed

The <terminal inline>responseStatus<terminal inline> and <terminal inline>responseObject<terminal inline>, in combination with the <terminal inline>annotations<terminal inline> key, provide insight into the response to the requests initiated by users or service accounts. The <terminal inline>annotations.authorization.k8s.io/decision<terminal inline> provides either an<terminal inline> allow<terminal inline> or <terminal inline>reject<terminal inline> value. These keys are very useful when performing audits on your Kubernetes cluster to detect anomalous behaviors.

Automatically Collect Anything Logged Inside Your Cluster

Monitor the health of your cluster and troubleshoot issues faster with pre-built dashboards that just work.

Learn More Book a Demo

Enabling and Accessing Audit Logs

Now that you know what audit logs look like and why you need them, you’ll learn about how you can enable and access them on your Kubernetes cluster.

Self-Hosted Kubernetes Cluster

Self-hosted Kubernetes clusters don’t have audit logs enabled by default. The following steps explain how to enable and view audit logs in your Kubernetes cluster. In this section, you’ll use Minikube to set up a Kubernetes cluster on your local machine so that you can follow along. Minikube is a free tool that allows you to run a single-node Kubernetes cluster on your local machine. It’s a platform to practice and learn Kubernetes, or to use for daily development work. It’s important to note that you can’t use Minikube to host production-based applications.

• Follow this guide to install <terminal inline>minikube<terminal inline> on your local machine.
• Once installed, run the command below to start your cluster

minikube start

Minikube creates a cluster consisting of the Kubernetes primary and worker components on your local machine.

To enable audit logs, you’ll have to perform some configuration, such as updating the kube-apiserver config file and creating the directory path for the log file. The kube-apiserver is part of the control plane components that are hosted on the primary node, which means that its config file is located in the primary node. Because of this, you have to SSH into the primary node to update its config file, as you can’t do it imperatively using the kubectl command.

• To access the <terminal inline>kube-apiserver<terminal inline> config file in your minikube Kubernetes cluster, run the command below.

minikube ssh

• Create an <terminal inline>audit-policy.yaml<terminal inline> file.

sudo touch /etc/kubernetes/audit-policy.yaml

• Open the file using the Vim editor.

sudo vi /etc/kubernetes/audit-policy.yaml

• Paste the following example rules into the file. This is a basic policy file modeled on the example in the Kubernetes documentation, and can be modified as indicated in the comments to suit your logging needs. Kubernetes uses this policy file to identify if events should be logged or excluded.

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy

# Prevent requests in the RequestReceived stage from generating audit events.
omitStages:
  - "RequestReceived"

rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]
  # Exclude logging requests to a configmap called "controller-config"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-config"]
  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]
  # Log deployment changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["deployments"]
  # Log service changes at metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["services"]
  # Log the request body of configmap changes in the kube-system namespace.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # You can use an empty string [""] to select resources not associated with a namespace.
    namespaces: ["kube-system"]
  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]
  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.
  # A wild-card rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

• Create <terminal inline>audit.log<terminal inline> in the following directory. This is where Kubernetes will save your audit logs.

sudo mkdir /var/log/kubernetes/ && cd /var/log/kubernetes/
sudo mkdir audit/ && sudo touch audit.log

• Run the command below to edit the <terminal inline>kube-apiserver<terminal inline> config file.

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

• Update the volume mount section of the config file.

# ...

volumeMounts:
- mountPath: /etc/kubernetes/audit-policy.yaml
     name: audit
     readOnly: true
- mountPath: /var/log/kubernetes/audit/audit.log
     name: audit-log
     readOnly: false
 # ...

• Update the volumes section.

 # ...
 volumes:

 - hostPath:
      path: /etc/kubernetes/audit-policy.yaml
      type: File
   name: audit
 - hostPath:
      path: /var/log/kubernetes/audit/audit.log
      type: FileOrCreate
   name: audit-log
 # ... 

• Finally, update the command section to enable auditing in your cluster.

 --audit-policy-file=/etc/kubernetes/audit-policy.yaml
 --audit-log-path=/var/log/kubernetes/audit/audit.log

Test the Auditing Feature

Give Kubernetes a few seconds to apply the changes you made to the <terminal inline>kube-apiserver<terminal inline> config file, and you can test your auditing to make sure it works.

• Open a new terminal and run the command below to check that your API server is up and running.

kubectl get pods -n kube-system -w

• Create a new pod to see if Kubernetes will log the request in the log file

kubectl run nginx --image=nginx

• Return to the <terminal inline>minikube ssh<terminal inline> terminal and open the log file to see if it has logs in it by running the command below.

sudo cat /var/log/kubernetes/audit/audit.log | grep -i "nginx"

• If you see content in the log file that’s similar to the image below, auditing is now enabled and configured on your Kubernetes cluster

Managed Kubernetes Cluster by Cloud Providers

Cloud providers offering managed Kubernetes as a service provide advanced auditing for your Kubernetes cluster—that is, they’ve integrated Kubernetes auditing with their auditing and logging tools. Google Kubernetes Engine (GKE) enables audit logging out of the box, while Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) require that you enable logging before you can start viewing logs generated on your cluster.

To enable or disable audit logging on a managed Kubernetes service, follow the guides in the official documentation, linked below.

• Amazon EKS Logging
• Accessing Audit logs in GKE
• Configuring Logging and Monitoring in AKS

Final Thoughts

Auditing is very important to the security of your Kubernetes cluster and, in this article, you’ve learned more about Kubernetes audit logs, the importance of enabling audit logs in your Kubernetes cluster, and how to enable and configure audit logs on both self-hosted and managed Kubernetes clusters.

You also learned how to supplement Kubernetes audit logs with third-party tools.