Start your free 14-day ContainIQ trial

Kubernetes Container Lifecycle Events and Hooks | Ultimate Guide

June 27, 2022

In Kubernetes, it can be useful to run code in response to the pod lifecycle. This guide will teach you about lifecycle events and hooks: what they are, what they do, and why you need them.

James Walker
Software Engineer

Kubernetes lifecycle events and hooks let you run scripts in response to the changing phases of a pod’s lifecycle. The framework can be used to record new container creations, send notifications to other parts of your infrastructure, and perform cleanups after a pod is removed.

Utilizing the built-in hooks is the best way to be informed when a pod’s lifecycle changes. Events are issued by the kubelet worker process in real time as the state of each container evolves. Containers move through three distinct phases: <terminal inline>Waiting<terminal inline>, <terminal inline>Running<terminal inline>, and <terminal inline>Terminated<terminal inline>. Hooks let you plug code in at the transition points before and after <terminal inline>Running<terminal inline>.

In this article, you’ll learn about how hooks are executed, when they can be useful, and how you can attach your own scripts to your Kubernetes containers. You’ll also look at some of the issues you might face when configuring your hook handlers.

What Are Hooks For?

The primary purpose of lifecycle hooks is to provide a mechanism for detecting and responding to container state changes. The surfaced events track each container’s progress through its linear lifecycle. Before starting to use hooks, it’s important to understand what each of the three lifecycle phases means.

  • <terminal inline bold>Waiting<terminal inline bold>: The container is preparing to start, but is not currently running. This is most commonly seen when the node is pulling the container image needed to start the container.
  • <terminal inline bold>Running<terminal inline bold>: These containers are actively executing. Containers are usually in this state for most of their life—they’re the operational workloads within your cluster.
  • <terminal inline bold>Terminated<terminal inline bold>: Terminated containers are end-of-life instances that are now stopped. They either exited after running to completion, or terminated prematurely due to a failure.

Being able to track transitions between these phases gives you more insights into the status of your cluster. Registered hook handlers run within the container, so they can prepare or clean up its environment as it moves in and out of the <terminal inline>Running<terminal inline> state.

As hooks are managed by Kubernetes, they’re guaranteed to be executed, even in a failure scenario. Your hooks will still run if a container becomes <terminal inline>Terminated<terminal inline> because Kubernetes evicted its pod. You can reliably handle terminations due to resource constraints and cluster-level errors using lifecycle event handlers.

Available Hooks

Kubernetes currently supports two container lifecycle hooks, <terminal inline>PostStart<terminal inline> and <terminal inline>PreStop<terminal inline>. They sit on either side of the <terminal inline>Running<terminal inline> lifecycle phase.

The flow of container lifecycle phases and hooks

Here’s how they work.

PostStart

The <terminal inline>PostStart<terminal inline> hook is called immediately after a container is created. This occurs as it transitions into the <terminal inline>Running<terminal inline> phase after its <terminal inline>Waiting<terminal inline> period.

<terminal inline>PostStart<terminal inline> is normally used to configure the container, set up dependencies, and record the new creation. You could use this event to check that a required API is available before the container’s main work begins. Kubernetes will not change the container’s state to <terminal inline>Running<terminal inline> until the <terminal inline>PostStart<terminal inline> script has executed successfully.

As the event fires after container creation, the container’s [<terminal inline>ENTRYPOINT<terminal inline> script may run before your handler gets called. The handler blocks management of your container until it completes, but is executed asynchronously relative to your container. This means the container will be operational while Kubernetes waits for your handler to finish.

PreStop

The <terminal inline>PreStop<terminal inline> hook is called during a container’s termination sequence. The container will still be running at the time the event fires, and will enter the <terminal inline>Terminated<terminal inline> state after your hook handler executes. In the meantime, the container will show as <terminal inline>Terminating<terminal inline>.

Kubernetes includes safeguards to ensure faulty hook handlers don’t indefinitely prevent container termination. A grace period applied to each pod defines the maximum execution time of <terminal inline>PreStop<terminal inline> handlers. Kubernetes will kill the container if it’s been <terminal inline>Terminating<terminal inline> for longer than the grace period, even if a <terminal inline>PreStop<terminal inline> hook is running.

There’s another important “gotcha”, too. <terminal inline>PreStop<terminal inline> is only called when a container is terminated due to a Kubernetes API request or a cluster-level management event. Examples include pod deletion commands you’ve issued and Kubernetes-triggered evictions due to resource contention. The hook will not be invoked when a container is stopped because its pod successfully exited and became complete. You can’t currently clean up after finished jobs using your hook handlers.

K8s Metrics, Logging, and Tracing
Monitor the health of your cluster and troubleshoot issues faster with pre-built dashboards that just work.
Start Free Trial Book a Demo

Hook Handlers

Handlers are the second foundational component of the lifecycle hook system. There are two different types, <terminal inline>Exec<terminal inline> and <terminal inline>HTTP<terminal inline>.

An <terminal inline>Exec<terminal inline> handler runs a command within the container. This can be any executable process that’s available inside the container’s filesystem. Resources consumed by the command will count against the container like any other process.

<terminal inline>HTTP<terminal inline> handlers make an HTTP request against a URL exposed by the container. This gives you an alternative to launching a standalone process when you want to interact with an existing application component. Only <terminal inline>GET<terminal inline> requests are supported; if you need more advanced functionality, use the <terminal inline>Exec<terminal inline> handler to run a utility such as <terminal inline>curl<terminal inline> or <terminal inline>wget<terminal inline> instead.

Whichever handler you use, it’s best to keep your scripts as short and simple as possible. Long-running hook handlers will slow down container starts and stops, reducing the agility and efficiency of your cluster. If your handlers are likely to take more than a few seconds to run, it could be best to incorporate handler implementations into your container images, instead.

A different approach must also be used if you need a guarantee that your handler will only be called once. On rare occasions, Kubernetes may call handlers more than once for a single event. This may happen if, for example, a node’s kubelet process restarts midway through hook delivery. Handlers should be idempotent to avoid the possibility of any issues caused by this.

Failed hook handlers cause their container to be killed. In the case of <terminal inline>PostStart<terminal inline>, it means the container will never enter the <terminal inline>Running<terminal inline> state. A failed <terminal inline>PreStop<terminal inline> hook will be less serious, as the container would have terminated anyway. Kubernetes will not retry hooks or repeat event deliveries upon failure. This extends to treatment of fundamental issues such as the URL for an HTTP hook becoming unreachable—the hook will be treated as failed, and the container will be killed.

Adding Container Lifecycle Hooks

Now let’s use the available lifecycle hooks to respond to container creations and terminations. Hook handlers are attached to containers via their <terminal inline>lifecycle.postStart<terminal inline> and <terminal inline>lifecycle.preStop<terminal inline> manifest fields. Add the correct field for the hook you want to use. Nest an <terminal inline>exec<terminal inline> or <terminal inline>http<terminal inline> section within to configure your handler.

Here’s an example that applies hook handlers to a container running the NGINX web server. The implementation of the <terminal inline>PostStart<terminal inline> hook is trivial—it writes a file containing the time it was fired. The <terminal inline>PreStop<terminal inline> hook is used to gracefully stop NGINX when the container’s about to terminate, allowing it to finish serving existing clients. As this may take some time, the pod’s termination grace period is set to thirty seconds. This gives the combination of the <terminal inline>preStop<terminal inline> hook and the regular container termination process up to thirty seconds to complete.


apiVersion: v1
kind: Pod
metadata:
  name: hooks-demo
  namespace: demo
spec:
  containers:
    - name: hooks-demo
    image: nginx:latest
    ports:
        - containerPort: 80
    lifecycle:
        postStart:
        exec:
            command: ["/bin/sh", "-c", "date > /container-started.txt"]
        preStop:
        exec:
            command: ["/usr/sbin/nginx", "-s", "quit"]
  terminationGracePeriodSeconds: 30

Apply the manifest to your Kubernetes cluster using a tool like kubectl:


kubectl apply -f manifest.yml

Check the pod is <terminal inline>Running<terminal inline>:

$ kubectl --namespace demo get pod hooks-demo


NAMESPACE NAME READY STATUS RESTARTS
demo hooks-demo 1/1 Running 0

If the pod’s <terminal inline>Running<terminal inline>, the container must be, too! The <terminal inline>PostStart<terminal inline> script will have executed, so you can get a shell to the container and inspect the file that was created:


$ kubectl --namespace demo exec -it hooks-demo -- /bin/sh
$ cat /container-started.txt

Mon 28 Feb 09:30:00 GMT 2022

Here’s another example that makes an HTTP request to the container’s <terminal inline>/startup<terminal inline> URL upon creation:


apiVersion: v1
kind: Pod
metadata:
  name: hooks-demo
  namespace: demo
spec:
  containers:
    - name: hooks-demo
    image: nginx:latest
    ports:
        - containerPort: 80
    lifecycle:
        postStart:
        httpGet:
            path: /startup
            port: 80

Debugging Failed Hooks

Things can become difficult when a hook handler fails or behaves unexpectedly. Kubernetes doesn’t directly log what happens inside your handlers. Failures will be reported as <terminal inline>FailedPostStartHook<terminal inline> and <terminal inline>FailedPreStopHook<terminal inline> events you can view on your pods. The event’s message will describe what went wrong.

Here’s a pod that tries to use a non-existing command as a <terminal inline>PostStart<terminal inline> hook:


apiVersion: v1
kind: Pod
metadata:
  name: hooks-demo
  namespace: demo
spec:
  containers:
    - name: hooks-demo
    image: nginx:latest
    ports:
        - containerPort: 80
    lifecycle:
        postStart:
        exec:
            command: ["broken"]

Applying this manifest to your cluster will create a faulty container that never starts <terminal inline>Running<terminal inline>. It’ll be repeatedly restarted on a back-off loop each time the <terminal inline>PostStart<terminal inline> script fails. View the container’s events to see what’s causing the problem:

$ kubectl --namespace demo describe pod hooks-demo


<omitted>

Events:

TYPE REASON AGE FROM MESSAGE
---- ------ --- ---- -------
Normal Scheduled 30s default-scheduler Successfully assigned demo/hooks-demo...
Normal Created 10s (x2 over 11s) kubelet Created container hooks-demo
Normal Started 10s (x2 over 11s) kubelet Started container hooks-demo
Warning FailedPostStartHook 10s (x2 over 11s) kubelet Exec lifecycle hook ([broken]) for Container "hooks-demo" in Pod "hooks-demo" failed - error: command 'broken' exited with 126: , message: "OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: \"broken\": executable file not found in $PATH: unknown\r\n"
Normal Killing 10s (x2 over 11s) kubelet FailedPostStartHook
Warning Killing 10s (x2 over 11s) kubelet Back-off restarting failed container

The event log shows that the container was created and started successfully. The procedure went awry when the broken <terminal inline>PostStart<terminal inline> hook was executed. The handler’s failure to complete caused the container to be killed, entering a back-off loop that, in this example, is doomed to perpetual failure. The references to <terminal inline>x2 over 11s<terminal inline> in the log indicate multiple occurrences of each event due to the retry looping.

Final Thoughts

In this article, you’ve looked at what Kubernetes container lifecycle hooks are, and detailed some of the reasons they’re used. You’ve also seen how you can add hook handlers to your containers using the <terminal inline>lifecycle.postStart<terminal inline> and <terminal inline>lifecycle.preStop<terminal inline> manifest fields.

The available hooks let you respond to changes in a container’s lifecycle as they occur. They remove the need to set up custom tooling around your cluster and inside your containers to do things like make new creations wait for dependent services or perform cleanups upon termination. With both shell-based and HTTP hook handlers available, you’ve got flexibility to add the extra behaviors you need.

Start your free 14-day ContainIQ trial
Start Free TrialBook a Demo
No card required
James Walker
Software Engineer

James Walker is the founder of Heron Web, a UK-based digital agency providing bespoke software development services to SMEs. He has experience managing complete end-to-end web development workflows with DevOps, CI/CD, Docker, and Kubernetes. James also writes technical articles on programming and the software development lifecycle, using the insights acquired from his industry career. He's currently a regular contributor to CloudSavvy IT and has previously written for DigitalJournal.com, OnMSFT.com, and other technology-oriented publications.

READ MORE