Kubernetes ImagePullBackOff: Troubleshooting With Examples

Troubleshooting an issue on a distributed system like Kubernetes can be challenging at times. There are just so many things that can go wrong with a distributed system. It can be even more challenging when a particular error has multiple reasons for occurring.

One such error is <terminal inline>ImagePullBackOff<terminal inline>. It typically shows up when the kubelet agent instructs the container runtime and can’t pull the image from the container registry for various reasons.

This article will provide an in-depth overview of possible causes for your pod entering into <terminal inline>ImagePullBackOff<terminal inline> state while starting your container. More importantly, you’ll learn how to troubleshoot and solve this notorious error.

What Does an ImagePullBackOff Error Mean?

The <terminal inline>ImagePull<terminal inline> part of the <terminal inline>ImagePullBackOff<terminal inline> error primarily relates to your Kubernetes container runtime being unable to pull the image from a private or public container registry. The <terminal inline>Backoff<terminal inline> part indicates that Kubernetes will continuously pull the image with an increasing backoff delay. Kubernetes will keep on increasing the delay with each attempt until it reaches the limit of five minutes.

It seems like a generalized statement to say that container runtime (be it Docker, containerd, etc.) fails to pull the image from the registry, but let’s try to understand the possible causes for this issue.

Here are some of the possible causes behind your pod getting stuck in the <terminal inline>ImagePullBackOff<terminal inline> state:

• Image doesn’t exist.
• Image tag or name is incorrect.
• Image is private, and there is an authentication failure.
• Network issue.
• Registry name is incorrect.
• Container registry rate limits.

How Can You Troubleshoot ImagePullBackOff?

Let’s try to troubleshoot each of the possible causes in that bulleted list.

Image Doesn’t Exist, or Name Is Incorrect

In most cases, the error could be either from a typo or the image was not pushed to the container registry, and you’re referring to an image that doesn’t exist. Let’s try to replicate this by creating a pod with a fake image name.

$ kubectl run myapp --image=myimage/myimage:latest
pod/myapp created

As you can see, the pod is stuck in an <terminal inline>ImagePullBackOff<terminal inline> because the image doesn’t exist and we cannot pull the image.

To understand the root cause and find more details about this error, use the <terminal inline>kubectl describe<terminal inline> command. The command itself gives a verbose output, so we’ll just show the parts of output that are relevant to our discussion.

In the following output under Events in the Message column, you can see the actual error message:

$ kubectl run myapp --image=myimage/myimage:latest
pod/myapp created

Which confirms that the image doesn’t exist.

Monitor Kubernetes Events in Real-Time

Monitor the health of your cluster and troubleshoot issues faster with pre-built dashboards that just work.

Learn More Book a Demo

Tag Doesn’t Exist

There could be cases where the image tag you’re trying to pull is retired, or you entered the wrong tag name. In those cases, your pod will again get stuck in the <terminal inline>ImagePullBackOff<terminal inline> state, as seen in the following code snippet.

We have deliberately entered the wrong tag name, <terminal inline>lates<terminal inline> instead of <terminal inline>latest<terminal inline>, to replicate this issue.

kubectl run nginx --image=nginx:lates
pod/nginx created

In the following output, the message indicates that tag <terminal inline>lates<terminal inline> doesn’t exist for image <terminal inline>nginx<terminal inline>.

Failed to pull image "nginx:lates": rpc error: code = Unknown desc = Error response from daemon: manifest for nginx:lates not found: manifest unknown: manifest unknown

Hence the image pull is unsuccessful.

Private Image Registry and Wrong Credentials Provided

Most enterprises typically use an internal private container registry instead of DockerHub because they don’t want to push their internal applications to someone outside their organization. Even with DockerHub or any other publicly accessible password-protected registry, you must provide proper credentials to Kubernetes using the secret to pull the image from the registry.

In the following example, we’re trying to replicate this issue by spinning up a pod that uses an image from a private registry.

$ kubectl run mypod --image=docker.io/hiyou/imagename

We have neither added a secret to Kubernetes nor reference of the secret in pod definition. The pod will again get stuck in the ImagePullBackOff status and the message confirms that access is denied to pull an image from the registry:

Failed to pull image "docker.io/hiyou/image": rpc error: code = Unknown desc = Error response from daemon: pull access denied for hiyou/image, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

To resolve this error, create a secret using the following <terminal inline>kubectl<terminal inline> command. The following <terminal inline>kubectl<terminal inline> command creates a secret for a private Docker registry.

kubectl create secret docker-registry dockersecret --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

Add your secret to your pod definition, as explained in the following snippet.

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: dockersecret

Network Issue

There could be a widespread network issue on all the nodes of your Kubernetes cluster, and the container runtime will not be able to pull the image from the container registry. Let’s try to replicate that scenario.

kubectl run nginx --image=nginx:latest
pod/nginx created

In the preceding output, the message indicates that there is a network issue.

Failed to pull image "nginx:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head https://registry-1.docker.io/v2/library/nginx/manifests/latest: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: server misbehaving

Container Registry Rate Limits

Most container registries have implemented some rate limits (i.e., number of images you can pull) to protect their infrastructure. For example, with Docker Hub, anonymous and free Docker Hub users can only request 100 and 200 container image pull requests per six hours. If you exceed your maximum download limit, you’ll be blocked, resulting in <terminal inline>ImagePullBackOff<terminal inline> error.

To resolve this for Docker Hub, you would need to upgrade to a Pro or Team account. Many other popular container image registries like GCR or ECR propose similar limitations.

Final Thoughts

In this article, you learned some possible reasons why a pod would get stuck in an <terminal inline>ImagePullBackOff<terminal inline> state. You checked out some different examples to understand the error better and troubleshoot it with commands like <terminal inline>kubectl describe<terminal inline>.

If you’re confident there is no typo in the image, registry, or tag name, then <terminal inline>kubectl describe<terminal inline> will reveal the chain of events that led to the failure. In some cases, you may be able to pull the image using <terminal inline>docker pull<terminal inline>, but your cluster can’t, then that probably means there’s a network issue.