Resources

Debugging Your Kubernetes Nodes in the ‘Not Ready’ State | nodenotready

November 30, 2021

Kubernetes clusters typically run on multiple “nodes” each having its own state. In this article, you’ll learn a few possible reasons a node might enter the NotReady state and how you can debug it.

Aniket Bhattacharyea
Software Engineer

Nodes are a vital component of a Kubernetes cluster and are responsible for running the pods. Depending on your cluster setup, a node can be a physical or a virtual machine. A cluster typically has one or multiple nodes, which are managed by the control plane.

Because nodes do the heavy lifting of managing the workload, you want to make sure all your nodes are running correctly. The <terminal inline>kubectl get nodes<terminal inline> command can be used to check the state of your nodes.

Output of kubectl get nodes
Output of kubectl get nodes

A node with a <terminal inline>NotReady<terminal inline> status means it can’t be used to run a pod because of an underlying issue. It’s essentially used to debug a node in the <terminal inline>NotReady<terminal inline> state so that it doesn’t lie unused.

In this article, you’ll learn a few possible reasons why a node might enter the <terminal inline>NotReady<terminal inline> state and how you can debug it.

The <terminal inline>NotReady<terminal inline> State

As mentioned earlier, each node in a cluster is used to run pods. Before a pod is scheduled on a node, Kubernetes checks whether the node is capable of running the pod or not. The <terminal inline>STATUS<terminal inline> column in the output of <terminal inline>kubectl get nodes<terminal inline> represents the status. The possible values in this column are:

  1. <terminal inline>Ready<terminal inline>: The node is healthy and ready to accept pods.
  2. <terminal inline>NotReady<terminal inline>: The node has encountered some issue and a pod cannot be scheduled on it.
  3. <terminal inline>SchedulingDisabled<terminal inline>: The node is marked as unschedulable. This can be done using the kubectl cordon command.
  4. <terminal inline>Unknown<terminal inline>: The node is unreachable by the control plane.

Having a node in the <terminal inline>NotReady<terminal inline> state means that the node is effectively unused and will accumulate costs without participating in running pods. Furthermore, losing a node can negatively impact your production workload.

In order for your application to run smoothly, you must debug them quickly.

Possible Causes of the <terminal inline>NotReady<terminal inline> State

There can be various reasons why a node might enter the <terminal inline>NotReady<terminal inline> state. This section will review some of the most common reasons for this error.

Scarcity of Resources

To operate normally, a node must have sufficient disk space, memory, and sufficient processing ability. If a node is running low on disk space or the available memory is low, it will go into the <terminal inline>NotReady<terminal inline> state. If pressure exists on the processes, eg too many processes are running on the node, it will also change to the <terminal inline>NotReady<terminal inline> state.

Network Misconfiguration

If the network has not been correctly configured on the node or it can’t reach the internet, the node will be unable to communicate with the master node and will be listed as <terminal inline>NotReady<terminal inline>.

Issue with kubelet Process

kubelet is an agent that runs on each node. It is responsible for communicating with the Kubernetes API server and registering the nodes. If kubelet crashes or stops on the node, it will not be able to communicate with the API Server and will be in the <terminal inline>NotReady<terminal inline> state.

Issue with kube-proxy

kube-proxy is a network proxy that runs on each node and maintains the network rules. These rules allow network communication to your pods from inside or outside your cluster. If kube-proxy crashes or stops, the node will be in the <terminal inline>NotReady<terminal inline> state.

Vendor Specific Issues

Suppose you’re using a cloud-hosted solution like GKE or EKS. In that case, some vendor-specific issues may be preventing your nodes from operating normally and communicating with the control plane. These issues could be IAM misconfiguration, misconfigured network rules, etc.

Debugging the <terminal inline>NotReady<terminal inline> State

As you can see, the <terminal inline>NotReady<terminal inline> status can be caused by a multitude of issues. This section will help you identify the root cause of the problem. However, it’s essential to understand that how you go about fixing these issues depends on the exact cause and your cluster setup. There are no one-size-fits-all solutions. But, once you identify the root cause, it should be easier to resolve it.

Check the kube-proxy Pod

First, ensure that each node has exactly one <terminal inline>kube-proxy<terminal inline> pod and is in the <terminal inline>Running<terminal inline> state.


kubectl get pods -n kube-system -o wide

The output might look like this:

NAME READY STATUS AGE IP NODE NOMINATED NODE READINESS GATES
kube-proxy-nhbtp 1/1 Running 2 (11h ago) 2d16h 192.168.99.10 1 my-cluster <none> <none>
kube-proxy-tkmsk 1/1 Running 2 (11h ago) 2d16h 192.168.99.10 3 my-cluster-m03 <none> <none>
kube-proxy-vk4ch 1/1 Running 2 (11h ago) 2d16h 192.168.99.10 2 my-cluster-m02 <none> <none>

If any one pod is in some state other than <terminal inline>Running<terminal inline>, use the following command to get more information:


kubectl describe pod yourPodName -n kube-system

The Events section logs the various events on the pod, and it could be an excellent place to start looking for any mishaps.

The events section in the output
The events section in the output

You can get access to the pod logs by running the following command:


kubectl logs yourPodName -n kube-system

The logs and the events list is a good place to start looking for any issues.

If your node does not have a <terminal inline>kube-proxy<terminal inline> pod, then you need to inspect the <terminal inline>kube-proxy<terminal inline> daemonset, which is responsible for running one kube-proxy pod on each node.


kubectl describe daemonset kube-proxy -n kube-system

The output of this command might reveal any possible issue with the daemonset.

Verify Resources are Available

Run the following command to get detailed information about a node that is not ready:


kubectl describe node nodeName

In the output, the <terminal inline>Conditions<terminal inline> section shows if the node is running out of resources or not.

The conditions section in the output
The conditions section in the output

The following conditions are available:

  1. <terminal inline>MemoryPressure<terminal inline>: If <terminal inline>True<terminal inline>, it indicates that the node is running out of memory.
  2. <terminal inline>DiskPressure<terminal inline>: A <terminal inline>True<terminal inline> value in this field indicates that the node lacks enough space.
  3. <terminal inline>PIDPressure<terminal inline>: If too many processes are running on the node, this field will be <terminal inline>True<terminal inline>.
  4. <terminal inline>NetworkUnavailable<terminal inline>: If the network for the node is not correctly configured, this will be <terminal inline>True<terminal inline>.
  5. <terminal inline>Ready<terminal inline>: If the node is healthy and ready to accept pods, this will be <terminal inline>True<terminal inline>. In this field, a <terminal inline>False<terminal inline> is equivalent to the <terminal inline>NotReady<terminal inline> status in the <terminal inline>get nodes<terminal inline> output. It can also have the <terminal inline>Unknown<terminal inline> value, which means the node controller has not heard from the node in the last <terminal inline>node-monitor-grace-period<terminal inline> (defaults to 40 seconds).

If any one of the first four conditions is <terminal inline>True<terminal inline>, you have identified the problem.

Verify kubelet is Running

If all the <terminal inline>Conditions<terminal inline> fields show <terminal inline>Unknown<terminal inline>, it might hint that the kubelet process on the node has run into some issues.

The conditions field shows unknown
The conditions field shows unknown

To debug this, first SSH into the node and check the status of the kubelet process. If it’s running as a systemd service, use the following command:


systemctl status kubelet

If the <terminal inline>Active<terminal inline> field shows <terminal inline>inactive (dead)<terminal inline>, it means the kubelet process has stopped.

The active field of the output
The active field of the output

To reveal the possible reason for the crash, check the logs with the following command:


journalctl -u kubelet

Once the issue is fixed, restart kubelet with:


systemctl restart kubelet

Verify Network Communication with the Control Plane

If the <terminal inline>Conditions<terminal inline> field shows <terminal inline>NetworkUnavailable<terminal inline>, it indicates an issue in the network communication between the node and the control plane.

A few possible fixes:

  • If the node is configured to use a proxy, verify that the proxy allows access to the API server endpoints.
  • Ensure that the route tables are appropriately configured to avoid blocking communication with the API server.
  • If you’re using a cloud provider like AWS, verify that no VPC network rules block communication between the control plane and the node.

You can run the following command from within the node to verify that it can reach the API server.


nc -vz <your-api-server-endpoint> 443

If the output shows <terminal inline>succeeded<terminal inline>, then network communication is working correctly.

Vendor Specific Debugging

If you’re using a cloud provider like EKS, or GKE, sometimes it’s worth looking into vendor-specific issues if you’ve exhausted all other debugging techniques. EKS has an extremely detailed guide that you can follow.

GKE provides an auto repair feature that can attempt to repair a node that has been in the <terminal inline>NotReady<terminal inline> state for a given amount of time. If all else fails, you can always get in touch with your cloud provider for more assistance.

Final Thoughts

Having a node in the <terminal inline>NotReady<terminal inline> state is undesirable and needs to be fixed immediately. However, there are multiple reasons this might occur, and it can be challenging to pinpoint the exact cause. This article discussed some common reasons you may encounter the <terminal inline>NotReady<terminal inline> command and solutions for it.

The earlier you can catch nodes entering the <terminal inline>NotReady<terminal inline> state, the higher your chances of quickly debugging it. ContainIQ is a Kubernetes monitoring platform that comes with an extensive events dashboard that can monitor and alert you when a node enters the <terminal inline>NotReady<terminal inline> state. You can also see all the events leading up to this, allowing you to quickly identify and solve the issue.

Looking for an out-of-the-box monitoring solution?

With a simple one-line install, ContainIQ allows you to monitor the health of your cluster with pre-built dashboards and easy-to-set alerts.

Article by

Aniket Bhattacharyea

Software Engineer

Aniket is a software engineer currently working towards a Master's in Mathematics from Ramakrishna Mission Vivekananda Educational and Research Institute. Previously, Aniket was Head of Technology at PiParadox, where he worked in C, C++, Python, JavaScript, and many other languages. Prior to that, in 2016, Aniket participated in Google Code where he finished in 6th position in TopCoder. Aniket has a Bachelor of Science degree in Mathematics from St. Xavier's College, Kolkata.

Read More