Start your free 14-day ContainIQ trial

An Introduction to Kubernetes Networking

June 27, 2022

In this article, we explore the concept of Kubernetes networking with 10 in-depth diagrams, and additional context for beginners to advanced readers alike.

Lee Atchison
Author, Architecting for Scale

Introduction

In the world of Kubernetes, networking is an important topic. It’s not just because applications running in Kubernetes need access to resources outside the cluster to operate, but also because Kubernetes networking gives you flexibility and helps drive the adoption of loosely coupled service architectures.

In this article, we’ll introduce you to the basics of Kubernetes networking. We’ll talk about how loosely coupled services can work together to create large, advanced applications, how these same services can communicate to external services, and how external services can communicate with our services.

Basic Networking Terminology

Before we can start talking about Kubernetes, we need to define a few networking terms. In nearly all networked systems, there are a few fundamental ways in which one system talks to another system.

In Figure 1, Computer “A” on the left is talking to an application in Computer “B” on the right. In order for that communication to occur, several things must happen.

First, Computer “A” needs to find where the application is located, using two pieces of information:

  1. Hostname. This is the unique name given to the Computer “B.” In this case, we are using app.myapp.com.
  2. Port number. This is a unique address on Computer “B” that specifies which application to talk to on that computer. In this example, the port number is “12953.”

To locate the desired application, Computer “A” takes the hostname and performs a lookup to determine the computer’s IP address. The IP address is the address that the underlying network protocols use to actually locate Computer “B.” It does this by contacting a service called DNS, or Domain Name System. The DNS service performs a lookup to map the hostname to the computer’s IP address. This is very analogous to how a phone book is used to find a person’s name and phone number.

Once Computer “A” knows the IP address of Computer “B,” it can send a message to “B” by sending it to the IP address. This IP address is associated with a particular network interface on computer “B.” The message is sent to a particular port number on Computer “B,” which allows the message to be sent to the desired application. Each application on the computer is assigned a network interface/port number pair, so all messages arriving on a particular network interface (specified by the IP address) and a given port number will be routed to the desired application.

Many port numbers are standardized across common applications so that it’s easier to find the desired application. For example, port 80 is for the “HTTP service,” which processes web page requests. Port 21 is for the “FTP service” for file transfer. Port 25 is a standard port for the “SMTP service,” used to send email.

A computer can have multiple network interfaces (for instance, a cell phone typically has one for WiFi networking and one for cellular networking). They also can have one or more virtual network interfaces. This allows a single computer to have many IP addresses associated with it.

Figure 1. Basic networking
Figure 1. Basic networking

This is the basis for all networking between applications and between users and applications on the internet today.

Basic Kubernetes Cluster

Things get a bit more involved when you are talking about Kubernetes clusters. Let’s take a look at the basic building blocks of a Kubernetes cluster.

Our applications and application services run in containers, such as those created using Docker. These containers run in a Kubernetes object known as a pod. A pod is the smallest addressable component of a Kubernetes system. A single pod contains a single container (although it can contain more than one container working together) and performs the work of a single service instance.

An application typically does not have a single instance of a particular service running, it will have many instances to distribute the workload and provide a higher level of availability. In a Kubernetes system, this means there will typically be many instances of a pod running simultaneously for the same application. These pods, taken together, form a Kubernetes service. A Kubernetes service is a group of deployed pods running together, all performing the same function. A service will distribute the workload it has to perform across all of the pods in the service.

Each pod in each service needs to run on some form of computer instance. In Kubernetes, these computer instances are called nodes. A Kubernetes node may be a single server or some form of virtualized server instance. Typically, the nodes of a single service are distributed across multiple nodes, to increase availability. That way if a node crashes—and the pods on that node terminate—the service can still recover by using other pods on other nodes to handle the workload.

Figure 2. Basic Kubernetes cluster
Figure 2. Basic Kubernetes cluster

Together, all the nodes running all the pods in a Kubernetes system create a Kubernetes cluster. Figure 2 shows the components that make up a single cluster.

Talking Between Pods

We mentioned network interfaces previously, but one aspect of network interfaces is that they allow a single computer to split processes into multiple network namespaces, each assigned to a different ethernet interface. The namespaces typically are isolated and can’t talk to each other without going through the ethernet interface. In a Kubernetes cluster, each pod is given a unique namespace, so in order to talk to the code running in the pod, you need to talk to it through the assigned ethernet interface.

The node itself also has a namespace, called the root namespace. The node sets up a network tunnel between each pod and the root namespace. Because all pods have a tunnel to the root namespace, pods can talk to each other by sending messages to the root namespace through its tunnel, then to the other pod via its tunnel. This is illustrated in Figure 3.

Figure 3. Pod-to-pod communications
Figure 3. Pod-to-pod communications

The root namespace gives each pod a unique IP address (actually, a unique range of IP addresses, called a CIDR block). That way, messages sent from one pod are routed by the root namespace to the correct pod by using the destination IP address of the message. This is exactly the same way that messages are sent in regular networking from one computer to another. In this case, the messages go from one pod to another pod.

This same model works for communications between two pods on the same node and between two pods on different nodes. In the different node case, when a message is sent to an IP address representing a pod on a different node, the root namespace sends the message over the normal network connecting the nodes together to the node that has the destination pod. The message is sent to the root namespace of that destination node, which routes the message to the correct pod. This is illustrated in Figure 4.

Figure 4. Pod to pod communications across nodes
Figure 4. Pod to pod communications across nodes

An interesting question that comes up is, “How does the root namespace know which node to send a message to?” Each node may have hundreds of pods, each with one or more unique IP addresses (CIDR blocks), and there can be hundreds of nodes within a given Kubernetes cluster. How does any given node know where to send a message to reach the right pod?

Kubernetes makes sure each and every pod in the cluster has a unique set of IP addresses, so each pod is uniquely identifiable. Then, a standard network protocol known as ARP is used to broadcast the IP addresses and how to get to them to each node in the system. This protocol creates a map, or route, from any given node to any given IP address in any other node. This protocol dynamically creates this map and it is automatically updated continuously because the exact makeup of pods in the system can change often.

Incidentally, the ARP protocol is the same protocol used in normal host-based networking. Traditional networking raises the same question, “How do you know where to send a message that is destined for a given IP address?” The same ARP protocol, along with other standard routing protocols and default routes, automatically and dynamically figures out how any given computer on the internet can talk to any other computer on the internet. It’s dynamic, fast, scalable to an extremely large number of hosts, and extremely stable. It’s been powering the internet since the 1960s.

K8s Metrics, Logging, and Tracing
Monitor the health of your cluster and troubleshoot issues faster with pre-built dashboards that just work.
Start Free Trial Book a Demo

Using Services to Communicate

Even though each pod has an IP address that allows it to be directly addressable, there is one major problem with this model. Pods tend to come and go. If a pod has a problem, it’s terminated and a new pod is started. If traffic increases to a service and more instances are needed to handle the traffic, new pods are created. Essentially, pods are ephemeral, and as such, so are their IP addresses. How can you reliably send a message to a pod if the IP addresses for the pods keep changing?

The answer to that problem is the service. A Kubernetes service provides a front end to a series of pods that are performing the same capabilities. All pods in a system that perform the same functions are part of the same service. As pods are launched and terminated, they are added and removed to the corresponding service, so the service always has a list of valid pods and their IP address.

Now, if you need the capabilities of a pod, rather than send a message directly to the pod, you send the message to the service. The service then forwards the request to one of the pods that it knows is active. The service can load balance the requests it receives across a series of pods that are performing the same capability, even if those pods are on different nodes.

The service has a fixed, assigned IP address associated with it. This IP address typically never changes, so anyone who wishes to engage a pod to perform some action, can talk to the service at a well-known IP address and not worry about the details regarding which pod is implementing the request. Figure 5 shows a service named “ABC” and the pods it is connected to. One pod is dying in the diagram and a new pod is being created. The service maintains the list of pods that are currently active and makes sure requests are forwarded to an appropriate active pod.

Figure 5. A service creates a virtual address in front of the changing pods
Figure 5. A service creates a virtual address in front of the changing pods

Pod to Service

The network message routing for one pod talking to another is slightly different when services are involved. As shown in Figure 6, Pod A sends a message to Service ABC. This service, in the root namespace of the node, determines which pod to send the request to. It then updates the IP address of the message to contain the correct IP address of the chosen pod and forwards the request to that pod (Pod D in this example). The routing for the message from the service to the pod is identical as previously shown in Figure 4. If a pod is terminated, and a new one takes its place, the new pod will receive requests sent to the service in place of the old pod. The service to pod mapping is completely dynamic.

Figure 6. Pod to pod communications via service
Figure 6. Pod to pod communications via service

External Internet to Service

Services allow the external internet to easily access the capabilities of the pods within the cluster without knowing any of the details of how the pods are laid out in the cluster. When a service is created that is designed to accept traffic from the external internet, a load balancer is attached to the front end of the system facing the internet. It routes traffic to the nodes that contain the service. The service then picks an appropriate node to send the message to. As such, a request from the external internet can be routed to a particular pod without the requestor knowing the address of the pod. They only know the IP address of the service load balancer that is fronting this request.

Figure 7 shows the routing of a message from the external internet (bottom of image) through the service load balancer, to the service on a node, and on to the destination pod.

Figure 7. Service loadbalancer for external internet access
Figure 7. Service loadbalancer for external internet access

DNS and Kubernetes Services

Now that the service has an externally visible IP address to receive traffic, we can put this address into DNS, so that we can refer to the service by its hostname, rather than its IP address. Figure 8 shows the hostname “app.myapp.com” as it maps the request to the IP address of the service load balancer that handles this service.

Figure 8. DNS enables application discovery
Figure 8. DNS enables application discovery

Inter-cluster Communications

Communicating between pods in two different clusters is handled just like communications to the external internet. The pod sending the request looks up the hostname in DNS and gets the IP address of the service load balancer. The load balancer forwards the request to one of the nodes within the destination cluster. This node forwards the request to an appropriate pod within the destination cluster.

Large Service-based Applications

Building a large service-based application using Kubernetes is an exercise in creating multiple Kubernetes services, with each service operating multiple pods. The pods are spread among multiple nodes to provide appropriate load balancing and service-level availability guarantees.

Figure 9 shows a typical service-oriented application (top), and how Kubernetes services and pods map onto that application.

Figure 9. Service-based application mapped to Kubernetes services and nodes
Figure 9. Service-based application mapped to Kubernetes services and nodes

These nodes are distributed across multiple nodes within the cluster, with the pods for a given service spread across multiple nodes for availability purposes. Figure 10 shows the application from the perspective of pods running on nodes.

Figure 10. Service-based application from node perspective
Figure 10. Service-based application from node perspective

With this sort of application distribution, it is easy to imagine why pod-to-pod communications, via services, across node boundaries is so essential. Kubernetes provides the networking infrastructure for complex service-oriented architectures to exist and thrive in a Kubernetes cluster environment.

Final Thoughts

This was a simple overview of how networking works within a Kubernetes cluster. In general, Kubernetes is very flexible in how it allows communication between pods within the cluster, as well as how it communicates with the internet. While the details can get quite technical, the high-level overview is quite straightforward. Kubernetes networking provides a sophisticated, flexible model for networking within a distributed containerized environment.

Start your free 14-day ContainIQ trial
Start Free TrialBook a Demo
No card required
Lee Atchison
Author, Architecting for Scale

Lee Atchison is a recognized thought leader in cloud computing and application modernization. With more than three decades of experience in product development, architecting, scaling, and modernization, Lee has worked at Amazon, Amazon Web Services (AWS), New Relic, and other modern application organizations. He is widely quoted in many publications and has been a featured speaker across the globe. Lee’s most recent book is Architecting for Scale (O’Reilly Media). You can check out his books, courses, articles, and speaking sessions at leeatchison.com, and follow him on Twitter and LinkedIn.

READ MORE