Using Kubernetes Admission Controllers & Webhooks

November 30, 2021

Kubernetes offers rich deployment for containerized services but sometimes you need more to fully optimize infrastructure. This guide offers tips for using admission controllers and webhooks effectively.

Tyler Charboneau
Software Engineer

Kubernetes provides organizations with rich deployment options for containerized services, yet additional configurations are sometimes needed to fully optimize infrastructure. Deeper controls situated between sender and receiver—in this latter case, the Kubernetes API server—are required to unlock all of Kubernetes’s powerful features.

The Kubernetes API undertakes a number of key operations within the overall system. This includes querying and manipulating API objects including namespaces, pods, events, and ConfigMaps. Since the Kubernetes API server is central to the Kubernetes control plane, the methods through which you can utilize the API are numerous. It’s what makes REST calls and <terminal inline>kubectl<terminal inline> commands so vital within a microservices architecture.

Admission webhooks and controllers provide the extensibility that many administrators require. Discussed in unison, webhooks are specialized types of controllers that either “validate” or “mutate” objects. Admission webhooks are HTTP callbacks, meaning they receive and route admission requests accordingly. These plugins ultimately have dominion over clusters and determine how Kubernetes uses them. Anadmissions controller is a code block that intercepts Kubernetes server requests and performs custom operations.

This article thoroughly reviews admissions controllers and webhooks. You’ll learn why each mechanism is important, how they exist throughout the system, and how they’re deployed or configured.

Why Do You Need Admission Webhooks and Controllers?

We’ve touched on how webhooks help determine the way API requests travel between different components of the Kubernetes system. Because these link to the cluster control plane, leveraging them is immensely beneficial for those seeking ultimate configuration control. However, proceed with caution. This level of control is useful, yet missteps can cause larger problems that impact your overall system.

Webhooks are convenient because you don’t have to compile them into your builds. Their plugin nature lets them behave like extensions—which you can essentially bolt on to your infrastructure with minimal developmental work. After deployment, the mutate webhook type executes first, followed by the validate webhook. These mutations modify objects, while validations help decide which custom policies are enforced or rejected.

Controllers and webhooks exist separately within Kubernetes. Controllers effectively live within the <terminal inline>kube-apiserver<terminal inline> binary and are compiled accordingly. Webhooks are fittingly configured within the API itself, since they directly influence HTTP activity.

Controllers have elevated privileges within Kubernetes, given that they impact requests after authentication and authorization are completed.

Admission controllers can limit requests to the following:

  • Create
  • Delete
  • Modify
  • Connect to proxy

However, admission controllers cannot handle read requests. While requests may be successful, controllers can return error responses to users upon rejecting requests.

How to Implement

Getting started with these powerful components of Kubernetes requires some prep work. Controllers must be activated before you can begin using them. Activation is easy to do with a quick <terminal inline>kubectl<terminal inline> command:

kube-apiserver --enable-admission-plugins=CertificateApproval

Turning off an admissions controller is equally simple. Simply enter the following prompt:

kube-apiserver --disable-admission-plugins=CertificateApproval

This same command works for any of Kubernetes 30+ admission controllers. Just replace <terminal inline>CertificateApproval<terminal inline> with your preferred controller. You can also simultaneously activate multiple controllers by sequentially writing in a comma-list format.

However, Kubernetes does save you the work of manually enabling default controllers. Because these controllers are already on, you can verify their status using the following command:

Kube-apiserver -h | grep enable-admission-plugins

It’s important to note that over half of these controllers are automatically enabled. Kubernetes already has these controllers up and running:

CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook

As time passes, the team behind Kubernetes will deprecate various controllers through the development cycle. These changes are documented accordingly.

Kubernetes’s documentation also denotes which controllers are actively being developed (alpha) and which may not be stable enough (or supported for) production use. While some work out of the box, others require you to specify appropriate fields and values within a YAML configuration file.

Overview of Admission Controller Types

There’s an impressive variety of controllers that you can use to enhance your Kubernetes deployments and achieve fine-grained control. It’s important to determine if these supplemental controllers can benefit you before implementing. Each non-default controller is broken down briefly and explained below.

  • <terminal inline>AlwaysPullImages<terminal inline>. Every new pod automatically forces the image pull policy to “Always” (useful for credential-dependent multi-tenant clusters, for privacy reasons).
  • <terminal inline>DenyServiceExternalIPs<terminal inline>. All net-new usages of the <terminal inline>Service<terminal inline> field <terminal inline>externalIPs<terminal inline> are rejected.
  • <terminal inline>EventRateLimit<terminal inline>. The controller prevents flurries of requests from overwhelming the server, through enforced rate-limiting controls.
  • <terminal inline>ExtendedResourceToleration<terminal inline>. Allows the creation of new nodes with extended resources.
  • <terminal inline>ImagePolicyWebhook<terminal inline>. Permits a backend webhook to make admission decisions.
  • <terminal inline>LimitPodHardAntiAffinityTopology<terminal inline>. Denies any pods that define an <terminal inline>AntiAffinity<terminal inline> topology key aside from <terminal inline><terminal inline>.
  • <terminal inline>NamespaceAutoProvision<terminal inline>. Checks all incoming requests for the existence of referenced namespaces, otherwise creating an associated namespace.
  • <terminal inline>NamespaceExists<terminal inline>. Checks all requests on namespaced resources other than <terminal inline>Namespace<terminal inline>.
  • <terminal inline>NodeRestriction<terminal inline>. Limits the <terminal inline>Node<terminal inline> and <terminal inline>Pod<terminal inline> objects a kubelet can modify.
  • <terminal inline>OwnerReferencesPermissionEnforcement<terminal inline>. Protects access to an object’s <terminal inline>metadata.ownerReferences<terminal inline> file, so only owners with <terminal inline>delete<terminal inline> permissions can modify it.
  • <terminal inline>PodNodeSelector<terminal inline>. Determines and limits which node selectors are usable within a namespace.
  • <terminal inline>PodSecurity<terminal inline>. Determines on pod modification and creation if that pod should be admitted based on security controls (alpha).
  • <terminal inline>PodTolerationRestriction<terminal inline>. Verifies conflicts between pod and namespace tolerations, either rejecting pods or merging tolerances accordingly.
  • <terminal inline>SecurityContextDeny<terminal inline>. Denies any pod which attempts to escalate certain <terminal inline>SecurityContext<terminal inline> fields outside of scope.

As you can see, admission controllers can directly impact horizontal scaling, proper resource consumption, security, tolerations, and more. While these aren’t inherently required to leverage Kubernetes, the evolution of your deployment (and use cases) may make them relevant. Some are more challenging to enable than others. However, the Kuberenetes controllers documentation is extremely handy for exploring best practices.

Deploying Webhooks for Mutation and Validation

Before deploying and experimenting with webhooks, the following prerequisites must be met:

  • Your Kubernetes cluster(s) must be running v1.16 or newer, or v1.9 when using <terminal inline><terminal inline>.
  • Both the <terminal inline>MutatingAdmissionWebhook<terminal inline> and <terminal inline>ValidatingAdmissionWebhook<terminal inline> controllers must be enabled.
  • Either the <terminal inline><terminal inline> or <terminal inline><terminal inline> API must be enabled.

You’ll also have to include or write an intermediary admission webhook server to handle these specialized requests as they traverse the network. The <terminal inline>AdmissionReview<terminal inline> object relays any decisions made regarding those requests once they’re sent, thus determining whether they’re valid or denied. It’s not always necessary to authenticate clients or other fields (for example), and you can configure your server protocols accordingly. You may also opt for or against mutual TLS (Transport Layer Security).

Deployments also enjoy some flexibility. You can deploy your webhooks either within or outside your clusters depending on preference or functional requirements. This is true for both test cases and production deployments. The former relies on the Deployment API. This creates a service, which acts as the frontend component to your webhooks server. The code is quite extensive, yet openly available via GitHub.

Deploying externally, by contrast, may require some additional configurations to ensure everything works properly. This is where <terminal inline>ValidatingWebhookConfiguration<terminal inline> and <terminal inline>MutatingWebhookConfiguration<terminal inline> come in handy. The Kubernetes documentation outlines the following example:

kind: ValidatingWebhookConfiguration
  name: ""
- name: ""
  - apiGroups:   [""]
    apiVersions: ["v1"]
    operations:  ["CREATE"]
    resources:   ["pods"]
    scope:       "Namespaced"
      namespace: "example-namespace"
      name: "example-service"
    caBundle: "Ci0tLS0tQk...<`caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate.>...tLS0K"
  admissionReviewVersions: ["v1", "v1beta1"]
  sideEffects: None
  timeoutSeconds: 5

It’s also expected that you define your webhook requests and responses via YAML files. This includes any matching request and policy configurations, which are widely varied akin to controller types. URLs, references, timeouts, failures, and reinvocations will be essential elements in this equation. Each piece is configurable and customizable to your deployment. Kubernetes provides two separate YAML configurations depending on your <terminal inline>admissionregistration<terminal inline> version.

Be sure to follow best practices as defined by Kubernetes, especially when avoiding any problematic side effects. You can also monitor your webhooks in production through metrics collection.

In addition, consider how deep your webhooks deployments will extend. Those that touch the control plane need the <terminal inline>kube-system<terminal inline> namespace to be intercepted prior to modification. If your deployment ignores the control plane, use the <terminal inline>namespaceSelector<terminal inline> field to exclude <terminal inline>kube-system<terminal inline>. This prevents unpredictability and ensures that everything functions normally.

Final Thoughts

Kubernetes’s default functionality is undoubtedly impressive, yet ambitious administrators might favor extensibility for their use cases. Admission controllers and webhooks give teams added flexibility, while unlocking Kubernetes’s complete granularity. These separate basic deployments form purpose-built deployments.

However, remember that configuration changes aren’t geared toward Kubernetes novices—they’re mostly advanced operations with careful considerations attached.

Are you looking to better monitor your Kubernetes events and metrics? ContainIQ simplifies Kubernetes monitoring through centralized visualization, easy core-metrics gathering, and streamlined cluster health monitoring.

Article by

Tyler Charboneau

Software Engineer

Tyler is a hardware-software devotee and researcher. He specializes in simplifying the complex while speaking effectively to all audiences. Tyler has a Bachelor of Arts degree from the University of Michigan and is a self-taught software engineer. His work has been published on a number of leading technology blogs from companies including Veeam, Launch Darkly, and others.

Read More