Start your free 14-day ContainIQ trial

On-Prem Kubernetes Monitoring: Best Practices, Tools to Use

July 3, 2022

For companies with highly secure environments, using a K8s monitoring toolset on-prem, or in-cloud, can offer peace of mind. We explore the challenges, some best practices, and tools to help.

Nate Matherson
Co-founder

Kubernetes has become an increasingly important tool for companies running workloads with containers. Small and large organizations have migrated to K8s for a number of reasons, including cost savings, autoscaling, and improved productivity.

These days, most organizations are also using a cloud-hosted or managed Kubernetes service. Cloud providers like AWS, GCP, and Azure all offer managed Kubernetes services. However, some engineering teams are still working in a physical on-prem environment, where they are responsible for supporting Kubernetes and maintaining the hardware.

While Kubernetes has many benefits, there is often a need for teams to deploy a monitoring and observability stack to troubleshoot issues that happen within the cluster and the applications themselves.

For companies with highly secure environments, like those in financial services or healthcare, using a SaaS offering for monitoring and observability might not be feasible. The risk of sensitive data leaving a secure environment for a third-party environment keeps engineering leaders up at night.

For these companies, whether they are using a managed Kubernetes service or running K8s with physical hardware on-prem, choosing to run their observability tools in-cluster, in-cloud, or “on-prem” offers peace of mind. In this article, we’ll explore some of the challenges of using an on-prem, or in-cloud, set of observability tools, introduce some best practices, and highlight a handful of tools that companies are using.

Challenges of Introducing Monitoring On-Prem or In-Cloud

While running your monitoring and observability toolset on-prem has many benefits, there are without a doubt a number of challenges that organizations might face. These challenges include the added complexity of self-managing a toolset, additional direct costs, and the limited availability of tooling that works on-prem or in an in-cloud environment.

Self-Management

One of the biggest challenges of running a monitoring solution in an on-prem environment is that it requires self-management. This introduces another variable for engineering teams to be aware of. And it can create a scenario wherein your monitoring toolset needs, itself, to be monitored.

Fortunately, today, there are a number of open-source tools with an emphasis on Kubernetes. And many of these tools have become reliable and secure, due in large part to robust contributor communities.

While self-managing your toolset on-prem or in-cloud will take additional engineering time, it may be worthwhile for companies in highly secure industries.

Additional Direct Costs

When using a managed solution or SaaS offering, you are able to pass along the costs of hosting to the vendor itself. However, when using a toolset on-prem, or in-cloud specifically, you will be tasked with hosting and managing this resource on your own.

By self-hosting your toolset, you might find that the direct costs of hosting and maintaining the environment will increase. For example, if you are self-hosting Prometheus in-cloud, there will be meaningful cloud costs associated with receiving and storing metrics in the time series database.

Limited Availability of Tooling

Unfortunately, for companies looking to run their toolset in an on-prem environment, there are a smaller number of vendors to choose from. Popular solutions like Datadog and New Relic are not available for on-prem or in-cloud offerings. And while Datadog, for example, allows you to run their agent in an on-prem cluster, it still requires that the data then leave.

Tools For K8s Monitoring On-Prem

As discussed in the previous section, a limited number of tools are available for companies looking to run their monitoring on-prem. Vendors like Datadog, New Relic, and others do not offer their services to customers on-prem. In this section, we’ll highlight three different tools and platforms that are available for companies looking to keep their monitoring toolset on-prem: two third-party offerings and a popular open-source solution.

Using ContainIQ | A K8s Native Solution

ContainIQ Kubernetes platform

ContainIQ, a Kubernetes native monitoring and tracing platform, is available both as a SaaS and as an on-premise solution. ContainIQ users are able to run ContainIQ on-prem on their physical machines, as well as through one of the cloud providers as an in-cloud offering.

ContainIQ provides a number of features out of the box, including metrics, logging, and tracing. And ContainIQ makes it easy to troubleshoot when issues arise, with its simple one-click correlations functionality. For example, users are able to click to correlate everything from logs to individual requests, Kubernetes events to logs and metrics, and individual requests to metrics at points in time. ContainIQ also allows users to alert on most pieces of information collected by the platform, including metrics, events, latency, and log messages.

ContainIQ is often compared to a number of open-source tools, like Prometheus and Grafana. However, ContainIQ is able to deliver in one solution what would otherwise take a number of disparate open-source tools. ContainIQ is also able to deliver APM-like features, including tracing, without actually needing to be configured on each microservice manually.

ContainIQ’s UX/UI is more approachable and allows for the entire engineering team to get and stay engaged.

Getting started with ContainIQ’s on-prem offering requires that users first book a demo with a member of the team. For the on-prem solution, ContainIQ charges $20 per node per month and typically requires an annual commitment. ContainIQ does not charge per seat, and an unlimited number of seats are included.

K8s Metrics, Logging, and Tracing
Monitor the health of your cluster and troubleshoot issues faster with pre-built dashboards that just work.
Start Free Trial Book a Demo

Using Prometheus and Grafana | Popular Open-Source Tools

Prometheus, a popular open-source monitoring solution, and a graduated CNCF project, is available for companies running workloads on-prem. Prometheus is a time series database known for its ability to scale to massive environments and for its efficiency.

Prometheus is often used in conjunction with Grafana, which is an open-source tool for visualization. Today, both Prometheus and Grafana can be run as managed SaaS offerings, through the cloud providers or through Grafana Enterprise. However, many organizations still run these tools together in on-prem environments, too.

Prometheus is known for its large number of integrations, which make it easy to bring third-party data into Prometheus. It’s also known for its popular PromQL query language, and for its alerting capabilities with Alertmanager.

Grafana, when it’s used to visualize the data collected in Prometheus, can be helpful for teams when they are troubleshooting. Grafana Enterprise is an increasingly popular toolset for the largest enterprises looking to use Grafana on-prem. Included with Grafana Enterprise is access to Enterprise Logs, Enterprise Metrics, and Enterprise Traces. There are a number of additional enterprise features like authentication, usage insights, and scalability guarantees.

Grafana Enterprise pricing is not publicly available, and you must request a demo in order to learn more.

AppDynamics | An APM Ideal for On-Prem and Hybrid Environments

AppDynamics, now part of Cisco, has been a leading provider of application performance monitoring since 2008.

AppDynamics has a Kubernetes Monitoring suite that can be run on-prem, hybrid, and across multiple cloud providers. This is particularly helpful for companies that are looking to aggregate data between on-prem and cloud environments in one tool.

While there is quite a bit of overlap between what AppDynamics offers and what you can get from another tooling, there are a couple of places where AppDynamics excels.

AppDynamics offers automatic root cause analysis, which uses historical performance to identify and alert on actual issues that need to be addressed. This can help to reduce alert fatigue, a common complaint with Promtheus’s AlertManager.

AppDynamics also offers users a toolset to tie monitoring to actual business performance. For example, with AppDynamics, users are able to correlate latency to the conversion rate on a given path. For example, if you were running an e-commerce website, you could view the performance of the checkout services, like Add To Cart, to checkout conversion rates at points in time.

Users can access a 15-day free trial, or book a demo with a member of the sales team to get started.

Best Practices to Implement

In this section, we’ll explore some best practices for organizations of all sizes. As part of this, we’ll discuss which metrics are important, tips for troubleshooting more effectively, and which critical pieces of data might be providing clues about the end-user experience.

Keep a Close Eye on Metrics

Monitoring metrics is often the first step for companies standing up a monitoring stack. Keeping track of pod and node CPU and memory, among other things like disk space, is important to ensure that overall cluster health is strong.

One of the biggest benefits of using Kubernetes is that it is efficiently able to scale resources, or the number of pods and nodes, as demand on your applications grows. By tracking resource usage, as well as the allocation of resources, teams are able to ensure a more performant experience. And by tracking historical usage alongside limits, teams are often able to rightsize and reduce expenses.

Use Kubernetes Events When Troubleshooting

The Kubernetes events feed can be a highly valuable source of information, especially when it comes to troubleshooting and remediation.

The Kubernetes events feed captures a historical record of everything happening inside of the cluster including both Normal and Warning events. When issues arise, Warning events can act as a guide to what went wrong and when. And when used alongside logs and metrics, Kubernetes events can be a helpful starting point for gathering additional context and data.

Correlations Make Troubleshooting Faster

We’ve all been there. Endlessly scrolling through logs trying to figure out what happened, and then matching timestamps up in order to get the full picture.

Teams can avoid doing this manual matching and correlating by finding a toolset that correlates automatically. ContainIQ, Grafana Enterprise, and AppDynamics all have the ability to correlate between multiple metrics and pieces of information, for example, logs, at points in time.

Correlations make troubleshooting a more efficient process, and teams should use a toolset that enables this behavior by default.

Track End-User Experiences Too

While infrastructure metrics are important, it’s also wise to spend time thinking about end-user performance metrics. By implementing an APM, or APM-like features, engineering teams can track the performance of end-user metrics. For example, teams may find it valuable to track latency for particular microservices, or for particular paths, where a customer exists.

Teams might also find it useful to track status codes by request and to alert when problematic status codes are identified.

While engineering teams likely aren’t making decisions for the end-user, it is important for the team to work alongside other parts of the organization, like the product and marketing teams, to implement monitoring that ensures a performant end-user experience.

Be Aware of Alert Fatigue

Whether using a SaaS offering or one of the on-prem offerings mentioned above, alert fatigue is going to be an issue to get out in front of.

As the name suggests, alert fatigue is the result of inadequately set alerts and monitoring, which leads to wasted time and energy for the engineering team. Alert fatigue is typically caused by setting too many alerts, or by using the wrong thresholds. Instead, engineering teams should work to set alerts only on the actions that have a direct impact or could lead to a compounded issue.

Both ContainIQ and Prometheus come, by default, with preset alerting capabilities. Before automatically turning all of these alerts on, engineers should double-check that each alert has an actual business use case.

Finally, teams should work to deliver the alerts to the right place and person. By using a paging system like PagerDuty or K6s, teams can forward the alerts to specific people or teams depending on the date and time. Knowing who is responsible for which issues can also help to avoid overfatiguing an entire team.

Final Thoughts

In this article, we explored the concept of running a Kubernetes monitoring toolset on-prem or in-cloud. Included, we highlighted some of the challenges of going on-prem, including the limited availability of some tools, and the added costs associated with self-managing your monitoring stack.

We also explored some popular tools for going on-prem with your Kubernetes monitoring toolset. ContainIQ, Prometheus, and AppDynamics are good examples of monitoring tools that can be run either on-prem or in-cloud.

And finally, we explored some best practices to implement to make troubleshooting a more efficient process, improve end-user experience, and avoid the alert fatigue that is a byproduct of observability tools.

Start your free 14-day ContainIQ trial
Start Free TrialBook a Demo
No card required
Nate Matherson
Co-founder

Nate Matherson is the Co-founder & CEO of ContainIQ. An experienced entrepreneur and technologist, he has founded multiple venture-backed companies and is a two-time Y Combinator Alum. Nate is also an active angel investor.

READ MORE