Resources

Grafana vs. Prometheus for Kubernetes Monitoring

October 18, 2021

Two common options for monitoring your Kubernetes cluster are Prometheus and Grafana. Let’s talk about when you’d use each tool and when you’d use both together.

Ashish Choudhary
Software Engineer

At the same time, Kubernetes provides liveness and readiness probes or metrics via cAdvisor through which you can see the current state of your system. However, they cannot be relied upon entirely for production scenarios.

Metrics-based tools like Grafana or Prometheus can give you continuous time-series data on hundreds of metrics exposed by your system. So let’s talk about Prometheus and Grafana in the context of Kubernetes monitoring—when you’d use each one and when you’d use them together.

Getting Started with Grafana

Grafana is an open-source data visualization tool that helps users understand complex data through some great visualization. It comes with capabilities to query, visualize, and alert on metrics from different data sources, such as Prometheus (which I’ll discuss more in-depth later in this article), MySQL, MSSQL, PostgreSQL, InfluxDB, Graphite, and Elasticsearch. Grafana can also pull data from cloud-hosted services like AWS CloudWatch, Google Cloud Monitoring, and Azure Monitor.

Running Grafana on your local machine is as easy as executing a <terminal inline>docker run<terminal inline> command or running the <terminal inline>helm install grafana bitnami/grafana<terminal inline> Helm command. For example, you can use the following <terminal inline>docker run<terminal inline> command to start Grafana on your local:


docker run -d -p 3000:3000 grafana/grafana

Verify that the Grafana container image is running using the <terminal inline>docker ps<terminal inline> command.

$ docker ps

Container ID Image Command Created Status Ports Name
fcbd1815b4a4 grafana/grafana "/run.sh" 3 minutes ago Up 3 minutes 0.0.0.0:3000->3000/tcp dreamy_brattain

Now browse the URL <terminal inline>http://localhost:3000<terminal inline> and you will be redirected to the Grafana login page.

Grafana login
Grafana login

Enter default credentials admin/admin, and you will see the following screen with options given to create dashboard and data sources.

Grafana home
Grafana home

To verify further, you can even add a data source named as <terminal inline>TestData DB<terminal inline> which is a fake data source that comes with Grafana. On the sidebar, click Configuration, then click Add data source to add <terminal inline>TestData DB<terminal inline> as a data source.

Grafana configuration
Grafana configuration

Click Save & test. Under the Dashboards tab, you can import the Simple Streaming Example.

Grafana configuration
Grafana configuration

Navigate to your dashboard home and choose the Simple Streaming Example dashboard. You will see the following dashboard.

Grafana dashboard
Grafana dashboard

Grafana Features

Some of Grafana’s highlighted features include:

  • Data analytics through dashboards
  • Application monitoring
  • Integration with variety of data sources, ie, Prometheus, MySQL
  • Visualize system resources such as CPU and memory
  • Create, edit, and share dashboards
  • Alerting capabilities
  • Query your metrics

Grafana Support and Pricing

Grafana currently comes with three different support models:

  1. Grafana Open Source: The open-source and self-managed offering of Grafana. You have to manage your instances yourself, and if you run into any issues, you have to raise the problem with the Grafana community. However, if you want to contribute to the community, you can raise issues and fix them.
  2. Grafana Cloud: This is the fully managed service offered by Grafana. With Grafana Cloud, you don’t have to worry about managing infrastructure for the Grafana instances. Grafana Cloud comes in three flavors: Free ($0), Pro ($49 +usage), and Advanced (custom pricing).
  3. Grafana Enterprise: This offering caters to company-specific needs around data and privacy. It comes with features such as advanced security and compliance, 24/7 hands-on support teams, and access to enterprise plugins like New Relic, AppDynamics, Oracle, Dynatrace, ServiceNow, and Datadog.

Getting Started with Prometheus

Prometheus is an open-source metrics-based monitoring and alerting solution, and the de facto standard when it comes to monitoring Kubernetes. You can install Prometheus to your Kubernetes cluster with one command: <terminal inline>helm install Prometheus bitnami/kube-prometheus<terminal inline> using Helm.

In Prometheus terminology, the process to gather data from the cluster or application is called scraping. It scrapes the metrics by using an HTTP connection on a predefined port to get metrics from your applications and stores the data in time-series format in its database.

You can run query data using its PromQL query language. The Prometheus server uses a pull-based approach to collect metrics data from applications.

Prometheus has the following core components:

  • Server: Collects and stores metrics.
  • Alert manager: Receives alerts from various data sources and processes them. Alerts are typically sent using Slack or email.
  • Client libraries: Supports client libraries for different languages that can be used to instrument your applications.

Prometheus and Grafana Feature Comparison

PrometheusGrafanaSome options to view metrics and basic trends.Maximum data visualization options using charts and graphs.Provides storage for time-series data.Doesn’t have its own storage.Pull-based, scraping HTTP endpoints.Provides plugins to get data from different data sources.

Grafana and Prometheus: Match Made in Heaven

However, capable as Grafana and Prometheus are on their own, a common use case for each is to work in tandem. Prometheus collects and stores data, and Grafana offers developers an elegant way to visualize that data, making it easier to analyze and act on.

In this section, you’ll be installing Prometheus and Grafana on an existing Kubernetes cluster for monitoring the cluster. You’ll use a minikube local Kubernetes cluster for this setup.

First, install Prometheus on your local Kubernetes cluster using Helm. You’ll use the <terminal inline>helm repo add bitnami https://charts.bitnami.com/bitnami<terminal inline> command to add a Bitnami Helm repository.

For this demo, you’ll install Prometheus with its default configuration using the <terminal inline>helm install prometheus bitnami/kube-prometheus<terminal inline> command. It takes a while for all the pods to come into RUNNING status. After the install, you should see the following pods in the output of <terminal inline>kubectl get all<terminal inline> command:

$ docker ps

Name Ready Status
pod/alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running
pod/prometheus-kube-prometheus-operator-86bbf564b8-rzx2h 1/1 Running
pod/prometheus-kube-state-metrics-78667fbf77-wj4w4 1/1 Running
pod/prometheus-node-exporter-8zpd4 1/1 Running
pod/prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running

In this default setup, you cannot access Prometheus outside the cluster as the service type is <terminal inline>ClusterIP<terminal inline>. To access Prometheus from outside the cluster, execute the following commands:


kubectl port-forward --namespace default svc/prometheus-kube-prometheus-prometheus 9090:9090

After that you can access it via <terminal inline>http://127.0.0.1:9090/<terminal inline>.

Next install Grafana using the <terminal inline>helm install grafana bitnami/grafana<terminal inline> command. You should see the following output on your terminal:


NAME: grafana
LAST DEPLOYED: Wed Jun 16 22:58:46 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

1. Get the application URL by running these commands:
 echo "Browse to http://127.0.0.1:8080"
 kubectl port-forward svc/grafana 8080:3000 &

2. Get the admin credentials:
 echo "User: admin"
 echo "Password: $(kubectl get secret grafana-admin --namespace default -o jsonpath="{.data.GF_SECURITY_ADMIN_PASSWORD}" | base64 --decode)"

After the Grafana pod is created, you can run the <terminal inline>kubectl port-forward<terminal inline> command as mentioned in the output and navigate to <terminal inline>http://127.0.0.1:8080<terminal inline>.

In this setup, default admin/admin credentials don’t work, as the Helm chart generates a random password stored in Kubernetes secret. You can retrieve it using the <terminal inline>kubectl get secret<terminal inline> command as mentioned in the output. Log in using this password. At this point, both Grafana and Prometheus are up and running inside your Kubernetes cluster.

Prometheus has already started collecting metrics from your Kubernetes cluster, but you haven’t added it as a data source in Grafana. Let’s do that.

The previous section about Grafana already covered how to add a data source in Grafana, so the process is not much different for Prometheus. However, while adding Prometheus as a data source, you have to provide its server URL. Both Prometheus and Grafana are running in the same cluster, so use the <terminal inline>prometheus-kube-prometheus-prometheus.default.svc.cluster.local<terminal inline> DNS name mentioned in the following screenshot as server URL and port <terminal inline>9090<terminal inline>.

Grafana prometheus
Grafana prometheus

For visualization, you can also import ready-made Grafana dashboards from its vast repository of dashboards. Use a dashboard named <terminal inline>Cluster Monitoring<terminal inline> for Kubernetes and have ID <terminal inline>10000<terminal inline> for this demo.

Grafana dashboard import
Grafana dashboard import

After the successful import, you should see the following dashboard:

Grafana dashboard import
Grafana dashboard import

Conclusion

While Prometheus is good at scraping metrics and storing time-series data, Grafana scores in visualization using charts and graphs. Together, they’re a strong combo for your team’s ability to access and work from data that’s relevant and thorough. At this point, you should feel armed with some practical examples for combining these tools to monitor your Kubernetes clusters.

Observability and application monitoring are essential components of the DevOps philosophy. We tend to interchange these terms when used together, but in reality, they have different meanings. While observability helps you understand why your application/system is not working, monitoring tells how it is working. When we talk about monitoring distributed systems like Kubernetes, it’s hard to know if something is up or down through traditional monitoring.

Article by

Ashish Choudhary

Software Engineer

Java, BigData and Cloud enthusiast, Ashish likes learning and sharing about emerging new technologies

Read More