Prometheus, a popular open-source project, is used by thousands of organizations as a time-series database and monitoring system. Prometheus has a number of benefits including a large number of integrations, efficient storage, powerful alerting with Alertmanager, as well as the ability to run queries using PromQL.
While Prometheus does have some visualization capabilities, it is most often paired with another tool for visualization, like Grafana. Grafana, originally launched as an open-source project, is a popular visualization toolset and a popular integration with Prometheus.
Both of these tools have their own set of benefits, use cases, and capabilities. And for companies using Kubernetes, the leading container orchestration system, it makes a lot of sense to use these tools together.
In this article, we highlight both Prometheus and Grafana as tools to monitor workloads running on Kubernetes. We start by highlighting the setup and implementation process for each tool and then finish by discussing best practices for using these tools together.
Grafana and Prometheus Feature Comparison
Getting Started with Grafana
Grafana is an open-source data visualization tool that helps users understand complex data through some great visualization. It comes with capabilities to query, visualize, and alert on metrics from different data sources, such as Prometheus (which I’ll discuss more in-depth later in this article), MySQL, MSSQL, PostgreSQL, InfluxDB, Graphite, and Elasticsearch. Grafana can also pull data from cloud-hosted services like AWS CloudWatch, Google Cloud Monitoring, and Azure Monitor.
Running Grafana on your local machine is as easy as executing a <terminal inline>docker run<terminal inline> command or running the <terminal inline>helm install grafana bitnami/grafana<terminal inline> Helm command. For example, you can use the following <terminal inline>docker run<terminal inline> command to start Grafana on your local:
Verify that the Grafana container image is running using the <terminal inline>docker ps<terminal inline> command.
Now browse the URL <terminal inline>http://localhost:3000<terminal inline> and you will be redirected to the Grafana login page.
Enter default credentials admin/admin, and you will see the following screen with options given to create dashboard and data sources.
To verify further, you can even add a data source named as <terminal inline>TestData DB<terminal inline> which is a fake data source that comes with Grafana. On the sidebar, click Configuration, then click Add data source to add <terminal inline>TestData DB<terminal inline> as a data source.
Click Save & test. Under the Dashboards tab, you can import the Simple Streaming Example.
Navigate to your dashboard home and choose the Simple Streaming Example dashboard. You will see the following dashboard.
Some of Grafana’s highlighted features include:
- Data analytics through dashboards
- Application monitoring
- Integration with variety of data sources, ie, Prometheus, MySQL
- Visualize system resources such as CPU and memory
- Create, edit, and share dashboards
- Alerting capabilities
- Query your metrics
Grafana Support and Pricing
Grafana currently comes with three different support models:
- Grafana Open Source: The open-source and self-managed offering of Grafana. You have to manage your instances yourself, and if you run into any issues, you have to raise the problem with the Grafana community. However, if you want to contribute to the community, you can raise issues and fix them.
- Grafana Cloud: This is the fully managed service offered by Grafana. With Grafana Cloud, you don’t have to worry about managing infrastructure for the Grafana instances. Grafana Cloud comes in three flavors: Free ($0), Pro ($49 +usage), and Advanced (custom pricing).
- Grafana Enterprise: This offering caters to company-specific needs around data and privacy. It comes with features such as advanced security and compliance, 24/7 hands-on support teams, and access to enterprise plugins like New Relic, AppDynamics, Oracle, Dynatrace, ServiceNow, and Datadog.
Getting Started with Prometheus
Prometheus is an open-source metrics-based monitoring and alerting solution, and the de facto standard when it comes to monitoring Kubernetes. You can install Prometheus to your Kubernetes cluster with one command: <terminal inline>helm install Prometheus bitnami/kube-prometheus<terminal inline> using Helm. Prometheus is often self-hosted, but it has become increasingly popular to used a managed service.
In Prometheus terminology, the process to gather data from the cluster or application is called scraping. It scrapes the metrics by using an HTTP connection on a predefined port to get metrics from your applications and stores the data in time-series format in its database.
You can run query data using its PromQL query language. The Prometheus server uses a pull-based approach to collect metrics data from applications.
Prometheus has the following core components:
- Server: Collects and stores metrics.
- Alertmanager: Receives alerts from various data sources and processes them. Alerts are typically sent using Slack or email.
- Client libraries: Supports client libraries for different languages that can be used to instrument your applications.
Prometheus and Grafana Feature Comparison
Grafana and Prometheus: Match Made in Heaven
However, capable as Grafana and Prometheus are on their own, a common use case for each is to work in tandem. Prometheus collects and stores data, and Grafana offers developers an elegant way to visualize that data, making it easier to analyze and act on.
In this section, you’ll be installing Prometheus and Grafana on an existing Kubernetes cluster for monitoring the cluster. You’ll use a minikube local Kubernetes cluster for this setup.
First, install Prometheus on your local Kubernetes cluster using Helm. You’ll use the <terminal inline>helm repo add bitnami https://charts.bitnami.com/bitnami<terminal inline> command to add a Bitnami Helm repository.
For this demo, you’ll install Prometheus with its default configuration using the <terminal inline>helm install prometheus bitnami/kube-prometheus<terminal inline> command. It takes a while for all the pods to come into RUNNING status. After the install, you should see the following pods in the output of <terminal inline>kubectl get all<terminal inline> command:
In this default setup, you cannot access Prometheus outside the cluster as the service type is <terminal inline>ClusterIP<terminal inline>. To access Prometheus from outside the cluster, execute the following commands:
After that you can access it via <terminal inline>http://127.0.0.1:9090/<terminal inline>.
Next install Grafana using the <terminal inline>helm install grafana bitnami/grafana<terminal inline> command. You should see the following output on your terminal:
After the Grafana pod is created, you can run the <terminal inline>kubectl port-forward<terminal inline> command as mentioned in the output and navigate to <terminal inline>http://127.0.0.1:8080<terminal inline>.
In this setup, default admin/admin credentials don’t work, as the Helm chart generates a random password stored in Kubernetes secret. You can retrieve it using the <terminal inline>kubectl get secret<terminal inline> command as mentioned in the output. Log in using this password. At this point, both Grafana and Prometheus are up and running inside your Kubernetes cluster.
Prometheus has already started collecting metrics from your Kubernetes cluster, but you haven’t added it as a data source in Grafana. Let’s do that.
The previous section about Grafana already covered how to add a data source in Grafana, so the process is not much different for Prometheus. However, while adding Prometheus as a data source, you have to provide its server URL. Both Prometheus and Grafana are running in the same cluster, so use the <terminal inline>prometheus-kube-prometheus-prometheus.default.svc.cluster.local<terminal inline> DNS name mentioned in the following screenshot as server URL and port <terminal inline>9090<terminal inline>.
For visualization, you can also import ready-made Grafana dashboards from its vast repository of dashboards. Use a dashboard named <terminal inline>Cluster Monitoring<terminal inline> for Kubernetes and have ID <terminal inline>10000<terminal inline> for this demo.
After the successful import, you should see the following dashboard:
While Prometheus is good at scraping metrics and storing time-series data, Grafana scores in visualization using charts and graphs. Together, they’re a strong combo for your team’s ability to access and work from data that’s relevant and thorough. At this point, you should feel armed with some practical examples for combining these tools to monitor your Kubernetes clusters.
Observability and application monitoring are essential components of the DevOps philosophy. We tend to interchange these terms when used together, but in reality, they have different meanings. While observability helps you understand why your application/system is not working, monitoring tells how it is working. When we talk about monitoring distributed systems like Kubernetes, it’s hard to know if something is up or down through traditional monitoring.