Install Prometheus on metalstack.cloud

This developer guide was kindly written by iSecNG. The security specialists at iSecNG use metalstack.cloud to securely host Wazuh, an open source XDR and SIEM. Learn more about their services and solutions on their website.

In this guide, we’ll walk you through installing Prometheus on metalstack.cloud to monitor your Kubernetes cluster and applications.

Introduction

Prometheus is an open-source monitoring tool widely used in cloud-native environments for its scalability and reliability. It utilizes a pull-based model to collect metrics from systems, stores them in a time-series database, and offers powerful querying and alerting capabilities via PromQL.

In Kubernetes, Prometheus is essential for monitoring cluster health, application performance, and resource usage. This guide explains how to install Prometheus on metalstack.cloud using the kube-prometheus-stack Helm chart, which integrates Prometheus, Grafana, and alerting tools for Kubernetes.

What is kube-prometheus-stack?

The kube-prometheus-stack is a Helm chart that provides a comprehensive monitoring solution for Kubernetes clusters. It includes Prometheus for metrics collection, Grafana for metric visualization, and Alertmanager for managing alerts. The stack comes pre-configured with dashboards, recording rules, and alerting rules, making it easy to monitor your clusters and applications from day one. This integrated setup simplifies observability in Kubernetes environments, allowing you to deploy a full monitoring stack quickly.

Prerequisites

Before starting the installation, ensure you meet the following prerequisites:

Kubernetes 1.19+: For this guide, we are using metalstack.cloud with Kubernetes version 1.28.13.
Helm 3.x: You’ll need Helm to manage the chart installation.

Installation

The following installation steps are adapted from the official kube-prometheus-stack documentation.

Add the Helm Repository

To begin, add the prometheus-community repository to Helm, which contains the kube-prometheus-stack chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

This ensures that your local Helm installation has the latest repository updates. For further information on how Helm repositories work, refer to the Helm repository documentation.

Install the kube-prometheus-stack Helm Chart

Once the repository is added, you can install the kube-prometheus-stack by running the following command (replace [RELEASE_NAME] with your preferred name):

helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack

This command deploys Prometheus, Grafana, and additional monitoring components into your Kubernetes cluster. For more advanced installation options, check the Helm install documentation.

Chart Dependencies

By default, the kube-prometheus-stack chart includes several dependent charts:

prometheus-community/kube-state-metrics
prometheus-community/prometheus-node-exporter
grafana/grafana

These dependencies help ensure comprehensive monitoring. However, if you need to disable any of these dependencies during installation, you can refer to the configuration settings for multiple releases. Detailed instructions on managing dependencies are available in the Helm dependency documentation.

Uninstalling the Helm Chart

To completely remove the kube-prometheus-stack from your cluster, run the following command:

helm uninstall [RELEASE_NAME]

This will remove all Kubernetes components associated with the release. However, keep in mind that the CustomResourceDefinitions (CRDs) created by this chart are not deleted automatically. To manually clean up the CRDs, use these commands:

kubectl delete crd alertmanagerconfigs.monitoring.coreos.com
kubectl delete crd alertmanagers.monitoring.coreos.com
kubectl delete crd podmonitors.monitoring.coreos.com
kubectl delete crd probes.monitoring.coreos.com
kubectl delete crd prometheusagents.monitoring.coreos.com
kubectl delete crd prometheuses.monitoring.coreos.com
kubectl delete crd prometheusrules.monitoring.coreos.com
kubectl delete crd scrapeconfigs.monitoring.coreos.com
kubectl delete crd servicemonitors.monitoring.coreos.com
kubectl delete crd thanosrulers.monitoring.coreos.com

For more information on uninstalling, you can refer to the Helm uninstall documentation.

Upgrading the Chart

To upgrade an existing installation, use the following command:

helm upgrade [RELEASE_NAME] prometheus-community/kube-prometheus-stack

Be aware that CRDs are not automatically updated by Helm v3, so you will need to handle CRD updates manually. For guidance on upgrading, refer to the Helm upgrade documentation and their notes on managing CRDs.

Considerations for Using Prometheus on metalstack.cloud

The kube-prometheus-stack Helm chart is highly flexible, offering numerous configuration options via its values.yaml file to suit various scenarios. Below, we’ll highlight some recommended configurations for deploying Prometheus effectively on metalstack.cloud.

Enabling the Prometheus Operator

One of the key components to activate is the Prometheus Operator, which manages the lifecycle of Prometheus, automating tasks such as upgrades, scaling, and ensuring high availability. This simplifies managing Prometheus as a stateful application.

prometheusOperator:
    enabled: true

Retention and Replication

To ensure Prometheus retains enough metrics data for meaningful analysis, we recommend setting a retention period of 14 days. Additionally, in production environments, it’s important to distribute Prometheus instances across different physical nodes for redundancy. This can be achieved by configuring a podAntiAffinity rule, ensuring that multiple replicas are scheduled on different nodes, enhancing fault tolerance.

prometheus:
  prometheusSpec:
    retention: 14d

    replicas: 3

    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: prometheus

            topologyKey: kubernetes.io/hostname

Storage Considerations

For optimal storage performance, we recommend using the csi-lvm storage class provided by metalstack.cloud. This class utilizes the local SSDs available on each physical node, offering two key advantages:

Cost efficiency: Since local SSDs are included with each node, using them doesn’t incur additional storage costs.
High performance: Local SSDs are based on NVMe technology, providing ample I/O throughput to handle Prometheus’ demands.

storageSpec:
  volumeClaimTemplate:
    spec:
      storageClassName: csi-lvm

These configurations leverage metalstack.cloud’s infrastructure, ensuring Prometheus runs efficiently while maintaining cost-effectiveness and high availability.

Conclusion

In summary, the kube-prometheus-stack is an excellent choice for monitoring Kubernetes environments due to its flexibility and seamless integration with Prometheus, Grafana, and Alertmanager. Its versatility allows us to efficiently monitor our infrastructure on metalstack.cloud.

At iSecNG, we rely on metalstack.cloud to host our Managed SIEM, a key component of our Managed Detection and Response (MDR) Service. Metalstack.cloud’s high-performance infrastructure, including local NVMe SSDs and scalable storage options, ensures optimal performance and cost-efficiency. With our commitment to secure and reliable operations, we are able to deliver industry-leading services at scale, keeping our customers’ environments safe and resilient.