Platform Engineers

How AlertManager and Prometheus statefulsets are created in prometheus-operator helm chart?

Platform Engineers's photo
Platform Engineers
·

3 min read

The Prometheus Operator, a helm chart in Kubernetes, simplifies deploying and managing Prometheus and Alertmanager. This blog delves into the technical details of how it creates statefulsets for these crucial monitoring components.

Understanding StatefulSets

Statefulsets ensure that pods within a deployment maintain their identity and persistent storage across restarts. This is essential for Prometheus and Alertmanager, as they need to retain data like metrics and alerts.

The Helm Chart at Work

The prometheus-operator helm chart utilizes Custom Resource Definitions (CRDs) to configure desired deployments. These CRDs define resources like Prometheus and Alertmanager. Here's a breakdown of the process:

  1. CRD Creation:

    • The helm chart installs the necessary CRDs for Prometheus and Alertmanager into the Kubernetes cluster. These CRDs define the schema for these resources and allow users to specify configuration details.

  2. CRD Manifestation:

    • The chart includes YAML manifests containing the desired configuration for Prometheus and Alertmanager. These manifests leverage the CRD schema to define parameters like:

      • replicas: Number of desired pods in the deployment.

      • resources: Resource requests and limits for CPU and memory.

      • storage: Persistent storage configuration for data persistence.

      • monitoring: Configuration for scraping targets and rule evaluation (Prometheus).

      • route: Configuration for routing alerts to receivers (Alertmanager).

YAML

# Example Prometheus CRD manifest
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: my-prometheus
spec:
  replicas: 2
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
  storage:
    selector:
      matchLabels:
        prometheus: my-prometheus
    volumeClaimTemplate:
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: "10Gi"
  monitoring:
    # ... monitoring configuration options ...

  1. StatefulSet Creation:

    • The Prometheus Operator acts as a controller, watching for changes to these CRDs.

    • When a Prometheus or Alertmanager CRD is created or modified, the operator translates the configuration into Kubernetes deployment and service manifests.

    • These manifests define a statefulset with the specified number of replicas.

    • The statefulset ensures pods maintain a unique and persistent identity across restarts.

    • Persistent volume claims (PVCs) are also created based on the storage configuration in the CRD. These PVCs request storage from the underlying storage provisioner in the Kubernetes cluster.

  2. Persistent Storage Mounting:

    • The deployment pods for Prometheus and Alertmanager are configured to mount the persistent volumes claimed by the PVCs. This allows them to store and retrieve data across restarts.

Additional Considerations

  • Prometheus Operator Configuration: The helm chart itself can be further configured to define default values for resources, storage, and other parameters used in the generated manifests.

  • Horizontal Pod Autoscaler (HPA): The chart can be extended to deploy HPAs alongside the statefulsets. HPAs dynamically scale the number of pods based on resource utilization.

Conclusion

By leveraging CRDs and the statefulset construct, the Prometheus Operator automates the deployment and management of Prometheus and Alertmanager with persistent storage. This simplifies monitoring infrastructure management in Kubernetes environments. Understanding this mechanism empowers users to effectively configure and manage their monitoring setups.

#prometheus