Platform Engineers

Introduction to PromQL Queries for Custom Service Level Indicators in Grafana

Platform Engineers's photo
Platform Engineers
·

4 min read

Service Level Indicators (SLIs) are crucial metrics used to measure the performance and health of services over time. In Grafana, SLIs can be created using PromQL queries, which provide a flexible way to define custom metrics. This article will delve into the process of using PromQL queries to create custom SLIs in Grafana, focusing on the technical aspects of query construction and integration.

Understanding PromQL

PromQL is a query language used by Prometheus, a popular monitoring system. It allows users to query time-series data stored in Prometheus. PromQL supports various functions and operators that can be used to manipulate and aggregate data. Common PromQL functions include rate(), increase(), and histogram_quantile(), which are essential for creating SLIs.

Creating Custom SLIs with PromQL

To create a custom SLI using PromQL in Grafana, you need to define a query that returns a ratio between 0 and 1. This ratio typically represents the proportion of successful events to total events.

Example: Successful Requests Ratio

A common SLI is the ratio of successful requests to total requests. This can be achieved using the following PromQL query:

sum(rate(http_request_duration_seconds_bucket{le="0.3"}[$__rate_interval])) by (job) /
sum(rate(http_request_duration_seconds_count[$__rate_interval])) by (job)

This query calculates the rate of requests that completed within 300 milliseconds (http_request_duration_seconds_bucket{le="0.3"}) divided by the total rate of requests (http_request_duration_seconds_count), grouped by job.

Using the Advanced Query Builder in Grafana

Grafana provides an advanced query builder that allows users to write custom PromQL queries directly. This feature is useful for users familiar with PromQL or those who need more complex queries than what the ratio query builder offers.

  1. Accessing the Advanced Query Builder:

    • Navigate to the SLO setup in Grafana.

    • Choose the "Advanced" query option.

    • Enter your custom PromQL query in the text box provided.

  2. Example Query:

    • Suppose you want to measure the availability of a service based on the number of successful responses compared to total requests. You can use a query similar to the one above, adjusting it according to your specific metrics.

Integrating Custom SLIs into Grafana SLOs

Once you have defined your custom SLI using PromQL, you can integrate it into a Service Level Objective (SLO) in Grafana.

  1. Defining an SLO:

    • An SLO consists of an SLI, a target, and an error budget.

    • The SLI is the metric you want to measure, which in this case is your custom PromQL query.

    • The target is the desired level of service (e.g., 99.9% availability).

    • The error budget is the amount of deviation from the target that is acceptable before alerts are triggered.

  2. Setting Up Alerts:

    • Grafana allows you to set up alert rules based on your SLOs.

    • You can configure notifications for when your error budget is being consumed at a rate that might lead to missing your target.

Multi-Dimensional SLIs

Grafana supports multi-dimensional SLIs, which allow you to group your metrics by different labels. This feature is useful for tracking performance across different regions or clusters.

  1. Grouping Labels:

    • When creating an SLI, you can specify grouping labels.

    • For example, if you want to track the performance of a service by region, you can group your SLI by a "region" label.

  2. Example Query with Grouping:

    • Suppose you want to measure the successful request ratio for each region. You can modify the previous query to include a grouping label:
    sum(rate(http_request_duration_seconds_bucket{le="0.3", region="us"}[$__rate_interval])) by (job) /
    sum(rate(http_request_duration_seconds_count{region="us"}[$__rate_interval])) by (job)

This query would need to be repeated for each region or dynamically handled using label matching.

Conclusion

Creating custom Service Level Indicators using PromQL queries in Grafana provides a powerful way to monitor and manage service performance. By defining precise metrics that reflect the health of your services, you can set realistic targets and alert on deviations, ensuring high-quality service delivery. The flexibility of PromQL allows for complex queries that can be tailored to specific use cases, making it a valuable tool in service monitoring and management.

For more technical blogs and in-depth information related to platform engineering, please check out the resources available at “www.platformengineers.io/blogs".