Enabling PromQL queries with Erlenmeyer

erlenmeyer-promql banner
You are using Clever Cloud to deploy applications runtimes or add-ons. But do you know that all Clever Cloud services are producing metrics? And that you already have access to all of them? Indeed, we register an agent on our side, whose job is to collect all local data.Once it’s done, they are stored in a Time Series database called Warp10.
Erlenmeyer logo

To simplify the adoption of Warp10, the OVHcloud Metrics team built the Erlenmeyer tool. In the first place, this tool enables the use of several open source protocols as PromQL, OpenTSDB, InfluxQL or Graphite to query Warp10 stored metrics.
Secondly, Erlenmeyer works as a proxy in front of a Warp10 cluster. Then we deployed a couple of Erlenmeyer instances in front of our Warp10 infrastructure, naturally providing new ways to query your application metrics.

In this post, we will review what Erlenmeyer brings and how to handle Clever Cloud services metrics with PromQL.

Erlenmeyer protocols

Erlenmeyer almost entirely enables PromQL queries, OpenTSDB, InfluxQL and some of the Graphite functions. Moreover, you can retrieve the documentation about each protocol implementation and usage on Erlenmeyer’s GitHub:

On Clever Cloud, we deployed an Erlenmeyer in front of our Warp10 backend. This means all these open source protocols can be used to query our application Metrics. The host is https://PROTO-c1-warp10-clevercloud-customers.services.clever-cloud.com/PROTO with PROTO being:

  • for PromQL: prometheus,
  • for OpenTSDB: opentsdb,
  • for InfluxQL: influxql,
  • for Graphite: graphite.
To authenticate your request, you need to use the Warp10 READ TOKEN as the password of the basic authentication. You can find it in the metrics panel of any Clever Cloud application:
Metrics token in the console

Retrieve the memory data with PromQL

As a first step, let’s store the Warp10 READ TOKEN and an APP_ID as an environment variable:
export TOKEN=MY_AWESOME_TOKEN # Replace the value by your Clever Cloud Warp10 token
export APP_ID=app_test # Replace the value by one of your own Clever Cloud application

Get raw data

To retrieve the mem.used_percent in PromQL, we need to send an HTTP GET request to Erlenmeyer, with the TOKEN as basic AUTH. In that case, the PromQL GET request requires four parameters: query, start, end and step. Similarly, to learn more about the PromQL language, you can read the Prometheus documentation. In our case we will create a simple query to get the raw data:
api/v1/query_range?
query=mem.used_percent{app_id="$APP_ID"}&
start=1623289800&
end=1623311400&
step=120s
The query parameter is used for the PromQL queries: start and end are milliseconds timestamps corresponding to the time limit to get data. And the step one is the sampling parameter.
As data are collected around every minute, let’s set at least twice the duration: 120s.
Now, to get the result, execute the following cURL url encoded:
curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'

Group by example

A group by query can be useful in the Clever Cloud case to merge some series (multiple deployments or instances). Thus we will group data based on the app_id labels in the following example.

We will re-use the stored variable READ_TOKEN  and APP_ID. To compute a group by in our case we apply the max PromQL function grouped by the series app_id label:
api/v1/query_range?
query=max(mem.used_percent{app_id="$APP_ID"}) by (app_id)&
start=1623289800&
end=1623311400&
step=120s
Finally to get the result, execute the following cURL url encoded:
curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=max(mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D)%20by%20(app_id)&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'

Grafana PromQL data source

You may use Grafana to create you own dashboards, which is an open source interactive visualisation web application (to deploy a Grafana as a Clever Cloud application, we provide a github example and you can also follow this French blog post). Of course you can configure a PromQL data source to get Clever Cloud applications metrics:

  • create a Prometheus source,
  • then set Prometheus source host to https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus,
  • in addition activate the basic authentication with a user set to metrics and a password containing your read Warp10 token,
  • to conclude, click on Save and Test, your Prometheus data source should now be working!
PromQL Grafana data source configuration

Prometheus UI with remote_read endpoint

Like with Grafana, you can use Prometheus and plot Clever Cloud metrics in the Web UI. And as a matter of fact, simply fill your Prometheus configuration with a remote_read source:

api/v1/query_range?
remote_read:
  - url: "https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/remote_read/"
  basic_auth:
    username: "metrics"
    password: "TOKEN"
Restart Prometheus, then simply execute a Prometheus query in the UI as below:
{ __name__= "mem.used_percent", app_id="$APP_ID" }
Our backend is able to correctly resolve .  which are invalid prometheus characters. Nonetheless the backend response will always translate those . values in series data (name or labels) into valid _.

To Sum Up

As you can see with this post, you can get metrics of Clever Cloud applications and add-ons. Presently you now have one more way to query them, as Erlenmeyer allows the use of PromQL. However the drawback is that you will have to update regularly your token, as they expire in a few days. We are working on a solution to improve the user experience. First, we will soon provide our customers with access to a Grafana with configured data source and already pre-filled dashboards. The token will be handled on our side and you will then be able to build your own graphs with PromQL or WarpScript queries.

Blog

À lire également

SuperBOL: The COBOL revolution in the Cloud

COBOL, a programming language that is over 60 years old, continues to power a large proportion of the IT systems of the world's major companies, particularly in the financial and insurance sectors.
Features

Clever Cloud welcomes the first startups to the UP Programme

Clever Cloud is proud to announce the arrival of the first five startups selected to join its UP Programme, an initiative dedicated to supporting young technology companies in their growth phase.
Company

A minor update resulted in a cascade of errors: how it went wrong, what we’ve learnt

On Friday, August 2nd, 2024 Clever Cloud’s platform became very unstable, leading to downtime of varying duration and scope, for customers using services on the EU-FR-1 (PAR) region, and remote zones depending on the EU-FR-1 control plane (OVHcloud, Scaleway, and Oracle). Privates and on-premise zones weren’t impacted.
Company Engineering