Copying Alert Rules From One Percona Monitoring and Management Server to Another

Since Percona Monitoring and Management (PMM) 2.40.0, Grafana uses PostgreSQL instead of SQLite database to store users, dashboards, and other persistent data. This aids in getting some useful information related to [configured alerts] directly via querying from the Postgres backend.

Sometimes we need to migrate only alert rules from one PMM server to another for some requirement or testing/troubleshooting purpose. So in order to cater to such requirements, we can now directly copy such alerts and restore them to the target PMM instance via using cURL request and the PostgreSQL database.

This will be a short blog post just focusing on the steps to perform the export/import of Grafana alert rules.

Let’s see the steps in detail.

1. We need to connect to the source PMM server and get the alert rule  “uid” detail from the Postgres backend to pass to the cURL option. The below single command will do all this.

So, here we have exported all the alert rules in the *.dump.json file via the cURL GET request.

For each alert rule, we see an individual file with a “uid” name prefix. If we are interested in only a specific alert, then we can modify the Postgres query accordingly OR put only the needed [uid] in the curl option.

2. Then, from the host machine, we can copy the target file generated inside the PMM container.

3.  We can also pass the above fetched [uid] details in the mentioned PMM Url in order to view the alert rules.

Output:

4. Finally, to import these rules to the target instance, we can simply run the below cUR POST method. This will upload/provision alerts for all the existing files; however, we can restrict it by defining the exact JSON file if we want to import only a specific file.

OR

For more secure way interaction, you can generate the PMM API Key [eyJrIjoiZE9EOTF5UFJZZnJmSThXMnI3TmJHeTYxU0NibHg2c2kiLCJuIjoiYWRtaW5fYXBpIiwiaWQiOjF9] using this link. 

Output:

There is one caveat here: provisioned alerts are not editable via PMM UI. That means we can view and export them, however, we won’t be able to perform any changes via UI. We have to rely on doing manual changes in the Alert files and then updating them via a PUT request.

To address this issue and to allow editing for provisioned alerts as well as a Header option [“X-Disable-Provenance”] implemented in [“Grafana v9.3”]:- https://github.com/grafana/grafana/pull/58410 and https://github.com/grafana/grafana/issues/57911which can be used with below request formats.

Reference:- https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/http-api-provisioning/#edit-resources-in-the-grafana-ui

Since Grafana 9.4, it’s possible to export the alert rule for provisioning directly via the UI itself. 

Reference:- https://grafana.com/docs/grafana/latest/whatsnew/whats-new-in-v9-4/#export-alert-rules-to-use-in-the-provisioning-api-or-files

Unfortunately, PMM’s latest version [2.41.2] is still based on old Grafana v9.2.20, so such a solution is still not feasible from the PMM UI. However, the Grafana version change is planned for [PMM v3], so hopefully, we will see improvement in this area. 

Note:  The scenario we have shown above will not copy the [Contact points] and any other policies [notification policies] etc. Only [Alert Rule] will be affected. So, in order to change or migrate other parts of the Alerting process, you can use separate URI/endpoint

E.g.,

For Contact points, the below APIs/Method can be used.

More references

Conclusion

So, here we discussed one of the approaches to simply migrate the alert rule from one PMM server to another. However, we can always perform the full migration by using backups from the old instance and restoring them to the new instance. Specifically for the latest PMM Version [2.40.0 or above], we can also use some workaround like copying the alert* specific tables from Postgres and restoring them to the target instance, but this is usually not the best-recommended approach and could affect/break the existing alerting in case using any or if there’s any change in the table formats. So, better to test before using it!!

Percona Monitoring and Management is a best-of-breed open source database monitoring solution tool for use with MySQL, PostgreSQL, MongoDB, and the servers on which they run. Monitor, manage, and improve the performance of your databases no matter where they are located or deployed.

Download Percona Monitoring and Management Today

Subscribe

Connect with

Login

Label

0 Comments