Securing Patroni REST API End Points - Part 1 - Percona Database Performance Blo...

Securing Patroni REST API End Points

In recent years, Patroni emerged as the number one HA framework for PostgreSQL, currently with 5K stars in its git repository. We blogged about how some of the extraordinary capabilities of Patroni to solve problems like Logical Replication Slot Failover.

One outstanding feature of Patroni is its powerful set of REST APIs. These APIs can be used for further automation, monitoring/healthchecks, and for integration with other systems. At the very least, the same APIs are useful for HAProxy integration for automatic connection routing, which is widely used.

Since the Patroni APIs won’t expose any data inside the PostgreSQL database and generally inbound traffic to database hosts is secured in most organizations, a large percentage of the Patroni users are happy with the default REST APIs over HTTP without any additional authentication. However, that may not be the case if the database hosts are potentially exposed to a broader network. A few organizations have blanket rules for raising any HTTP traffic as red flags. Sometimes, additional certificate verifications also become mandatory as per organization policies.

Safe and unsafe APIs

Patroni APIs can be broadly classified as Safe and Unsafe APIs.

Those APIs used for understanding the current state and topology of the Patroni cluster are widely used by monitoring and integration with other tools like load balancers or connection routers. They are basically “GET” requests. Those are considered “Safe APIs”.

There are other sets of APIs that can be used for administering the Patroni. They can be used for changing some of the PostgreSQL parameters, Triggering a switchover, etc. They are PUT, POST, PATCH, and DELETE requests, which can affect the state of the nodes. Those are considered unsafe APIs.

Protecting unsafe APIs

In this article, I am going to use the REST API : /reload as an example of an unsafe API that reloads the configuration.

This is the bare minimum to be achieved if traffic to the database/patroni host machine is not restricted by other means. Patroni provides a simple username and password mechanism by which all unsafe APIs can be protected.

Overall, the restapi: section of patroni configuration (yaml file) can have an entry like:

An external unsafe request will result in an error:

But a Patroni admin can make a request by specifying the username and password:

A request with the wrong credentials will be rejected:

Unlike the curl command, patronictl from the database/patroni host will continue to function as normal (without prompting for passwords) because it has access to the password from the patroni configuration.

Caveats:

All nodes should select either HTTP or HTTPS. Mix and match of protocols can result in patronictl failures like:

There is no restriction like each node having the same username and password. However, if passwords are different, patronictl from other nodes will fail with the message:

So it is safe to use the same credentials across all nodes.

Switch to HTTPS

Patroni automatically switches to HTTPS if a certificate file is specified.  Please see the below sample configuration.

The meaning of each of these parameters is part of official Patroni documentation.

• certfile: (optional): Specifies the file with the certificate in the PEM format. If the certfile is not specified or is left empty, the API server will work without SSL.
• keyfile: (optional): Specifies the file with the secret key in the PEM format.
• keyfile_password: (optional): Specifies a password for decrypting the keyfile.
• cafile: (optional): Specifies the file with the CA_BUNDLE with certificates of trusted CAs to use while verifying client certs.

This change requires a restart of the Patroni service. Once restarted, Patroni comes up with listening to HTTPS on port 8008 instead of an HTTP request.

Obviously, the cafile (CA Certificate) is required for the client to verify the server certificate. The client, including the patronictl, can choose not to verify the server certificates.

This patronictl behavior can be specified using another ctl section in Patroni configuration:

However, please expect warning messages from python libraries about skipping the certificate verification of patroni server, as follows.

So, it is always good to have the server certificate verification on the client side.

Please remember that only the “Unsafe APIs” are protected using password authentication. Safe APIs will continue to function as it is. So the impact of this change on other tools like HAProxy which use those safe APIs is almost nil.

In this blog post, we discussed setting up password authentication and changing the REST API communication to HTTPS, which should safeguard the REST APIs and satisfy many organizations’ requirements. In the next part of this, we shall discuss how to set up a certificate authentication instead of password authentication and a few other aspects of security.