Run PostgreSQL on Kubernetes with Percona Operator & Pulumi

Run PostgreSQL on Kubernetes with Percona Operator & Pulumi

Avoid vendor lock-in, provide a private Database-as-a-Service for internal teams, quickly deploy-test-destroy databases with CI/CD pipeline – these are some of the most common use cases for running databases on Kubernetes with operators. Percona Distribution for PostgreSQL Operator enables users to do exactly that and more.

Pulumi is an infrastructure-as-a-code tool, which enables developers to write code in their favorite language (Python, Golang, JavaScript, etc.) to deploy infrastructure and applications easily to public clouds and platforms such as Kubernetes.

This blog post is a step-by-step guide on how to deploy a highly-available PostgreSQL cluster on Kubernetes with our Percona Operator and Pulumi.

Desired State

We are going to provision the following resources with Pulumi:

• Google Kubernetes Engine cluster with three nodes. It can be any Kubernetes flavor.
• Percona Operator for PostgreSQL
• Highly available PostgreSQL cluster with one primary and two hot standby nodes
• Highly available pgBouncer deployment with the Load Balancer in front of it
• pgBackRest for local backups

Pulumi code can be found in this git repository.

Prepare

I will use the Ubuntu box to run Pulumi, but almost the same steps would work on macOS.

Pre-install Packages

gcloud and kubectl

python3

Pulumi allows developers to use the language of their choice to describe infrastructure and applications. I’m going to use python. We will also pip (python package-management system) and venv (virtual environment module).

Pulumi

Install Pulumi:

On macOS, this can be installed view Homebrew with brew install pulumi 

You will need to add .pulumi/bin to the $PATH:

Authentication

gcloud

You will need to provide access to Google Cloud to provision Google Kubernetes Engine.

Pulumi

Generate Pulumi token at app.pulumi.com. You will need it later to init Pulumi stack:

Action

This repo has the following files:

• Pulumi.yaml – identifies that it is a folder with Pulumi project
• __main__.py – python code used by Pulumi to provision everything we need
• requirements.txt – to install required python packages

Clone the repo and go to the pg-k8s-pulumi folder:

Init the stack with:

You will need the key here generated before on app.pulumi.com.

__main__.py

Python code that Pulumi is going to process is in __main__.py file. 

Lines 1-6: importing python packages

Lines 8-31: configuration parameters for this Pulumi stack. It consists of two parts:

• Kubernetes cluster configuration. For example, the number of nodes.
• Operator and PostgreSQL cluster configuration – namespace to be deployed to, service type to expose pgBouncer, etc.

Lines 33-80: deploy GKE cluster and export its configuration

Lines 82-88: create the namespace for Operator and PostgreSQL cluster

Lines 91-426: deploy the Operator. In reality, it just mirrors the operator.yaml from our Operator.

Lines 429-444: create the secret object that allows you to set the password for pguser to connect to the database

Lines 445-557: deploy PostgreSQL cluster. It is a JSON version of cr.yaml from our Operator repository

Line 560: exports Kubernetes configuration so that it can be reused later 

Deploy

At first, we will set the configuration for this stack. Execute the following commands:

These commands set the following:

• GCP project where GKE is going to be deployed
• GCP zone 
• Number of nodes in a GKE cluster
• Kubernetes version
• Namespace to run PostgreSQL cluster
• The name of the cluster
• Expose pgBouncer with LoadBalancer object

Deploy with the following command:

Verify

Get kubeconfig first:

Check if Pods of your PG cluster are up and running:

Get the IP-address of pgBouncer LoadBalancer:

You can connect to your PostgreSQL cluster through this IP-address. Use pguser password that was set earlier with pulumi config set pg_user_password:

Clean up

To delete everything it is enough to run the following commands:

Tricks and Quirks

Pulumi Converter

kube2pulumi is a huge help if you already have YAML manifests. You don’t need to rewrite the whole code, but just convert YAMLs to Pulumi code. This is what I did for operator.yaml.

apiextensions.CustomResource

There are two ways for Custom Resource management in Pulumi:

• apiextensions.CustomResource
• crd2pulumi

crd2pulumi generates libraries/classes out of Custom Resource Definitions and allows you to create custom resources later using these. I found it a bit complicated and it also lacks documentation.

apiextensions.CustomResource on the other hand allows you to create Custom Resources by specifying them as JSON. It is much easier and requires less manipulation. See lines 446-557 in my __main__.py.

True/False in JSON

I have the following in my Custom Resource definition in Pulumi code:

Be sure that you use boolean of the language of your choice and not the “true”/”false” strings. For me using the strings turned into a failure as the Operator was expecting boolean, not the strings.

Depends On…

Pulumi makes its own decisions on the ordering of provisioning resources. You can enforce the order by specifying dependencies

For example, I’m ensuring that Operator and Secret are created before the Custom Resource: