Securing a Zookeeper ensemble

Securing a Zookeeper ensemble

In the previous post, we looked at how to build a three cluster Zookeeper ensemble. However, the ensemble was not secured in any way. This would allow unauthorised clients to query Zookeeper and to push data to znodes. It also allows unauthorised Zookeeper instances to join the ensemble and potentially even instruct the cluster to shut down.

Even in secured networks, it’s a good idea to use some of the security features available in Zookeeper. In this post we’ll look at two security mechanisms: mutual TLS (mTLS) and SASL authentication. We’ll set up these security features on the server-server communication (leader election protocols) and client-server communication (Kafta to Zookeeper).

mTLS for quorum protocol

Mutual TLS (mTLS) is a mechanism to allow communication only where both parties trust each other. If this is enabled for the leader election protocol (Zookeeper server to Zookeeper server) then only trusted servers can join the ensemble. To enable mTLS, every Zookeeper server must have a signed certificate and a trust store containing the certificates of other servers in the ensemble.

Let’s start by creating the certificates, keystores and the trust store. Assuming we have Zookeeper running on three servers called:

• zookeeper-1.europe-north1-a.c.zookeeper-12345.internal
• zookeeper-2.europe-north1-b.c.zookeeper-12345.internal
• zookeeper-3.europe-north1-c.c.zookeeper-12345.internal

Use the Java keytool to create a self-signed keystore each server and a trust store containing all three certificates:

Run the above for each HOSTNAME to produce four files:

• zookeeper-1.europe-north1-a.c.zookeeper-12345.internal-zk-keystore.jks
• zookeeper-2.europe-north1-b.c.zookeeper-12345.internal-zk-keystore.jks
• zookeeper-3.europe-north1-c.c.zookeeper-12345.internal-zk-keystore.jks
• zk-truststore.jks

Copy the files to the three Zookeeper servers and then add the following to each zoo.cfg:

(adjust the ssl.quorum.keyStore.location property to pick the keystore for this server)

Restart the three Zookeeper instances to enable (and enforce) mTLS for server-server communication.

SASL Authentication for quorum protocol

Zookeeper supports server-server mutual authentication using Simple Authentication and Security Layer (SASL). It supports Kerberos and Digest-MD5 schemes. Kerberos is the stronger authentication scheme but requires additional infrastructure. This example uses the simpler Digest-MD5 scheme which just requires some usernames and passwords to be set up.

First, add some config to zoo.cfg to enable (and require) SASL:

Then create a jaas.conf file containing the our username and password:

In this case our our Quorum Learner has username zkquorumlearner and password zkqlpa55. The Quorum Server is configured to accept that credential.

Finally, enable the JAAS config by adding it to Zookeeper’s SERVER_JVMFLAGS. The easiest way is to add this to java.env in /usr/local/zookeeper/conf/ (create the file if it doesn’t exist):

Again, restart the three Zookeeper instances for this to take effect.

mTLS for client connections

Zookeeper supports the same security mechanisms for client-server communications. In this example we want to secure communication with a Kafka client.

To enforce mTLS for client communication, add the following to the zoo.cfg on all three servers:

Note that we’re using the same keystore / truststore files that we created earlier. Again, make sure that the ssl.keyStore.location is the keystore for this server.

Add the corresponding configuration to the Kafka server.properties

SASL authentication for client connections

Finally, we’ve already enabled SASL authentication in Zookeeper so using it for client connections is as simple as adding a new entry to the existing jaas.conf file:

Create a corresponding kafka_jaas.conf file for Kafka, this one contains the credentials that the client will authenticate with:

Use the KAFKA_OPTS environment variable to enable this in Kafka:

Now restart all three Zookeeper instances and Kafka. The Kafka logs should show that it’s using Digest-MD5 authentication: