Deploying a Production Druid Cluster in Google Cloud Platform

The purpose of this article is to guide through a set up process of an Apache Druid Cluster using GCP

Apache Druid (incubating)

Druid is an open-source analytics data store designed for business intelligence ( OLAP ) queries on event data. Druid provides low latency (real-time) data ingestion, flexible data exploration, and fast data aggregation.

A Scalable Architecture

Requirements

• For Basic Druid Cluster: 8vCPUS, 30GB RAM and 200GB disk size (E.G. custom-6–30720 or n1-standard-8) per node.
• Have a Google Cloud Storage enable
• Have a MySQL instance or Cloud SQL active

Setting up

Log in with your GCP account and create three virtual machines with SO Debian.

Donwload druid:

run for each node

wget https://www-us.apache.org/dist/incubator/druid/0.15.1-incubating/apache-druid-0.15.1-incubating-bin.tar.gz
tar -xzf apache-druid-0.15.1-incubating-bin.tar.gzexport $PATH_GCP = /path/druid/

Install components

Enter each node with SSH and run:

#Update libraries 
sudo apt-get update#Install Java JDK 
sudo apt install default-jdk -y #Install Perl 
sudo apt-get install perl -y #Donwload MySQL JAR 
sudo apt install libmysql-java #Install MySQL server 
sudo apt-get install mysql-server -y #Move MySQL JAR to Druid folder 
cp /usr/share/java/mysql-connector-java-5.1.42.jar $PATH_GCP/apache-druid-0.15.1-incubating/extensions/mysql-metadata-storage

Install Zookeeper

Zookeeper may be installed in an independent node, although in this case, we are going to install in Master node. Log in using SSH and run the following script.

#Download
wget https://www-eu.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz tar -zxf zookeeper-3.4.14.tar.gz #create folder and move
sudo mkdir -p /usr/local/zookeeper
sudo mv zookeeper-3.4.14 /usr/local/zookeeper#create folder
sudo mkdir -p /var/lib/zookeeper #create config file 
vi /usr/local/zookeeper/conf/zoo.cfg  #add properties inside config file
tickTime=2000 
dataDir=/var/lib/zookeeper clientPort=2181

Edit common runtime properties

Edit file located in the route: apache-druid-0.15.1-incubating/conf/druid/cluster/_common/common.runtime.properties , replicate for each node making the suggested changes.

Edit file using sudo vi

sudo vi apache-druid-0.15.1-incubating/conf/druid/cluster/_common/common.runtime.properties

The following changes are written in order.

Enabling basic security

If you need to set up a basic authentication system, add these properties in the common file for each node. After doing this you’ll have a admin user with the password [PASSWORD_1] with access to all segments in the cluster.

add the following properties inside the file: apache-druid-0.15.1-incubating/conf/druid/cluster/_common/common.runtime.properties