Deploying the ELK Stack on Kubernetes with Helm
source link: https://www.tuicool.com/articles/Vbm2Uzn
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
ELK and Kubernetes are used in the same sentence usually in the context of describing a monitoring stack. ELK integrates natively with Kubernetes and is a popular open-source solution for collecting, storing and analyzing Kubernetes telemetry data.
However, ELK and Kubernetes are increasingly being used in another context — that of a method for deploying and managing the former. While deploying the ELK Stack using Kubernetes might seem like a complex task, there are more and more best practices around this scenario as well as Kubernetes-native solutions.
What’s Helm?
Maintained by CNCF, Helm is increasingly becoming a standard way for managing applications on Kubernetes. The easiest way to think about Helm is as a package manager for Kubernetes. It’s actually a bit more than just a package manager though as it allows users to create, publish and share applications on Kubernetes.
Each Helm chart contains all the specifications needed to be deployed on Kubernetes in the form of files describing a set of Kubernetes resources and configurations. Charts can be used to deploy very basic applications but also more complex systems such as…the ELK Stack!
Earlier this year, the folks at Elastic published Helm charts for Elasticsearch, Kibana, Filebeat and Metricbeat, making the deployment of these components on Kubernetes extremely simple.
Let’s take a closer look.
The setup
For the sake of this tutorial, I used Minikube installed on my Mac. You’ll also need Kubectl set up and configured.
Step 1: Setting Up Kubernetes
Obviously, we first need to make sure we have a Kubernetes cluster to install the ELK Stack on.
When starting Minikube, you’ll need to allocate some extra firepower as the plan is to deploy a multi-node Elasticsearch cluster:
minikube start --cpus 4 --memory 8192
You should see output similar to this:
Starting local Kubernetes v1.10.0 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components... Kubectl is now configured to use the cluster.
Just to verify your single-node Kubernetes cluster is up and running, use:
kubectl cluster-info Kubernetes master is running at https://192.168.99.106:8443 KubeDNS is running at https://192.168.99.106:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Step 2: Installing Helm
Your next step is to install Helm. Again, if you’ve got Helm setup and initialized already, great, you can skip to deploying the ELK Stack in the following steps.
To install Helm. execute the following 3 commands:
curl https://raw.githubusercontent.com/kubernetes/Helm/master/scripts/get > get_Helm.sh chmod 700 get_Helm.sh ./get_Helm.sh
You should see the following output:
Downloading https://get.Helm.sh/Helm-v2.14.3-darwin-amd64.tar.gz Preparing to install Helm and tiller into /usr/local/bin Helm installed into /usr/local/bin/Helm tiller installed into /usr/local/bin/tiller Run 'Helm init' to configure Helm.
To start Helm, enter:
Helm init
To verify the Tiller server is running properly, use:
kubectl get pods -n kube-system | grep tiller
And the output:
tiller-deploy-77b79fcbfc-hmqj8 1/1 Running 0 50s
Step 3: Deploying an Elasticsearch Cluster with Helm
It’s time to start deploying the different components of the ELK Stack. Let’s start with Elasticsearch.
As mentioned above, we’ll be using Elastic’s Helm repository so let’s start with adding it:
Helm repo add elastic https://Helm.elastic.co "elastic" has been added to your repositories
Next, download the Helm configuration for installing a multi-node Elasticsearch cluster on Minikube:
curl -O https://raw.githubusercontent.com/elastic/Helm-charts/master/elasticsearch/examples/minikube/values.yaml
Install the Elasticsearch Helm chart using the configuration you just downloaded:
Helm install --name elasticsearch elastic/elasticsearch -f ./values.yaml
The output you should be seeing looks something like this:
NAME: elasticsearch LAST DEPLOYED: Mon Sep 16 17:28:20 2019 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE elasticsearch-master-0 0/1 Pending 0 0s ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE elasticsearch-master ClusterIP 10.101.239.94 9200/TCP,9300/TCP 0s elasticsearch-master-headless ClusterIP None 9200/TCP,9300/TCP 0s ==> v1beta1/PodDisruptionBudget NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE elasticsearch-master-pdb N/A 1 0 0s ==> v1beta1/StatefulSet NAME READY AGE elasticsearch-master 0/3 0s NOTES: 1. Watch all cluster members come up. $ kubectl get pods --namespace=default -l app=elasticsearch-master -w 2. Test cluster health using Helm test. $ Helm test elasticsearch
As noted at the end of the output, you can verify your Elasticsearch pods status with:
kubectl get pods --namespace=default -l app=elasticsearch-master -w
It might take a minute or two, but eventually, three Elasticsearch pods will be shown as running:
NAME READY STATUS RESTARTS AGE elasticsearch-master-0 1/1 Running 0 1m elasticsearch-master-2 1/1 Running 0 1m elasticsearch-master-1 1/1 Running 0 1m
Our last step for deploying Elasticsearch is to set up port forwarding:
kubectl port-forward svc/elasticsearch-master 9200
And the output:
Step 4: Deploying Kibana with Helm
Next up — Kibana. As before, we’re going to use Elastic’s Helm chart for Kibana:
Helm install --name kibana elastic/kibana
And the output:
NAME: kibana LAST DEPLOYED: Wed Sep 18 09:52:21 2019 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/Deployment NAME READY UP-TO-DATE AVAILABLE AGE kibana-kibana 0/1 1 0 0s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE kibana-kibana-6d7466b9b9-fbmsz 0/1 ContainerCreating 0 0s ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kibana-kibana ClusterIP 10.96.37.129 5601/TCP 0s
Verify your Kibana pod is running (it might take a minute or two until the status turns to “Running”):
kubectl get pods NAME READY STATUS RESTARTS AGE elasticsearch-master-0 1/1 Running 0 15m elasticsearch-master-1 1/1 Running 0 15m elasticsearch-master-2 1/1 Running 0 15m kibana-kibana-6d7466b9b9-fbmsz 1/1 Running 0 2m
And last but not least, set up port forwarding for Kibana with:
kubectl port-forward deployment/kibana-kibana 5601
You can now access Kibana from your browser at: http://localhost:5601:
Step 5: Deploying Metricbeat with Helm
To set up a data pipeline, we’re going to end this tutorial with deploying the Metricbeat Helm chart:
Helm install --name metricbeat elastic/metricbeat
Within a minute or two, your Kubernetes cluster will display Metricbeat pods running alongside your Elasticsearch and Kibana pods:
kubectl get pods NAME READY STATUS RESTARTS AGE elasticsearch-master-0 1/1 Running 0 11m elasticsearch-master-1 1/1 Running 0 11m elasticsearch-master-2 1/1 Running 0 11m kibana-kibana-6d7466b9b9-bsfd5 1/1 Running 0 6m metricbeat-kube-state-metrics-bd55f95cc-8654c 1/1 Running 0 1m metricbeat-metricbeat-kjj6z 1/1 Running 0 1m metricbeat-metricbeat-metrics-699db67c5c-b2fzs 1/1 Running 0 1m
If you curl Elasticsearch, you’ll see that metrics have already begun to be indexed in Elasticsearch:
curl localhost:9200/_cat/indices green open .kibana_task_manager QxPJtK5rQtGGguLRv5h9OQ 1 1 2 4 87.7kb 44.8kb green open metricbeat-7.3.0-2019.09.18-000001 DeXaNAnMTWiwrQKNHSL0FQ 1 1 291 0 1.1mb 544.1kb green open .kibana_1 gk0OHIZDQWCNcjgb-uCBeg 1 1 4 0 30.3kb 15.1kb
All that’s left to do now is define the index pattern in Kibana and begin analyzing your data. In Kibana, go to the Management → Kibana → Index Patterns page, and click Create index pattern . Kibana will automatically identify and display the Metricbeat index:
Enter ‘ metricbeat-* ’ and on the next step select the @timestamp field to finalize the creation of the index pattern in Kibana.
Hop on over to the Discover page. You’ll see all the metrics being collected from your Kubernetes cluster by Metricbeat displayed:
Endnotes
These Helm charts are a great way to get started with ELK on Kubernetes but will require tweaking to be able to handle large payloads. Maintaining an ELK Stack in production is not an easy task to start out with and managing a multi-node, large Elasticsearch cluster on Kubernetes will require both engineering resources and strong infrastructure. I expect that as Helm becomes the standard way to build and deploy applications on Kubernetes, best practices will emerge for handling large scale ELK deployments as well. Looking forward to it!
Recommend
-
12
E(4L)K: My Journey through AWS ELK StackJanuary 04, 2021Introduction About 5 years ago there was several places to check for information while debugging issues. Some errors would trigger email to be sent to a specific...
-
8
ELK Stack单节点 本分支使用ElasticSearch官方的镜像和Docker-Compose来创建单节点的ELK Stack; 在ELK Stack中同时包括了Elastic Search、LogStash、Kibana以及Filebeat; 各个组件的作用如下: Filebeat:采集文件等日志...
-
7
在上一篇文章《使用Docker-Compose部署单节点ELK》中讲述了如何使用Docker-Compose创建一个单节点的ELK服务;但是目前在整个ELK-Stack中还包括了Filebeat进行日志采集; 本文在上一篇文章的基础之上在服务中添加了Filebeat,形成了目前业界比较常用的完...
-
11
How To Install ELK Stack on CentOS 7Search ComputingForGeeksHow can I install ELK Stack on CentOS 7 / Fedora...
-
4
An introduction to monitoring using the ELK Stack If you need centralized, comprehensive monitoring, putting Elasticsearch,...
-
4
How to Install ELK Stack on RHEL 8Search ComputingForGeeksIn this blog post, I’ll show you how to install an...
-
7
The Cost of Doing the ELK Stack on Your OwnSo, you’ve decided to go with ELK to centralize, manage, and analyze your logs.Wise decision.The
-
5
Deploy EFK stack with Helm 3 in Kubernetes Skip to content Centralized logging is one of the essential part in Kubernetes environment. In thi...
-
5
In this fast-moving world, businesses need to be nimble and be able to deliver changes to their customers faster than ever before. To keep up with that, software systems and teams should find ways to build things faster and deploy quicker. Enter m...
-
2
Argo CD provides numerous ways to deploy resources from a Helm chart. In this article, you will learn about three patterns used to manage and deploy Helm charts, including when and where to use each pattern in your
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK