How I monitor my web server with the ELK Stack
source link: https://www.redhat.com/sysadmin/web-server-monitor-ELK
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
How I monitor my web server with the ELK Stack
Posted: September 30, 2021 | by Evans Amoany (Sudoer)
Photo from Pexels
In a previous article, I introduced Elasticsearch, Logstash, and Kibana (the ELK Stack) and the various components that make up this monitoring system. In this article, I'll look at how I use the ELK Stack to monitor my Nginx web server. This requires approximately 16GB of memory to operate.
As I wrote before, "Elasticsearch is the engine of the Elastic Stack, which provides analytics and search functionalities. Logstash is responsible for collecting, aggregating, and storing data to be used by Elasticsearch. Kibana provides the user interface and insights into data previously collected and analyzed by Elasticsearch."
Here, I'll introduce the concepts and basic configurations for how I use the ELK Stack to monitor my web server. Please note that these steps are not very detailed; I use this for development and demonstration rather than production. Running ELK in production would involve multiple instances in a cluster.
This tutorial uses Elasticsearch and Kibana; Logstash supports many modules by default, and you can tap into this information.
Step 1: Deploy Elasticsearch and Kibana
To make deployment easy, I created an application stalk with Elasticsearch and Kibana using Podman. Here is the pod and two containers:
podman pod create --name elastic -p 9200:9200 -p 9300:9300 -p 5601:5601
podman run --pod elastic --name elasticsearch -d -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.14.0
podman run --pod elastic --name kibana -d -e "ELASTICSEARCH_HOSTS=http://127.0.0.1:9200" docker.elastic.co/kibana/kibana:7.14.0
This creates a pod named elastic
and two containers within the pod:
- An
elasticsearch
container, which runs the imagedocker.elastic.co/elasticsearch/elasticsearch:7.14.0
. - A
kibana
container, which runs the imagedocker.elastic.co/kibana/kibana:7.14.0
and connects to theelasticsearch
container onport 9200
.
If these run successfully, the Kibana dashboard is accessible from the host browser. The firewall must allow port 5601
, which is used for accessing Kibana, for external access.
To run this tutorial on a local machine, use http://localhost:5601
to access the dashboard; to run it inside a virtual machine (VM), use the VM's IP address. Port forwarding uses the same steps as running it on localhost.
[ Free online course: Developing cloud-native applications with microservices architectures. ]
I use this path to access the Nginx logs from the main page:
Home page -> Add data -> Logs -> nginx logs
Step 2: Configure the Filebeat and Nginx module
According to Elastic, "Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing." The Nginx logs page explains how to configure Filebeat and the Nginx module. This configuration displays the Kibana entries on the server where Nginx is installed and sends the Nginx logs to Elasticsearch:
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.14.0-x86_64.rpm
rpm -vi filebeat-7.14.0-x86_64.rpm
I modify /etc/filebeat/filebeat.yml
to set the connection information:
output.elasticsearch:
hosts: ["10.233.208.8:9200"] #This is server in where elasticsearch is running
setup.kibana:
host: "10.233.208.8:5601" #This is the server where kabana is running
#check if filebeat file has the correct syntax:
filebeat -e -c /etc/filebeat/filebeat.yml
#Enable the nginx module
filebeat modules enable nginx
#configure filebeat to start and persist reboot:
filebeat setup
systemctl enable filebeat
systemctl start filebeat
Step 3: Create an index pattern on Elasticsearch
Kibana requires an index pattern in order to search the data that Elasticsearch processes. An index pattern identifies the data to use and the metadata or properties of the data. This is analogous to selecting specific data from a database.
On Kibana's main page, I use this path to create an index pattern:
Management -> Stack Management -> index patterns -> create index pattern
I enter the index pattern, such as filebeat-*
. It suggests choices, and a wildcard works to match multiple sources.
Click Next.
If Kibana detects an index with a timestamp, I expand the Time field menu and specify the default field for filtering data by time.
[ Learn more about server and configuration management by downloading Ansible for DevOps. ]
Step 4: Create a dashboard to visualize data
I follow this path to display a data visualization:
Main Page -> Analytics -> Dashboard -> create visualization
On the left, I select the Available fields and use the dropdown on the right to create a dashboard.
Learn more
This container-based deployment option for the ELK Stack is particularly useful in a lab or learning scenario. There are plenty of additional configurations available to monitor servers.
The ELK Stack is a comprehensive tool that sysadmins may find useful for real-time monitoring and analytics. It can also be integrated into other systems. If you want to go beyond this introduction of these basic concepts and configurations and use it in a production deployment, consult the documentation.
Recommend
-
38
Health Monitor Laravel Server & App Health Monitor and Notifier This package checks if the application resources are running as they should and creates a service status panel. It has the followin...
-
19
README.rst PHP Server Monitor
-
11
Monitor SQL Server Table Size with PowerShell By: Alejandro Cobar   |  Updated: 2020-12-15   | Â
-
6
4000 XP Monitor and optimize operational resources in SQL Server 3 hr 10 min Learning Path ...
-
4
With HTTP/2 it’s possible to deliver data at a client before the client even asks for the data. This will significantly improve latency and perceived download speed. Last hack day at JPoint we spent some time with HTTP/2 server push. I will s...
-
7
Tutorial How To Monitor Server Health with Checkmk on Ubuntu 20.04 Monitoring
-
5
Tutorial How To Monitor Server Health with Checkmk on Ubuntu 18.04 Monitoring
-
11
Monitor your SQL Server databases in the cloud and on-premises with one monitoring tool
-
10
Full-Stack Observability with Grafana and Azure Monitor May 10, 2022...
-
8
Article How to monitor workloads using OpenShift monitoring stack
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK