An introduction to monitoring using the ELK Stack

IT system monitoring is a proactive means of observing systems with the goal of preventing outages and downtime. It involves measuring current behavior against predetermined baselines. Some of the commonly monitored devices are CPU usage, memory usage, network traffic over routers and switches, and application performance, which helps a lot when performing root-cause analysis.

There are many systems out there, but not all can provide centralized, comprehensive monitoring. Sysadmins sometimes monitor their systems with scripting. Some use cron jobs when they write and configure their Bash scripts so that they receive an email when there is a change to the baseline. As someone exploring monitoring systems, I looked into the ELK Stack's set of applications as an option.

[ You might also like to read Using Cerebro as WebUI to manage an ELK cluster. ]

ELK is an acronym for several open source tools: Elasticsearch, Logstash, and Kibana. Elasticsearch is the engine of the Elastic Stack, which provides analytics and search functionalities. Logstash is responsible for collecting, aggregating, and storing data to be used by Elasticsearch. Kibana provides the user interface and insights into data previously collected and analyzed by Elasticsearch.

Elasticsearch

Elasticsearch provides real-time search and analytics for all data types, whether structured, unstructured, or numerical. It can efficiently store and index data in a way that enhances quick search and retrieval. It can also aggregate data to discover trends and patterns as the data and query volume grow. Elasticsearch is scalable, and as the data expands, the program deploys additional nodes to meet the demand. Because of its speed, Elasticsearch is a tremendous help in many instances, such as searching for a website or analyzing security events.

Logstash

Logstash collects data from multiple sources, transforms it, and then sends it to Elasticsearch for analysis. In short, it receives inputs and filters them down into helpful outputs for the other engines. Inputs primarily come from files, syslogs, and some lightweight log shippers called beats. Beats are small packages that are installed on target devices to feed information to Logstash. They may be in the form of Filebeats (logs and files), Packetbeats (network packet), Winlogbeats (Windows event logs), Metricbeats (system and service statistics), and others.

Logstash receives the files, filters the data, puts the files in a supported format, and then outputs them to Elasticsearch. It can also generate output to a file, graphic, or several other formats.

Kibana

Kibana, the last tool in the stack, is responsible for visualizing the data stored in Elasticsearch. Kibana allows you to explore the data as well as manage and monitor the entire ELK Stack. Kibana gives shape to your data and provides the means to navigate the ELK Stack. Kibana helps you search for hidden insights, then visualize what you find in charts, gauges, maps, and more. You can then combine this information into a dashboard. Kibana also monitors the ELK Stack's health. Finally, it controls users and their level of access in the ecosystem.

By default, Kibana comes with histograms, line graphs, pie charts, sunbursts, and more. It also supports highly available, scalable alerting via email, webhooks, Jira, Microsoft Teams, Slack, and other tools.

[ Free course: Red Hat Satellite Technical Overview. ]

Wrapping up

ELK is a comprehensive system that can help the sysadmin and the entire IT department. In my next article, I'll look at installing and configuring the stack and describe several real-world test scenarios.