About Logs
Often, we need to comb through various system logs to find the cause of errors. An unavoidable reality is: logs come in many types (OS / application server / business logic) and are very large (generated daily, weekly, or monthly).
Faced with such massive and scattered data, manually checking each one is impractical. That’s where centralized logging systems come in, with the following characteristics:
- Collection — Ability to gather log data from multiple sources
- Transport — Reliably transfer log data to a central system
- Storage — How to store log data
- Analysis — Support UI-based analysis
- Alerting — Provide error reporting and monitoring mechanisms
There are many centralized logging systems: simple ones like Rsyslog and Syslog-ng, commercial ones like Splunk, and open-source options such as Facebook’s Scribe, Apache’s Chukwa, LinkedIn’s Kafka, Cloudera’s Fluentd, ELK, and so on.
Here, let’s briefly introduce the open-source ELK.
ELK
ELK is not a single piece of software, but a complete solution — an acronym for three software products: Elasticsearch, Logstash, and Kibana. All three are open-source, typically used together, and are now all under Elastic.co, hence the shorthand “ELK Stack.”
Elasticsearch
Elasticsearch is a real-time distributed search and analytics engine. It can be used for full-text search, structured search, and analytics. It’s a search engine built on the full-text search engine Apache Lucene, written in Java.
Key Features:
- Real-time analytics
- Distributed real-time file storage, with every field indexed
- Document-oriented — all objects are documents
- High availability, easy to scale, supports Clusters, Shards, and Replicas
- Developer-friendly interface, supports JSON
Logstash
Logstash is a data collection engine with real-time pipeline capabilities, written in JRuby. Its author is the world-renowned ops engineer Jordan Sissel.
Key Features:
- Access to virtually any data
- Integration with various external applications
- Elastic scalability
It consists of three main components:
Shipper — Sends log data Broker — Collects data, default built-in Redis Indexer — Writes data
Kibana
Kibana is an open-source web platform based on the Apache license, written in JavaScript, that provides analytics and visualization for Elasticsearch. It can search and interact with data in Elasticsearch indices and generate multidimensional charts and graphs.
ELK Stack Architecture
The basic workflow: Shipper collects data from various data sources and sends it to Broker. Indexer writes the data stored in Broker into Elasticsearch. Elasticsearch indexes this data, and Kibana analyzes it and displays it in chart form.
The three ELK tools work together seamlessly, efficiently meeting the needs of many scenarios.