ELK logstash是如何工作的?

How Logstash Works

---

The Logstash event processing pipeline has three stages: inputs → filters → outputs. Inputs generate events, filters modify them, and outputs ship them elsewhere. Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.

Logstash事件处理管道包括三个阶段：输入→过滤器→输出。输入会生成事件，过滤器会对其进行修改，而输出会将其发送到其他地方。输入和输出支持编解码器，使您可以在数据进入或退出管道时对其进行编码或解码，而不必使用单独的过滤器。 

Inputs

You use inputs to get data into Logstash. Some of the more commonly-used inputs are:

• file: reads from a file on the filesystem, much like the UNIX command tail -0F
• syslog: listens on the well-known port 514 for syslog messages and parses according to the RFC3164 format
• redis: reads from a redis server, using both redis channels and redis lists. Redis is often used as a "broker" in a centralized Logstash installation, which queues Logstash events from remote Logstash "shippers".
• beats: processes events sent by Beats.

For more information about the available inputs, see Input Plugins.

Filters

Filters are intermediary processing devices in the Logstash pipeline. You can combine filters with conditionals to perform an action on an event if it meets certain criteria. Some useful filters include:

• grok: parse and structure arbitrary text. Grok is currently the best way in Logstash to parse unstructured log data into something structured and queryable. With 120 patterns built-in to Logstash, it’s more than likely you’ll find one that meets your needs!
• mutate: perform general transformations on event fields. You can rename, remove, replace, and modify fields in your events.
• drop: drop an event completely, for example, debug events.
• clone: make a copy of an event, possibly adding or removing fields.
• geoip: add information about geographical location of IP addresses (also displays amazing charts in Kibana!)

For more information about the available filters, see Filter Plugins.

Outputs

Outputs are the final phase of the Logstash pipeline. An event can pass through multiple outputs, but once all output processing is complete, the event has finished its execution. Some commonly used outputs include:(输出是Logstash管道的最后阶段。一个事件可以通过多个输出，但是一旦完成所有输出处理，该事件就完成了执行。一些常用的输出包括)

• elasticsearch: send event data to Elasticsearch. If you’re planning to save your data in an efficient, convenient, and easily queryable format…​Elasticsearch is the way to go. Period. Yes, we’re biased :)
• file: write event data to a file on disk.
• graphite: send event data to graphite, a popular open source tool for storing and graphing metrics. http://graphite.readthedocs.io/en/latest/
• statsd: send event data to statsd, a service that "listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services". If you’re already using statsd, this could be useful for you!

Codecs

Codecs are basically stream filters that can operate as part of an input or output. Codecs enable you to easily separate the transport of your messages from the serialization process. Popular codecs include json, msgpack, and plain (text).(编解码器基本上是流过滤器，可以作为输入或输出的一部分进行操作。编解码器使您可以轻松地将消息的传输与序列化过程分开。流行的编解码器包括json，msgpack和plain （文本）。)

• json: encode or decode data in the JSON format.
• multiline: merge multiple-line text events such as java exception and stacktrace 8messages into a single event.(将多行文本事件（例如java异常和stacktrace消息）合并为一个事件。)

 

Execution Model

---

The Logstash event processing pipeline coordinates the execution of inputs, filters, and outputs.

Each input stage in the Logstash pipeline runs in its own thread. Inputs write events to a central queue that is either in memory (default) or on disk. Each pipeline worker thread takes a batch of events off this queue, runs the batch of events through the configured filters, and then runs the filtered events through any outputs. The size of the batch and number of pipeline worker threads are configurable (see Tuning and Profiling Logstash Performance).

(Logstash管道中的每个输入阶段都在其自己的线程中运行。输入将事件写入到内存（默认）或磁盘中的中央队列。每个管道工作程序线程都会从该队列中除去一批事件，通过配置的筛选器运行这批事件，然后通过任何输出运行经过筛选的事件。批处理的大小和管道工作线程的数量是可配置的)

By default, Logstash uses in-memory bounded queues between pipeline stages (input → filter and filter → output) to buffer events. If Logstash terminates unsafely, any events that are stored in memory will be lost. To help prevent data loss, you can enable Logstash to persist in-flight events to disk. See Persistent Queues for more information.

 (默认情况下，Logstash在管道阶段（输入→过滤器和过滤器→输出）之间使用内存中有界队列来缓冲事件。如果Logstash不安全地终止，则存储在内存中的所有事件都将丢失。为了帮助防止数据丢失，可以启用Logstash将正在进行的事件持久保存到磁盘。)

 

Reserved fields in Logstash events

---

Some fields in Logstash events are reserved, or are required to adhere to a certain shape. Using these fields can cause runtime exceptions when the event API or plugins encounter incompatible values.

@metadata	A key/value map. Ruby-based Plugin API: value is an org.jruby.RubyHash. Java-based Plugin API: value is an org.logstash.ConvertedMap. In serialized form (such as JSON): a key/value map where the keys must be strings and the values are not constrained to a particular type.
@timestamp	An object holding representation of a specific moment in time. Ruby-based Plugin API: value is an org.jruby.RubyTime. Java-based Plugin API: value is a java.time.Instant. In serialized form (such as JSON) or when setting with Event#set: an ISO8601-compliant String value is acceptable.
@version	A string, holding an integer value.
tags	An array of distinct strings