【笔记】Logstash环境搭建和安装配置

Logstash介绍：

概述

Logstash 是 Elastic Stack 的中央数据流引擎，用于收集、丰富和统一所有数据，而不管格式或模式。当与Elasticsearch，Kibana，及 Beats 共同使用的时候便会拥有特别强大的实时处理能力。在这段视频中，Elastic 技术布道师曾勇将会对如何开始 Logstash 进行了概述和演示。

Logstash 是免费且开放的服务器端数据处理管道，能够从多个来源采集数据，转换数据，然后将数据发送到您最喜欢的“存储库”中。

Logstash 能够动态地采集、转换和传输数据，不受格式或复杂度的影响。利用 Grok 从非结构化数据中派生出结构，从 IP 地址解码出地理坐标，匿名化或排除敏感字段，并简化整体处理过程。Logstash主要由三部分组成：

• input：从一个或多个数据源获取数据，常用插件如file、syslog、redis、beats等。
• filter：用来数据过滤、格式转换等，常见插件如grok、mute、drop、geoip等
• output：数据输出，常用插件如elastcisearch、file、statsd等

Logstash下载：

提示：因为是使用Ruby写的，所以需要先安装JAVA JDK，这里不细说怎么安装JDK

建议kibana、es、logstash的版本一致否则会出现兼容性问题

• 历史版本下载地址
  

• 官网最新版下载地址
  

---

Logstash安装：

解压压缩包即可非常简单

$ tar -zxvf logstash-7.10.2-darwin-x86_64.tar.gz 

vim logstash-sample.conf 

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

---

Logstash启动：

提示：这里 -e 指的是启动通过命令行进行配置

 $ logstash -e 'input { stdin {} } output { stdout {} }'
Using JAVA_HOME defined java: /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK

Sending Logstash logs to /Users/xiaojialiang/module/logstash-7.10.2/logs which is now configured via log4j2.properties
[2022-07-18T19:24:33,639][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.291-b10 on 1.8.0_291-b10 +indy +jit [darwin-x86_64]"}
[2022-07-18T19:24:33,934][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-07-18T19:24:35,409][INFO ][org.reflections.Reflections] Reflections took 55 ms to scan 1 urls, producing 23 keys and 47 values 
[2022-07-18T19:24:36,585][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x74a21a6f run>"}
[2022-07-18T19:24:37,477][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.88}
[2022-07-18T19:24:37,520][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2022-07-18T19:24:37,598][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-07-18T19:24:37,970][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
       "message" => "",
      "@version" => "1",
    "@timestamp" => 2022-07-18T11:24:37.582Z,
          "host" => "xiaoDe-MacBook-Pro.local"
}