logstash

简介

Logstash是一个接收，处理，转发日志的工具。支持系统日志，webserver日志，错误日志，应用日志，总之包括所有可以抛出来的日志类型。怎么样听起来挺厉害的吧？
在一个典型的使用场景下(ELK)：用Elasticsearch作为后台数据的存储，kibana用来前端的报表展示。Logstash在其过程中担任搬运工的角色，它为数据存储，报表查询和日志解析创建了一个功能强大的管道链。Logstash提供了多种多样的 input,filters,codecs和output组件，让使用者轻松实现强大的功能。好了让我们开始吧

安装

laradock 安装

docker compose build logstash

遇到了错误

 => ERROR [6/7] RUN logstash-plugin install logstash-input-jdbc 

ERROR: Installation aborted, plugin 'logstash-input-jdbc' is already provided by 'logstash-integration-jdbc'

这里提示的意思是logstash-input-jdbc 插件已经由logstash-integration-jdbc 提供，不需要重复安装，所以将logstash的dockerfile进行修改，删除了 logstash-plugin install logstash-input-jdbc，删除后重新构建。

启动logstash

docker compose up -d logstash

学习

使用终端行进入logstash

docker compose exec logstash bash

执行命令

 logstash  -e 'input { stdin { } } output { stdout {} }'

报错

ERROR: Configuration reloading can't be used with 'config.string' (-e).

所以将命令改为

logstash --path.settings=  -e 'input { stdin { } } output { stdout {} }'

运行后又报

[FATAL] 2023-03-17 09:22:15.912 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.

查阅资料后说需要删除==.lock== 文件

[root@c4c80be4ad84 logstash]# find ./ -name .lock
./data/.lock
[root@c4c80be4ad84 logstash]# rm ./data/.lock 
rm: remove regular empty file ‘./data/.lock’? yes

开始学习

客户端执行命令

logstash --path.settings=  -e 'input { stdin { } } output { stdout {} }'

[root@c4c80be4ad84 logstash]# logstash --path.settings=  -e 'input { stdin { } } output { stdout {} }'
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-180/jruby9708044941576158965jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /log4j2.properties. Using default config which logs errors to the console
[INFO ] 2023-03-17 09:53:48.700 [main] runner - Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +indy +jit [linux-x86_64]"}
[WARN ] 2023-03-17 09:53:49.273 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-03-17 09:53:51.126 [Converge PipelineAction::Create<main>] Reflections - Reflections took 55 ms to scan 1 urls, producing 22 keys and 45 values 
[INFO ] 2023-03-17 09:53:51.859 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x6bbb59b9 run>"}
[INFO ] 2023-03-17 09:53:52.870 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.01}
[INFO ] 2023-03-17 09:53:52.928 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2023-03-17 09:53:53.021 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2023-03-17 09:53:53.313 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
hello
{
    "@timestamp" => 2023-03-17T09:54:47.239Z,
          "host" => "c4c80be4ad84",
      "@version" => "1",
       "message" => "hello"
}

总结

文档地址

elasticsearch 中国社区

logstash 快速入门

高效实现mysql 与 elasticsearch 的数据同步