Logstash Docker 部署 安装 logstash-output-jdbc
前置步骤参考:https://blog.csdn.net/weixin_44121790/article/details/141305720
问题:
今天使用docker 部署logstash,遇到无法运行的问题,原因是因为配置问题使用了 logstash-output-jdbc ,但是镜像默认没有安装。
配置文件如下:
#注释方法#####
input {
syslog {
port => "5044"
}
}
filter {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)
"
}
if("WinFileService" in [message] or "FtpFileTransfer" in [message]){
grok{
match =>{"message"=>"%{WORD:opt_type} Event: %{DATA:opt_name}, Path: %{DATA:path}, File/Folder: %{WORD:file_type}, Size: %{BASE16FLOAT:size} %{WORD:size_unit}, User: %{DATA:username}, IP: %{IPV4:opt_ip}"}
}
grok{
match=>{"path"=>"%{GREEDYDATA}/%{GREEDYDATA:file_name}"}
}
if("KB" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i * 1024)"
}
}
if("MB" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024)"
}
}
if("GB" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024 * 1024)"
}
}
if("Bytes" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i)"
}
}
}
}
output {
stdout {}
if("FtpFileTransfer" in [opt_type]){
jdbc{
driver_jar_path => "/usr/share/logstash/config/jar/mysql-connector-j-8.4.0.jar"
driver_class => "com.mysql.cj.jdbc.Driver"
connection_string => "jdbc:mysql://192.168.10.23:3306/database_name?user=root&password=xxxxxxx&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true"
statement => ["insert into t_ftp_log(level, event, full_path, file_size, event_time,app_source,user_name,source_ip,file_name) VALUES (?,?,?,?,?,?,?,?,?)","[log][syslog][priority]","opt_name","path","size_byte","@timestamp","[host][hostname]","username","opt_ip","file_name"]
}
}
}
解决方法:
编辑logstash.yml
指定 config路径
http.host: "0.0.0.0"
# xpack.monitoring.elasticsearch.hosts: [ "http://127.0.0.1:9200" ]
path.logs: /usr/share/logstash/logs
path.config: /usr/share/logstash/config/conf.d/*.conf
重点
在此config路径下创建一个简单的config文件logstash.conf,内容如下:
input {
syslog {
port => 5044
}
}
output {
stdout {
codec => rubydebug
}
}
然后启动容器
docker run --name logstash-sample -p 5044:5044 -p 9600:9600 -v /opt/docker/logstash/config:/usr/share/logstash/config -v /opt/docker/logstash/data:/usr/share/logstash/data -v /opt/docker/logstash/pipeline:/usr/share/logstash/pipeline -d docker.elastic.co/logstash/logstash:8.14.1
使用docker logs [container]查看是否启动成功,如果启动成功,则使用如下命令进入容器
docker exec -it --user root logstash-syslog-nas /bin/bash
进入容器后,需修改Gemfile source 源为国内镜像,否则可能会无法获取安装包
source "https://gems.ruby-china.com"
然后执行命令安装 logstash-output-jdbc
bin/logstash-plugin install logstash-output-jdbc
安装成功后,退出容器。记得配置 mysql-connector-j-8.4.0.jar路径,参考文章开头的配置文件
exit
重点
进入映射配置文件路径
cd /usr/share/logstash/config/conf.d/
将一开始配置的文件logstash.conf 替换为最初的配置:
#注释方法#####
input {
syslog {
port => "5044"
}
}
filter {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)
"
}
if("WinFileService" in [message] or "FtpFileTransfer" in [message]){
grok{
match =>{"message"=>"%{WORD:opt_type} Event: %{DATA:opt_name}, Path: %{DATA:path}, File/Folder: %{WORD:file_type}, Size: %{BASE16FLOAT:size} %{WORD:size_unit}, User: %{DATA:username}, IP: %{IPV4:opt_ip}"}
}
grok{
match=>{"path"=>"%{GREEDYDATA}/%{GREEDYDATA:file_name}"}
}
if("KB" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i * 1024)"
}
}
if("MB" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024)"
}
}
if("GB" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024 * 1024)"
}
}
if("Bytes" in [size_unit]){
ruby{
code =>"event.set('size_byte', event.get('size').to_i)"
}
}
}
}
output {
stdout {}
if("FtpFileTransfer" in [opt_type]){
jdbc{
driver_jar_path => "/usr/share/logstash/config/jar/mysql-connector-j-8.4.0.jar"
driver_class => "com.mysql.cj.jdbc.Driver"
connection_string => "jdbc:mysql://192.168.10.23:3306/database_name?user=root&password=xxxxxxx&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true"
statement => ["insert into t_ftp_log(level, event, full_path, file_size, event_time,app_source,user_name,source_ip,file_name) VALUES (?,?,?,?,?,?,?,?,?)","[log][syslog][priority]","opt_name","path","size_byte","@timestamp","[host][hostname]","username","opt_ip","file_name"]
}
}
}
然后重启容器
docker restart logstash
恭喜
大功告成!