Logstash Docker 部署 安装 logstash-output-jdbc

Logstash Docker 部署 安装 logstash-output-jdbc

前置步骤参考：https://blog.csdn.net/weixin_44121790/article/details/141305720

问题：

今天使用docker 部署logstash，遇到无法运行的问题，原因是因为配置问题使用了 logstash-output-jdbc ，但是镜像默认没有安装。

配置文件如下：

#注释方法#####
input {
	syslog {
    		port => "5044"
  	}	
}
filter {
    ruby {
        code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)
		"
    }
    if("WinFileService" in [message] or "FtpFileTransfer" in [message]){
	grok{
	    match =>{"message"=>"%{WORD:opt_type} Event: %{DATA:opt_name}, Path: %{DATA:path}, File/Folder: %{WORD:file_type}, Size: %{BASE16FLOAT:size} %{WORD:size_unit}, User: %{DATA:username}, IP: %{IPV4:opt_ip}"}
	}
	grok{
	    match=>{"path"=>"%{GREEDYDATA}/%{GREEDYDATA:file_name}"}
	}
	if("KB" in [size_unit]){
            ruby{
			code =>"event.set('size_byte', event.get('size').to_i * 1024)"
		}
	}
	if("MB" in [size_unit]){
		ruby{
                        code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024)" 
                }
	} 
	if("GB" in [size_unit]){
		ruby{
			code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024 * 1024)"
                }
	}
	if("Bytes" in [size_unit]){
                ruby{
                        code =>"event.set('size_byte', event.get('size').to_i)"
                }
        }
		
    }		 
}

output {
    stdout {}
    if("FtpFileTransfer" in [opt_type]){
        jdbc{
            driver_jar_path => "/usr/share/logstash/config/jar/mysql-connector-j-8.4.0.jar"
            driver_class => "com.mysql.cj.jdbc.Driver"
            connection_string => "jdbc:mysql://192.168.10.23:3306/database_name?user=root&password=xxxxxxx&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true"
            statement => ["insert into t_ftp_log(level, event, full_path, file_size, event_time,app_source,user_name,source_ip,file_name) VALUES (?,?,?,?,?,?,?,?,?)","[log][syslog][priority]","opt_name","path","size_byte","@timestamp","[host][hostname]","username","opt_ip","file_name"]
        }   
    }
}

解决方法：

编辑logstash.yml

指定 config路径

http.host: "0.0.0.0"
# xpack.monitoring.elasticsearch.hosts: [ "http://127.0.0.1:9200" ]
path.logs: /usr/share/logstash/logs
path.config: /usr/share/logstash/config/conf.d/*.conf

重点
在此config路径下创建一个简单的config文件logstash.conf，内容如下：

input {
  syslog {
    port => 5044
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

然后启动容器

docker run --name logstash-sample -p 5044:5044 -p 9600:9600  -v /opt/docker/logstash/config:/usr/share/logstash/config -v /opt/docker/logstash/data:/usr/share/logstash/data -v /opt/docker/logstash/pipeline:/usr/share/logstash/pipeline -d docker.elastic.co/logstash/logstash:8.14.1

使用docker logs [container]查看是否启动成功，如果启动成功，则使用如下命令进入容器

docker exec -it --user root logstash-syslog-nas /bin/bash

进入容器后，需修改Gemfile source 源为国内镜像，否则可能会无法获取安装包

source "https://gems.ruby-china.com"

然后执行命令安装 logstash-output-jdbc

bin/logstash-plugin install logstash-output-jdbc

安装成功后，退出容器。记得配置 mysql-connector-j-8.4.0.jar路径，参考文章开头的配置文件

exit

重点
进入映射配置文件路径

cd /usr/share/logstash/config/conf.d/

将一开始配置的文件logstash.conf 替换为最初的配置：

#注释方法#####
input {
	syslog {
    		port => "5044"
  	}	
}
filter {
    ruby {
        code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)
		"
    }
    if("WinFileService" in [message] or "FtpFileTransfer" in [message]){
	grok{
	    match =>{"message"=>"%{WORD:opt_type} Event: %{DATA:opt_name}, Path: %{DATA:path}, File/Folder: %{WORD:file_type}, Size: %{BASE16FLOAT:size} %{WORD:size_unit}, User: %{DATA:username}, IP: %{IPV4:opt_ip}"}
	}
	grok{
	    match=>{"path"=>"%{GREEDYDATA}/%{GREEDYDATA:file_name}"}
	}
	if("KB" in [size_unit]){
            ruby{
			code =>"event.set('size_byte', event.get('size').to_i * 1024)"
		}
	}
	if("MB" in [size_unit]){
		ruby{
                        code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024)" 
                }
	} 
	if("GB" in [size_unit]){
		ruby{
			code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024 * 1024)"
                }
	}
	if("Bytes" in [size_unit]){
                ruby{
                        code =>"event.set('size_byte', event.get('size').to_i)"
                }
        }
		
    }		 
}

output {
    stdout {}
    if("FtpFileTransfer" in [opt_type]){
        jdbc{
            driver_jar_path => "/usr/share/logstash/config/jar/mysql-connector-j-8.4.0.jar"
            driver_class => "com.mysql.cj.jdbc.Driver"
            connection_string => "jdbc:mysql://192.168.10.23:3306/database_name?user=root&password=xxxxxxx&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true"
            statement => ["insert into t_ftp_log(level, event, full_path, file_size, event_time,app_source,user_name,source_ip,file_name) VALUES (?,?,?,?,?,?,?,?,?)","[log][syslog][priority]","opt_name","path","size_byte","@timestamp","[host][hostname]","username","opt_ip","file_name"]
        }   
    }
}

然后重启容器

docker restart logstash

恭喜
大功告成！