Logstash8 从 MYSQL 导入数据到 ES

拉取镜像

docker pull logstash:8.0.0

运行进行并把配置文件拷贝出来，Logstash 有两种配置文件，管道配置文件（/usr/share/logstash/pipeline/）和Logstash启动配置文件（/usr/share/logstash/config/）

docker run --rm --name logstash -d logstash:8.0.0

docker cp logstash:/usr/share/logstash/pipeline/ .
docker cp logstash:/usr/share/logstash/config/ .

下载mysql驱动包，根据你使用的mysql版本下载

修改config目录中的 logstash.yml 文件，只需要很少的配置，具体配置参考：https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html

http.host: "0.0.0.0"
node.name: logstash

修改pipeline目录中的 logstash.conf，只配置两个插件，jdbc插件文档地址：https://www.elastic.co/guide/en/logstash/8.0/plugins-inputs-jdbc.html#_scheduling_2

这里es输出插件我启用了SSL验证，因为我的ES开启了SSL，配置证书的时候，如果证书没有密码，要将 truststore_password 设置为空字符串，ssl_certificate_verification => false 不验证主机名。

方括号中的配置需要自己填写

input{
	jdbc{
		jdbc_driver_library => "/usr/share/logstash/lib/jars/mysql-connector-java-8.0.11.jar"
		jdbc_driver_class => "com.mysql.jdbc.Driver"
		jdbc_connection_string => "jdbc:mysql://XX.XX.XX.XX:3030/test?useUnicode=true&characterEncoding=utf8&serverTimezone=Asia/Shanghai&useSSL=false"
		jdbc_user => [username]
		jdbc_password => [password]
		statement => "select * from student where id > :sql_last_value"
		use_column_value => true
		tracking_column => "id"
		tracking_column_type => "numeric"
		schedule => "0/5 * * * * *"
		
		jdbc_paging_enabled => true
		jdbc_page_size => 1000
		
	}
}

output{
	elasticsearch{
		hosts => "https://XX.XX.XX.XX:9200"
		index => "demo"
		user => [username]
		password => [password]
		ssl => true
		ssl_certificate_verification => false
		truststore => "/usr/share/logstash/cert/http.p12"
		truststore_password => ''
	}
}

最终的文件目录如下

• /d/IT/docker/logstash/image/lib/jars/：mysql驱动包
• /d/IT/docker/logstash/image/config/：logstash启动配置文件
• /d/IT/docker/logstash/image/pipeline/：logstash管道配置文件
• /d/IT/docker/logstash/image/cert/：证书文件

然后docker启动

docker run --rm --name logstash-mysql -it -v /d/IT/docker/logstash/image/config/:/usr/share/logstash/config/ -v /d/IT/docker/logstash/image/pipeline/:/usr/share/logstash/pipeline/ -v /d/IT/docker/logstash/image/cert/:/usr/share/logstash/cert/ -v /d/IT/docker/logstash/image/lib/jars/:/usr/share/logstash/lib/jars/ logstash-mysql:8.0.0

如果在jdbc input 插件中配置了 schedule，logstash 会循环调度，如果没有配置，logstash 导入一次就会自动停止。

参考：
https://blog.csdn.net/neweastsun/article/details/96581340

https://blog.csdn.net/u010361276/article/details/107919576

https://blog.csdn.net/u014568072/article/details/115638260

https://elasticstack.blog.csdn.net/article/details/101691238

https://www.elastic.co/guide/en/logstash/8.0/plugins-inputs-jdbc.html#_scheduling_2