Docker 搭建ELK集群(二)Logstash7.6.2 同步Mysql8.0.24数据到ES

最新推荐文章于 2024-07-03 21:27:40 发布

一个名叫豆包的码农

最新推荐文章于 2024-07-03 21:27:40 发布

阅读量706

点赞数

分类专栏： docker ElasticSearch MySql 文章标签： mysql elasticsearch 数据库

本文链接：https://blog.csdn.net/qq_38360901/article/details/116204757

版权

ElasticSearch 同时被 3 个专栏收录

4 篇文章 0 订阅

订阅专栏

docker

3 篇文章 0 订阅

订阅专栏

MySql

3 篇文章 2 订阅

订阅专栏

该博客详细介绍了如何使用Docker部署Logstash，从MySQL数据库中抽取数据，并实时同步到Elasticsearch的过程。涉及内容包括Docker镜像拉取、配置文件修改、JDBC连接设置以及Logstash容器的启动和日志查看。通过这个配置，可以实现每30分钟更新一次数据库记录，并将数据存储在指定的Elasticsearch索引中。

摘要由CSDN通过智能技术生成

拉取镜像并启动

docker pull logstash:7.6.2

docker run -d --name=logstash logstash:7.6.2

拷贝 logstash 配置文件

docker cp logstash:/usr/share/logstash/config/ /home/elasticsearch/logstash/
docker cp logstash:/usr/share/logstash/pipeline/ /home/elasticsearch/logstash/
docker cp logstash:/usr/share/logstash/logstash-core/lib/jars/ /home/elasticsearch/logstash/

对logstash文件夹授权

chmod 777 /home/elasticsearch/logstash/

将Mysql JAR包上传到 /logstash/jars/

在这里插入图片描述

修改配置文件

vim /home/elasticsearch/logstash/config/logstash.yml 
# 改为如下内容
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://192.168.1.114:9200" ]

修改 logstash.conf 文件

vim /home/elasticsearch/logstash/pipeline/logstash.conf

input{
     jdbc{
		  type => "merchant_commodity"
          jdbc_connection_string => "jdbc:mysql://你的数据库地址:端口号/数据库?serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false"
          jdbc_user => ""
          jdbc_password => ""
          jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-8.0.24.jar"
          jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
          jdbc_paging_enabled => "true"
          jdbc_page_size => "50000"
          jdbc_default_timezone => "Asia/Shanghai"  
          statement => "select mc.commodity_id as id, mc.commodity_name as name ,mc.commodity_number as number ,mc.commodity_picture as picture ,mc.commodity_selling_price as price,mc.commodity_unit as unit, mc.commodity_description as description  
		 ,UNIX_TIMESTAMP(mc.update_time) as update_time FROM merchant_commodity mc WHERE mc.update_time > :sql_last_value AND mc.update_time < NOW() "
          clean_run => false
          # 每30分钟更新一次 默认为1分钟一次
          schedule => "0 */30 * * * *"
     }
	 jdbc{
		  type => "merchant_shop"
          jdbc_connection_string => "jdbc:mysql://地址:端口/数据库?serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false"
          jdbc_user => ""
          jdbc_password => ""
          jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-8.0.24.jar"
          jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
          jdbc_paging_enabled => "true"
          jdbc_page_size => "50000"
          jdbc_default_timezone => "Asia/Shanghai"  
          statement => "select ms.shop_id as id , ms.shop_name as name ,ms.shop_address as address ,ms.shop_tel as tel ,ms.shop_introduction as introduction from merchant_shop ms
		    WHERE ms.update_time > :sql_last_value AND ms.update_time < NOW() "
          clean_run => false
          schedule => "0 */30 * * * *"
     }
}
output{
	if [type]=="merchant_commodity" {
		  elasticsearch {
		  hosts => "139.186.162.76:9200"
		  # 索引名
          index => "merchant_commodity"
          #索引类型
          document_type => "commodity"
          document_id => "%{id}"
		  }
		}
	
	if [type]=="merchant_shop" {
      elasticsearch{
        hosts => "139.186.162.76:9200"
        index => "merchant_shop"
        document_type => "shop"
        document_id => "%{id}"
       }
	  }
      stdout{
            codec => json_lines
      } 
}

启动Logstash容器

docker run -d --privileged=true --network es-net -v /home/elasticsearch/logstash/config/:/usr/share/logstash/config/ -v /home/elasticsearch/logstash/pipeline/:/usr/share/logstash/pipeline/ -v /home/elasticsearch/logstash/jars/:/usr/share/logstash/logstash-core/lib/jars/ --name=logstash   logstash:7.6.2

–network es-net 加入通信否则不能连接Kibana

查看日志