logstash同步mysql数据到elasticsearch

最新推荐文章于 2024-04-23 10:33:19 发布

爪蛙没有爪

最新推荐文章于 2024-04-23 10:33:19 发布

阅读量189

点赞数

分类专栏： Elasticsearch Ubuntu Mysql 文章标签： ubuntu mysql logstash jdbc inpu-jdbc

本文链接：https://blog.csdn.net/qq_32013641/article/details/102924739

版权

Ubuntu 同时被 3 个专栏收录

8 篇文章 0 订阅

订阅专栏

Elasticsearch

5 篇文章 0 订阅

订阅专栏

Mysql

4 篇文章 0 订阅

订阅专栏

安装 jdbc 和 elasticsearch 插件

/usr/share/logstash/bin/logstash-plugin install logstash-input-jdbc
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

下载mysql-connector-java

cd /usr/share/logstash/logstash-core/lib/jars
wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar

编写配置文件 logstash.conf
tips:数据库中删除的数据无法同步到ES中，只能同步insert,update数据

input {
  stdin{
  }
  jdbc {
    type=>"task_desc_info"
    jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/db_dev?useUnicode=true&characterEncoding=utf-8&useSSL=false"
    jdbc_user => "root"
    jdbc_password => "147258369"
    jdbc_driver_library => ""
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_paging_enabled => true
    jdbc_page_size => "50000"
    jdbc_default_timezone =>"Asia/Shanghai"

    tracking_column => "auto_id"
    record_last_run => "true"
    use_column_value => "true"
    last_run_metadata_path => "task_desc"
    clean_run => "false"

    statement => "select * from task_description where auto_id > :sql_last_value"
    schedule => "* * * * *"
  }
}
output{
  if [type]=="task_desc_info" {
    stdout {
      codec => json_lines
    }
    elasticsearch {
      hosts => "127.0.0.1:9200"
      index => "task_desc"
      document_id => "%{task_id}"
    }
  }
}

启动

/usr/share/logstash/bin/logstash/ -f logstash.conf

同步多张表
编写/pipelines.yml

- pipeline.id: table1
  path.config: "/path/tabale1.conf"
- pipeline.id: table2
  path.config: "/path/table2.conf"
- pipeline.id: table3
  path.config: "/path/table3.conf"

参考：https://www.cnblogs.com/qdhxhz/p/9520685.html