logstash同步mysql数据到elasticsearch

5 篇文章 0 订阅
4 篇文章 0 订阅

安装 jdbc 和 elasticsearch 插件

/usr/share/logstash/bin/logstash-plugin install logstash-input-jdbc
/usr/share/logstash/bin/logstash-plugin install logstash-output-elasticsearch

下载mysql-connector-java

cd /usr/share/logstash/logstash-core/lib/jars
wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar

编写配置文件 logstash.conf
tips:数据库中删除的数据无法同步到ES中,只能同步insert,update数据

input {
  stdin{
  }
  jdbc {
    type=>"task_desc_info"
    jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/db_dev?useUnicode=true&characterEncoding=utf-8&useSSL=false"
    jdbc_user => "root"
    jdbc_password => "147258369"
    jdbc_driver_library => ""
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_paging_enabled => true
    jdbc_page_size => "50000"
    jdbc_default_timezone =>"Asia/Shanghai"

    tracking_column => "auto_id"
    record_last_run => "true"
    use_column_value => "true"
    last_run_metadata_path => "task_desc"
    clean_run => "false"

    statement => "select * from task_description where auto_id > :sql_last_value"
    schedule => "* * * * *"
  }
}
output{
  if [type]=="task_desc_info" {
    stdout {
      codec => json_lines
    }
    elasticsearch {
      hosts => "127.0.0.1:9200"
      index => "task_desc"
      document_id => "%{task_id}"
    }
  }
}                                                                                                   

启动

/usr/share/logstash/bin/logstash/ -f logstash.conf

同步多张表
编写/pipelines.yml

- pipeline.id: table1
  path.config: "/path/tabale1.conf"
- pipeline.id: table2
  path.config: "/path/table2.conf"
- pipeline.id: table3
  path.config: "/path/table3.conf"

参考:https://www.cnblogs.com/qdhxhz/p/9520685.html

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值