logstash同步mysql数据，多表同步配制

最新推荐文章于 2024-09-18 20:41:48 发布

dewffgqd

最新推荐文章于 2024-09-18 20:41:48 发布

阅读量4.3k

点赞数

分类专栏：全文检索

本文链接：https://blog.csdn.net/dewffgqd/article/details/73741354

版权

全文检索专栏收录该内容

3 篇文章 0 订阅

订阅专栏

本文介绍了一种使用 Logstash 从 MySQL 数据库提取数据并同步到 Elasticsearch 的配置方案。通过两个 JDBC 输入插件连接到同一数据库，分别设置不同的类型标记，并使用 JSON 过滤器进行数据格式转换，最后根据不同类型将数据写入同一个 Elasticsearch 索引中。

摘要由CSDN通过智能技术生成

input {
stdin {
}
jdbc {
# mysql jdbc connection string to our backup databse
jdbc_connection_string => "jdbc:mysql://192.168.42.147:3306/test"
# the user we wish to excute our statement as
jdbc_user => "root"
jdbc_password => "root"
# the path to our downloaded jdbc driver
jdbc_driver_library => "/home/hadoop/opt/logstash-2.3.2/mysql-connector-java-5.1.29.jar"
# the name of the driver class for mysql
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement_filepath => "jdbc.sql"
schedule => "* * * * *"
type => "gjhz"
}
jdbc {
# mysql jdbc connection string to our backup databse
jdbc_connection_string => "jdbc:mysql://192.168.42.147:3306/test"
# the user we wish to excute our statement as
jdbc_user => "root"
jdbc_password => "root"
# the path to our downloaded jdbc driver
jdbc_driver_library => "/home/hadoop/opt/logstash-2.3.2/mysql-connector-java-5.1.29.jar"
# the name of the driver class for mysql
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement_filepath => "jdbc2.sql"
schedule => "* * * * *"
type => "rjfw"
}
}

filter {
json {
source => "message"
remove_field => ["message"]
}
}

output {
if [type] == "gjhz"{
elasticsearch {
hosts => "192.168.42.190:9200"
index => "zkr"

document_id => "%{a_id}"
}
}else{
elasticsearch {
hosts => "192.168.42.190:9200"
index => "zkr"
document_id => "%{d_id}"
}
}

stdout {
codec => json_lines
}
}