1、下载官方Logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
2、配置logstash-input-jdbc插件环境
#查看gem环境
gem
RubyGems is a sophisticated package manager for Ruby. This is a
basic help message containing pointers to more information.
Usage:
gem -h/--help
gem -v/--version
gem command [arguments...] [options...]
Examples:
gem install rake
gem list --local
gem build package.gemspec
gem help install
Further help:
gem help commands list all 'gem' commands
gem help examples show some examples of usage
gem help platforms show information about platforms
gem help <COMMAND> show help on COMMAND
(e.g. 'gem help install')
gem server present a web page at
http://localhost:8808/
with info about installed gems
Further information:
http://guides.rubygems.org
显示出gem帮助信息,说明已经安装。
3、如果没有安装gem
yum install gem
4、gem源修改
gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/
5、查看gem源
gem sources -l
6、修改logstash目录中的Gemfile文件
vim Gemfile
source "https://gems.ruby-china.org/"
7、修改logstash目录中的Gemfile.lock文件
vim Gemfile.lock
remote https://gems.ruby-china.org/
8、安装gem bundler
gem install bundler
9、安装logstash-input-jdbc插件
bin/logstash-plugin install logstash-input-jdbc
10、上传数据库驱动文件mysql-connector-java-5.1.44.jar到logstash目录。
11、编写logstash-input-jdbc-mysql.conf文件
input {
jdbc {
jdbc_driver_library => "/usr/elasticsearch/mysql-connector-java-5.1.44.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://IP:3306/xxx"
jdbc_user => "xxx"
jdbc_password => "xxx"
# sql 语句文件
# statement_filepath => "filename.sql"
statement => "SELECT * from sys_user_log where record_date> :sql_last_value"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
type => "jdbc"
tracking_column => "update_date"
use_column_value => false
# 设置监听间隔 各字段含义(由左至右)分、时、天、月、年,全部为*默认含义为每分钟都更新
schedule => "* * * * *"
}
}
output {
stdout {
codec => json_lines
}
elasticsearch_http {
host => "192.168.1.210:9200"
index => "sys_user_log"
}
}
需要注意,网上给出的测试例子有问题,会让初学者刚开始使用误认为同步组件出错了。类似问题
12、执行同步
/bin/logstash -f logstash-input-jdbc-mysql.conf