logstash安装

​
 

下载
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.13.1-linux-x86_64.tar.gz

解压
tar -zxvf logstash-7.13.1-linux-x86_64.tar.gz -C ./

修改允许外网访问和启动端口
http.host: 0.0.0.0
http.port: 8880

启动
./bin/logstash -e 'input { stdin { } } output { stdout { } }'

下载mysql-connector-java
wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.18/mysql-connector-java-8.0.18.jar

创建mysql配置文件夹
mkdir ./mysql_config

生成数据连接配置文件
vi ./mysql_config/jdbc.conf
写入
input {
  jdbc {
    jdbc_driver_library => "/mnt/main_disk/elastic_search/mysql-connector-java-8.0.18.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://******:****?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true"
    jdbc_user => "root"
    jdbc_password => "******"
    schedule => "* * * * *"
    statement => "SELECT CONTENT_ID id,TITLE title ,SUMMARY_MEDIUM description FROM CCMS_CONTENT_PROGRAM"
    use_column_value => true
    tracking_column_type => "timestamp"
    tracking_column => "update_time"
    last_run_metadata_path => "syncpoint_table"
  }
}
output {
    elasticsearch {
        # ES的IP地址及端口
        hosts => ["111.111.111.111:8877"]
        # 索引名称 可自定义
        index => "program"
        # 需要关联的数据库中有有一个id字段，对应类型中的id
        document_id => "%{id}"
    }
    stdout {
        # JSON格式输出
        codec => json_lines
    }
}

测试配置是否正确
./bin/logstash -f ./mysql_config/jdbc.conf -t

通过后启动，提前做好索引的类型设计，有中文需要对中文做IK分词设置
./bin/logstash -f ./mysql_config/jdbc.conf