logstash消费kafka输出至mysql

1、安装jdk

若没有安装wget  则执行命令：

yum -y install wget

yum install -y java-1.8.0-openjdk-devel.x86_64

2、安装logstash

解压至 /tmp/package

tar -xvzf logstash-6.4.1.tar.gz

修改配置

测试：

vim ps -aux | grep logstash

input {

stdin{

}

}

output {

stdout{

}

}

进入 bin目录

./logstash -f ../config/logstash-sample.conf

输入helloworld

3、链接mysql

进入 /tmp/package/logstash-6.4.1

mkdir vender

cd vender/

mkdir jar

cd jar/

mkdir jdbc

将mysql放进该文件夹

将插件zip放入文件夹

/tmp/package/plugin

进入 /tmp/package/logstash-6.4.1/bin

./logstash-plugin install file:///tmp/package/plugin/logstash-output-jdbc.zip

进入config

vim syslog.conf

编辑syslog.conf ：

input {

kafka {

bootstrap_servers => "ip:port"

topics => ["audit"]

codec => json {

charset => "UTF-8"

}

}

}

output {

jdbc {

driver_jar_path => "/tmp/package/logstash-6.4.1/vender/jar/jdbc/mysql-connector-java-5.1.47-bin.jar"

driver_class => "com.mysql.jdbc.Driver"

connection_string => "jdbc:mysql://ip:port/crcloudportal?user=&password="

statement => [ "INSERT INTO table (name,age) VALUES(?,?)", "name","age"]

}

}

4、启动

启动logstash

进入目录 /tmp/package/logstash-6.4.1

nohup bin/logstash -f config/syslog.conf >null 2>&1 &

查看端口

ps -aux | grep logstash

netstat -tunlp | grep 9600

netstat -nlp | grep :9600 | awk '{print $7}'