kafka消息通过logstash导入HDFS

1.安装Logstash
1、下载安装包
2、解压
tar -xzf 文件名
3、下载插件logstash-output-webhdfs-master
4、解压到logstash
5、 安装Gem并更新

# yum install -y gem
# gem -v
# gem update --system
# gem -v

6、编辑Gemfile
添加：gem “logstash-output-webhdfs”, :path => “/home/es/logstash-6.4.2/logstash-output-webhdfs-master”
7、到logstash 目录下执行
bin / logstash-plugin install logstash-output-webhdf --no-verify
./bin/logstash-plugin install --local logstash-output-webhdfs
8、配置/logstash-6.4.2/config 中的 kafka-logstash-hdfs.conf文件：

input {
    kafka {
        bootstrap_servers => ["10.2.5.37:9092"]  //集群地址
        group_id => "logstash-group1"  
                topics => ["ISEVehicleToBMS"] //主题
                consumer_threads => 1
                decorate_events => true
                codec => "json"
                }
}
output {