logstash-out-hdfs

logstash从kafka拉取数据，并插入到hdfs中

配置文件如下：

input {
  kafka {
    type => "test"
    auto_offset_reset => "smallest"
    group_id => "mon2"
    topic_id => "db100"
    zk_connect => "192.168.199.6:2181,192.168.199.7:2181,192.168.199.8:2181"
  }
}


output {
  webhdfs {
    server => "192.168.199.6:50070"        
    path => "/logstash/dt=%{+YYYY-MM-dd}/logstash-%{+HH}.log"  
    user => "root"                      
  }
}

注意：

需要安装插件。bin/logstash-plugin install logstash-output-webhdfs