Logstash--与Kafka集成

采集文件到kafka

• 注意：
  • 启动zookeeper、kafka要用root用户

vim usercase/output-kafka.json

input {
        file{
        path => "/home/es/tomcat.log"
        type => "log"
        start_position => "beginning"
    }
}
output {
    kafka {
        topic_id => "bigdata"
        bootstrap_servers => "node-01:9092,node-02:9092,node-03:9092"
        batch_size => 5
   }
}

采集kafka数据到Es

vim usercase/kafka-es.json

input{
    kafka {
        group_id => "testLogstash"
        auto_offset_reset => "earliest"
        topics => ["bigdata"]
        bootstrap_servers => "node-01:9092,node-02:9092,node-03:9092"
   }
}
output {
    elasticsearch {
        hosts => ["node-01:9200"]
        index => "kakfatoes"
    }
}