一、任务描述:将本地目录~/testdata/logs.件夹下的所有.本.件通过Kafka Sink写入Kafka中的flume topic(topic名称为:flume-topic)
数据流
~/testdata/logs -> flume -> kafka
二、版本信息:
flume:1.7.0
zookeeper:3.4.5
kafka:2.10-0.10.1.1
节点数:3
三、相关配置
flume agent配置
LogAgent.sources = mysource LogAgent.channels = mychannel LogAgent.sinks = mysink LogAgent.sources.mysource.type = spooldir LogAgent.sources.mysource.channels = mychannel LogAgent.sources.mysource.spoolDir =/home/zkpk/testdata/logs LogAgent.sinks.mysink.channel = mychannel LogAgent.sinks.mysink.type = org.apache.flume.sink.kafka.KafkaSink LogAgent.sinks.mysink.kafka.bootstrap.servers=master:9092,slave1:9092,slave2:9092 LogAgent.sinks.mysink.kafka.topic=flume-topic LogAgent.sinks.mysink.kafka.flumeBatchSize=20 LogAgent.sinks.mysink.kafka.producer.acks=1 Lo