Kafka 与flume的整合

flume与kafka的整合

第一步：配置flume的conf文件

TaildirSource-kafka.conf

---

agent1.sources = source1agent1.sinks = sink1agent1.channels = channel1

agent1.sources.source1.type = TAILDIRagent1.sources.source1.positionFile = /home/hadoop/bigdatasoftware/datas/nginx/taildir_position.jsonagent1.sources.source1.filegroups = f1agent1.sources.source1.filegroups.f1 =/home/hadoop/bigdatasoftware/datas/nginx/access_*.log

agent1.sources.source1.interceptors = i1agent1.sources.source1.interceptors.i1.type = hostagent1.sources.source1.interceptors.i1.hostHeader = hostname

#配置sink组件为hdfsagent1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSinkagent1.sinks.sink1.kafka.topic = thirdagent1.sinks.sink1.kafka.bootstrap.servers = hadoop-001:9092,hadoop-002:9092,hadoop-003:9092agent1.sinks.sink1.kafka.flumeBatchSize = 20agent1.sinks.sink1.kafka.producer.acks = 1

#指定文件名前缀agent1.sinks.sink1.hdfs.filePrefix = access_log#指定每批下沉数据的记录条数agent1.sinks.sink1.hdfs.batchSize= 100agent1.sinks.sink1.hdfs.fileType = DataStreamagent1.sinks.sink1.hdfs.writeFormat =Text#指定下沉文件按1G大小滚动agent1.sinks.sink1.hdfs.rollSize = 1024*1024*1024#指定下沉文件按1000000条数滚动agent1.sinks.sink1.hdfs.rollCount = 1000000#指定下沉文件按30分钟滚动#agent1.sinks.sink1.hdfs.rollInterval = 30#agent1.sinks.sink1.hdfs.round = trueagent1.sinks.sink1.hdfs.roundValue = 1#agent1.sinks.sink1.hdfs.roundUnit = minuteagent1.sinks.sink1.hdfs.useLocalTimeStamp = true

#使用memory类型channelagent1.channels.channel1.type = memoryagent1.channels.channel1.capacity = 500000

agent1.channels.channel1.transactionCapacity = 600

# Bind the source and sink to the channelagent1.sources.source1.channels = channel1agent1.sinks.sink1.channel = channel1

---

启动flume

---

bin/flume-ng agent --conf conf --conf-file conf/TaildirSource-kafka.conf --name agent1 -Dflume.root.logger=INFO,console

---

创建消费者

---

kafka-console-consumer.sh --zookeeper hadoop-001:2181 --from-beginning --topic third

---

对access.log文件增加内容

消费者中会出现消息