flume与kafka的整合
第一步:配置flume的conf文件
TaildirSource-kafka.conf
agent1.sources = source1agent1.sinks = sink1agent1.channels = channel1
agent1.sources.source1.type = TAILDIRagent1.sources.source1.positionFile = /home/hadoop/bigdatasoftware/datas/nginx/taildir_position.jsonagent1.sources.source1.filegroups = f1agent1.sources.source1.filegroups.f1 =/home/hadoop/bigdatasoftware/datas/nginx/access_*.log
agent1.sources.source1.interceptors = i1agent1.sources.source1.interceptors.i1.type = hostagent1.sources.source1.interceptors.i1.hostHeader = hostname
#配置sink组件为hdfsagent1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSinkagent1.sinks.sink1.kafka.topic = thirdagent1.sinks.sink1.kafka.bootstrap.servers = hadoop-001:9092,hadoop-002:9092,hadoop-003:9092agent1.sinks.sink1.kafka.flumeBatchSize = 20agent1.sinks.sink1.kafka.producer.acks = 1
#指定文件名前缀agent1.sinks.sink1.hdfs.filePrefix = access_log#指定每批下沉数据的记录条数agent1.sinks.sink1.hdfs.batchSize= 100agent1.sinks.sink1.hdfs.fileType = DataStreamagent1.sinks.sink1.hdfs.writeFormat =Text#指定下沉文件按1G大小滚动agent1.sinks.sink1.hdfs.rollSize = 1024*1024*1024#指定下沉文件按1000000条数滚动agent1.sinks.sink1.hdfs.rollCount = 1000000#指定下沉文件按30分钟滚动#agent1.sinks.sink1.hdfs.rollInterval = 30#agent1.sinks.sink1.hdfs.round = trueagent1.sinks.sink1.hdfs.roundValue = 1#agent1.sinks.sink1.hdfs.roundUnit = minuteagent1.sinks.sink1.hdfs.useLocalTimeStamp = true
#使用memory类型channelagent1.channels.channel1.type = memoryagent1.channels.channel1.capacity = 500000
agent1.channels.channel1.transactionCapacity = 600
# Bind the source and sink to the channelagent1.sources.source1.channels = channel1agent1.sinks.sink1.channel = channel1
启动flume
bin/flume-ng agent --conf conf --conf-file conf/TaildirSource-kafka.conf --name agent1 -Dflume.root.logger=INFO,console
创建消费者
kafka-console-consumer.sh --zookeeper hadoop-001:2181 --from-beginning --topic third
对access.log文件增加内容
![](https://i-blog.csdnimg.cn/blog_migrate/91acdbfb7bd36cadfd5cfbe5d452e4f7.png)
消费者中会出现消息