本想搭建一个 flume+hdfs+kafka+storm+mysql 的日志实时分析和存储的系统,但是flume日志收集这块一直不通,查看flume的日志也没有报错,不知道该怎么解决了,求大家帮帮忙,贴出集群配置和配置文件如下:
共5台机器:node1~node5,其中node3~node5为日志收集的agent,node1~node2为flume的collector,最终存储两份,一份到kafka,一份到hdfs。
agent的配置文件如下:
#def
agent.sources = src_spooldir
agent.channels = file memory
agent.sinks = collector_avro1 collector_avro2
# sources
agent.sources.src_spooldir.type = spooldir
agent.sources.src_spooldir.channels = file memory
agent.sources.src_spooldir.spoolDir = /data/flume/spoolDir
agent.sources.src_spooldir.selector.type = multiplexing
agent.sources.src_spooldir.fileHeader = true
# channels
agent.channels.file.type = file
agent.channels.file.checkpointDir = /data/flume/checkpoint
agent.channels.file.dataDirs = /data/flume/data
agent.channels.memory.type = memory
agent.channels.memory.capacity = 10000
agent.channels.memory.transactionCapacity = 10000
agent.channels.memory.byteCapacityBufferPercentage = 20
agent.channels.memory.byteCapacity = 800000
# sinks
agent.sinks.collector_avro1.type = avro
agent.sinks.collector_avro1.channel = file
agent.sinks.collector_avro1.hostname = node1
agent.sinks.collector_avro1.port = 45456
agent.sinks.collector_avro2.type = avro
agent.sinks.collector_avro2.channel = memory
agent.sinks.collector_avro2.hostname = node2
agent.sinks.collector_avro2.port = 4545
collector端的配置文件如下:
#def
agent.sources = src_avro
agent.channels = file memory
agent.sinks = hdfs kafka
# sources
agent.sources.src_avro.type = avro
agent.sources.src_avro.channels = file memory
agent.sources.src_avro.bind = node1
agent.sources.src_avro.port = 45456
agent.sources.src_avro.selector.type = replicating
# channels
agent.channels.file.type = file
agent.channels.file.checkpointDir = /data/flume/checkpoint
agent.channels.file.dataDirs = /data/flume/data
agent.channels.memory.type = memory
agent.channels.memory.capacity = 10000
agent.channels.memory.transactionCapacity = 10000
agent.channels.memory.byteCapacityBufferPercentage = 20
agent.channels.memory.byteCapacity = 800000
# sinks
agent.sinks.hdfs.type = hdfs
agent.sinks.hdfs.channel = file
agent.sinks.hdfs.hdfs.path = hdfs://node1/flume/events/%y-%m-%d/%H%M/%S
agent.sinks.hdfs.hdfs.filePrefix = log_%Y%m%d_%H
agent.sinks.hdfs.hdfs.fileSuffix = .txt
agent.sinks.hdfs.hdfs.useLocalTimeStamp = true
agent.sinks.hdfs.hdfs.writeFormat = Text
agent.sinks.hdfs.hdfs.rollCount = 0
agent.sinks.hdfs.hdfs.rollSize = 1024
agent.sinks.hdfs.hdfs.rollInterval = 0
agent.sinks.kafka.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.kafka.channel = memory
agent.sinks.kafka.kafka.topic = test
agent.sinks.kafka.kafka.bootstrap.servers = node3:9092,node4:9092,node5:9092
agent.sinks.kafka.kafka.flumeBatchSize = 20
agent.sinks.kafka.kafka.producer.acks = 1
agent.sinks.kafka.kafka.producer.linger.ms = 1
agent.sinks.kafka.kafka.producer.compression.type = snappy
最终 hdfs和kafka都没有接收到数据。