flume配置hdfs sink时没有按照滚动规则滚动生成文件

flume配置hdfs sink时没有按照滚动规则滚动生成文件

问题描述

# ------------------- 定义数据流----------------------
# source的名字
flume2HDFS_agent.sources = source_from_kafka  
# channels的名字，建议按照type来命名
flume2HDFS_agent.channels = mem_channel  
# sink的名字，建议按照目标来命名
flume2HDFS_agent.sinks = hdfs_sink  

#auto.commit.enable = true  

## kerberos config ##  
#flume2HDFS_agent.sinks.hdfs_sink.hdfs.kerberosPrincipal = flume/datanode2.hdfs.alpha.com@OMGHADOOP.COM  
#flume2HDFS_agent.sinks.hdfs_sink.hdfs.kerberosKeytab = /root/apache-flume-1.6.0-bin/conf/flume.keytab   


#-------- kafkaSource相关配置-----------------
# 定义消息源类型
# For each one of the sources, the type is defined  
flume2HDFS_agent.sources.source_from_kafka.type = org.apache.flume.source.kafka.KafkaSource  
flume2HDFS_agent.sources.source_from_kafka.channels = mem_channel  
flume2HDFS_agent.sources.source_from_kafka.batchSize = 5000  

# 定义kafka所在的地址  
flume2HDFS_agent.sources.source_from_kafka.kafka.bootstrap.servers = localhost:6667

# 配置消费的kafka topic
flume2HDFS_agent.sources.source_from_kafka.kafka.topics = spark2hdfs

# 配置消费的kafka groupid
flume2HDFS_agent.sources.source_from_kafka.kafka.consumer.group.id = 222

#---------hdfsSink 相关配置------------------
# The channel can be defined as follows.  
flume2HDFS_agent.sinks.hdfs_sink.type = hdfs 
# 指定sink需要使用的channel的名字,注意这里是channel
#Specify the channel the sink should use  
flume2HDFS_agent.sinks.hdfs_sink.channel = mem_channel 
flume2HDFS_agent.sinks.hdfs_sink.hdfs.filePrefix = %Y-%m-%d 
flume2HDFS_agent.sinks.hdfs_sink.hdfs.fileSuffix = .log
flume2HDFS_agent.sinks.hdfs_sink.hdfs.path = hdfs://localhost:8020/环境数据监测/大气环境数据/空气加密站数据/2021/
flume2HDFS_agent.sinks.hdfs_sink.hdfs.rollSize = 0    
flume2HDFS_agent.sinks.hdfs_sink.hdfs.rollCount = 0  
#设置滚动时间1天
flume2HDFS_agent.sinks.hdfs_sink.hdfs.rollInterval = 86400 
flume2HDFS_agent.sinks.hdfs_sink.hdfs.threadsPoolSize = 30
#flume2HDFS_agent.sinks.hdfs_sink.hdfs.codeC = gzip  
#flume2HDFS_agent.sinks.hdfs_sink.hdfs.fileType = CompressedStream  
flume2HDFS_agent.sinks.hdfs_sink.hdfs.fileType=DataStream    
flume2HDFS_agent.sinks.hdfs_sink.hdfs.writeFormat=Text   

#------- memoryChannel相关配置-------------------------
# channel类型
# Each channel's type is defined.  
flume2HDFS_agent.channels.mem_channel.type = memory  
# Other config values specific to each type of channel(sink or source)  
# can be defined as well  
# channel存储的事件容量
# In this case, it specifies the capacity of the memory channel  
flume2HDFS_agent.channels.mem_channel.capacity = 100000  
# 事务容量
flume2HDFS_agent.channels.mem_channel.transactionCapacity = 10000

效错误果展示

生成很多小文件，并没有按照我配置的1天进行滚动

解决办法

给hdfssink加两条配置

a1.sinks.s1.hdfs.useLocalTimeStamp=true 
a1.sinks.s1.hdfs.minBlockReplicas=1