frume avro支持断点续传 实现从centos虚拟机 传文件到hdfs上

新建配置文件:/opt/software/flume190/flume-conf-files/test_avro_file_hdfs.conf

#组件声明
a1.sources = s1
a1.channels = c1
a1.sinks = k1

#初始化数据源
a1.sources.s1.type = avro	#avro文件类型
a1.sources.s1.bind = 192.168.171.200	#ip
a1.sources.s1.port = 7777	#端口号 和命令对应
a1.sources.s1.threads = 5	#线程数


#初始化通道
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /opt/software/flume190/mydata/checkpoint
a1.channels.dataDirs = /opt/software/flume190/mydata/data
a1.channels.capacity = 100000
a1.channels.transactionCapacity = 10000

#初始化数据槽
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://192.168.171.200:9820/flume/events/avroevent/%Y-%m-%d/%H
a1.sinks.k1.hdfs.round=true
a1.sinks.k1.hdfs.roundValue=10
a1.sinks.k1.hdfs.roundUnit=minute
a1.sinks.k1.hdfs.filePrefix=log_%Y%m%d_%H
a1.sinks.k1.hdfs.fileSuffix= .log
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.sinks.k1.hdfs.writeFormat=Text
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.rollSize=134217728
a1.sinks.k1.hdfs.rollInterval=0
a1.sinks.k1.hdfs.batchSize=100
a1.sinks.k1.hdfs.threadsPoolSize=4
a1.sinks.k1.hdfs.idleTimeout=0
a1.sinks.k1.hdfs.minBlockReplicas=1

#关联组件
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1

运行命令2个:

第一个开服务侦听: 后面的文件路径是配置文件路径

flume-ng agent --name a1 --conf /opt/software/flume190/conf/ --conf-file /opt/software/flume190/flume-conf-files/test_avro_file_hdfs.conf -Dflume.root.logger=INFO,console

第2个命令 后面是虚拟机文件路径

flume-ng avro-client -H 192.168.171.200 -p 7777 -c /conf -F /root/data/flume/prohead1000.log

测试

可以到ip:9870 上查看文件