flume 读取数据并写入到hdfs

1.在/root/software/flume160/conf/kb11job 下
vi userfriend-hdfs.conf

user_friends.sources=userFriendsSource
user_friends.channels=userFriendsChannel
user_friends.sinks=userFriendsSink

user_friends.sources.userFriendsSource.type=spooldir
#文件读取路径
user_friends.sources.userFriendsSource.spoolDir=/opt/flumelogfile/userfriend_hdfs
user_friends.sources.userFriendsSource.deserializer=LINE
#每行读取的最大长度
user_friends.sources.userFriendsSource.deserializer.maxLineLength=32000
#文件名格式：events_2021-05-24.csv
user_friends.sources.userFriendsSource.includePattern=userfriend_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv

user_friends.channels.userFriendsChannel.type=file
#检查点路径
user_friends.channels.userFriendsChannel.checkpointDir=/opt/flumelogfile/checkpoint/userfriend_hdfs
user_friends.channels.userFriendsChannel.dataDirs=/opt/flumelogfile/data/userfriend_hdfs

#sink 输出类型hdfs
user_friends.sinks.userFriendsSink.type=hdfs
#设置文件类型，可支持压缩
user_friends.sinks.userFriendsSink.hdfs.fileType=DataStrea
#上传文件的前缀
user_friends.sinks.userFriendsSink.hdfs.filePrefix=userfriend
#上传文件的后缀
user_friends.sinks.userFriendsSink.hdfs.fileSuffix=.csv
#hdfs路径
user_friends.sinks.userFriendsSink.hdfs.path=hdfs://192.168.107.103:9000/kb11file/userfriend/%Y-%m-%d
#是否使用本地时间戳
user_friends.sinks.userFriendsSink.hdfs.useLocalTimeStamp=true
#积攒对少个Event才flush到hdfs一次
user_friends.sinks.userFriendsSink.hdfs.batchSize=640
#文件的滚动与Event数量无关
user_friends.sinks.userFriendsSink.hdfs.rollCount=0
#设置每个文件的滚动大小
user_friends.sinks.userFriendsSink.hdfs.rollSize=6400000
#多久生成一个新的文件
user_friends.sinks.userFriendsSink.hdfs.rollInterval=30

user_friends.sources.userFriendsSource.channels=userFriendsChannel
user_friends.sinks.userFriendsSink.channel=userFriendsChannel

2.创建三个文件夹
1）新建读取文件的路径 并 将文件放入目录中
mkdir -p /opt/flumelogfile/userfriend_hdfs
cp /opt/flumelogfile/user_friends.csv /opt/flumelogfile/userfriend_hdfs/userfriend_2015-05-23.csv
2)新建checkpoint路径
/opt/flumelogfile/checkpoint/userfriend_hdfs
3)新建data路径
/opt/flumelogfile/data/userfriend_hdfs

3.执行命令（在flume160下）
./bin/flume-ng agent --name user_friends --conf ./conf/ --conf-file ./conf/kb11job/userfriend-hdfs.conf -Dflume.root.logger=INFO,console

4.执行完 查看hdfs