Spark实训3-1 flume数据采集和清洗

河南工业大学人工智能与大数据学院

对于该项目首先将不太干净的数据上传至本地虚拟机，本人用xftp上传

之后，需要转到flume的conf目录编写配置文件,我的路径是/usr/local/flume/conf

编写flume的conf文件,假设命名为flume_test.conf ,用vim编辑:vim flume_test.conf

内容如下:

#Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe the source
a1.sources.r1.type = TAILDIR
#保存的是采集进度，方便记录采集进度，不要也行(建议要)
a1.sources.r1.positionFile = /home/hadoop/taildir_position.json
a1.sources.r1.filegroups = f1
#采集文件的名称
a1.sources.r1.filegroups.f1 = /home/hadoop/15.1-17.2.txt               
a1.sources.r1.batchSize = 1000
a1.sources.r1.pollDelay = 5000
a1.sources.r1.decodeErrorPolicy = IGNORE
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = timestamp
# Describe the sink
a1.sinks.k1.type = file_roll
#存放到的文件目录
a1.sinks.k1.sink.directory = /home/hadoop
a1.sinks.k1.sink.rollInterval = 0
a1.sinks.k1.sink.rollSize = 0
a1.sinks.k1.sink.rollCount = 10
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 1000
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动之前最好多开几个窗口，数据如果过大，可能就采集过滤不完整

启动flume的命令:

flume-ng agent --name a1 --conf conf --conf-file文件路径

对于我就是:

flume-ng agent --name a1 --conf conf --conf-file /home/hadoop/flume/conf/flume_test.conf

 

采集完成后，按ctrl+z退出

最终会在/home/hadoop目录下产生一个名字一般是数字的文件,查看文件是否生成

 查看文件内容:cat 1686106789599-1

文件重命名:mv 1686106789599-1 spark_1.txt

其它数据异曲同工,过滤器改改即可