一.Flume架构
执行流程:
概念event:
event将传输的数据进行封装,是flume传输数据的基本单位,如果是文本文件,通常是一行记录,event也是事务的基本单位。event从source,流向channel,再到sink,本身为一个字节数组,并可携带headers(头信息)信息。event代表着一个数据的最小完整单元,从外部数据源来,向外部的目的地去。
二.通过netcat写入到控制台
写入文件example.conf:
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat //可用telnet来测试
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 44444 //访问44444端口就行
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
# Describe the sink
a1.sinks.k1.type = logger //显示到控制台
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
运行命令:
flume-ng agent \
--conf $FLUME_HOME/config \
--conf-file $FLUME_HOME/config/example.conf \
--name a1 \
-Dflume.root.logger=INFO,console
telnet命令:
[hadoop@hadoop000 config]$ telnet localhost 44444
Trying 192.168.198.128...
Connected to localhost.
Escape character is '^]'.
aa //传输字符串aa
OK
bb
OK
cc
OK
显示结果:
21/01/24 01:59:18 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:44444]
21/01/24 01:59:34 INFO sink.LoggerSink: Event: { headers:{} body: 61 61 0D aa. }
21/01/24 01:59:36 INFO sink.LoggerSink: Event: { headers:{} body: 62 62 0D bb. }
21/01/24 01:59:37 INFO sink.LoggerSink: Event: { headers:{} body: 63 63 0D cc. }
可见,每次输入都会构成一个event,event由headers和body构成。body包含数据信息。
三.通过exec写入到hdfs
# Name the components on this agent
exec-hdfs-agent.sources = exec-source
exec-hdfs-agent.sinks = hdfs-sink
exec-hdfs-agent.channels = memory-channel
# Describe/configure the source
exec-hdfs-agent.sources.exec-source.type = exec
exec-hdfs-agent.sources.exec-source.command = tail -F ~/data/data.log
exec-hdfs-agent.sources.exec-source.shell = /bin/bash -c
# Describe the sink
exec-hdfs-agent.sinks.hdfs-sink.type = hdfs
exec-hdfs-agent.sinks.hdfs-sink.hdfs.path = hdfs://hadoop000:8020/data/flume/tail
exec-hdfs-agent.sinks.hdfs-sink.hdfs.fileType=DataStream
exec-hdfs-agent.sinks.hdfs-sink.hdfs.writeFormat=Text
exec-hdfs-agent.sinks.hdfs-sink.hdfs.batchSize=10
# Use a channel which buffers events in memory
exec-hdfs-agent.channels.memory-channel.type = memory
# Bind the source and sink to the channel
exec-hdfs-agent.sources.exec-source.channels = memory-channel
exec-hdfs-agent.sinks.hdfs-sink.channel = memory-channel
运行命令:
flume-ng agent \
--conf $FLUME_HOME/config \
--conf-file $FLUME_HOME/config/exec-hdfs-agent.conf \
--name exec-hdfs-agent \
-Dflume.root.logger=INFO,console
输入数据:
[hadoop@hadoop000 data]$ echo aaa >> data.log
[hadoop@hadoop000 data]$ echo bbb >> data.log
[hadoop@hadoop000 data]$ echo ccc >> data.log
显示结果:
21/01/24 00:32:42 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false
21/01/24 00:32:42 INFO hdfs.BucketWriter: Creating hdfs://hadoop000:8020/data/flume/tail/FlumeData.1611419562576.tmp
21/01/24 00:33:12 INFO hdfs.BucketWriter: Closing hdfs://hadoop000:8020/data/flume/tail/FlumeData.1611419562576.tmp
21/01/24 00:33:12 INFO hdfs.BucketWriter: Renaming hdfs://hadoop000:8020/data/flume/tail/FlumeData.1611419562576.tmp to hdfs://hadoop000:8020/data/flume/tail/FlumeData.1611419562576
21/01/24 00:33:12 INFO hdfs.HDFSEventSink: Writer callback called.
hdfs.rollInterval:每30秒滚动一个文件。可见Creating hdfs到Closing hdfs就是30秒。
文件名为:前缀+文件名+后缀。
运行时文件名为:运行时前缀+前缀+文件名+后缀+运行时后缀。
hdfs.filePrefix:文件前缀为FlumeData。
hdfs.inUsePrefix:运行时FlumeData前面没有前缀。
hdfs.fileSuffix:文件无后缀。
hdfs.inUseSuffix:运行时有后缀tmp。运行完会有Renaming操作,去掉tmp。
HDFS Sink参数:
重要参数:
batchSize:从source写入到channel的批次大小或者sink从channel拉一次的大小。(一批批的写入比一个个写入,效率会增高很多)。一个批次以事务的形式写入,即中途写入失败会回滚。
transactionCapacity:一个事务的最大容量,即上方架构图的List集合的容量,所以batchSize要小于transactionCapacity。
Capacity:channel中最大能存放多少event。
缺点:
缺点一:通过tail -F的linux命令,可靠性不是太好,虽然有cmd命令挂了的restart机制,但是还是不靠谱。
缺点二:只能监控一个文件,不能监控文件夹。
四.Spooling Directory Source
官网解释:
这个source允许您通过将要接收的文件放置到磁盘上的“spooling”目录中来接收数据。该source将监视指定目录中的新文件,并在新文件出现时从新文件中解析事件。事件解析逻辑是可插入的。在将给定的文件完全读入channel后,默认情况下,通过重命名文件来指示完成,或者可以删除该文件,或者使用trackerDir来跟踪已处理的文件。
与Exec source不同,这个source是可靠的,即使Flume重新启动或被杀死,也不会丢失数据。为了获得这种可靠性,必须只将不可变的、唯一命名的文件放入spooling目录。Flume试图检测这些问题条件,如果它们被违反,将会大声失败:
- 如果一个文件在被放置到假脱机目录后被写入,Flume将在其日志文件中打印一个错误并停止处理。
- 如果某个文件名在以后被重用,Flume将在其日志文件中打印一个错误并停止处理。
为了避免上述问题,在将文件名移动到假脱机目录时,为它们添加一个惟一标识符(例如时间戳)可能是有用的。
尽管该source具有可靠性保证,但如果发生某些下游故障,仍然存在事件可能重复的情况。这与其他Flume组件提供的保证是一致的。
上方就是说,不能修改“spooling”目录中的文件,所以只能把整个写好的文件cp到“spooling”目录中,而不能cp到“spooling”目录中再去修改。
# Describe/configure the source
a3.sources.r3.type = spooldir
a3.sources.r3.spoolDir = /tmp/upload
# 给 spoolDir 目录中文件添加的后缀,区分记录与未记录(先记录后改名)
a3.sources.r3.fileSuffix = .COMPLETED //完成后会默认在文件后加上后缀
a3.sources.r3.fileHeader = true
# 忽略所有以.tmp 结尾的文件,不上传
a3.sources.r3.ignorePattern = ([^ ]*\.tmp)
优点:能够监控文件夹。
缺点:如上所示
五.Taildir Source(非常重要)
taildir-memory-logger.conf:
agent.sources = s1
agent.channels = c1
agent.sinks = r1
# 指定source使用的channel
agent.sources.s1.channels = c1
# 指定sink使用的channel
agent.sinks.r1.channel = c1
# source类型
agent.sources.s1.type = TAILDIR
# 元数据位置
agent.sources.s1.positionFile = /home/hadoop/tmp/flume/taildir_position.json
# 监控的目录f1和f2
agent.sources.s1.filegroups = f1 f2
agent.sources.s1.filegroups.f1=/home/hadoop/tmp/flume/test1/example.log
agent.sources.s1.headers.f1.headerKey1 = value1
agent.sources.s1.filegroups.f2=/home/hadoop/tmp/flume/test2/.*log.*
agent.sources.s1.headers.f2.headerKey1 = value2-1
agent.sources.s1.headers.f2.headerKey2 = value2-2
agent.sources.s1.fileHeader = true
agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100
agent.sinks.r1.type = logger
运行命令:
flume-ng agent \
--conf $FLUME_HOME/config \
--conf-file $FLUME_HOME/config/taildir-memory-logger.conf \
--name agent \
-Dflume.root.logger=INFO,console
f1下的example.log和 f2下的1.log,2.log:
[hadoop@hadoop000 test2]$ echo aaa>>1.log
[hadoop@hadoop000 test2]$ echo bbb>>1.log
[hadoop@hadoop000 test2]$ echo ccc>>2.log
[hadoop@hadoop000 test2]$ echo ddd>>2.log
[hadoop@hadoop000 test1]$ echo eee >> example.log
显示结果:
21/01/24 14:15:16 INFO taildir.ReliableTaildirEventReader: headerTable: {f1={headerKey1=value1}, f2={headerKey1=value2-1, headerKey2=value2-2}}
21/01/24 14:15:16 INFO taildir.ReliableTaildirEventReader: Opening file: /home/hadoop/tmp/flume/test1/example.log, inode: 102948027, pos: 0
21/01/24 14:15:16 INFO taildir.ReliableTaildirEventReader: Opening file: /home/hadoop/tmp/flume/test2/1.log, inode: 546644, pos: 0
21/01/24 14:15:16 INFO taildir.ReliableTaildirEventReader: Opening file: /home/hadoop/tmp/flume/test2/2.log, inode: 546645, pos: 0
21/01/24 14:15:16 INFO taildir.ReliableTaildirEventReader: Updating position from position file: /home/hadoop/tmp/flume/taildir_position.json
21/01/24 14:15:16 INFO taildir.ReliableTaildirEventReader: File not found: /home/hadoop/tmp/flume/taildir_position.json, not updating position
21/01/24 14:15:16 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: s1: Successfully registered new MBean.
21/01/24 14:15:16 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: s1 started
21/01/24 14:15:38 INFO sink.LoggerSink: Event: { headers:{headerKey1=value2-1, headerKey2=value2-2, file=/home/hadoop/tmp/flume/test2/1.log} body: 61 61 61 aaa }
21/01/24 14:15:43 INFO sink.LoggerSink: Event: { headers:{headerKey1=value2-1, headerKey2=value2-2, file=/home/hadoop/tmp/flume/test2/1.log} body: 62 62 62 bbb }
21/01/24 14:15:49 INFO sink.LoggerSink: Event: { headers:{headerKey1=value2-1, headerKey2=value2-2, file=/home/hadoop/tmp/flume/test2/2.log} body: 63 63 63 ccc }
21/01/24 14:15:53 INFO sink.LoggerSink: Event: { headers:{headerKey1=value2-1, headerKey2=value2-2, file=/home/hadoop/tmp/flume/test2/2.log} body: 64 64 64 ddd }
21/01/24 14:16:15 INFO sink.LoggerSink: Event: { headers:{headerKey1=value1, file=/home/hadoop/tmp/flume/test1/example.log} body: 65 65 65 eee }
21/01/24 14:17:47 INFO taildir.TaildirSource: Closed file: /home/hadoop/tmp/flume/test2/1.log, inode: 546644, pos: 8
21/01/24 14:17:57 INFO taildir.TaildirSource: Closed file: /home/hadoop/tmp/flume/test2/2.log, inode: 546645, pos: 8
21/01/24 14:18:17 INFO taildir.TaildirSource: Closed file: /home/hadoop/tmp/flume/test1/example.log, inode: 102948027, pos: 4
结果会显示偏移量,用来记录消费到的位子,就算Flume挂了,依然有数据进入文件,Flume重启后依然会从挂了的偏移量开始消费。不会丢失数据。
查看偏移量:
[hadoop@hadoop000 flume]$ cat /home/hadoop/tmp/flume/taildir_position.json
//结果:
[{"inode":102948027,"pos":4,"file":"/home/hadoop/tmp/flume/test1/example.log"},
{"inode":546644,"pos":8,"file":"/home/hadoop/tmp/flume/test2/1.log"},
{"inode":546645,"pos":8,"file":"/home/hadoop/tmp/flume/test2/2.log"}]
echo eee >> example.log 后 example.log的偏移量就为4。