Flume之——配置案例

转载请注明出处：https://blog.csdn.net/l1028386804/article/details/97934236

• 监听telnet 44444端口

myagent.sources = r1
myagent.sinks = k1
myagent.channels = c1
# Describe/configure the source
myagent.sources.r1.type = netcat
myagent.sources.r1.bind = 192.168.175.100
myagent.sources.r1.port = 44444
# Describe the sink
myagent.sinks.k1.type = logger
# Use a channel which buffers events in memory
myagent.channels.c1.type = memory
myagent.channels.c1.capacity = 1000
myagent.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
myagent.sources.r1.channels = c1
myagent.sinks.k1.channel = c1

之后在命令行输入telnet 192.168.175.100 44444命令，连接成功后，输入数据，会在Flume命令行打印出来。

• 基于内存的Channel

# 定义 agent
myagent.sources = s1
myagent.channels = c1
myagent.sinks = k1
# 定义 sources
myagent.sources.s1.batchsize=10
myagent.sources.s1.type = exec
myagent.sources.s1.command = tail -F /usr/local/nginx-1.17.2/logs/access.log
# 定义 channels
myagent.channels.c1.type = memory
myagent.channels.c1.capacity = 100
myagent.channels.c1.transactionCapacity = 20
# 定义 sinks
myagent.sinks.k1.type = logger
# bind source and sink to the channel
myagent.sources.s1.channels = c1
myagent.sinks.k1.channel = c1

• 基于文件的Channel

# 定义 agent
myagent.sources = s1
myagent.channels = c1
myagent.sinks = k1
# 定义 sources
myagent.sources.s1.batchsize=10
myagent.sources.s1.type = exec
myagent.sources.s1.command = tail -F /usr/local/nginx-1.17.2/logs/access.log
# 定义 channels
myagent.channels.c1.type = file
myagent.channels.c1.checkpointDir = /usr/local/flume-1.9.0/file/check
myagent.channels.c1.dataDirs = /usr/local/flume-1.9.0/file/data
myagent.channels.c1.transactionCapacity = 20
# 定义 sinks
myagent.sinks.k1.type = logger
# bind source and sink to the channel
myagent.sources.s1.channels = c1
myagent.sinks.k1.channel = c1

• 基于目录的Channel

配置Flume监听某个目录，当目录中有新文件创建时，Flume将文件写入Channel，之后会在文件名后加一个后缀.COMPLETED，同时，配置中会利用正则表达式忽略对临时文件的监听。

# 定义 agent
myagent.sources = s1
myagent.channels = c1
myagent.sinks = k1
# 定义 sources
myagent.sources.s1.batchsize=10
myagent.sources.s1.type = spooldir
#指定监听目录
myagent.sources.s1.spoolDir = /usr/local/flume-1.9.0/spooling
myagent.sources.s1.ignorePattern = ([^ ]*\.tmp$)
# 定义 channels
myagent.channels.c1.type = memory
myagent.channels.c1.capacity = 200
myagent.channels.c1.transactionCapacity = 150
# 定义 sinks
myagent.sinks.k1.type = logger
# bind source and sink to the channel
myagent.sources.s1.channels = c1
myagent.sinks.k1.channel = c1

• Flume写数据到HDFS

# 配置Agent
myagent.sources = r1
myagent.sinks = k1
myagent.channels = c1
# 配置Source
myagent.sources.r1.type = exec
myagent.sources.r1.channels = c1
myagent.sources.r1.deserializer.outputCharset = UTF-8
# 配置需要监控的日志输出目录
myagent.sources.r1.command = tail -F /usr/local/nginx-1.17.2/logs/access.log
# 配置Sink
myagent.sinks.k1.type = hdfs
myagent.sinks.k1.channel = c1
myagent.sinks.k1.hdfs.useLocalTimeStamp = true
myagent.sinks.k1.hdfs.path = hdfs://binghe100:9000/flume/events/%Y-%m
myagent.sinks.k1.hdfs.filePrefix = %Y-%m-%d-%H
myagent.sinks.k1.hdfs.fileSuffix = .log
myagent.sinks.k1.hdfs.minBlockReplicas = 1
myagent.sinks.k1.hdfs.fileType = DataStream
myagent.sinks.k1.hdfs.writeFormat = Text
myagent.sinks.k1.hdfs.rollInterval = 86400
myagent.sinks.k1.hdfs.rollSize = 1000000
myagent.sinks.k1.hdfs.rollCount = 10000
myagent.sinks.k1.hdfs.idleTimeout = 0
# 配置Channel
myagent.channels.c1.type = memory
myagent.channels.c1.capacity = 1000
myagent.channels.c1.transactionCapacity = 100
# 将三者连接
myagent.sources.r1.channel = c1
myagent.sinks.k1.channel = c1

• Flume写数据到Kafka

# 配置Agent
myagent.sources = r1
myagent.sinks = k1
myagent.channels = c1
# 配置Source
myagent.sources.r1.type = exec
myagent.sources.r1.command = tail -F /usr/local/nginx-1.17.2/logs/access.log
# 配置Sink
#myagent.sinks.k1.type = logger
myagent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
myagent.sinks.k1.topic = mytopic
myagent.sinks.k1.brokerList = 192.168.175.100:9092
myagent.sinks.k1.requiredAcks = 1
myagent.sinks.k1.batchSize = 20
myagent.channels.c1.type = memory
myagent.channels.c1.capacity = 1000
myagent.channels.c1.transactionCapacity = 100
# 将三者连接
myagent.sources.r1.channels = c1
myagent.sinks.k1.channel = c1

启动Flume命令如下：

flume-ng agent --conf conf --conf-file flume-xxx.conf --name myagent -Dflume.root.logger=INFO,console