Flume简单使用

版权声明：未经允许，随意转载，请附上本文链接谢谢（づ￣3￣）づ╭❤～
https://blog.csdn.net/xiaoduan_/article/details/79773883

Flume简单使用

netcat-memory-logger 模式配置（监听某个端口并且通过内存缓冲后输出为日志）

1. 安装目录conf文件夹新建一个配置文件（便于管理）
   # vim netcat-memory-logger.conf
   写入如下内容

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

1. 启动
   启动命令为

flume-ng agent \
--name a1  \
--conf $FLUME_HOME/conf  \
--conf-file $FLUME_HOME/conf/netcat-memory-logger.conf \
-Dflume.root.logger=INFO,console

启动后该窗口不要关闭

1. 测试
   使用telnet命令
   telnet localhost 44444
   然后输入消息 如果前面的窗口可以接受到就是成功了

exec-memory-logger 模式配置（监听某个文件并且通过内存缓冲后输出为日志）

1. 安装目录conf文件夹新建一个配置文件（便于管理）
   # vim exec-memory-logger.conf
   写入如下内容

#Agent选型：exec source + memory channel +logger sink
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /Users/duanjiaxing/data/flume-test.log
a1.sources.r1.shell = /bin/sh -c

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

1. 启动
   启动命令为

flume-ng agent \
--name a1  \
--conf $FLUME_HOME/conf  \
--conf-file $FLUME_HOME/conf/exec-memory-logger.conf \
-Dflume.root.logger=INFO,console

启动后该窗口不要关闭

1. 测试
   使用telnet命令
   echo hello >> flume-test.log
   然后输入消息 如果前面的窗口接受到 hello 就是成功了

exec-memory-avro + avro-memory-logger 模式配置（两个Flume对接，将一台机器上的Flume的输出作为另一个Flume的输入）

1. 安装目录conf文件夹新建两个配置文件（便于管理）
   # vim exec-memory-avro.conf
# vim avro-memory-logger.conf

exec-memory-avro.conf写入如下内容

# Name the components on this agent
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel


# Describe/configure the source
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /Users/duanjiaxing/data/flume-test.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c


# Describe the sink
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = localhost
exec-memory-avro.sinks.avro-sink.port = 44444


# Use a channel which buffers events in memory
exec-memory-avro.channels.memory-channel.type = memory


# Bind the source and sink to the channel
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

avro-memory-logger.conf 写入如下内容

# Name the components on this agent
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel

# Describe/configure the source
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /Users/duanjiaxing/data/flume-test.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c

# Describe the sink
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = localhost
exec-memory-avro.sinks.avro-sink.port = 44444

# Use a channel which buffers events in memory
exec-memory-avro.channels.memory-channel.type = memory

# Bind the source and sink to the channel
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

1. 启动
   
   1. 先启动avro-memory-logger

flume-ng agent \
--name avro-memory-logger  \
--conf $FLUME_HOME/conf  \
--conf-file $FLUME_HOME/conf/avro-memory-logger.conf \
-Dflume.root.logger=INFO,console

1. 再启动exec-memory-avro.conf

flume-ng agent \
--name exec-memory-avro  \
--conf $FLUME_HOME/conf  \
--conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
-Dflume.root.logger=INFO,console

1. 测试
   使用telnet命令
   echo hello223 >> flume-test.log
   然后输入消息 如果avro-memory-logger窗口接受到 hello223 就是成功了