flume-笔记（一）

1. 安装地址
   1. Flume 官网地：Welcome to Apache Flume — Apache Flume
   2. 文档查看地址：Flume 1.9.0 User Guide — Apache Flume
   3. 下载地址：Index of /dist/flume

安装

解压：tar -zxvf apache-flume-1.8.0-bin.tar.gz -C /usr/local/src/

重命名：mv /usr/local/src/apache-flume-1.8.0-bin/ /usr/local/src/flume

删除jar：rm -rf /usr/local/src/flume/lib/guava-11.0.2.jar

将 lib 文件夹下的guava-11.0.2.jar 删除以兼容 Hadoop 3.1.3

监控端口数据官方案例

# Name the components on this agent 
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source 
a1.sources.r1.type = netcat 
a1.sources.r1.bind = localhost 
a1.sources.r1.port = 44444

# Describe the sink 
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory 
a1.channels.c1.type = memory 
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel 
a1.sources.r1.channels = c1 
a1.sinks.k1.channel = c1

# 为该代理上的组件命名
# Name the components on this agent
a1.sources = r1    # r1:表示a1的Source的名称
a1.sinks = k1    # k1:表示a1的Sink的名称
a1.channels = c1  # c1:表示a1的Channel的名称

# 描述/配置源
# Describe/configure the source
a1.sources.r1.type = netcat      # 表示a1的输入源类型为netcat端口类型
a1.sources.r1.bind = localhost    # 表示a1的监听的主机
a1.sources.r1.prot = 44444      # 表示a1的监听的端口号

# 描述了沉
# Describe the sink
a1.sinks.k1.type = logger      # 表示a1的输出目地是控制台logger类型

# 使用一个在内存中缓冲事件的通道
# Use a channel which buffers events in memory
a1.channels.c1.type = memory        # 表示a1的channel类型是memory内存型
a1.channels.c1.capacity = 1000        # 表示a1的channel总容量1000个event
a1.channels.c1.transactionCapacity = 100  # 表示a1的channel传输时收集到了100条event以后再去提交事务

# 将源和接收器绑定到通道
# Bind the source and sink to the channel
a1.sources.r1.channels = c1    # 表示将r1和c1连接起来
a1.sinks.k1.channel = c1    # 表示将k1和c1连接起来

 在flume下创建文件夹job并进入文件夹

mkdir /usr/local/src/flume/job/
cd /usr/local/src/flume/job/

 创建 Flume Agent配置相关文件

vi flume-netcat-logger.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

打开监听端口

bin/flume-ng agent --conf conf/ --name a1 --conf-file job/flume-netcat-logger.conf - Dflume.root.logger=INFO,console

简写

bin/flume-ng agent -n a1 -c conf/ -f job/flume-netcat-logger.conf -Dflume.root.logger=INFO,console

--conf/-c：表示配置文件存储在 conf/目录

--name/-n：表示给 agent 起名为 a1

--conf-file/-f：flume 本次启动读取的配置文件是在 job 文件夹下的flume-telnet.conf

-Dflume.root.logger=INFO,console ：-D 表示flume 运行时动态修改 flume.root.logger参数属性值，并将控制台日志打印级别设置为 INFO 级别。日志级别包括:log、info、warn、error。

2021-09-07 23:28:17,052 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2021-09-07 23:28:17,054 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2021-09-07 23:28:17,477 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink k1
2021-09-07 23:28:17,478 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source r1
2021-09-07 23:28:17,479 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting
2021-09-07 23:28:17,517 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:169)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]

 监听127.0.0.1:44444

 用 netcat 工具向本机的 44444 端口发送内容

安装 netcat 工具：sudo yum install -y nc

1、检查端口 

sudo netstat -nlp | grep 44444

 2、打开netcat工具向本机44444端口发送数据

[root@node01 logs]# nc localhost 44444
1
OK
2
OK
Hello
OK

---

 监听端数据：

2021-09-07 23:32:47,922 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 31                                              1 }
2021-09-07 23:33:02,972 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 32                                              2 }
2021-09-07 23:33:57,119 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 48 65 6C 6C 6F                                  Hello }

2021-09-07 23:32:47,922 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 31                                              1 }
2021-09-07 23:33:02,972 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 32                                              2 }
2021-09-07 23:33:57,119 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 48 65 6C 6C 6F                                  Hello }