Flume的原理及安装（日志采集工具）

Flume的原理及安装（日志采集工具）

1 Flume原理

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-QbVOiE72-1647349435640)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image002.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xSm5FZ6d-1647349435641)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image004.jpg)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ASS0ufRl-1647349435641)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image006.jpg)]

2 Flume案例

2.1 监控端口数据

案例需求：

首先，flume监控本机4444端口，然后通过telnet工具向本机4444端口发送消息，最后flume将监听的数据实时显示在控制台。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-yudP4uSe-1647349435641)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image008.jpg)]

1 安装telnet工具：

[root@cent71 software]# yum install telnet-server

[root@cent71 software]# yum install telnet

[root@cent71 software]# yum install -y xinetd

[root@cent71 software]# yum install -y nc

2 开启端口4444

[root@cent71 software]# nc -lp 4444 &

#netstat -tunlp | grep 4444 (查看是否打开4444端口)

#telnet localhost 4444 （测试telnet是否成功，然后退出）

3 配置Flume采集方案

# example.conf: A single-node Flume configuration

# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = netcat

a1.sources.r1.bind = localhost

a1.sources.r1.port = 4444

# Describe the sink

a1.sinks.k1.type = logger

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

# The maximum number of events stored in the channel:channel中能容下的最大events个数

a1.channels.c1.capacity = 1000

#The maximum number of events the channel will take from a source or give to a sink per transaction:每次拉取或接收的最大events个数

a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

4 使用指定采集方案启动Flume

[root@cent71 apache-flume-1.8.0-bin]# bin/flume-ng agent --conf conf/ --conf-file conf/flume-netcat-logger.conf --name a1 -Dflume.root.logger=INFO,console

5 Flume采集数据测试

[root@cent71 ~]# tel