Flume的原理及安装(日志采集工具)
1 Flume原理
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-QbVOiE72-1647349435640)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image002.jpg)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xSm5FZ6d-1647349435641)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image004.jpg)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ASS0ufRl-1647349435641)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image006.jpg)]
2 Flume案例
2.1 监控端口数据
案例需求:
首先,flume监控本机4444端口,然后通过telnet工具向本机4444端口发送消息,最后flume将监听的数据实时显示在控制台。
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-yudP4uSe-1647349435641)(file:///C:/Users/MEETAG~1/AppData/Local/Temp/msohtmlclip1/01/clip_image008.jpg)]
1 安装telnet工具:
[root@cent71 software]# yum install telnet-server
[root@cent71 software]# yum install telnet
[root@cent71 software]# yum install -y xinetd
[root@cent71 software]# yum install -y nc
2 开启端口4444
[root@cent71 software]# nc -lp 4444 &
#netstat -tunlp | grep 4444 (查看是否打开4444端口)
#telnet localhost 4444 (测试telnet是否成功,然后退出)
3 配置Flume采集方案
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 4444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
# The maximum number of events stored in the channel:channel中能容下的最大events个数
a1.channels.c1.capacity = 1000
#The maximum number of events the channel will take from a source or give to a sink per transaction:每次拉取或接收的最大events个数
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
4 使用指定采集方案启动Flume
[root@cent71 apache-flume-1.8.0-bin]# bin/flume-ng agent --conf conf/ --conf-file conf/flume-netcat-logger.conf --name a1 -Dflume.root.logger=INFO,console
5 Flume采集数据测试
[root@cent71 ~]# tel