flume数据流监控

目录

flume数据流监控

Ganglia的安装与部署

操作flume测试监控

---

flume数据流监控

Ganglia的安装与部署

Ganglia由gmond、gmetad和gweb三部分组成。

gmond（Ganglia Monitoring Daemon）：是一种轻量级服务，安装在每台需要收集指标数据的节点主机上。使用gmond可以很容易收集很多系统指标数据，如CPU、内存、磁盘、网络和活跃进程的数据等。

gmetad（Ganglia Meta Daemon）：整合所有信息，并将其以RRD格式存储至磁盘的服务。

gweb（Ganglia Web）：Ganglia可视化工具，gweb是一种利用浏览器显示gmetad所存储数据的PHP前端。在Web界面中以图表方式展现集群的运行状态下收集的多种不同的指标数据。

（1）安装Ganglia

1）规划

hadoop01：gweb、gmetad、gmond

hadoop02、hadoop03、hadoop04：gmond

2）在所有节点安装epel-release

sudo yum -y install epel-release

3)在hadoop01安装gweb、gmetad、gmond

sudo yum -y install ganglia-gmetad
sudo yum -y install ganglia-gmond
sudo yum -y install ganglia-web

4)在Hadoop02、Hadoop03和Hadoop04上安装gmond

sudo yum -y install ganglia-gmond

 

（2）在Hadoop01上修改配置文件

1)ganglia.conf

sudo vim /etc/httpd/conf.d/ganglia.conf

# Ganglia monitoring system php web frontend
Alias /ganglia /usr/share/ganglia
<Location /ganglia>
 # Order deny,allow
  Require ip 192.168.17.1
 # Deny from all
 # Allow from 127.0.0.1
 # Allow from ::1
  # Allow from .example.com
</Location>

注意ip地址！！！ 

2)gmetad.conf

sudo vim /etc/ganglia/gmetad.conf

data_source "my cluster" hadoop01

（3）在Hadoop01、Hadoop02、Hadoop03、Hadoop04修改配置文件

1）gmond.conf

 sudo vim /etc/ganglia/gmond.conf

cluster {
 name = "my cluster"
 owner = "unspecified"
 latlong = "unspecified"
 url = "unspecified"
}
udp_send_channel {
 #bind_hostname = yes # Highly recommended, soon to be default.
 # This option tells gmond to use a source 
address
 # that resolves to the machine's hostname. 
Without
 # this, the metrics may appear to come from 
any
 # interface and the DNS names associated with
 # those IPs will be used to create the RRDs.
 # mcast_join = 239.2.11.71
 # 数据发送给 hadoop01
 host = hadoop01
 port = 8649
 ttl = 1
}

udp_recv_channel {
 # mcast_join = 239.2.11.71
 port = 8649
# 接收来自任意连接的数据
 bind = 0.0.0.0
 retry_bind = true
 # Size of the UDP buffer. If you are handling lots of metrics 
you really
 # should bump it up to e.g. 10MB or even higher.
 # buffer = 10485760
}

（4）在Hadoop01上修改config

sudo vim /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

注：selinux 生效需要重启，如果此时不想重启，可以临时生效。 

sudo setenforce 0

（5）启动ganglia

1)在Hadoop02、Hadoop03、Hadoop04启动

sudo systemctl start gmond

2）在Hadoop01启动

sudo systemctl start httpd

sudo systemctl start gmetad

（6）打开网页浏览ganglia页面

http://hadoop01/ganglia

注：如果完成以上操作依然出现权限不足错误，请修改/var/lib/ganglia 目录的权限。

sudo chmod -R 777 /var/lib/ganglia

 

操作flume测试监控

（1）启动flume任务

bin/flume-ng agent \ 
-c conf/ \ 
-n a1 \ 
-f conf/flume-netcat-logger.conf \ 
-Dflume.root.logger=INFO,console \ 
-Dflume.monitoring.type=ganglia \ 
-Dflume.monitoring.hosts=hadoop102:8649

（2）发送数据观察ganglia监控图

nc localhost 44444

说明：

字段	含义
EventPutAttemptCount	source 尝试写入 channel 的事件总数量
EventPutSuccessCount	成功写入 channel 且提交的事件总数量
EventTakeAttemptCount	sink 尝试从 channel 拉取事件的总数量。
EventTakeSuccessCount	sink 成功读取的事件的总数量
StartTime	channel 启动的时间（毫秒）
StopTime	channel 停止的时间（毫秒）
ChannelSize	目前 channel 中事件的总数量
ChannelFillPercentage	channel 占用百分比
ChannelCapacity	channel 的容量

本文为学习笔记！！！