flume 数据流监控 ganglia 安装

最新推荐文章于 2024-10-30 15:55:38 发布

春光静好

最新推荐文章于 2024-10-30 15:55:38 发布

阅读量349

点赞数

分类专栏： bigdata 文章标签： flume ganglia

本文链接：https://blog.csdn.net/ITresource/article/details/107815934

版权

bigdata 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

一. 安装配置 ganglia

安装 httpd 服务与 php

sudo yum -y install httpd php

安装其他依赖

sudo yum -y install rrdtool perl-rrdtool rrdtool-devel

sudo yum -y install apr-devel

安装 ganglia

sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6- 8.noarch.rpm

sudo yum -y install ganglia-gmetad

sudo yum -y install ganglia-web

sudo yum -y install ganglia-gmond

Ganglia 由 gmond、gmetad 和 gweb 三部分组成。
gmond(Ganglia Monitoring Daemon)是一种轻量级服务，安装在每台需要收集指标数据的节点主机上。使用 gmond，你可以很容易收集很多系统指标数据，如 CPU、内存、磁盘、网络和活跃进程的数据等。
gmetad(Ganglia Meta Daemon)整合所有信息，并将其以 RRD 格式存储至磁盘的服务。
gweb(Ganglia Web)Ganglia 可视化工具，gweb 是一种利用浏览器显示 gmetad 所存储数据的 PHP 前端。在 Web 界面中以图表方式展现集群的运行状态下收集的多种不同指标数据。

sudo vim /etc/httpd/conf.d/ganglia.conf

# Ganglia monitoring system php web frontend
Alias /ganglia /usr/share/ganglia <Location /ganglia>
  Order deny,allow
  #Deny from all
  Allow from all
  # Allow from 127.0.0.1
  # Allow from ::1
  # Allow from .example.com
</Location>

以上本人操作，访问报错

[root@laxin-rtstream-calc-18-78 flume-1.7.0]# curl http://172.28.18.78/ganglia
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /ganglia
on this server.</p>
</body></html>

解决方案：

#
# Ganglia monitoring system php web frontend
#
Alias /ganglia /usr/share/ganglia
<Location /ganglia>
  #Order deny,allow
  #Deny from all
  #Allow from all
  # Allow from 127.0.0.1
  # Allow from ::1
  # Allow from .example.com
  Require all granted
</Location>

修改配置文件/etc/ganglia/gmetad.conf

sudo vim /etc/ganglia/gmetad.conf

data_source "hadoop02" 自己IP

修改配置文件/etc/ganglia/gmond.conf

修改为:
cluster {
  name = "hadoop102"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}
udp_send_channel {
#bind_hostname = yes # Highly recommended, soon to be default.
# This option tells gmond to use a source
address
Without
# that resolves to the machine's hostname.
# this, the metrics may appear to come from any # interface and the DNS names associated with
# those IPs will be used to create the RRDs.
  # mcast_join = 239.2.11.71
  host = XXX.XXX.XXX.XXX
  port = 8649
  ttl = 1
}
udp_recv_channel {
  # mcast_join = 239.2.11.71
  port = 8649
  bind = XXX.XXX.XXX.XXX
  retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
  # buffer = 10485760
}

修改配置文件/etc/selinux/config

SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection. 
SELINUXTYPE=targeted

selinux 本次生效关闭必须重启，如果此时不想重启，可以临时生效之
sudo setenforce 0

启动 ganglia

sudo service httpd start
sudo service gmetad start
sudo service gmond start

检查和停止

sudo service httpd status / stop
sudo service gmetad status / stop
sudo service gmond status / stop

打开网页浏览 ganglia 页面
```
http://hadoop02/ganglia/.  
```
⚠️如果完成以上操作依然出现权限不足错误，请修改/var/lib/ganglia 目录的权限:

二. 操作 Flume 测试监控

修改/opt/module/flume/conf 目录下的 flume-env.sh 配置:

 JAVA_OPTS="-Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=xxx.xxx.xxx.xxx(IP):8649 -Xms100m
-Xmx200m"

启动 Flume 任务

job/flume-netcat-logger.conf

name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

bin/flume-ng agent \ --conf conf/ \
--name a1 \
--conf-file job/flume-netcat-logger.conf \ -Dflume.root.logger==INFO,console \ -Dflume.monitoring.type=ganglia \ -Dflume.monitoring.hosts=xxx.xxx.xxx.xxx:8649