ganglia-hadoop

hadoop的metrics2系统可以支持将metrics数据写入到本地文件或导出到外部系统；

• 默认情况下即并没有修改 hadoop-metrics2-properties文件时，监控的有load_one,cpu,mem,network等单个节点或整个集群的机器相关性能；
• 当修改 hadoop-metrics2-properties文件，将hadoop系统的metrics导入到 ganglia中，监控的有 dfs,jvm,rpc等单个node或aggregated数据；

hadoop-metrics2-properties 示例：

• 环境：hadoop2.6.5+ganglia3.6.0 单机伪分布式运行

#
# Below are for sending metrics to Ganglia
#
# for Ganglia 3.0 support
# *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink30
#
# for Ganglia 3.1 support
 *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31

 *.sink.ganglia.period=10

# default for supportsparse is false
# *.sink.ganglia.supportsparse=true

*.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both
*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40

# Tag values to use for the ganglia prefix. If not defined no tags are used.
# If '*' all tags are used. If specifiying multiple tags separate them with 
# commas. Note that the last segment of the property name is the context name.
#
#*.sink.ganglia.tagsForPrefix.jvm=ProcesName
#*.sink.ganglia.tagsForPrefix.dfs=
#*.sink.ganglia.tagsForPrefix.rpc=
#*.sink.ganglia.tagsForPrefix.mapred=

namenode.sink.ganglia.servers=localhost:8649
datanode.sink.ganglia.servers=localhost:8649
resourcemanager.sink.ganglia.servers=localhost:8649
nodemanager.sink.ganglia.servers=localhost:8649
mrappmaster.sink.ganglia.servers=localhost:8649
jobhistoryserver.sink.ganglia.servers=localhost:8649

当采用ganglia的gmond以组播方式发送和接收 udp 时，发现 在gweb中只能看到 有关机器性能的数据，后来通过 telnet localhost 8649 读出来的 xml 数据中也没有发现里面并没有 dfs,jvm,rpc 方面的数据，说明 metrics并没有真正被导出，后来想了下可能是 组播 和 单播的问题；便把 ganglia 的 gmond.conf 更改为单播即可，当然了都设置成组播也可以。

ref:

1. https://blog.cloudera.com/blog/2012/10/what-is-hadoop-metrics2/
2. https://blog.csdn.net/xxd851116/article/details/25109043
3. https://laoxiangchun.cn/

更多内容可关注本人博客：老香椿（https://laoxiangchun.cn/）