主机清单(es集群生产环境建议三节点以上)
hostname | ip | software | system os |
elk-node1 | 192.168.99.185 | elasticsearch-6.8.4 zabbix-agent-4.2.7 | CentOS 7.5.1804 |
elk-node2 | 192.168.99.186 | elasticsearch-6.8.4 zabbix-agent-4.2.7 | CentOS 7.5.1804 |
zabbix-server | 192.168.99.50 | zabbix-server-4.2.7 zabbix-agent-4.2.7 | CentOS 7.5.1804 |
es _cat/health?v 详解
[root@elk-node1 elasticsearch_monitor]# curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -s http://192.168.99.185:9200/_cat/health?vepoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent1578208577 07:16:17 elk-cluster green 2 2 998 499 0 0 0 0 - 100.0%
参数含义
cluster ,集群名称status,集群状态 green代表健康;yellow代表分配了所有主分片,但至少缺少一个副本,此时集群数据仍旧完整;red代表部分主分片不可用,可能已经丢失数据。node.total,代表在线的节点总数量node.data,代表在线的数据节点的数量shards, active_shards 存活的分片数量pri,active_primary_shards 存活的主分片数量 正常情况下 shards的数量是pri的两倍。relo, relocating_shards 迁移中的分片数量,正常情况为 0init, initializing_shards 初始化中的分片数量 正常情况为 0unassign, unassigned_shards 未分配的分片 正常情况为 0pending_tasks,准备中的任务,任务指迁移分片等 正常情况为 0max_task_wait_time,任务最长等待时间active_shards_percent,正常分片百分比 正常情况为 100%
_cat/health?h 检查集群状态
(由于es集群开启用户身份认证,user:elastic ,password:qZXo7EkxoxmKvDqQIwn5 )
[root@elk-node1 /]# curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://192.168.99.185:9200/_cat/health?h=node.total"2[root@elk-node1 /]# curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://192.168.99.185:9200/_cat/health?h=cluster"elk-cluster[root@elk-node1 /]# curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://192.168.99.185:9200/_cat/health?h=active_shards_percent"100.0%
es 环境变量配置脚本(es集群主机建议三台以上,防止单一节点宕机导致数据丢失)
[root@elk-node1 elasticsearch_monitor]# cat es_env.sh# es监控环境变量配置文件# es节点ip和端口ESADDR="192.168.99.185:9200, 192.168.99.186:9200"
es 集群健康检查脚本
[root@elk-node1 elasticsearch_monitor]# cat es_cluster_health.sh #!/bin/bashPATH=/bin:/sbin:/usr/bin:/usr/sbin#echo $0cd $(dirname $0)HOMEPATH=$(pwd)source es_env.sh[ $? -ne 0 ] && echo "ERROR: es_env.sh not not found." >&2 && exit 1[ X$1 = X ] && echo "ERROR: type empty." >&2 && exit 1for addr in $(echo ${ESADDR} |sed -e 's/,/ /g')do curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://$addr/_cat/health?h=$1" 2>/dev/null [ $? -eq 0 ] && breakdone
es node 节点自动发现脚本
[root@elk-node1 elasticsearch_monitor]# cat es_nodes_discovery.sh#!/bin/bashPATH=/bin:/sbin:/usr/bin:/usr/sbin#echo $0cd $(dirname $0)HOMEPATH=$(pwd)source es_env.sh[ $? -ne 0 ] && echo "ERROR: es_env.sh not not found." >&2 && exit 1# 连接es节点,获取索引,自动发现监控项for addr in $(echo ${ESADDR} |sed -e 's/,/ /g')do curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://$addr/" &>/dev/null [ $? -ne 0 ] && echo "WARN: connect to $addr failed." >&2 && continue # 获取索引列表 # 日索引 list=( $(curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://$addr/_cat/nodes?h=name" 2>/dev/null) ) #echo -e "${list[*]}\n\n" #[ x$list = x ] && echo "ERROR: no discovery data list found." >&2 && exit 1 # 将列表list,转换为发送给zabbx的json数据。length=${#list[@]} #printf "{\n" printf '{\"data\":[ ' for ((i=0;i<$length;i++)) do printf "{\"{#NAME}\":\"${list[$i]}\"}" if [ $i -lt $[ $length-1 ] ];then printf ',' fi done #printf "\n\t]\n" #printf "}\n" printf " ]}" breakdone
es node 节点监控脚本
[root@elk-node1 elasticsearch_monitor]# cat es_node_monitor.sh#!/bin/bashPATH=/bin:/sbin:/usr/bin:/usr/sbin#echo $0cd $(dirname $0)HOMEPATH=$(pwd)source es_env.sh[ $? -ne 0 ] && echo "ERROR: es_env.sh not not found." >&2 && exit 1[ X$1 = X ] && echo "ERROR: node name empty." >&2 && exit 1[ X$2 = X ] && echo "ERROR: type empty." >&2 && exit 1for addr in $(echo ${ESADDR} |sed -e 's/,/ /g')do curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://$addr/_cat/nodes?bytes=b&size=b&h=name,$2" 2>/dev/null |grep "^$1" |awk '{print $2}' [ $? -eq 0 ] && breakdone
检查 es 索引在周期是否正常更新
document count 表示可用文档的数量,不包括未被合并进程清理的已删除文档。
[root@elk-node1 elasticsearch_monitor]# cat es_index_monitor.sh#!/bin/bashPATH=/bin:/sbin:/usr/bin:/usr/sbin#echo $0cd $(dirname $0)HOMEPATH=$(pwd)source es_env.sh[ $? -ne 0 ] && echo "ERROR: es_env.sh not not found." >&2 && exit 1# index countindices_count() { local indexfilter=$1 for addr in $(echo ${ESADDR} |sed -e 's/,/ /g') do curl -u elastic:qZXo7EkxoxmKvDqQIwn5 -XGET "http://$addr/_cat/count/$indexfilter*?h=count" 2>/dev/null [ $? -eq 0 ] && break done}# 检查类型case $1 in count) indices_count $2 ;; *) echo "ERROR: monitor type $1 error." >&2esac
es 监控脚本测试
zabbix-agent 配置
[root@elk-node1 elasticsearch_monitor]# cat /etc/zabbix/zabbix_agentd.d/userparameter_elasticsearch.conf UserParameter=es_cluster_health[*],/usr/local/zabbix/scripts/elasticsearch_monitor/es_cluster_health.sh $1UserParameter=es_nodes_discovery,/usr/local/zabbix/scripts/elasticsearch_monitor/es_nodes_discovery.shUserParameter=es_node_monitor[*],/usr/local/zabbix/scripts/elasticsearch_monitor/es_node_monitor.sh $1 $2UserParameter=es_index_monitor[*],/usr/local/zabbix/scripts/elasticsearch_monitor/es_index_monitor.sh $1 $2
zabbix_get 测试
创建索引中文档计数监控项
创建索引中文档计数触发器
索引在周期内文档 count 差异数值为0触发告警,正常情况下索引的文档计数是不断增加累计。
zabbix 监控指标
参考链接
https://github.com/Vastxiao/zabbixMonitorES/tree/master/elasticsearch_monitor