Linux虚拟机 百度云 监控,使用钉钉完成对虚拟机监控告警

本篇文章不讲述如何使用prometheus监控VM,详细来说一下监控VM后续使用钉钉报警的讲解

前言：

之前一直使用的是prometheus-webhook-dingtalk0.3.0版本来进行告警的

效果图如下:

微信截图_20191209164428.png

作者关注了很久项目今天迎来了重大更新Click me迎来了prometheus-webhook-dingtalk1.2.2版本,整体有很大的革新,所以按奈不住躁动的心,前来尝试一番:

效果图如下:

微信截图_20191209165028.png

接下来就来说说是如何实现的

首先下载对应版本的压缩包

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.2.2/prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz

tar xf prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz

添加钉钉机器人的webhook

cd prometheus-webhook-dingtalk-1.2.2.linux-amd64

mv config.example.yml config.yml

vim config.yml

# 我这里使用的是webhook2来进行告警的

## Request timeout

# timeout: 5s

## Customizable templates path

# templates:

# - contrib/templates/legacy/template.tmpl

## Targets, previously was known as "profiles"

targets:

# webhook1:

# url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx

# secret for signature

# secret: SEC000000000000000000000

webhook2:

url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook

webhook_legacy:

url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook

# Customize template content

message:

# Use legacy template

title: '{{ template "default.title" . }}'

text: '{{ template "default.content" . }}'

webhook_mention_all:

url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook

mention:

all: true

webhook_mention_users:

url: https://oapi.dingtalk.com/robot/send?access_token=10bda98979ae2155b6822b699cde1841d4fbd8514c0441bbbb4485caddf3a388x #这是是钉钉机器人的webhook

mention:

mobiles: ['15xxxxxxxx0']

配置alertmanager

我这里使用的是webhook2来进行告警的

vim /etc/alertmanager/alertmanager.yml

global: #全局设置

resolve_timeout: 1m #没有报警情况下申明已解决的时间

route:

group_by: ['alertname'] #分组

group_wait: 1m #至少等待group_wait时间来初始化

group_interval: 1m #当第一个报警发送后，等待'group_interval'时间来发送新的一组报警信息

repeat_interval: 2m #如果一个报警信息已经发送成功了，等待'repeat_interval'时间来重新发送他们

receiver: 'webhook' #接收器

routes: #子路由

- receiver: webhook #接收器

group_wait: 1m

match:

team: node #匹配一个标签

receivers:

- name: 'webhook' #接收器

webhook_configs:

- url: http://localhost:8060/dingtalk/webhook2/send #钉钉的发送方式,如果你是webhook1方式启动的 这里要换一下

send_resolved: true #解决告警之后回复信息

#重启alertmanager

systemctl restart alertmanager

启动prometheus-webhook-dingtalk

--web.ui-enabled打开webUI界面

--config.file指定配置文件启动

nohup ./prometheus-webhook-dingtalk --web.ui-enabled --config.file=config.yml &

接下来就可以看到全新的报警方式了,看起来比之前的0.3.0版本清爽多了

微信截图_20191209170426.png

说明： 上图中[]不是异常也不是模板问题，是prometheus没取到值,然后传给alertmanager的时候那里是空数组,所以就这样显示了。

下面是我用的一些报警规则

groups:

- name: host_monitoring

rules:

- alert: 内存报警

expr: (1 - (node_memory_MemAvailable_bytes / (node_memory_MemTotal_bytes)))* 100 > 80

for: 2m

labels:

team: node

annotations:

Alert_type: 内存报警

Server: '{{$labels.instance}}'

description: "内存使用量超过80%，当前使用量为：{{ $value }}%"

- alert: CPU报警

expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80

for: 2m

labels:

team: node

annotations:

Alert_type: CPU报警

Server: '{{$labels.instance}}'

description: "CPU使用量超过80%，当前使用量为：{{ $value }}%"

- alert: 磁盘报警

expr: max by (instance, namespace, pod, device) ((node_filesystem_size_bytes{fstype=~"ext[234]|btrfs|xfs|zfs"} - node_filesystem_avail_bytes{fstype=~"ext[234]|btrfs|xfs|zfs"}) / node_filesystem_size_bytes{fstype=~"ext[234]|btrfs|xfs|zfs"}) * 100 >80

for: 2m

labels:

team: node

annotations:

Alert_type: 磁盘报警

Server: '{{$labels.instance}}'

description: "磁盘使用量超过80%，当前使用量为：{{ $value }}%"

- alert: 服务报警

expr: up == 0

for: 2m

labels:

team: node

annotations:

Alert_type: 服务报警

Server: '{{$labels.instance}}'

description: "{{$labels.instance}}服务已关闭"

- alert: 硬件温度报警

expr: node_hwmon_temp_celsius >= 63

for: 2m

labels:

team: node

annotations:

Alert_type: 硬件温度报警

Server: '{{$labels.instance}}'

description: "硬件温度超过55°，当前温度为：{{ $value }}"

- name: blackbox_network_stats

rules:

- alert: tcp端口检测/icmp主机检测

expr: probe_success == 0

for: 10s

labels:

severity: critical

annotations:

summary: "Instance {{ $labels.instance }} is down"

description: "This requires immediate action!"

总结:prometheus-webhook-dingtalk1.2.2比prometheus-webhook-dingtalk0.3.0告警信息看的更加清爽了,另外还迎来一个webUI

优化

#!/bin/sh

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.2.2/prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz

if [ $? -ne 0 ]; then

while true

do

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.2.2/prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz

if [ $? -eq 0 ]; then

break

fi

done

else

tar xf prometheus-webhook-dingtalk-1.2.2.linux-amd64.tar.gz

cp -R prometheus-webhook-dingtalk-1.2.2.linux-amd64/prometheus-webhook-dingtalk /usr/local/bin/

fi

#echo "---验证版本---"

#node_exporter --version

echo "---剪切并命名---"

mkdir /etc/prometheus-webhook-dingtalk/ -p

mv prometheus-webhook-dingtalk-1.2.2.linux-amd64/config.example.yml /etc/prometheus-webhook-dingtalk/config.yml

#创建工作目录

mkdir -p /var/lib/prometheus-webhook-dingtalk

#创建并编写配置启动项配置文件

cat > /usr/lib/systemd/system/prometheus-webhook-dingtalk.service <

[Unit]

Description=prometheus-webhook-dingtalk

[Service]

Type=simple

ExecStart=/usr/local/bin/prometheus-webhook-dingtalk \

--config.file /etc/prometheus-webhook-dingtalk/config.yml

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload && systemctl start prometheus-webhook-dingtalk && systemctl enable prometheus-webhook-dingtalk

echo "---验证---"

systemctl status prometheus-webhook-dingtalk

echo "---开机自启动---"

systemctl list-unit-files | grep prometheus-webhook-dingtalk

echo "---监听端口---"

ss -ntlp | grep 8060

#清除无用包

rm -rf ./prometheus-webhook-dingtalk-*