ELK+Sentinl日志监控邮件报警

一、部署环境
1.基础环境：

软件	版本
Linux	Centos7.1，16g
Jdk	1.8.0_151
Elasticsearch	5.5.0
Kibana	5.5.0
Sentinl	5.5.0

---

二、安装Sentinl

1. 软件下载
   链接:sentinl-v5.5.0.zip
2. 解压文件到指定目录
   /usr/local/etc/sentinl
3. 修改配置文件
   vi /config/kibana.yml在末尾增加如下参数

sentinl:
 settings:
   email:
     active: true
     user: XXX@163.com
     password: XXX  #163邮箱授权码
     host: smtp.163.com
     ssl: true
   report:
     active: true
     tmp_path: /tmp/

• 163邮箱授权：设置→邮箱安全设置→客户端授权密码→开启
  

4.安装命令
./bin/kibana-plugin install file:///usr/local/etc/sentinl/sentinl-v5.5.0.zip，
注意查看安装日志，当显示Plugin installation complete表示安装成功

Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete

5. 重启kibana
   ps -ef|grep node kill已启动的kibana进程
   ./kibana & 启动kibaba，注意查看控制台输出日志

 log   [00:51:25.233] [info][status][Sentinl][scheduler] server.sentinlStore.scheduled Watch: n7n5fpp7d1e-vfv6sj66b5k-upv171lvs every every 10 minutes
 log   [00:51:25.234] [info][status][Sentinl][scheduler] Clearing watcher: AWg3P0g_uUKuvElvcMZ5
 log   [00:51:25.235] [info][status][Sentinl][scheduler] server.sentinlStore.scheduled Watch: AWg3P0g_uUKuvElvcMZ5 every every 1 minutes
 log   [00:51:25.237] [info][status][Sentinl][scheduler] Clearing watcher: p3ngnebkbqs-4sk3ww25jze-6f3vkc6f5gc
 log   [00:51:25.239] [info][status][Sentinl][scheduler] server.sentinlStore.scheduled Watch: p3ngnebkbqs-4sk3ww25jze-6f3vkc6f5gc every every 5 minutes
 log   [00:51:25.239] [info][status][Sentinl][scheduler] Clearing watcher: 4vbknzuhxh7-gf7zeehc3a-t70g0ja2ax
 log   [00:51:25.240] [info][status][Sentinl][scheduler] server.sentinlStore.scheduled Watch: 4vbknzuhxh7-gf7zeehc3a-t70g0ja2ax every every 3 minutes

5.验证
重启成功后，打开kibana界面，会出现sentinl的菜单按钮

6.配置sentinl

• 新建

• 点击sentinl→watchers→New→Watcher,进入设置界面。
  

• General配置

• Title：名称
• Schedule：执行周期，如：every 5 minutes每5分钟执行一次，详细规则如下。

fires at 17:15 every day 
at 17:15

fires every 5 minutes every day 
every 5 mins

fires at 10:15am and 5:15pm every day except on Tuesday 
at 10:15 am also at 5:15pm except on Tuesday

fires every Sunday 
on the first day of the week

fires on the last day of every month 
on the last day of the month

fires every day between the 15th and 20th day every month 
on the 15th through 20th day of the month

fires every 5 minutes on Saturday and Sunday 
every 5 mins every weekend

fires on the 7th, 27th, and 47th minute every hour 
every 20 mins starting on the 7th min

fires every hour after noon every day 
after 12th hour

fires every hour before noon every day 
before 12th hour

fires at 5:00pm every day 
at 5:00 pm

fires at 5:00pm on Wednesday, Thursday, and Friday 
at 5:00 pm on Weds,Thurs and Fri

fires at 5:00pm every day of March in 2014 
at 5:00 pm every 1 day of March in 2014

• input配置

• Body：查询语句

{
 "search": {
   "request": {
     "index": [
       "truck-ka-prod*"
     ],
     "body": {
       "query": {
         "bool": {
           "must": [
             {
               "query_string": {
                 "query": "\"服务内部异常\""
               }
             }
           ],
           "filter": [
             {
               "range": {
                 "@timestamp": {
                   "gte": "now-5m",
                   "lt": "now",
                   "format": "epoch_millis"
                 }
               }
             }
           ]
         }
       },
       "sort": [
         {
           "@timestamp": {
             "order": "desc"
           }
         }
       ]
     }
   }
 }
}

• Condition

• Body：查询条件，大于等于1次就报警，一定要注意>=1和>1的区别，小细节也是天坑之一

{
 "script": {
   "script": "payload.hits.total > =1"
 }
}

• Actions
  点击Add action→email,进入邮件设置界面。

• Throttle：
  查询限流设置，三个表格的单位是时、分、秒，如下图设置60S查询限流设置，
  
  当60S内点击或触发第二次时，不会发邮件，但会在Alarms日志中记录下Action Throttled for 0h0m60s，如下图。
  

• Body：邮件内容，其中{{#payload.hits.hits}} {{/payload.hits.hits}}可以循环遍历

在生产环境发现了: {{payload.hits.total}} 个“服务内部异常”，

{{#payload.hits.hits}}
time：{{_source.time}}, logId:【 {{_source.logId}} 】, module：{{_source.module}} 
{{/payload.hits.hits}}

邮件实例如下：

附：
Sentinl插件Git地址：https://github.com/sirensolutions/sentinl
Sentinl文档地址：https://sentinl.readthedocs.io/en/latest/