EFK+sentinl报警机制

全栈工程师开发手册 （作者：栾鹏）
架构系列文章

---

EFK的部署可以参考https://blog.csdn.net/luanpeng825485697/article/details/83312662

fluentd日志采集的语法可以参考 ：https://blog.csdn.net/luanpeng825485697/article/details/83339985

EFK缺少一个报警机制

下面我们尝试几种方法来设置报警

sentinl

先构建镜像，Dockerfile文件内容如下

# 在当前文件夹执行 sudo docker build -t luanpeng/lp:kibana-oss-6.2.4 .

FROM registry.cn-beijing.aliyuncs.com/efk-install/kibana-oss:6.2.4

USER root
# 安装插件
RUN yum install vim -y
RUN /opt/kibana/bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.4/sentinl-v6.2.4.zip
# COPY ./sentinl-v6.2.4.zip /
# RUN /opt/kibana/bin/kibana-plugin install file:///sentinl-v6.2.4.zip
# RUN rm /sentinl-v6.2.4.zip

CMD /usr/local/bin/kibana-docker

安装过程中可能因为网络原因会中断，多build几次就行了。

也可以先离线下载好，

/opt/kibana/bin/kibana-plugin install file:///sentinl-v6.2.4.zip

安装，版本号要和kibana版本号一致

/opt/kibana/bin/kibana-plugin remove sentinl   卸载

如果使用docker运行镜像

docker run -it --name kibana --env ELASTICSEARCH_URL=http://192.168.2.177:31001 -p 5601:5601 luanpeng/lp:kibana-oss-6.2.4 .

进入容器修改/opt/kibana/config/kibana.yml

server.name: kibana
    server.host: "0"
    elasticsearch.url: http://elasticsearch-logging:9200      # 修改es的地址
    sentinl:
      settings:
        email:
          active: true
          user: luan.peng@intellif.com   # 设置发送邮箱
          password: xxxxxxxxxx
          host: smtp.exmail.qq.com
          port: 465      # 要加端口号才行
          ssl: true
        report:
          active: true

然后http://kibana-server-ip:5601，界面上会多一个sentinl

添加一个watcher

配置如下

{
  "actions": {
    "email_admin": {
      "throttle_period": "5m",
      "email_html": {
        "to": "luan.peng@intellif.com",
        "from": "luan.peng@intellif.com",
        "stateless": false,
        "subject": "{{payload.hits.total}} new results from watcher {{watcher.title}} error",
        "priority": "high",
        "html": "<p>Hi {{watcher.username}},</p>\n<p>There are {{payload.hits.total}} results found by the watcher <i>{{watcher.title}}</i>.</p>\n<div style=\"color: grey\">\n  <hr>\n  <p>log is: </p>\n{{#payload.my_data}}{{time}}<br/>{{namespace_name}}:{{container_name}}<br/>{{log}}<br/>----------------------------------<br/><br/>{{/payload.my_data}}\n</div>"
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "index": [
          "logstash-2018*"
        ],
        "type": [
          "fluentd"
        ],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "query_string": {
                    "fields": [
                      "log"
                    ],
                    "analyze_wildcard": true,
                    "query": "error"
                  }
                },
                {
                  "match": {
                    "kubernetes.namespace_name": "cloudai-2"
                  }
                },
                {
                  "match": {
                    "kubernetes.container_name": {
                      "query": "backend-traffic-container grpc-container backend-mqtt-container",
                      "operator": "or"
                    }
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m",
                      "lte": "now"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "script": "payload.hits.total >= 0"
    }
  },
  "transform": {
    "script": {
      "script": "payload.my_data=[];for(var i=0;i<payload.hits.hits.length;i++){payload.my_data.push({time:payload.hits.hits[i]._source['@timestamp'],namespace_name:payload.hits.hits[i]._source['kubernetes']['namespace_name'],container_name:payload.hits.hits[i]._source['kubernetes']['container_name'],log:payload.hits.hits[i]._source['log'],message:payload.hits.hits[i]._source.message});}"
    }
  },
  "trigger": {
    "schedule": {
      "later": "every 5 minutes"
    }
  },
  "disable": false,
  "report": false,
  "title": "yinli-aicloud",
  "save_payload": false,
  "spy": true,
  "impersonate": false,
  "username": "luanpeng"
}

其中disable表示是否关闭,spy表示是否在关闭网页后仍然监控运行.默认情况只有在打开网页的情况下才能周期报警.

如果使用k8s部署，则需要将kibana.yml文件创建成为configmap
configmap.yaml文件内容如下

kind: ConfigMap
apiVersion: v1
metadata:
  name: kibana-config
  namespace: logging
  labels:
    app: kibana
data:
  kibana.yml: |
    server.name: kibana
    server.host: "0"
    elasticsearch.url: http://elasticsearch-logging:9200  
    sentinl:
      settings:
        email:
          active: true
          user: luan.peng@intellif.com
          password: xxxxxx
          host: smtp.exmail.qq.com
          port: 465
          ssl: true
        report:
          active: true

创建configmap

kubectl create -f configmap.yaml

然后在pod里面挂载configmap成pod目录文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: logging
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      volumes:
      - name: config
        configMap:
          name: kibana-config
      containers:
      - name: kibana-logging
        image: luanpeng/lp:kibana-oss-6.2.4     
        imagePullPolicy: Always
        command: ["/usr/local/bin/kibana-docker"]   # 
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch-logging:9200   
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
        volumeMounts:
        - name: config
          mountPath: /opt/kibana/config/kibana.yml
          subPath: kibana.yml
          readOnly: False