ElastAlert告警搭建

ElastAlert告警搭建

ElastAlert钉钉告警

基础环境

服务器安装Python3.6.9

先查看下机器py版本，如果是3.6，则执行以下
 
apt -y install wget openssl openssl-devel gcc gcc-c++
 
wget https://www.python.org/ftp/python/3.6.9/Python-3.6.9.tgz
tar xf Python-3.6.9.tgz
 
cd Python-3.6.9
./configure --prefix=/usr/local/python --with-openssl
make && make install
 
mv /usr/bin/python /usr/bin/python_old
ln -s /usr/local/python/bin/python3 /usr/bin/python
ln -s /usr/local/python/bin/pip3 /usr/bin/pip
 
pip install --upgrade pip
sed -i '1s/python/python2.7/g' /usr/bin/yum
sed -i '1s/python/python2.7/g' /usr/libexec/urlgrabber-ext-down
 
python -V
显示为3.6.9

从GitHub上拉取源码至本地

进入opt文件夹创建Dingtalk_ElastAlert文件夹
mkdir -p /opt/Dingtalk_ElastAlert
cd /opt/Dingtalk_ElastAlert
从GitHub上拉取源码至本地
git clone https://github.com/Yelp/elastalert.git

下载钉钉报警模块至本地

wget https://github.com/xuyaoqiang/elastalert-dingtalk-plugin/archive/master.zip

配置钉钉机器人，复制其webhook

钉钉告警模块为elastalert_modules

安装ElastAlert

进入Dingtalk_ElastAlert文件后

编写项目配置脚本start.sh

#!/bin/bash
cd /opt/dingtalk_elastalert/elastalert/
python setup.py install
pip3 install --upgrade pip
pip3 install -r requirements.txt -i  https://pypi.tuna.tsinghua.edu.cn/simple
mkdir dingding
cd dingding
mv /opt/dingtalk_elastalert/master.zip .
apt install unzip
unzip master.zip
cd elastalert-dingtalk-plugin-master
pip3 install  pyOpenSSL==16.2.0 -i  https://pypi.tuna.tsinghua.edu.cn/simple
pip3 install  setuptools==46.1.3 -i  https://pypi.tuna.tsinghua.edu.cn/simple
cd /opt/dingtalk_elastalert/elastalert/
cp -r /opt/dingtalk_elastalert/elastalert/dingding/elastalert-dingtalk-plugin-master/elastalert_modules/ /opt/dingtalk_elastalert/elastalert/
cd /opt/dingtalk_elastalert/elastalert/
cp -r example_rules rules
cp config.yaml.example config.yaml

修改config配置文件

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
#指定告警文件存放位置
rules_folder: rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
#设置向ES发送请求的时间
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
#用来设置请求里时间字段的范围 时间为1分钟
buffer_time:
  minutes: 1

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
#设置ES地址
es_host: esip

# The Elasticsearch port
#设置ES的端口
es_port: 9200

# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
#  version: 1
#  incremental: false
#  disable_existing_loggers: false
#  formatters:
#    logline:
#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
#    handlers:
#      console:
#        class: logging.StreamHandler
#        formatter: logline
#        level: DEBUG
#        stream: ext://sys.stderr
#
#      file:
#        class : logging.FileHandler
#        formatter: logline
#        level: DEBUG
#        filename: elastalert.log
#
#    loggers:
#      elastalert:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch.trace:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      '':  # root logger
#        level: WARN
#          handlers:
#            - console
#            - file
#        propagate: false

编写钉钉告警文件的告警规则

#规则的唯一名称。如果相同，则elastalert不会启动。
name: API错误响应(status >= 400)
#数据验证方式(规则类型)
type: frequency
#要查询的索引名称。默认logstash-*
index: mimo-*
#定时向ES发请求
num_events: 1
timeframe:
        minutes: 24
#query查询语法，将需要匹配的信息给匹配
filter:
        - query:
                query_string:
                        query: "Message: 500.jsp"
#每个匹配项上运行的警报列表。
alert:
        - "elastalert_modules.dingtalk_alert.DingTalkAlerter"
#钉钉机器人的webhook值
dingtalk_webhook: "webhook"
dingtalk_msgtype: text

编写Dockerfile进行镜像封装

#将项目文件，启动脚本，钉钉告警模块共同大打包镜像
FROM python:3.6.9
COPY ./elastalert /opt/dingtalk_elastalert/elastalert
COPY ./start.sh /opt/dingtalk_elastalert/
COPY ./master.zip /opt/dingtalk_elastalert/
RUN sh /opt/dingtalk_elastalert/start.sh
WORKDIR /opt/dingtalk_elastalert/elastalert/
EXPOSE 3030

docker镜像打包

docker build -t dingtalk_elastalert .

编写docker-compose.yml文件启动dingtalk_elastalert容器

#使用主机网络
#在容器中通过elastalert-test-rule验证钉钉告警配置文件是否正确
#通过python -m elastalert.elastalert --config ./config.yaml --rule ./rules/api_error.yaml启动项目
#将rules（告警规则）挂载
#将congif配置文件进行挂载
version: '3'
services:
        dingtalk_elastalert:
                image: dingtalk_elastalert:latest
                container_name: dingtalk_elastalert
                command:
                        - sh
                        - -c
                        - |
                          #      tail -f /dev/null
                                pip3 install cryptography
                                elastalert-test-rule rules/api_error.yaml
                                python -m elastalert.elastalert --config ./config.yaml --rule ./rules/api_error.yaml
                volumes:
                        - ./rules:/opt/dingtalk_elastalert/elastalert/rules
                        - ./config.yaml:/opt/dingtalk_elastalert/elastalert/config.yaml
                network_mode: "host"

启动容器

docker-compose up -d && docker-compose logs -f 

观察日志是否有采集到日志，查看是否报错