日志监控--elastalert

最新推荐文章于 2023-05-05 09:32:28 发布

_0x00

最新推荐文章于 2023-05-05 09:32:28 发布

阅读量408

点赞数 1

分类专栏：云安全文章标签： python linux 开发语言

本文链接：https://blog.csdn.net/qq_39325340/article/details/126757810

版权

云安全专栏收录该内容

6 篇文章 0 订阅

订阅专栏

依赖环境

系统：Centos7
依赖： python3.6.9

安装python

cd /tmp
# 使用yum安装依赖
yum -y install gcc libffi-devel python-devel openssl-devel4 ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev 
yum -y install wget openssl openssl-devel gcc gcc-c++ zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel xz xz-devel make git vim -y
# 下载PYTHON源码包
wget https://www.python.org/ftp/python/3.6.9/Python-3.6.9.tgz
# 解压python源码包
tar -zxvf Python-3.6.9.tgz
# 切换到解压包目录下
cd Python-3.6.9
# 编译配置：指定编译包目录/usr/local/python3
./configure --prefix=/usr/local/python3 --with-openssl
# 编译安装
make && make install
# 软链接文件命令到bin目录
ln -s /usr/local/python3/bin/python3 /usr/bin/
ln -s /usr/local/python3/bin/pip3 /usr/bin/
# 查看python版本, 是否安装成功
python3 -V

安装Elastalert

# 切换到上级目录
cd ..

pip3 install --upgrade pip
# 源码安装elastalert
git clone https://github.com/Yelp/elastalert.git
cd elastalert
pip3 install "elasticsearch<7,>6"
pip3 install -r requirements.txt
python3 setup.py install
# 查看是否安装成功, 安装成功后可以看到四个命令
ll /usr/local/python3/bin/
/usr/local/python3/bin/elastalert
/usr/local/python3/bin/elastalert-create-index
/usr/local/python3/bin/elastalert-rule-from-kibana
/usr/local/python3/bin/elastalert-test-rule
# 软连接文件命令到bin目录
ln -s /usr/local/python3/bin/elastalert* /usr/bin/
# 报错SyntaxError: invalid syntax
pip3 uninstall jira 
pip3 install jira>2.0.0
# 执行命令,查看是否存在问题
elastalert -h

配置

# 配置全局文件，修改其中的es为自己的es地址;如:
cat config.yaml.example |grep -vE "^#" > config.yaml
    rules_folder: rules
    run_every:
      minutes: 1
    buffer_time:
      minutes: 15
    es_host: 11.2.2.80
    es_port: 9200
    writeback_index: elastalert_status
    writeback_alias: elastalert_alerts
    alert_time_limit:
      days: 2
      
# 执行elastalert-create-index命令在ES创建索引，这不是必须的步骤，但是强烈建议创建。因为对于审计和测试很有用，并且重启ES不影响计数和发送alert。
elastalert-create-index
    Elastic Version: 7.3.2
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!
# 看到这个输出，就说明创建成功了，也可以请求一下看看：
curl es_adress_ip:9200/_cat/indices?v
    health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    green  open   elastalert_status_status  lh8LL4iCQeSn0afyzxBX7w   1   1          0            0       460b           230b
    green  open   elastalert_status         i7B7IfCuSb2Sex8U5KoTZg   1   1          0            0       460b           230b
    green  open   elastalert_status_past    et2aF44VR4WQnxB8T7zD4Q   1   1          0            0       460b           230b
    green  open   elastalert_status_silence lhXHEsuUQeGZaW3cRLp5pQ   1   1          0            0       460b           230b
    green  open   elastalert_status_error   zykwk4KtSyyOY7ckxQTrkA   1   1          0            0       460b           230b

# 编写规则， 如:
vim nginx.yaml
    type: frequency
    index: mytest*
    num_events: 3
    timeframe:
      minutes: 1
    filter:
     - query:
        query_string:
          query: "message: sd"
    alert_text: "TTTTTTTtest"
    alert:
     - "post"
    http_post_url: "http://127.0.0.1:5000/elk_alarm"
# 测试规则是否可以运行, 没有报错，则说明可用
elastalert-test-rule --config config.yaml nginx.yaml
elastalert-test-rule --config config.yaml nginx.yaml --alert
# 启动方式有两种
# (1)指定规则文件路径
elastalert --verbose --config config.yaml --rule nginx.yaml
elastalert --verbose --config config.yaml --rule nginx.yaml --debug --es_debug --es_debug_trace test.log  # debug模式运行, 日志信息保存到test.log文件
# (2)在全局路径config.yaml下,配置规则存放在加载规则rules目录下
elastalert --debug  #启动debug模式会禁止报警

常见的一些问题

关于命中规则，不报警的问题

请参考官方说明，有三种方式导致不报警，其中一种是开启debug，另外两种是规则编写方式导致:
https://github.com/Yelp/elastalert#how-can-i-prevent-duplicate-alerts

关于时间格式问题

解决时间戳格式问题，修改全局配置config.yaml, 如下:
rules_folder: rules
run_every:
  minutes: 1
buffer_time:
  minutes: 15
es_host: 11.2.2.80
es_port: 9200
writeback_index: elastalert_status
writeback_alias: elastalert_alerts
alert_time_limit:
  days: 2
# 指定时间字段和对应的格式
timestamp_field: update_time
timestamp_format: "%Y-%m-%d"
timestamp_type: custom

测试，写入es数据

curl -XPOST 'http://1.1.1.1:9200/mytest_customer/_doc/9?pretty' -H 'Content-Type: application/json' -d '{ "customer_code": 22222243434434,"update_time" : "2020-12-09", "city_name" : "helloworld"}'
# mytest_customer 为索引名
# 9 为_id, 也可以不写
# -d 后面为测试数据内容

curl -XPOST 'http://1.1.1.1:9200/mytest_customer/_search?pretty'

场景

用于做态势感知
用于运维类日志分析
…

参考

https://segmentfault.com/a/1190000041943196
https://www.cnblogs.com/wzxmt/p/12180555.html
https://zhuanlan.zhihu.com/p/386723212
https://www.cnblogs.com/dance-walter/p/11194001.html
https://www.freebuf.com/sectool/164591.html
https://cloud.tencent.com/developer/article/1509404 – es操作
https://godleon.github.io/blog/Elasticsearch/Elasticsearch-getting-started/