rsyslog+elk 网络设备日志收集及钉钉报警

目录

目录

一.概要

二.实施

1.数据源

2.rsyslog

3.elasticsearch

4.logstash

5.kibana

 三.日志展示

1.打开kibana页面

四·钉钉报警

1.elastalert

2.dingtalk

3.rule

4.报警脚本

5.启动脚本

6.报警测试

五.附录

---

---

一.概要

二.实施

1.数据源

1.网络设备日志，可用模拟器如华为的ensp进行模拟，并发送所需日志到rsyslog服务端。实验环境为华为ar3260，如下命令开启info-center功能

info-center loghost 192.168.21.128 facility local5 //此处local5需要与rsyslog配置文件设置一致
info-center logfile channel 2

2.rsyslog

1.安装

yum install -y rsyslog-elasticsearch（注意该版本rsyslog可以利用omelasticsearch.so模块直接发送到elasticsearch6.0以下版本，但笔者所用为6.4.2，因此并未使用om模块）

2.配置文件，添加下面一行

3.rsyslog接收到的日志大致如下 ，后文所做的mypattern文件依此做正则匹配

3.elasticsearch

1.官方下载并解压

tar -zxvf elasticsearch-6.4.2.tar.gz

2.调整配置文件elasticsearch.yml，添加如下

network.host: 0.0.0.0
http.port: 9200

2.调整打开文件数及打开进程数，vim /etc/security/limits.conf，添加以下四行:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

3.调整/etc/sysctl.conf并用sysctl -p生效

vm.max_map_count = 655360

4. 启动elasticsearch

./elasticsearch

4.logstash

            1.官网下载logstash并解压

tar -zxvf logstash-6.4.3.tar.gz

2.建立文件，命名为mypatterns_file

#TIME \w+\s\d+\s\d+:\d+:\d+
#DEVICE_NAME \w+
#OUTLINE .*:
#CONTENT .*
#华为交换设备
TIME \w+\s+\d+\s\d+:\d+:\d+
DEVICE_NAME \w+-\w+-*\w*
OUTLINE %.*?:
CONTENT .*

3.cd到解压路径下，建立network.conf

input {
    file { path => "/var/log/network-rsy.log" type => "rsyslog"}    //数据源
}
filter {
    grok {
        partterns_dir => {"/home/elk/logstash-6.4.3/config/mypatterns_file"}    //mypatterns_file为上一步所建立
        match => {
            #"message"=>"%{TIME:time}\s%{DEVICE_NAME:device_name}\s%{OUTLINE:outline}\s%{CONTENT:content}"     
            "message"=>"%{TIME:time}\s.*\+08:00\s%{DEVICE_NAME:device_name}\s%{OUTLINE:outline}%{CONTENT:content}" //%{X:Y}这个格式X即为上一步文件中的名称，Y为json化后的key值
        }
    }
}
output {
    elasticsearch {
        host => "192.168.21.128:9200"    //elasticsearch服务端
        index => "network_device"    //elasticsearch中建立的索引名称
    }
    stdout {}    //输出到标准输出，可用于排查
}

 4.启动logstash

./logstash -f ../config/network.conf

5.kibana

1.官方下载并解压

tar -zxvf kibana-6.4.2-linux-x86_64.tar.gz

2.修改配置文件，config/kibana.yml，添加下面一行:

server.host: "0.0.0.0"

3 .启动kibana

./kibana

 三.日志展示

1.打开kibana页面

http://x.x.x.x:5601

四·钉钉报警

1.elastalert

GitHub - Yelp/elastalert: Easy & Flexible Alerting With ElasticSearch下载安装elastalert

2.dingtalk

elastalert自身不带钉钉报警可下载第三方插件，GitHub - xuyaoqiang/elastalert-dingtalk-plugin: elastalert 的钉钉报警插件下载钉钉报警插件后将elastalert-dingtalk-plugin中的elastalert_modules、rules和config.yaml（替换原elastalert的config.yaml）复制到elastalert下，并更改config.yaml:

es_host: 192.168.21.129 //es地址
es_port: 9200 //es端口

3.rule

于./elastalert/rules下添加yaml文件

name: 网络日志测试
type: any
es_host: 192.168.21.129
es_port: 9200
index: network_device
realert:
  minutes: 0 //设置为0可以指定监听的时间范围内 连续发送报警，举例若为1 则表示在发送报警后，将忽略匹配该规则的其他报警直到下一分钟重新在生效，英文释义如下
That means that after the first alert, it will ignore matches for that rule for the next minute.
alert:
- elastalert_modules.dingtalk_alert.DingTalkAlerter
dingtalk_webhook: 此处填写钉钉机器人的webhook
dingtalk_msgtype: markdown

4.报警脚本

修改dingtalk_alert.py，脚本启动命令为python -m elastalert.elastalert --verbose由于原脚本的发送格式为text，并不是我希望的发送格式，特意修改为markdown格式发送，如下

def alert(self,matches):
        headers = {
            "Content-Type": "application/json",
            "Accept": "application/json;charset=utf-8"
        }
        server_time=matches[0]["@timestamp"]
        device_time=matches[0]["time"]
        device_name=matches[0]["device_name"]
        summary=matches[0]["outline"]
        content=matches[0]["content"]
        body = u'#[服务器接收时间]%s\n- [设备发送时间]%s\n- [设备名称]%s\n- [概要]%s\n- [详细内容]%s' %(server_time,device_time,device_name,summary,content)
        payload = {
             'msgtype': self.dingtalk_msgtype,
             'markdown': {
                 'title':'test'+'\n',
                 'text': body
             },
             'at': {
                  'isAtAll': False
             }
        }
        try:
            response = requests.post(self.dingtalk_webhook_url,
                        data=json.dumps(payload, cls=DateTimeEncoder),
                        headers=headers)
            response.raise_for_status()
        except RequestException as e:
            raise EAException("Error request to Dingtalk: {0}".format(str(e)))

5.启动脚本

python -m elastalert.elastalert --verbose --config /root/elastalert-masterrules/network_device.yaml

6.报警测试

触发一条日志记录，按照rsyslog->logstash->elasticsearch->elastalert->ding_talk->钉钉端的递送逻辑，其测试结果如下（内容为测试内容，不必细究）

五.附录

这个logstash配置文件作为样版 以供以后语法参考

input {
	file { 
		path => "/var/log/network-rsys.log" 
		tags => ["syslog"]
	}
	file { 
		path => "/var/log/network-rsys-huawei-fw.log" 
		tags => "huawei-fw" 
	}
	file {
		path => "/var/log/USG6600/*.log"
		tags => "huawei-fw-session"
		ignore_older => "1 day"
		codec => plain { charset => "GBK" }
	}
}
filter {
	if "syslog" in [tags] {
		grok {
			patterns_dir => ["/root/logstash-6.4.3/config/mypatterns_file"]
				match => {
					"message"=>"%{TIME:time}\s.*\+08:00\s%{DEVICE_NAME:device_name}\s%{OUTLINE:outline}%{CONTENT:content}"
				}
			}
		}
	else if "huawei-fw" in [tags] {
		grok {
			patterns_dir => ["/root/logstash-6.4.3/config/mypatterns_file"]
				match => {
					"message"=>"%{TIME:time}\s%{DEVICE_FW_NAME:device_fw_name}\s%{OUTLINE:outline}%{CONTENT:content}"
				}
			}
	}
	else if "huawei-fw-session" in [tags] {
		grok {
			patterns_dir => ["/root/logstash-6.4.3/config/mypatterns_file"]
				match => {
					"message"=>"%{TIME:time}\s%{DEVICE_FW_NAME:device_fw_name}\s%{OUTLINE:outline}%{CONTENT:content}"
				}
			}
	}
}
output {
	if "syslog" in [tags] {
		elasticsearch {
			hosts => ["192.168.170.108:9200","192.168.170.109:9200","192.168.170.112:9200","192.168.170.113:9200","192.168.170.114:9200","192.168.170.115:9200","192.168.170.116:9200","192.168.170.117:9200","192.168.170.118:9200","192.168.170.119:9200","192.168.170.120:9200","192.168.170.121:9200","192.168.170.122:9200","192.168.170.123:9200","192.168.170.124:9200","192.168.170.125:9200","192.168.170.126:9200","192.168.170.127:9200","192.168.170.128:9200","192.168.170.129:9200"]
			index => "network_device_log"
		}
	}
	else if "huawei-fw" in [tags] {
		elasticsearch {
			hosts => ["192.168.170.108:9200","192.168.170.109:9200","192.168.170.112:9200","192.168.170.113:9200","192.168.170.114:9200","192.168.170.115:9200","192.168.170.116:9200","192.168.170.117:9200","192.168.170.118:9200","192.168.170.119:9200","192.168.170.120:9200","192.168.170.121:9200","192.168.170.122:9200","192.168.170.123:9200","192.168.170.124:9200","192.168.170.125:9200","192.168.170.126:9200","192.168.170.127:9200","192.168.170.128:9200","192.168.170.129:9200"]
			index => "network_device_huawei_fw"
		}
	}
	else if "huawei-fw-session" in [tags] {
		elasticsearch {
			hosts => ["192.168.170.108:9200","192.168.170.109:9200","192.168.170.112:9200","192.168.170.113:9200","192.168.170.114:9200","192.168.170.115:9200","192.168.170.116:9200","192.168.170.117:9200","192.168.170.118:9200","192.168.170.119:9200","192.168.170.120:9200","192.168.170.121:9200","192.168.170.122:9200","192.168.170.123:9200","192.168.170.124:9200","192.168.170.125:9200","192.168.170.126:9200","192.168.170.127:9200","192.168.170.128:9200","192.168.170.129:9200"]
			index => "network_device_huawei_fw_session"
		}
	}
}