通过filebeat向es发送日志

下载filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz

下载后解压缩

配置filebeat

#输入
filebeat.inputs:
- type: log
  # 启用
  enabled: true

  # 监控指定日志
  paths:
    - /uhealthy/logs/log.log
  # 匹配规则 如果不是数字开头的，会把数据合并到上一行，例如java报错
  multiline.pattern: ^2
  multiline.negate: true
  multiline.match: after

#输出到es服务
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["ip:9200"]
  enabled: true

启动

nohup ./filebeat -e -c filebeat.yml -d "Publish" & > nohup.out

校验

遇到问题

版本信息：filebeat-7.7.0-linux-x86_64.tar.gz
启动方式：nohup ./filebeat -c filebeat.yml &
导致问题：运行一段时候后自动关闭，日志如下

2020-05-26T07:20:52.806+0800	INFO	beater/filebeat.go:449	Stopping filebeat
2020-05-26T07:20:52.806+0800	INFO	beater/crawler.go:138	Stopping Crawler
2020-05-26T07:20:52.806+0800	INFO	beater/crawler.go:148	Stopping 1 inputs
2020-05-26T07:20:52.806+0800	INFO	cfgfile/reload.go:238	Dynamic config reloader stopped
2020-05-26T07:20:52.806+0800	INFO	input/input.go:149	input ticker stopped
2020-05-26T07:20:52.806+0800	INFO	input/input.go:167	Stopping Input: 4779968662065981764
2020-05-26T07:20:52.806+0800	INFO	harvester/forwarder.go:52	Input outlet closed
2020-05-26T07:20:52.806+0800	INFO	log/harvester.go:320	Reader was closed: /home/yezhou/applogs/eureka-server.log. Closing.
2020-05-26T07:20:52.806+0800	INFO	beater/crawler.go:164	Crawler stopped
2020-05-26T07:20:52.806+0800	INFO	registrar/registrar.go:367	Stopping Registrar
2020-05-26T07:20:52.806+0800	INFO	registrar/registrar.go:293	Ending Registrar
2020-05-26T07:20:52.835+0800	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":27060,"time":{"ms":27060}},"total":{"ticks":54630,"time":{"ms":54631},"value":54630},"user":{"ticks":27570,"time":{"ms":27571}}},"handles":{"limit":{"hard":100002,"soft":100001},"open":12},"info":{"ephemeral_id":"73be5fc1-5dff-4739-8e53-3c3fe50f3fef","uptime":{"ms":30851737}},"memstats":{"gc_next":12299920,"memory_alloc":7304424,"memory_total":8235809224,"rss":42262528},"runtime":{"goroutines":24}},"filebeat":{"events":{"added":23527,"done":23527},"harvester":{"closed":4,"open_files":0,"running":0,"started":4}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":23515,"batches":7054,"total":23515},"type":"kafka"},"outputs":{"kafka":{"bytes_read":385840,"bytes_write":6362352}},"pipeline":{"clients":0,"events":{"active":0,"filtered":12,"published":23515,"retry":2048,"total":23527},"queue":{"acked":23515}}},"registrar":{"states":{"cleanup":2,"current":4,"update":23527},"writes":{"success":7065,"total":7065}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.025,"5":0.005}}}}}}
2020-05-26T07:20:52.835+0800	INFO	[monitoring]	log/log.go:154	Uptime: 8h34m11.756256865s
2020-05-26T07:20:52.835+0800	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2020-05-26T07:20:52.835+0800	INFO	instance/beat.go:444	filebeat stopped.

解决方案

自定义Service方式启动

vim /usr/lib/systemd/system/filebeat.service

# 将如下信息copy到文件中

[Unit]
Description=Filebeat is a lightweight shipper for metrics.
Documentation=https://www.elastic.co/products/beats/filebeat
Wants=network-online.target
After=network-online.target

[Service]
Environment="LOG_OPTS=-e"
Environment="CONFIG_OPTS=-c /usr/local/filebeat/filebeat.yml"
Environment="PATH_OPTS=-path.home /usr/local/filebeat -path.config /usr/local/filebeat -path.data /usr/local/filebeat/data -path.logs /usr/local/filebeat/logs"
ExecStart=/usr/local/filebeat/filebeat $LOG_OPTS $CONFIG_OPTS $PATH_OPTS
Restart=always

[Install]
WantedBy=multi-user.target

 授权 加载 重启

chmod +x /usr/lib/systemd/system/filebeat.service
systemctl daemon-reload
systemctl enable filebeat
systemctl start filebeat

自动邮件预警功能

参考 https://blog.csdn.net/wk2yangyang/article/details/118638046