ELFK采集某业务系统日志

阿彪大大

已于 2024-03-06 15:18:15 修改

阅读量374

点赞数 5

文章标签： elk

于 2024-03-06 14:47:10 首次发布

本文链接：https://blog.csdn.net/u012173915/article/details/136505223

版权

本文介绍了如何在Windows服务器上安装Filebeat，配置其从多个日志文件中收集信息，并使用multiline模式解析。同时，详细说明了如何配置Logstash接收Kafka消息，过滤和格式化Sirm应用的日志，并将Sirm相关的日志发送到Elasticsearch集群。

摘要由CSDN通过智能技术生成

1.在业务系统服务器安装 filebeat

2.在 C:\ProgramData\Elastic\Beats\filebeat 目录添加配置文件 filebeat.yml，内容如下：

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\log\default.log
  encoding: GB2312 #应对Windows文件内容里边中文的编码
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  fields:
    topic: default
- type: log
  enabled: true
  paths:
    - C:\log\kettle\kettle.log
  encoding: GB2312
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  fields:
    topic: kettle
- type: log
  enabled: true
  paths:
    - C:\log\portfolio\portfolio.log
  encoding: GB2312
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  fields:
    topic: portfolio
- type: log
  enabled: true
  paths:
    - C:\log\sirm\sirm.log
  encoding: GB2312
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  fields:
    topic: sirm
fields:
  ip: "10.1.5.92"
output.kafka:
  enabled: true
  hosts: ["10.1.5.9:9092"]
  topic: '%{[fields.topic]}' 
  partition.round_robin:
    reachable_only: true
  worker: 2
  required_acks: 1
  compression: gzip
  max_message_bytes: 10000000
logging.level: info

3.安装并配置logstash，内容如下：

input {
    kafka {
        bootstrap_servers => "10.1.5.9:9092"
        group_id => "app_logs"
        client_id => "sirm1"
        id => "sirm1"
        topics => ["default","kettle","portfolio","sirm"]
        codec => json {
           charset => "UTF-8"
        }
        consumer_threads => 1
        add_field => { "[@metadata][appname]" => "sirm" }
    }
}
filter {
    if [@metadata][appname] == "sirm" {
      grok{
        match => ["message","\[%{DATA}\]\[%{DATA}\]\[%{TIMESTAMP_ISO8601:timestamp}\]%{GREEDYDATA:info}"]
      }
      date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss"]
      }
      mutate {
        gsub => ["message", "\\x", "\\\x"]
        remove_field => ["@version","agent","event","ecs","input","tags","timestamp","info"]
      }
    }
}
output {
	if [@metadata][appname] == "sirm" {
		elasticsearch {
			hosts => ["10.1.5.13:9200","10.1.5.14:9200","10.1.5.15:9200"]
			user => "elastic"
			password => "app!236"
			ssl_enabled => true
			ssl_certificate_authorities => "/home/app/logstash/config/elasticsearch-ca.pem"
			index => "sirm-%{+YYYY.MM}"
		}
	}
}