利用FileBeat、Logstash、ElasticSearch实现web日志存储

qq_34324703

已于 2023-12-15 15:27:50 修改

阅读量101

点赞数

文章标签： elasticsearch 大数据搜索引擎

于 2023-10-17 15:58:02 首次发布

本文链接：https://blog.csdn.net/qq_34324703/article/details/133885088

版权

1下载FileBeat,版本选择8.6.1

https://www.elastic.co/cn/downloads/past-releases#filebeat

2 上传到服务器并解压，打开目录中的filebeat.yml，配置如下

filebeat.inputs:
#指定log文件
- type: log
  enabled: true
  paths:
    - /home/admin/app/api/logs/*.log   //web的日志文件
  #多行日志的打印策略
  multiline.pattern: '^\d{4}'
  multiline.negate: true
  multiline.match: after 
#输出至logstash服务
output.logstash:
  hosts: ["172.22.96.247:5044"]

3 下载同版本的logstash，上传并解压到服务器

4 在config下新建logstash-beat.conf

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
     beats {
        port => 5044   //接收filebeat采集数据
    }
}

filter{
    grok{
        match => ['message','%{TIMESTAMP_ISO8601:logTime} \[(?<thread>.*)\] %{WORD:level} (?<msg>.*)']
    }
    date {
	match => ["logTime","yyyy-MM-dd HH:mm:ss Z"]
	timezone => "UTC"
    	target => "@timestamp"
    }
    mutate{
        remove_field => ["@version"]
        remove_field => ["logTime"]
	remove_field => ["type"]
	remove_field => ["event"]
	remove_field => ["message"]
	remove_field => ["log"]
	remove_field => ["tags"]
	remove_field => ["agent"]
	remove_field => ["ecs"]
	remove_field => ["input"]
    }
}

output {
    elasticsearch {
        hosts => ["http://172.17.0.1:9200"] //宿主机的es地址
	index => "icloud-api-prod-%{+YYYY.MM.dd}"
        user => "elastic"
        password => "123456"
    }
    stdout { codec => rubydebug}
}

先启动logstash，在启动filebeat

#启动logstash
./bin/logstash -f config/logstash-beat.conf


#启动filebeat
./filebeat -e -c filebeat.yml


#后台方式启动filebeat
./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

注意，如果是云服务器，需要开启9200和5044端口

logstash用docker创建步骤

1 拉取docker镜像

docker pull logstash:8.6.1

2 新建logstash容器

docker run -p 5044:5044 -p 9600:9600 \
--privileged=true --name logstash \
-v /root/logstash.conf:/usr/share/logstash/logstash.conf \
-v /root/logstash.yml:/usr/share/logstash/logstash.yml\
-d logstash:8.6.1