背景:在Logstash数据源为日志文件操作 基础上进行
一、 拉取镜像
docker pull elastic/filebeat:7.6.2
二、编写filebeat配置文件
2.1 在目录D:\usr\local\etc\filebeat下分别创建docker-compose-filebeat.yml、filebeat.yml文件
2.2 docker-compose-filebeat.yml文件内容
version: '3'
services:
filebeat:
image: elastic/filebeat:7.6.2
container_name: filebeat
volumes:
- /d/usr/local/logs/filebeat:/usr/share/filebeat/logs
- /d/usr/local/etc/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
- /d/usr/local/var/filebeat/data:/usr/share/filebeat/data
2.3 filebeat.yml文件内容
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/share/filebeat/logs/*.log
output.logstash:
hosts: ["192.168.1.110:5044"]
2.4 修改D:\usr\local\etc\logstash\pipeline目录下logstash配置文件logstash.conf,新增beats数据源
input {
stdin { }
beats {
port => 5044
type => "filebeat"
}
tcp {
mode => "server"
host => "0.0.0.0"
# 从5044端口取日志
port => 5044
# 需要安装logstash-codec-json_lines插件
codec => json_lines
type => "tcp"
}
file {
# 容器中日志所在目录的文件
path => ["/usr/share/logstash/logs/*.log"]
#codec => "json"
sincedb_path => "NUL"
type => "spring"
start_position => "beginning"
}
}
output {
if [type] == "spring" {
elasticsearch {
hosts => ["192.168.1.110:9200"]
index => "spring-%{+YYYY.MM.dd}"
}
}
if [type] == "tcp" {
elasticsearch {
hosts => ["192.168.1.110:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
if [type] == "filebeat" {
elasticsearch {
hosts => ["192.168.1.110:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
}
}
stdout { codec => rubydebug }
}
三、编写脚本
3.1 编写日志脚本save_filebeat_log.py和启动脚本docker-filebeat-up.sh
3.2 save_filebeat_log.py文件内容
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import datetime
import threading
import os
# 递归级联创建的目录
path = "D:/usr/local/logs/filebeat"
if not os.path.exists(path):
os.makedirs(path)
today = datetime.date.today()
# 方法1: 此方法文件夹不存在会报错:FileNotFoundError: [Errno 2] No such file or directory
with open(f'D:/usr/local/logs/filebeat/filebeat-{today}.log', mode='a+', encoding='utf-8') as f:
now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
thread_id = threading.currentThread().native_id
log_str = f'{now} DEBUG {thread_id} --- [scheduling-1] org.jooq.tools.LoggerListener : ' \
'Executing query : select id from user'
f.writelines(f'{log_str}\n')
f.close()
print('保存日志成功')
3.3 docker-filebeat-up.sh文件内容
#!/bin/bash
docker-compose -f D:/usr/local/etc/filebeat/docker-compose-filebeat.yml up -d
四、启动容器
4.1 重启logstash容器
docker restart logstash
4.2 执行docker-filebeat-up.sh脚本,启动filebeat容器
./docker-filebeat-up.sh
4.3 执行save_filebeat_log.py脚本,模拟生成日志
python3 save_filebeat_log.py
五、查看结果
5.1 打开浏览器,查看请求http://localhost:9100/
从结果可以看出filebeat日志采集成功