win10下docker操作filebeat做日志采集

背景：在Logstash数据源为日志文件操作 基础上进行

一、 拉取镜像

docker pull elastic/filebeat:7.6.2 

二、编写filebeat配置文件

2.1 在目录D:\usr\local\etc\filebeat下分别创建docker-compose-filebeat.yml、filebeat.yml文件

2.2 docker-compose-filebeat.yml文件内容

version: '3'
services:
  filebeat:
    image: elastic/filebeat:7.6.2
    container_name: filebeat
    volumes:
        - /d/usr/local/logs/filebeat:/usr/share/filebeat/logs
        - /d/usr/local/etc/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
        - /d/usr/local/var/filebeat/data:/usr/share/filebeat/data

2.3 filebeat.yml文件内容

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/logs/*.log
output.logstash:
  hosts: ["192.168.1.110:5044"]

 2.4 修改D:\usr\local\etc\logstash\pipeline目录下logstash配置文件logstash.conf，新增beats数据源

input { 
    stdin { } 
    beats {
        port => 5044
        type => "filebeat"
    }
    tcp {
        mode => "server"
        host => "0.0.0.0"
        # 从5044端口取日志
        port => 5044
        # 需要安装logstash-codec-json_lines插件
        codec => json_lines
        type => "tcp"
    }
    file {
        # 容器中日志所在目录的文件
        path => ["/usr/share/logstash/logs/*.log"]
        #codec => "json"
        sincedb_path => "NUL"
        type => "spring"
        start_position => "beginning"
    }
}

output {
    if [type] == "spring" {
        elasticsearch {
            hosts => ["192.168.1.110:9200"]
            index => "spring-%{+YYYY.MM.dd}"
        }
    }
    if [type] == "tcp" {
        elasticsearch { 
            hosts => ["192.168.1.110:9200"] 
            index => "logstash-%{+YYYY.MM.dd}"  
        }
    }
    if [type] == "filebeat" {
        elasticsearch { 
            hosts => ["192.168.1.110:9200"] 
            index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        }
    }
	stdout { codec => rubydebug }
}

三、编写脚本

3.1 编写日志脚本save_filebeat_log.py和启动脚本docker-filebeat-up.sh

3.2 save_filebeat_log.py文件内容

#!/usr/bin/python
# -*- coding: UTF-8 -*-
import datetime
import threading

import os

# 递归级联创建的目录
path = "D:/usr/local/logs/filebeat"
if not os.path.exists(path):
    os.makedirs(path)
today = datetime.date.today()
# 方法1： 此方法文件夹不存在会报错：FileNotFoundError: [Errno 2] No such file or directory
with open(f'D:/usr/local/logs/filebeat/filebeat-{today}.log', mode='a+', encoding='utf-8') as f:
    now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
    thread_id = threading.currentThread().native_id
    log_str = f'{now} DEBUG {thread_id} --- [scheduling-1] org.jooq.tools.LoggerListener            : ' \
              'Executing query          : select id from user'
    f.writelines(f'{log_str}\n')
    f.close()
    print('保存日志成功')

3.3 docker-filebeat-up.sh文件内容

#!/bin/bash

docker-compose -f D:/usr/local/etc/filebeat/docker-compose-filebeat.yml up -d

四、启动容器

4.1 重启logstash容器

docker restart logstash

4.2 执行docker-filebeat-up.sh脚本，启动filebeat容器

./docker-filebeat-up.sh

4.3 执行save_filebeat_log.py脚本，模拟生成日志

python3 save_filebeat_log.py

五、查看结果

5.1 打开浏览器，查看请求http://localhost:9100/

从结果可以看出filebeat日志采集成功