前言
后续开发项目要用到 Filebeat 对日志做收集和处理。本文介绍了 ELK 技术中的 Filebeat,用于轻量级的日志收集和分析
参考资料:
官方文档:Filebeat overview | Filebeat Reference [8.14] | Elastic
一、安装部署
环境:
root@Andrew:~# cat /proc/version
Linux version 5.15.153.1-microsoft-standard-WSL2 (root@941d701f84f1) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Fri Mar 29 23:14:13 UTC 2024
Linux环境下安装命令:
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.14.3-linux-x86_64.tar.gz
tar xzvf filebeat-8.14.3-linux-x86_64.tar.gz
解压后将文件夹重命名,进入目录:
mv filebeat-8.14.3-linux-x86_64 filebeat
cd filebeat/
压缩包解压安装的filebeat文件结构与deb/rpm等安装方式不同,详见:Directory layout | Filebeat Reference [8.14] | Elastic
二、启动示例
在解压后的根目录下新建测试的配置文件:
vi test.yml
从控制台输入的配置如下:
filebeat.inputs:
- type: stdin
enable: true # default true
# include_lines/exclude_lines support regular expresions
include_lines: ["^andrew"] # starts with "andrew"
exclude_lines: ["www"] # exclude which contains "www"
tags: ["stdin"] # define tags in output
fields: # define specific fields in output
author: andrew
date: 2024-7-26
fields_in_root: false # default false
output.console:
enable: true # default true
pretty: true # default false
上述配置文件表示:
从标准输入读取输入
读取包括开头为"andrew",且不包括"www"的行
为输出的数据添加tags为"stdin"
为输出的数据添加fields字段 author为"andrew" date为"2024-7-26"
fields添加的字段不位于根路径下,
输出到控制台,开启pretty美化输出
输出以下命令启动filebeat
./filebeat -e -c test.yml
-e 参数指定输出到stderr而不是配置文件指定的输出(仍然需要配置,否则无法启动)
-c 指定启动的配置文件,未指定时默认为filebeat.yml
在标准输入中输入,“andrew”、“andreww”、“andrewww”,观察输出情况
andrew
andreww
andrewww
{
"@timestamp": "2024-07-26T03:15:21.470Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.14.3"
},
"log": {
"file": {
"path": ""
},
"offset": 0
},
"message": "andrew",
"tags": [
"stdin"
],
"input": {
"type": "stdin"
},
"fields": {
"author": "andrew",
"date": "2024-7-26"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "Andrew"
},
"agent": {
"id": "57b956ef-540a-45a4-bc04-dc192b75c801",
"name": "Andrew",
"type": "filebeat",
"version": "8.14.3",
"ephemeral_id": "b4c352d9-1ed1-4369-95ec-47455c21e272"
}
}
{
"@timestamp": "2024-07-26T03:15:22.628Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.14.3"
},
"log": {
"offset": 0,
"file": {
"path": ""
}
},
"message": "andreww",
"tags": [
"stdin"
],
"input": {
"type": "stdin"
},
"fields": {
"author": "andrew",
"date": "2024-7-26"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "Andrew"
},
"agent": {
"ephemeral_id": "b4c352d9-1ed1-4369-95ec-47455c21e272",
"id": "57b956ef-540a-45a4-bc04-dc192b75c801",
"name": "Andrew",
"type": "filebeat",
"version": "8.14.3"
}
}
观察到filebeat只输出了两个json,message分别为"andrew"、“andreww”,说明include_lines和exclude_lines生效,看到tags存在"stdin",以及fields下有两个字段"author: “andrew”, “date”: “2024-7-26”
从日志文件输入的配置如下:
filebeat.inputs:
- type: log
paths:
- /root/filebeat/logs/*.log
include_lines: ['^andrew']
exclude_lines: ['ww']
encoding: utf-8
tags: ['log']
fields:
author: andrew
fields_under_root: true
output.console:
pretty: true
enable: true
测试结果
root@Andrew:~/filebeat/logs# echo andrew >> a.log
root@Andrew:~/filebeat/logs# echo andreww >> a.log
root@Andrew:~/filebeat/logs# echo andrewww >> a.log
{
"@timestamp": "2024-07-27T08:02:55.742Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.14.3"
},
"host": {
"name": "Andrew"
},
"agent": {
"id": "57b956ef-540a-45a4-bc04-dc192b75c801",
"name": "Andrew",
"type": "filebeat",
"version": "8.14.3",
"ephemeral_id": "428921c4-3a41-4225-80b5-3d2a3c6aa4bb"
},
"log": {
"offset": 31,
"file": {
"path": "/root/filebeat/logs/a.log"
}
},
"message": "andrew",
"tags": [
"log"
],
"input": {
"type": "log"
},
"author": "andrew",
"ecs": {
"version": "8.0.0"
}
}
指定输出的配置(如Kafka、Elasticsearch等)参考官方文档
三、使用modules
通过 filebeat modules 命令操作模块化的配置
filebeat modules list # 查看所有modules
filebeat modules enable nginx # 启动nginx模块
filebeat modules disable nginx # 关闭nginx模块
编辑 modules.d 目录下的文件更改配置:
- module: nginx
# Access logs
access:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Error logs
error:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
ingress_controller:
enabled: false
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths: