使用fluentbit采集日志

使用fluentbit采集日志

最近尝试在本地搭建一个elk的demo，由于最新的ES现在是商业收费的，就考虑采用开源的opensearch作为后台的数据存储，在opensearch的安装页面，看到了fluentbit这个日志采集方案（最新的filebeat和logstash似乎也商业收费，没有仔细研究它们的license），之前研究filebeat 7.x版本也有一些非常不好的回忆，借此机会玩一玩fluentbit这个东西。

安装

强烈推荐使用docker进行安装，fluentbit提供了一些安装脚本，但是我在aliyunos和mac上都失败了，最终还是采用了docker-compose的方式，简单省力

version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./data.log:/fluent-bit/log/data.log
      - .logs/:/fluent-bit/log/telemetry/

这里挂载了两个文件，第一个是fluent-bit.conf的配置文件，另一个是一个测试的log文件，第三个是我写了一个demo程序，产生的日志放在了这个文件下。
强烈建议将这个目录挂载进来，最初我只挂载了一个log文件，但是当log发生rollout的时候，不知道是不是因为挂载的原因，fluentBit容器里inode未发生变化导致fluentBit检测不到rollout的发生，不会将最新的日志收集上去。

配置

[SERVICE]
    flush        1
    log_level    debug
    parsers_file parsers.conf
    Time_Zone     "+0800"

[INPUT]
    name        tail
    path        /fluent-bit/log/data.log
    parser      json

[INPUT]
    name        tail
    path        /fluent-bit/log/telemetry/telemetry.log
    DB          /var/log/flb_metrics.db
    parser      json
    Path_Key    path

[OUTPUT]
    Name  opensearch
    Match *
    Host  host.docker.internal
    Port  9200
    Suppress_Type_Name On
    HTTP_User fluentbit
    HTTP_Passwd ZhangTest12
    TLS On
    tls.verify Off

    Logstash_Format On
    Logstash_Prefix logs-metrics
   

这里先看INPUT

• Path_Key，这个参数会把文件路径输出到output，设置为path就会输出{“path”:“/fluent-bit/log/telemetry/telemetry.log”}
• name，name表明你使用的插件类型，tail就是从文件末尾开始检测
• parser 解析器，这里我在打印日志的时候直接使用了json
• DB fluentbit内置DB的存放路径

如有多种类型的来源，可以设置多个INPUT标签，彼此之间通过参数tag做区分

再看output

• name，同input种的name，这里使用opensearch
• Match，用于匹配Input
• Host，我的opensearch也是使用docker启动的，正常为你的IP
• HTTP_User， HTTP_Passwd，opensearch的用户密码
• TLS，当前版本opensearch是启用TLS的
• tls.verify Off，不校验证书
• Suppress_Type_Name On，现在无论opensearch还是ES都不在支持指定type，不打开的话会报错
• Logstash_Format On，启用logstash的格式
• Logstash_Prefix logs-metrics，配置index的格式，该项配置产生的index name为logs-metrics-yyyyMMdd

启动

使用

docker-compose up

即可启动

Attaching to fluent-bit-1
fluent-bit-1  | Fluent Bit v3.0.3
fluent-bit-1  | * Copyright (C) 2015-2024 The Fluent Bit Authors
fluent-bit-1  | * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
fluent-bit-1  | * https://fluentbit.io
fluent-bit-1  | 
fluent-bit-1  | ___________.__                        __    __________.__  __          ________  
fluent-bit-1  | \_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
fluent-bit-1  |  |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
fluent-bit-1  |  |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
fluent-bit-1  |  \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
fluent-bit-1  |      \/                     \/     \/               \/                        \/ 
fluent-bit-1  | 
fluent-bit-1  | [2024/05/22 09:49:39] [ info] Configuration:
fluent-bit-1  | [2024/05/22 09:49:39] [ info]  flush time     | 1.000000 seconds
fluent-bit-1  | [2024/05/22 09:49:39] [ info]  grace          | 5 seconds
fluent-bit-1  | [2024/05/22 09:49:39] [ info]  daemon         | 0
fluent-bit-1  | [2024/05/22 09:49:39] [ info] ___________
fluent-bit-1  | [2024/05/22 09:49:39] [ info]  inputs:
fluent-bit-1  | [2024/05/22 09:49:39] [ info]      tail
fluent-bit-1  | [2024/05/22 09:49:39] [ info]      tail
fluent-bit-1  | [2024/05/22 09:49:39] [ info] ___________
fluent-bit-1  | [2024/05/22 09:49:39] [ info]  filters:
fluent-bit-1  | [2024/05/22 09:49:39] [ info] ___________
fluent-bit-1  | [2024/05/22 09:49:39] [ info]  outputs:
fluent-bit-1  | [2024/05/22 09:49:39] [ info]      stdout.0
fluent-bit-1  | [2024/05/22 09:49:39] [ info]      opensearch.1
fluent-bit-1  | [2024/05/22 09:49:39] [ info] ___________
fluent-bit-1  | [2024/05/22 09:49:39] [ info]  collectors:
fluent-bit-1  | [2024/05/22 09:49:39] [ info] [fluent bit] version=3.0.3, commit=3529bbb132, pid=1
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
fluent-bit-1  | [2024/05/22 09:49:39] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
fluent-bit-1  | [2024/05/22 09:49:39] [ info] [cmetrics] version=0.9.0
fluent-bit-1  | [2024/05/22 09:49:39] [ info] [ctraces ] version=0.5.1
fluent-bit-1  | [2024/05/22 09:49:39] [ info] [input:tail:tail.0] initializing
fluent-bit-1  | [2024/05/22 09:49:39] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [tail:tail.0] created event channels: read=21 write=22
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [input:tail:tail.0] inotify watch fd=27
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [input:tail:tail.0] scanning path /fluent-bit/log/data.log
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [input:tail:tail.0] inode=9 with offset=927 appended as /fluent-bit/log/data.log
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [input:tail:tail.0] scan_glob add(): /fluent-bit/log/data.log, inode 9
fluent-bit-1  | [2024/05/22 09:49:39] [debug] [input:tail:tail.0] 1 new files found on path '/fluent-bit/log/data.log'
fluent-bit-1  | [2024/05/22 09:49:39] [ info] [input:tail:tail.1] initializing

整个配置的git地址，文中提到的opensearch和一个grafana的配置也在其中
https://gitee.com/xiiiao/docker-compose-example