云原生链路观测平台 openobserve + fluent-bit,日志收集

gitxuzan_

于 2024-09-18 11:15:25 发布

阅读量172

点赞数 1

文章标签：云原生 openobserve fluent-bit

本文链接：https://blog.csdn.net/qq_36517296/article/details/142134014

版权

grpc-opentracing
https://github.com/grpc-ecosystem/grpc-opentracing

openobserve + fluent-bit
为啥会选择这个组合一个 rust 写的一个是c写的，性能和内存方面不用担心，比java 那套好太多了

openobserve 文档：https://openobserve.ai/docs/user-guide/users/

fluent-bit 文档: https://docs.fluentbit.io/manual/installation/linux/ubuntu

openobserve 和 fluent-bit 都是二进制文件启动很简单，fluent-bit 负责传输，目前给出fluent-bit 配置
配置文件

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
[INPUT]
    Name        tail
    Path        /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*.log
    Tag         logs
    Parser      json_parser
    DB          ./flb_logs.db
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
    Rotate_Wait 10
    Path_Key    filename
    Exclude_Path /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*.log
# [FILTER]
#     Name        lua
#     Match       logs
#     Script      process_time.lua
#     Call        extract_time

[OUTPUT]
    Name        http
    Match       *
    Host        127.0.0.1
    Port        5080
    URI         /api/default/test4/_json
    tls         Off
    Format      json
#     Json_date_key    _timestamp
#     Json_date_format iso8601
    HTTP_User   admin@example.com
    HTTP_Passwd 2rlpVuHNxjDkceZd
    compress    gzip

# [OUTPUT]
#     Name        stdout
#     Match       *
#     Format      json_lines

还可以直接脚本过滤数据，或者添加字段
process_time.lua

function extract_time(tag, timestamp, record)
   return 1, timestamp, record
--     if record.time then
--         -- Parse the time string into a Unix timestamp
--         local year, month, day, hour, min, sec = record.time:match("(%d+)-(%d+)-(%d+) (%d+):(%d+):(%d+)")
--         if year then
--             local time = os.time({year=year, month=month, day=day, hour=hour, min=min, sec=sec})
--             record.timestamp = os.date("!%Y-%m-%dT%H:%M:%SZ", time)
--         end
--     end
--     return 1, timestamp, record
end

启动命令：
fluent-bit -c config.conf

openobserve 启动更简单，直接设置一个启动或者登录直接可以启动了
例如我docker 启动

version: '3.8'

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.109.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver

    depends_on:
      - openobserve

  openobserve:
    image: openobserve/openobserve:latest
    container_name: openobserve
    ports:
      - "5080:5080"  # OpenObserve Web UI
      - "9200:9200"  # OpenObserve Ingest API
      - "5081:5081"  # OpenObserve gRPC API
    environment:
      ZO_ROOT_USER_EMAIL: "admin@example.com"  # 设置 root 用户的电子邮件
      ZO_ROOT_USER_PASSWORD: "123456"    # 设置 root 用户的密码