k8s使用fluent-bit连接外部ES

 helm安装

kubectl create ns fluent-bit
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit -n fluent-bit

 修改kubeConfig----fluent-bit内的fluent-bit.conf（直接替换）

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File parsers.conf
    Parsers_File custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[FILTER]
    Name kubernetes
    Match kube.*
    Annotations Off
    Labels Off
    Merge_Log On
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On

[OUTPUT]
    Name es
    Host 192.168.31.253
    Port 9200
    HTTP_User elastic
    HTTP_Passwd 6ygVP8r7QfGvW9jqyQGk
    Logstash_Format On
    Replace_Dots On
    Logstash_Prefix kubesphere
    Retry_Limit False

 下面为连接ES8.X用

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File parsers.conf
    Parsers_File custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Exclude_Path /var/log/containers/*_kube*-system_*.log
    Docker_Mode On
    Docker_Mode_Flush   5
    Docker_Mode_Parser java_multi_line
    Parser docker
    #multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
[FILTER]
    Name kubernetes
    Match kube.*
    Annotations Off
    Labels Off
    Merge_Log Off
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On

[FILTER]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_ 

[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_docker_id
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
    Rename    kubernetes_namespace_name namespace_name
    Rename    kubernetes_pod_name pod_name
    Rename    kubernetes_container_name container_name
    Rename    kubernetes_container_image container_image

[OUTPUT]
    Name es
    Host 10.8.67.105
    Port 9200
    HTTP_User elastic
    HTTP_Passwd tmWWCRjA5BvE*0JX1KaQ
    Logstash_Format On
    Suppress_Type_Name On
    Replace_Dots On
    Logstash_Prefix kubesphere
    Retry_Limit False

 

注释：

    Annotations Off    #关闭收集Annotations内容
    Labels Off            #关闭收集Labels 内容

    Logstash_Format On   #Logstash的格式

    Suppress_Type_Name On  #ES8版本index type 完全去除了，无法兼容，这个必须打开
    Replace_Dots On        #把点换成下短杠解决fluent内报错，（[ warn] [engine] failed to flush chunk '1-1670924415.565622347.flb', retry i）
    Logstash_Prefix kubesphere   #索引名字前缀，（kubesphere-20XX.XX.XX）

遇到的问题

分片被占完后，也会出现（[ warn] [engine] failed to flush chunk '1-1670924415.565622347.flb', retry i）的报错，

设置分片数

PUT /_cluster/settings
{
"transient":{
"cluster":{
"max_shards_per_node":5000
}
}
}