处理K8S中日志收集filebeat报异常

处理K8S中日志收集filebeat报异常

错误信息

INFO producer/broker/51 maximum request accumulated, waiting for space

解决办法

1、修改filebeat的配置文件添加如下：

      topic: '%{[fields][log_type]}'   
      partition.round_robin:
        reachable_only: false
      required_acks: 1
      compression: gzip
      max_message_bytes: 100000000
 

参考：
https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#kafka-max_message_bytes
2、加大容器的内存

            "resources": {
              "limits": {
                "cpu": "500m",
                "memory": "524Mi"
              },
              "requests": {
                "cpu": "100m",
                "memory": "100Mi"
              }
            },

            "resources": {
              "limits": {
                "cpu": "1500m",
                "memory": "1524Mi"
              },
              "requests": {
                "cpu": "100m",
                "memory": "100Mi"
              }
            },